check_pbsnodes

FULL DESCRIPTION: This plugin is for testing the presence of crashed nodes in a high performance computing cluster. In such clusters, it is not uncommon for load to reach very very high levels on compute nodes. Under such load, many parts of the system may bog down and become unresponsive. For example, SSH logins may no longer work. Polling via Gangila or Cacti may cease. And yet, this does not mean that the compute node has crashed or isn't still doing the work assigned to it by the cluster scheduler. Under such circumstances, the only way to know if a node is really down is if a job goes negative. Torque has a higher nice level than the jobs it runs, so it is always guranteed a processor time slice. If walltime is exceeded and Torque is able to get a slice it will kill the job. If it can't, then it's because the node has crashed and we'll see showq show negative time in the (time) REMAINING column. Therefore, this plugin is designed to be run on the Cluster Service Node, calling the showq command, parsing the output, and searching for values in the REMAINING column that are negative numbers. When it finds them, it should report the problem using correct Nagios syntax, and provide the crashed node names to the output string. It needs to be called from a remote plugin executor such as NRPE, or MRPE if using Matthias Kettner's Check_MK.

You must be logged in to submit a review.

Your review has been submitted and is pending approval.

Your recommendation has been sent.

Project Overview Project Files Project Notes Reviews