Bulk CLI command tool using node-command

 

The node-command CLI tool is used to run commands on managed console servers, allowing administrators to easily run a single CLI command in bulk, on all or on a range of their console server deployment.

This feature is also demonstrated in this video.

Note: To run node-* commands, you must be authorized as an admin group user

Getting Started

To get started with any of the node-* tools, you can get quick information on how to use it from the command line:

node-command --help

To see a list of all the registered console servers that the tool can operate on:

node-command --list-nodes

Selecting Console Servers

There are a number of ways to select console servers (aka “nodes”) as targets on which to run a command, listed below. These can be used multiple times, or together, to select a range of console servers:

Select individually by name, address, Call Home address or numeric config index (as per --list-nodes output):

node-command --node-name BNE-R01-IM4248
node-command --node-address 192.168.0.33:22
node-command --node-address localhost:40121
node-command --node-index 17

Select all: 

node-command --all

Select multiple using regular expression pattern matches or inverse pattern match, against console server attributes defined under in the Lighthouse web UI under Manage -> Access Console Servers:

node-command --select-match 'Model=ACM.*'
node-command --deselect-match 'Version=3.11.1'

Running Commands

Once console servers have been selected, the commands to be run for each can be given. These are run on each managed console server, in parallel.  This command can be any command you can run from the console server CLI, commands are run as root.

For example to check the version on the first three configured console servers: 

node-command --node-index 1 --node-index 2 --node-index 3 cat /etc/version

Note: When using non-trivial selection arguments, you can check which target console servers have been selected an initial pass specifying --list-nodes rather than the final command.

Copying Files

node-copy is a variation of node-command that can be used to copy a file to selected managed console servers, by specifying a source and destination, e.g. to copy a Cisco IOS system image that has been pre-copied onto the Lighthouse CMS’s /var/nvlog directory, to all managed IM family console servers’ TFTP server:

node-copy --select-match ‘Model=IM.*’ --source-file /var/nvlog/ios.img /var/tmp/usbdisk/tftpboot/

Output Format: CLI

The command outputs the result of each command run on each remote console server. For example, the example node-command from the Running Commands section gives the following result:

node-command --node-index 1 --node-index 2 --node-index 3 cat /etc/version

== node-command ID 2014-03-12T05:10:29.360164_29534 ==
15:10:29 [SUCCESS] BNE-R01-IM4248 10.10.0.1:22
OpenGear/IM42xx Version 3.11.0 --  Tue Jun 24 03:23:59 EST 2014
15:10:29 [SUCCESS] BNE-R01-IM4248 10.10.0.2:22
OpenGear/IM42xx Version 3.11.0 --  Tue Jun 24 03:23:59 EST 2014
15:10:29 [SUCCESS] BNE-R01-IM4248 10.10.0.3:22
OpenGear/IM42xx Version 3.11.0 --  Tue Jun 24 03:23:59 EST 2014

There are a number of components to this output:

1. The first line displays the run ID.  This is the unique ID for this command including a timestamp of when the command was run, used to locate historical logs (discussed below).
2. Each node command result has a result header line.  This contains the time the command completed, if the command succeeded or failed, the node name, and the node address.
3. The output (stdout) of the command being run is listed for each node on which the command was run.  If there is no output, only the header line is listed.

There are a few ways to modify the output of the command, useful for batch operation or noisy commands.  To hide the command output results, use the --quiet argument and only the headers will be shown.  To suppress headers and display command output only, use the --batch argument.  Combine both arguments to hide all output.

Output Format: Logs

Information about each run is logged to the filesystem, by default.  Filesystem logging can be disabled by using the --disable-fslog argument.  The logs are stored in /var/nvlog/node-command/, and are indexed using the run ID of each command (as detailed above in the Output Format: CLI section).

A new directory is created for each run, and contains 3 things:

1. A targets.txt file, listing the addresses on which the command was run.
2. A stdout directory, containing the output to stdout for the command printed by each console server.
3. A stderr directory, containing the output to stderr for the command printed by each console server.

By default, a history of the last 30 commands are kept logged to the file system before being removed.

Syslog Information

In addition to the output and file logging, the running of commands is also recorded in syslog (Status -> Syslog in the UI or /var/log/messages from the CLI).

<11>Mar 12 15:10:29 node-command[29534]: User 'root' ran command 'cat /etc/version' on node '10.10.0.1:22'
<11>Mar 12 15:10:29 node-command[29534]: User 'root' ran command 'cat /etc/version' on node '10.10.0.2:22'
<11>Mar 12 15:10:29 node-command[29534]: User 'root' ran command 'cat /etc/version' on node '10.10.0.3:22'