Batch node CLI commands with node-command

The node-command CLI tool allows Lighthouse Administrators to easily run a remote CLI command on a Managed Node, or in parallel on many or all Nodes.

Getting Started

To get started with any of the node-* tools, you can get quick information on how to use it from the CLI:

sudo node-command --help

To see a list of all the enrolled Nodes that the tool can operate on:

sudo node-info --all

This command displays the following information:

<Node's name in Lighthouse>
	address: <Node's Lighthouse VPN address>
	id: <Node's unique identifier in Lighthouse>
	ssh port: <Node's SSH server port>
	description: <Optional description of node in Lighthouse>
	enrollment status: <Whether a node is "Registered" pending enrollment, or "Enrolled">
	connection status: <Whether a node is currently "Connected" or "Disconnected" from Lighthouse VPN>

Selecting Nodes

There are a number of ways to select Nodes as targets on which to run a command, listed below. These can be used multiple times, or together, to select a multiple Nodes.

Select individually by name, Lighthouse VPN address or config ID, as per node-info output:

sudo node-command --node-name BNE-R01-IM4248
sudo node-command --node-address 192.168.128.50
sudo node-command --node-index nodes-49
sudo node-command --node-index nodes-1 --node-index nodes-2 --node-index nodes-3

Select all:

sudo node-command --all

Select a subset of nodes, using a pre-configured Smart Group:

sudo node-command --smartgroup "Brisbane Nodes"

Running Commands

Once Nodes have been selected, the commands to be run for each can be given. These are run on each Node in parallel. This command can be any command you can run from the Node CLI, commands are run as root.

For example to check the version on the first three configured console servers:

sudo node-command --node-index nodes-1 --node-index nodes-2 --node-index nodes-3 uptime

Note: When running a command on many or all Nodes, we recommend testing the command or one or just a few Nodes first.

Output Format: CLI

The command outputs the result of each command run on each Node. For example, the example node-command from the Running Commands section gives the following result:

sudo node-command --node-index nodes-1 --node-index nodes-2 --node-index nodes-3 uptime

== node-command ID 2017-05-29_24937 ==
11:41:13 [SUCCESS] demo-rg 192.168.128.2:22
 11:41:04 up 57 min, load average: 0.04, 0.11, 0.13

11:41:13 [SUCCESS] demo-im 192.168.128.3:22
 11:41:03 up 9 days, 23:42, load average: 0.23, 0.31, 0.33

11:41:13 [SUCCESS] demo-cm 192.168.128.4:22
 11:41:03 up 14 days, 20:10, load average: 0.13, 0.20, 0.22

There are a few components to this output:

1. The first line displays the run ID. This is the unique ID for this command including a timestamp of when the command was run, used to locate historical logs (discussed below).
2. Each node command result has a result header line. This contains the time the command completed, if the command succeeded or failed, the node name, and the node address.
3. The output (stdout) of the command being run is listed for each node on which the command was run. If there is no output, only the header line is listed.

There are a few ways to modify the output of the command, useful for batch operation or noisy commands. To hide the command output results, use the --quiet argument and only the headers will be shown. To suppress headers and display command output only, use the --batch argument. Combine both arguments to hide all output.

Output Format: Logs

Information about each run is logged to the filesystem by default. Filesystem logging can be disabled by using the --disable-fslog argument. The logs are stored in /var/log/node-command/, and are indexed using the run ID of each command (as detailed above in the Output Format: CLI section).

A new directory is created for each run, and contains 3 items:

1. A targets.txt file, listing the addresses on which the command was run.
2. A stdout directory, containing the output to stdout for the command printed by each Node.
3. A stderr directory, containing the output to stderr for the command printed by each Node.

By default, a history of the last 30 commands are kept logged to the file system before being removed.

Syslog Information

In addition to the output and file logging, the running of commands is also recorded in syslog (/var/log/messages from the CLI).

2017-05-29T11:50:12.069313+01:00 oglh-octo sudo:  robertw : TTY=pts/0 ; PWD=/var/volatile/log/node-command ; USER=root ; COMMAND=/usr/bin/node-command -i nodes-1 uptime
2017-05-29T11:50:13.971365+01:00 oglh-octo node-command[26837]: User root ran command uptime on node 192.168.128.2:22