Zero Touch Provisioning (ZTP), automating config and firmware using DHCP

Zero Touch Provisioning (ZTP) allows Opengear appliances to be provisioned with configuration and/or  firmware during their initial boot, from a DHCP and file server.

Note: ZTP using .opg and .xml was introduced in firmware 3.15.1, it was extended with .sh support in 3.16.4u6 and .flash support in 3.16.6.

Preparation

These are typical steps for configuration over a trusted network:

1. Create an .opg, .xml or .sh configuration file.
2. Obtain the latest .flash firmware upgrade file from: https://ftp.opengear.com/download/opengear_appliances/end_of_sale_products/
3. Publish these files on a fileserver that understands one of the HTTPS, HTTP, FTP or TFTP protocols.
4. Configure your DHCP server to include "vendor specific" sub-options for Opengear devices.
• The "vendor specific" option is option 43
• Option 43 sub-option 1 is the URL of the configuration file, the config URL must end in either .opg .xml or .sh
• Option 43 sub-option 2 is the URL of the firmware file
• The sub-option strings combined should not exceed 250 characters in length
• This configuration will be DHCP server-specific, a sample ISC DHCP server config snippet is included later in this document
5. Connect a new Opengear device (either at defaults from the factory, or config erased) to the network and apply power.
6. It may take 5 - 10 minutes for for the entire process to complete.

Create the configuration file

There are two main approaches to creating a configuration file to serve via ZTP. 

Backup method: Take an already configured Opengear device and backup its configuration – preferably same the model, and same version that you are targeting.

Optionally use the Bulk Provisioning wizard scripts to remove any appliance-specific settings (i.e. create a template configuration) and/or prepare the configuration for automated Lighthouse enrollment, see this article for details.

The configuration may be backed up in OPG (.opg) or XML (.xml) format:

• TXT  (2000 and 8100 Family) – This backup can be applied as a bash script or the import command. This article details the import/export options. 
• OPG (7000 Family) – this is a full snapshot of /etc/config/, available in the GUI under System: Configuration Backup in the web UI, or via config -e in the CLI – see this article for details about manipulating this format
• XML – this is the top-level /etc/config/config.xml configuration file only, available in the GUI under System: Configuration Backup: XML Configuration or from the CLI by copying /etc/config/config.xml off-box

Should you use OPG or XML?  The XML configuration generates the contents of /etc/config/ when the configurators run automatically at the end of the ZTP process, however some files such as SSL certificates and SSH keys are not included in the XML.  To backup and restore these files, use OPG.

Note: When creating backups to apply Opengear products of a different model, it is recommended to run make-template to scrub any model-specific configuration, which may cause compatibility issues.

Script method: Using a bash shell script, you may automate the same commands you would use to manually configure at the CLI.

For configuration, these will generally be config -s commands, however any non-interactive command is available allowing for advanced provisioning, e.g. wget, curl, scp, nslookup, etc.

Note: Refer to this article for details on configuring from the CLI, including a one-liner to convert an existing configuration into a series of config -s commands.

Please note this is an advanced method with a few potential gotchas:

• Ensure you configure networking (i.e. at least set config.interfaces.wan.mode to static or dhcp), to flag the device as configured and prevent a ZTP-loop.
• Do not run configurators to apply config from inside the script, i.e. do not include config -r or config -a commands in your script.  This may cause issues in the pre-boot state in which the script is running.
• Unlike the backup method, the script method does not automatically apply configuration or reboot when complete.  See the example script below for how this is done.

Therefore a trivial configuration script may look like:

2000 and 8100 Family:

https://ftp.opengear.com/download/documentation/manual/current/CM8100_and_OM/

7000 Family:

#!/bin/bash

# Configure networking
config -s config.interfaces.wan.mode=static
config -s config.interfaces.wan.address=192.168.0.1
config -s config.interfaces.wan.netmask=255.255.255.0
config -s config.interfaces.wan.gateway=192.168.0.254

# Configure hostname
config -s config.system.name=ztp-demo

# Set root's password
config -s config.users.user1.password='$1$eLRzeRxt$zgHgqEKId1i/1ZhwGzdE80'

# Schedule configurators and reboot
touch /etc/config/.run_configurators
reboot

Example ISC DHCP server configuration

The following is an example of an ISC DHCP server configuration fragment for serving an .opg configuration image and a firmware upgrade file:

option space opengear code width 1 length width 1;
option opengear.config-url code 1 = text;
option opengear.firmware-url code 2 = text;
class "opengear-ztp" {
    match if option vendor-class-identifier ~~ "Opengear/";
    vendor-option-space opengear;
    option opengear.config-url "https://example.com/opg/${class}.opg";
    option opengear.firmware-url "https://example.com/firmware/${class}.flash";
}

For other DHCP servers, please consult their documentation on specifying "Vendor Specific" option fields. We use sub-option 1 to hold the URL text.

Setup for an untrusted LAN

If network security is a concern, and you can have remote hands insert a trusted USB flash drive into the Opengear device during provisioning, then follows are a summary of the steps required for deploying configuration in an untrusted network:

1. Generate an X.509 certificate for the client. Place it and its private key file onto a USB flash drive (concatenated as a single file, client.pem).
2. Set up a HTTPS server that restricts access to the .opg or .xml file for HTTPS onnections providing the client certificate.
3. Put a copy of the CA cert (that signed the HTTP server's certificate) onto the USB flash drive as well (ca-bundle.crt).
4. Insert the USB flash drive into the Opengear device before attaching power or network.
5. Continue with the steps above, but using only a https URL.
6. A detailed step-by-step document for preparing a USB flash drive and using OpenSSL to create keys bound be found in this article.

How it works

This section explains in detail how the Opengear device uses DHCP to obtain initial setup.

First, an Opengear console manager is either configured or unconfigured. ZTP needs it to be in an unconfigured state, which is only obtained in the following ways:

• Firmware programming at factory
• Pressing the Config Erase button twice during operation
• Selecting Config Erase under System: Administration in the web UI, and rebooting
• Creating the file /etc/config/.init and then rebooting (command-line)

When an unconfigured Opengear boots, it performs these steps to find a configuration:

• The Opengear device transmits a DHCP DISCOVER request onto its primary Network Interface (wan). This DHCP request will carry a Vendor Class Identifier of the form Opengear/model-name (for example, Opengear/ACM5003-M) and its parameter request list will include option 43 (Vendor-Specific Information).
• On receipt of a DHCP OFFER, the device will use the information in the offer to assign an IPv4 address to its primary Network Interface, add a default route, and prepare its DNS resolver.
• If the offer also contained an option 43 with sub-option 1, the device interprets the sub-option as a whitespace-separated list of URLs to configuration files to try to restore.
• If an NTP server option was provided in the DHCP offer, the system clock is (quickly) synchronized with the NTP server.
• If firmware image URLs were provided (sub-option 2) the system now runs /etc/scripts/backup-url loadimage for each URL until one of the URLs succeeds (the image was flashed or matched to existing flash):
  1. The same secure rules as described below are used. That is, certificates found on an attached USB device are used to authenticate HTTPS URLs and other non-HTTS URLs are treated as errors.
  2. If the URL ends with options in the form ;nf=<options> these will be passed to netflash as options. When not specified, iE is the default. In secure mode, the nf options must not be specified.
  3. The firmware image is downloaded and passed to netflash on its standard input.
  4. netflash will checksum and flash the firmware image. If netflash reboots at this stage, /etc/config/.init will be present to re-initialize config after a potential downgrade.
  5. Otherwise, netflash (with -Z option) may have determined that the image has already been flashed and processing can continue.
• The system now searches all attached USB storage devices for two optional certificate files. The first file is named ca-bundle.crt and the second one is whichever one of the following filenames is found first:
  • client-AABBCCDDEEFF.pem (where AABBCCDDEEFF is the MAC address of the primary network interace); or
  • client-MODEL.pem (where MODEL is the (vendor class) model name in lowercase, truncated to before the first hyphen); or
  • client.pem
  • If both files are found (ca-bundle.crt and a client.pem), then secure mode is enabled for the next section.
  • Each URL in the list obtained from option 43 sub-option 1 is tried in sequence until one succeeds:
    • The URL undergoes substring replacement from the following table:

Substring	Replaced by	Example
${mac}	the 12-digit MAC address of the device, lowercase	0013b600b669
${model}	the full model name, in lowercase	om2248-l
${class}	the firmware hardware class	ACM500x ACM550x ACM7004-5 ACM700x CM710x CM7196 IM42xx IM72xx OM22xx
${version}	the firmware version number	23.03.0

• The resulting URL must end in .opg, .xml or .sh (an optional ?query-string is permitted). If it doesn't, then it is skipped and the next URL is tried.
• In secure mode, the URL must use the https scheme or it is skipped.
• Otherwise the available schemes are: http https tftp ftp ftps
• The curl program is used to download the URL.
• In secure mode, the server's certificate must validate against the ca-bundle.crt. The (reqiured) client.pem file is provided to authenticate the client to the server. Please see the curl documentation for the format of these files.
• The URL is downloaded. For .opg files its header is checked to see if it is compatible with the current device. For .xml files, a parse check is made. If the check fails, the downloaded file is abandoned and the next URL is tried.
• The file is imported into the current configuration.
• The system checks to see if a hostname has been set in the config. If not, it is set to ${model}-${mac}.
• The system checks to see if it is still in an unconfigured state. If it is, then the network interface mode is set to DHCP. This effectively forces the system into a configured state, preventing a future reboot loop.
• The system reboots

Note:  If all the URLs were skipped or failed, the system will wait for 30 seconds before retrying again. It will retry all the URLs up to 10 times. After the 10th retry, the system reboots. If the system has been manually configured in the meantime, the retries stop and ZTP is disabled.

Note:  If no option 43 is received over DHCP, no URLs are downloaded and no reboots occur: the system must be manually configured. Once configured (manually or by ZTP), an Opengear will no longer request option 43 from the DHCP server, and it will ignore any option 43 configuration URLs presented to it.