Gathering Data: Inventory File Format

The inventory file that the poller uses describes a set of sources used to gather the list of devices, credentials to authenticate in the devices, default settings and eventually puts all together defining namespaces. An extensive explanation of each secation is provided on this page.

The new inventory is structured in 4 major pieces, explained in its own section:

• sources: a list of source to gather the list of devices
• devices: a list of default settings to be applied in case it is not possible to deduce them from the sources
• auths: a list of credential sources
• namespaces: where you put together all the above. A namespace is be defined by a source, an auth and a device

Here is an example of a complete inventory file:

sources:
- name: netbox-instance-123
  token: af8717c89ec0ff420c19d89e6c20646ad55dd54e
  url: http://127.0.0.1:8000
  tag: suzieq-demo
  period: 3600

- name: dc-02-suzieq-native
  hosts:
  - url: ssh://vagrant@10.0.0.1:22 keyfile=/path/to/private_key
  - url: ssh://vagrant@10.0.0.2:22 devtype=eos keyfile=/path/to/private_key

- name: ansible-01
  type: ansible
  path: /path/to/ansible/list

devices:
- name: devices-with-jump-hosts
  transport: ssh
  jump-host: 127.0.0.1
  jump-host-key-file: /path/to/jump/key
  ignore-known-hosts: true
  port: 22
  devtype: eos

- name: devices-using-rest
  transport: rest

auths:
- name: credentials-from-file-0
  type: cred-file
  path: /path/to/device/credentials.yaml

- name: suzieq-user-01
  username: suzieq
  password: plain:pass

- name: suzieq-user-02
  username: suzieq
  password: env:PASSWORD_ENV_VAR

- name: suzieq-user-03
  username: suzieq
  password: ask

- name: suzieq-user-04
  key-passphrase: ask
  keyfile: path/to/key

namespaces:
- name: testing
  source: netbox-instance-123
  device: devices-00
  auth: credentials-from-file-0

Sources

The device sources currently supported are:

• Host list (the same used with the old option -D in Suzieq 0.15.x or lower)
• Ansible inventory, specifing a path to a file that has to be the output of ansible-inventory --list command
• Netbox

Each source must have a name so that it can be referred to in the namespace section.

Whenever a source has many fields in common with another, you don't have to rewrite it. With the copy: <source_name> all the fields of <source-name> are automatically replicated, and it is possible to specify only the ones which changes:

- name: netbox-orig
  type: netbox
  token: your-api-token-here
  url: http://127.0.0.1:8000
  tag: suzieq-demo
  period: 3600

- name: netbox-copy     # This source will use the same set of parameters of 'netbox-orig'
  copy: netbox-orig     # and only overrides the 'tag' field.
  tag: suzieq-copy

Host list

The host list contains the IP address, the access method (SSH or REST), the IP address of the node, the user name, the type of OS if using REST and the access token such as a private key file. Here is an example of a native suzieq source type. For example (all possible combinations are shown for illustration):

- name: dc-01-native
  type: native # optional, if type is not present this is the default value
  hosts:
    - url: https://vagrant@192.168.123.252 devtype=eos
    - url: ssh://vagrant@192.168.123.232  keyfile=/home/netenglabs/cloud-native-data-center-networking/topologies/dual-attach/.vagrant/machines/internet/libvirt/private_key
    - url: https://vagrant@192.168.123.164 devtype=eos
    - url: ssh://192.168.123.70 username=admin password=admin
    - url: ssh://vagrant@192.168.123.230  keyfile=/home/netenglabs/cloud-native-data-center-networking/topologies/dual-attach/.vagrant/machines/server101/libvirt/private_key
    - url: ssh://vagrant@192.168.123.54:2023  keyfile=/home/netenglabs/cloud-native-data-center-networking/topologies/dual-attach/.vagrant/machines/server104/libvirt/private_key
    - url: https://vagrant@192.168.123.123 password=vagrant

There's a template in the docs directory called hosts-template.yml. You can copy that file as the template and fill out the values for namespace and url (remember to delete the empty URLs and to not use TABS, some editors add them automatically if the filename extension isn't right). The URL is the standard URL format: <transport>://[username:password]@<hostname or IP>:<port>. For example, ssh://dinesh:dinesh@myvx or ssh://dinesh:dinesh@172.1.1.23.

Ansible

If you are using Ansible to configure the devices, it is possible to set the output of the ansible-inventory --list command as an input source. Once you created a json file containing the result of the command, with:

ansible-inventory --list > ansible.json

Now you can set the path of the ansible inventory in the source:

- name: ansible-01
  type: ansible
  path: /path/to/ansible.json

Netbox

Netbox is often used to store devices type, management address and other useful information to be used in network automation. Suzieq can pull device data from Netbox selecting them by tag (currently only one per each source). To do so a token to access the netbox API is required as well as the netbox instance url. Since Netbox is a dynamic source, the data are periodically pulled, the period can be set to any desired number in seconds (default is 3600).

Here is an example of the configuration of a netbox type source:

- name: netbox-dc-01
  type: netbox
  token: your-api-token-here
  url: https://127.0.0.1:8000
  tag: suzieq-demo    # if not present, default is "suzieq"
  period: 3600        # How frequently Netbox should be polled
  ssl-verify: false   # Netbox certificate validation will be skipped

Moreover, netbox type source is capable to assign each device to a namespace which corresponds to the device's sitename. To obtain this behaviour, we need to declare a namespace object with name: netbox-sitename.

Here is an example:

sources:
- name: netbox-dc-01
  type: netbox
  token: your-api-token-here
  url: http://127.0.0.1:8000
  tag: suzieq-demo

- name: netbox-dc-02
  type: netbox
  token: your-api-token-here
  url: http://127.0.0.1:9000
  tag: suzieq

auths:
- name: auth-st
  username: user
  passowrd: my-password

namespaces:
- name: netbox-sitename # devices namespaces equal to their site names
  source: netbox-dc-01
  auth: auth-st

- name: namespace01     # devices namespaces equal to 'namespace01'
  source: netbox-dc-02
  auth: auth-st

Devices

In this section you can specify some default settings to be applied to set of devices. You can bind this settings to a source in the namespaces section.

For example, if a set of devices is only reachable with a ssh jump, in devices you can define the jump host and the path to the jump-host key file. This way you define it once and possibly use it in multiple namespaces.

- name: devices-with-jump-hosts
  transport: ssh
  jump-host: 10.0.0.1
  jump-host-key-file: /path/to/jump/key
  port: 22

In case you want to ignore the check of the device's key against the known_hosts file you can set:

- name: devices-ignoring-known-hosts
  ignore-known-hosts: true

Moreover if all the devices inside a namespace run the same NOS, it is possible to specify it via the devtype option:

- name: eos-devices
  devtype: eos

Auths

This section is optional in case Suzieq native and ansible source types. Here a set of default authentication sources can be defined. Currently a cred-file type and a static default type are defined. This way if credentials are not defined in the sources, default values can be applied.

Currently for both SSH and REST API, the only supported is username and password, therefore you will not be able to set api keys.

The simplest method is defining either username and password/private key.

- name: suzieq-user
  username: suzieq
  password: plain:pass

where password can be specified in plaintext, as an environment variable or to be asked to the user:

• plain:<password-in-plaintext>
• env:<ENV_VARIABLE>
• ask

In case a private key is used to authenticate:

- name: suzieq-user
  keyfile: path/to/private/key
  key-passphrase: ask

Where key-passphrase is the passphrase of the private key. As the password field, it can be set as plaintext, env variable or to be asked to the user.

Credential file

All the fields specified directly inside an authentication source (i.e. username, password, keyfile, etc.) are used as default if the values are not provided. When the source do not provide the credentials, such as Netbox, we might want to specify different credentials for each device. This can be easily achieved via a cred-file, referenced in the auths section like in the following example:

- name: credentials-from-file-0
  type: cred-file
  path: /path/to/device/credentials.yaml

A cred-file is an external file where you store credentials for all the devices. Each device credentials can be specified via its hostname or its address (with Netbox, it's encouraged the usage of hostname). The credential file should look like this:

- namespace: testing
  devices:
  - hostname: leaf01
    password: my-password
    username: vagrant
  - hostname: leaf02
    keyfile: /path/to/private/key
    username: vagrant
  - hostname: leaf03
    keyfile: /path/to/private/key
    username: vagrant
    key-passphrase: my-passphrase
  - address: 10.0.0.1
    username: vagrant
    password: my-password

Namespaces

In the namespaces section sources, auths and devices can be put together to define namespaces. For example the following namespace will be defined by the source named netbox-1, the auths named dc-01-credentials, and the device named ssh-jump-devs:

namespaces:
- name: example
  source: netbox-1
  device: ssh-jump-devs
  auth: dc-01-credentials

In case you are using the Suzieq native or ansible source types, auth field is optional since the settings can be defined per-device in the source.

The device field is always optional since it only contains common configurations for all the devices in the namespace.

Running the poller

Once you have generated the inventory file you can launch the poller with the following command:

sq-poller -I inventory.yaml

Migrating Suzieq Native Inventory to new format

Starting with version 0.16.0, the Suzieq Native inventory format is no longer supported as is. We need to do some small changes to use it with the new version. Here is an example of creating a new inventory.yml from an old suzieq native inventory format.

Suppose we have this inventory valid for version 0.15.x:

- namespace: eos
  hosts:
    - url: https://vagrant@192.168.123.252 devtype=eos
    - url: ssh://vagrant@192.168.123.232  keyfile=/home/netenglabs/cloud-native-data-center-networking/topologies/dual-attach/.vagrant/machines/internet/libvirt/private_key
    - url: https://vagrant@192.168.123.164 devtype=eos
    - url: ssh://192.168.123.70 username=admin password=admin
    - url: ssh://vagrant@192.168.123.230  keyfile=/home/netenglabs/cloud-native-data-center-networking/topologies/dual-attach/.vagrant/machines/server101/libvirt/private_key
    - url: ssh://vagrant@192.168.123.54:2023  keyfile=/home/netenglabs/cloud-native-data-center-networking/topologies/dual-attach/.vagrant/machines/server104/libvirt/private_key
    - url: https://vagrant@192.168.123.123 password=vagrant

The new inventory format consists of four sections (sources, auths, devices, namespaces) which are described above. We need to add the devices specified in the old inventory format in a new source inside the sources section and link it to a namespace.

Here is how the new format will look like:

sources:
- name: eos-source # namespace is defined below, this is only a name to be used as reference
  hosts:
    - url: https://vagrant@192.168.123.252 devtype=eos
    - url: ssh://vagrant@192.168.123.232  keyfile=/home/netenglabs/cloud-native-data-center-networking/topologies/dual-attach/.vagrant/machines/internet/libvirt/private_key
    - url: https://vagrant@192.168.123.164 devtype=eos
    - url: ssh://192.168.123.70 username=admin password=admin
    - url: ssh://vagrant@192.168.123.230  keyfile=/home/netenglabs/cloud-native-data-center-networking/topologies/dual-attach/.vagrant/machines/server101/libvirt/private_key
    - url: ssh://vagrant@192.168.123.54:2023  keyfile=/home/netenglabs/cloud-native-data-center-networking/topologies/dual-attach/.vagrant/machines/server104/libvirt/private_key
    - url: https://vagrant@192.168.123.123 password=vagrant

namespaces:
- name: eos
  source: eos-source