Working with the command line on a YAML backend

When we use a backend based on files such as JSON or YAML, which are the most commonly used, we have to recreate on the filesystem the hierarchy defined in our hiera.yaml file; the files that contain Hiera data must be placed in these directories.

Let's see Hiera in action. Provide a sample hierarchy configuration as follows:

:hierarchy:
  - "nodes/%{::fqdn}"
  - "env/%{::env}"
  - common

:yaml:
  :datadir: /etc/puppet/hieradata

We have to create a directory structure as follows:

mkdir -p /etc/puppet/hieradata/nodes
mkdir -p /etc/puppet/hieradata/env

Then, work on the YAML files as shown:

vi /etc/puppet/hieradata/nodes/web01.example42.com.yaml
vi /etc/puppet/hieradata/env/production.yaml
vi /etc/puppet/hieradata/env/test.yaml
vi /etc/puppet/hieradata/common.yaml

The previous files are plain YAML files where we can specify the values for any Hiera-managed variable. These values can be strings, arrays, or hashes.

We can place our default settings in /etc/puppet/hieradata/common.yaml as follows:

---
# A simple string assigned to a key
timezone: 'Europe/Rome'

# A string with variable interpolation
nagios_server: "nagios.%{::domain}"

# A string with another variable defined in Hiera (!)
dns_nameservers: "%{hiera('dns_servers')}"

# A string assigned to a key that maps to the
# template parameter of the openssh class (on Puppet3)
openssh::template: 'site/common/openssh/sshd_config.erb'

# An array of values
ldap_servers:
  - 10.42.10.31
  - 10.42.10.32

# An array with a single value
ntp::ntp_servers:
  - 10.42.10.71

# A hash of values
users:
  al:
    home: '/home/al'
    comment: 'Al'
  jenkins:
    password: '!'
    comment: 'Jenkins'

Given the previous example, execute a Hiera invocation as follows:

hiera ldap_servers

It will return the following array:

["10.42.10.31", "10.42.10.32"]

If we define a different value for a key in a data source that is higher in the hierarchy, that value is returned. Let's create a data source for the test environment by editing the file /etc/puppet/hieradata/env/test.yaml as follows:

---
ldap_servers:
- 192.168.0.31
users:
  qa:
    home: '/home/qa'
    comment: 'QA Tester'

Consider a normal Hiera lookup for the ldap_servers key as follows:

hiera ldap_servers

It will still return the common value ["10.42.10.31", "10.42.10.32"].

However, we can explicitly pass the env variable as follows:

hiera ldap_servers env=test

The returned value is the one for the test env as follows:

["192.168.0.31"]

If we have a more specific setting for a given node, that value is returned. Suppose we edit the file ldap.example42.com.yaml in /etc/puppet/hieradata/nodes/ with the following code:

---
ldap_servers:
- 127.0.0.1

When we query Hiera and refer to a specific node's FQDN using the following command:

hiera ldap_servers fqdn=ldap.example42.com

The result is the one from the higher source in the hierarchy:

["127.0.0.1"]

Hiera also provides some alternate lookup options. When we deal with arrays, for example, we can decide to merge all the values found in the hierarchy instead of returning the first one found. We make our query to specify the -a or --array option as follows:

hiera -a ldap_servers fqdn=ldap.example42.com

The result contains all the entries for the ldap_servers key in all the data sources of our hierarchy as follows:

["127.0.0.1", "10.42.10.31", "10.42.10.32"]

Note that the value defined for the env=test case is not returned unless we specify it using the following command:

hiera -a ldap_servers fqdn=ldap.example42.com env=test

In this case, the output would be as follows:

["127.0.0.1", "192.168.0.31", "10.42.10.31", "10.42.10.32"]

When working with hashes, interesting things can be done.

Let's see what's the value of the users key for the test environment:

hiera users env=test

The output is the hash we configured in test.yaml:

{"qa"=>{"home"=>"/home/qa", "comment"=>"QA Tester"}}

As it normally does, Hiera returns the first value it encounters while traversing the hierarchy, but in this case, we might prefer to have a hash that contains all the values found. Similar to the -a option for arrays, we have at our disposal the -h (--hash) option for hashes:

hiera -h users env=test

The result is a hash that contains all the users defined in different data sources:

{"al"=>{"home"=>"/home/al", "comment"=>"Al"},
 "jenkins"=>{"password"=>"!", "comment"=>"Jenkins"},
 "qa"=>{"home"=>"/home/qa", "comment"=>"QA Tester"}}

Note that hashes are not ordered according to the matching order as arrays are.

Let's perform one more experiment. Let's add a new user specific to our ldap.example42.com node and give different values to a parameter that is already defined in the common data source. We edit the file ldap.example42.com.yaml in vi /etc/puppet/hieradata/nodes/ with the following code:

users:
  openldap:
    groups: 'apps'
  jenkins:
    ensure: absent

Consider a hash lookup as follows:

hiera -h users fqdn=ldap.example42.com env=test

As expected, this would return all the users found in all the hierarchy levels:

{"al"=>{"home"=>"/home/al", "comment"=>"Al"},
 "jenkins"=>{"ensure"=>"absent"},
 "qa"=>{"home"=>"/home/qa", "comment"=>"QA Tester"},
 "openldap"=>{"groups"=>"apps"}}

Let's take a look at the parameters of the jenkins user; being defined both at the node level and in the common data source, the returned value is the one for the higher data source in the hierarchy.

Hiera's management of hashes can be quite powerful, and we can make optimal use of it. For example, we can use them inside Puppet manifests with the create_resources function, with the hash of users data and a single line of code as follows:

create_resources(user, hiera_hash($users))

Based on highly customizable Hiera data, we can manage all the users of our nodes.

Quite often, we need to understand where a given key is set in our hierarchy and what values will be computed for it. The -d (debug) option is rather useful for this. The previous line will return an output as follows:

hiera -d -h users fqdn=ldap.example42.com env=test
DEBUG: 2013-12-07 13:11:07 +0100: Hiera YAML backend starting
DEBUG: <datetime>: Looking up users in YAML backend
DEBUG: <datetime>: Looking for data source nodes/ldap.example42.com
DEBUG: <datetime>: Found users in nodes/ldap.example42.com
DEBUG: <datetime>: Looking for data source env/test
DEBUG: <datetime>: Found users in env/test
DEBUG: <datetime>: Looking for data source common
DEBUG: <datetime>: Found users in common
{"al"=>{"home"=>"/home/al", "comment"=>"Al"},
 "jenkins"=>{"ensure"=>"absent"},
 "qa"=>{"home"=>"/home/qa", "comment"=>"QA Tester"},
 "openldap"=>{"groups"=>"apps"}}

This output also tells us where Hiera is actually looking for data sources.

In a real Puppet environment, it is quite useful to use the --yaml option, which, when used with a real facts file of a node, allows us to evaluate exactly how Hiera computes its keys for real servers.

On the Puppet Master, the facts of all the managed clients are collected in $vardir/yaml/facts, so this is the best place to see how Hiera evaluates keys for different clients:

hiera --yaml /var/lib/puppet/yaml/facts/<node>.yaml ldap_servers

Hiera can use other sources to retrieve the facts of a node and return its key values accordingly. We can interrogate the Puppet Master's inventory service with the following command:

hiera -i ldap.example42.com ldap_servers

As an alternative, we can query mcollective (from a machine where the mco client is installed):

hiera -m ldap.example42.com ldap_servers