In hiera backend, add ability to use "deep" _nodequery

[vincentbernat]

Hey!

It would be convenient if the hiera backend was able to substitute a node query when returning a structure from hiera. For example, if we have this:

something:
  sources::_nodequery: "Class[...]"

It would be convient if the hiera backend was able to return:

{ "something" => { "sources" => [ "host1", "host2" ] } }

[gregsio]

+1

[gregsio]

I am happy to give it a try but I would need a couple guidelines before I start writing this...
It looks like this can be done by editing the lookup function in puppetdb_backend.rb. I am not sure how to make sure this will retrun the whole hash and run the puppetdb query only for the inner deep key which contain the "_noquery" ...

"profile::elasticsearch::config":
  bootstrap.mlockall: true
  cluster:
    name: usersearch
  discovery:
    zen.minimum_master_nodes: 2
    zen.ping.multicast.enabled: false
    zen.ping.unicast.hosts::_nodequery: ['ec2_tag_role=usmaster', 'ipaddress']
  index:
    search.slowlog.threshold.fetch.debug: 500ms
    search.slowlog.threshold.fetch.info: 800ms
    search.slowlog.threshold.fetch.trace: 200ms
    search.slowlog.threshold.fetch.warn: 1s
    search.slowlog.threshold.query.debug: 2s
    search.slowlog.threshold.query.info: 5s
    search.slowlog.threshold.query.trace: 500ms
    search.slowlog.threshold.query.warn: 10s
  node:
    data: false
    master: true
    name: "%{::hostname}"

[gregsio]

Actually we don't need this feature because we can just interpolate the hiera function and let the magic happen:

"profile::elasticsearch::masternodes::_nodequery": ['ec2_tag_role=usmaster', 'ipaddress']

"profile::elasticsearch::config":
  bootstrap.mlockall: true
  cluster:
    name: usersearch
  discovery:
    zen.minimum_master_nodes: 2
    zen.ping.multicast.enabled: false
    zen.ping.unicast.hosts: "%{hiera('profile::elasticsearch::masternodes')}"
  index:
    search.slowlog.threshold.fetch.debug: 500ms
    search.slowlog.threshold.fetch.info: 800ms
    search.slowlog.threshold.fetch.trace: 200ms
    search.slowlog.threshold.fetch.warn: 1s
    search.slowlog.threshold.query.debug: 2s
    search.slowlog.threshold.query.info: 5s
    search.slowlog.threshold.query.trace: 500ms
    search.slowlog.threshold.query.warn: 10s
  node:
    data: true
    master: false
    name: "%{::hostname}"

Amazing!

[vincentbernat]

Unfortunately, this only works with Hiera 2+ as otherwise, everything hiera can fetch is a string. Right?

[gregsio]

I see this behavior too. With my example above I end up with:

hosts: 10.200.31.45610.200.40.12210.200.60.39

but base on the doc (https://docs.puppet.com/hiera/1/puppet.html#hiera-lookup-functions) the hiera function returns arrays and hashes as well...

So it looks like this is a problem with the puppetdb backend not returning an array.

Using hiera CLI or the query_nodes function I am retrieving the array as expected though...

DEBUG: 2016-06-16 00:23:17 +0000: Hiera PuppetDB backend starting
DEBUG: 2016-06-16 00:23:17 +0000: Looking up profile::elasticsearch::masternodes in PuppetDB backend
DEBUG: 2016-06-16 00:23:17 +0000: Hiera YAML backend starting
DEBUG: 2016-06-16 00:23:17 +0000: Looking up profile::elasticsearch::masternodes::_nodequery in YAML backend
DEBUG: 2016-06-16 00:23:17 +0000: Looking for data source node/i-1fc568c2
DEBUG: 2016-06-16 00:23:17 +0000: Looking for data source aws/usdata
DEBUG: 2016-06-16 00:23:17 +0000: Found profile::elasticsearch::masternodes::_nodequery in aws/usdata
DEBUG: 2016-06-16 00:23:17 +0000: Found nodequery ["ec2_tag_role=usmaster", "ipaddress"]
["10.200.31.456", "10.200.40.122", "10.200.60.39"]

[pietervogelaar]

I'm also trying to accomplish this exact same thing for elastic search. Too bad a string is returned instead of an array.

[pietervogelaar]

Any update on this?

[pietervogelaar]

It's a limitation of the hiera function used within a hiera yaml: https://docs.puppet.com/hiera/3.3/variables.html#the-hiera-lookup-function

The hiera() lookup function performs a Hiera lookup, using its input as the lookup key. The result of the lookup must be a string; any other result will cause an error.