Puppetdbquery support for unicast.hosts

[pietervogelaar]

I would like to search for other hosts of the cluster based on application, role and environment. This is very nicely possible with puppetdbquery.

Hiera yaml:

my_elasticsearch::unicast_hosts::_nodequery: [
  "application=%{::application} and role=elasticsearch and environment=%{::environment}",
  'ipaddress'
]

elasticsearch::instances:
  default:
    config:
      discovery.zen.ping.unicast.hosts: "%{hiera('my_elasticsearch::unicast_hosts')}"
      discovery.zen.minimum_master_nodes: 2

However the _nodequery puppetdbquery suffix doesn't work inside a deeper hash. So as a workaround I retrieve the hosts array with an extra parameter. With a hiera lookup inside Hiera I retrieve the value. But that hiera lookup always returns a string. This will result in the config file as discovery.zen.ping.unicast.hosts: "['192.168.50.52', '192.168.50.53']" instead of discovery.zen.ping.unicast.hosts: ['192.168.50.52', '192.168.50.53'].

Can this be corrected in the erb template?

[pietervogelaar]

Relates to dalen/puppet-puppetdbquery#84

[tylerjl]

Unfortunately I don't think the erb template can be amended to fix that case. The config template uses to_yaml to generate the yaml file, and the hiera() lookup function inherently converts looked up arrays into strings (see this note on the hiera documentation page.) Thus the call to "%{hiera('my_elasticsearch::unicast_hosts')}" is just a string, and to_yaml just represents what's present in the configuration hash.

It does make your use case somewhat difficult to solve, but I can't think of a way to get around what hiera() is returning. If you could join the array on commas, Elasticsearch will recognize the parameter as having multiple values and process it accordingly, but I don't know whether that's possible with hiera.

[pietervogelaar]

Today I created the following solution. All parameters are defined in Hiera except the ones below:

my_elasticsearch/manifests/init.pp

class { '::elasticsearch':
    instances             => enhance_instances(hiera_hash('elasticsearch::instances', {})),
    instances_hiera_merge => false,
  } ->

my_elasticsearch/lib/puppet/parser/functions/enhance_instances.rb

Puppet::Parser::Functions.newfunction(:enhance_instances, :type => :rvalue, :arity => 1, :doc => <<-EOT

  accepts one arguments, an instances hash. This function converts unicast hosts JSON array string
  to an actual array to support the use of puppetdbquery inside the instances hash in Hiera. Because the extra
  Hiera lookup in that hash always returns a string.

  Related issue:
  https://github.com/elastic/puppet-elasticsearch/issues/849

EOT
                                     ) do |args|

  require 'json'

  instances = args[0]

  instances.each do |key, instance|
    if !instance['config'].nil? and
      !instance['config']['discovery.zen.ping.unicast.hosts'].nil? and
      instance['config']['discovery.zen.ping.unicast.hosts'].is_a?(String)

      # Parse JSON string into array
      hosts = JSON.parse(instance['config']['discovery.zen.ping.unicast.hosts'])

      # Update hash with hosts of data type array
      instances[key]['config']['discovery.zen.ping.unicast.hosts'] = hosts
    end
  end

  return instances
end

[tylerjl]

Thanks @pietervogelaar, glad you found a workaround - hopefully this issue can help out someone down the road.