Update to Hiera 5 format?

[fatmcgav]

@lynxman Have you got any plans to update this Hiera backend to support Hiera 5.x?

If not, I might be able to take a stab...

[luyseyal]

+1

[lynxman]

Hey guys,
Definitely! Need to get onto it.
If you want to send a pull request though it'll be nice ;)

[petems]

@fatmcgav Have you had a chance to try updating the module for Hiera 5? I might give it a go if you haven't 😄

Docs are here: https://docs.puppet.com/puppet/5.0/hiera_custom_lookup_key.html

[fatmcgav]

@petems No, haven't had time to get to this one yet... Feel free to give it a stab :)

[elisiano]

Hi All, has there been any movement on this? We're currently at a crossroad and we're pondering whether waiting for this or find some other solution. Thanks!

[petems]

@elisiano I'm planning to pair up with some of the hiera whiz-kid engineers to implement this when he's back from PTO, should have something whipped up before end of August. What kinda time frame you looking for?

[elisiano]

@petems thanks for the update. We will wait until the end of August then!

[eric-aldinger]

@elisiano can you use a module specific version three hiera file for this module and retain your version five file at the global scope for everything else?

[elisiano]

How would I go about that?

[petems]

Ok, been a bit busy recently, but a colleague (the awesome @LMacchi) has made a hiera 5 backend for MSSQL, so it should be fairly easy to re-adapt for consul! 😄

https://github.com/LMacchi/hiera_mssql

Reminder 👍

[eric-aldinger]

That is very nice work. Makes me want to steal it and make one for PostgreSQL.

Is the change to the Consul backed module just the following?

1. Change the rakefile to add require 'puppet-syntax/tasks/puppet-syntax')
2. Add the puppetserver_gem shown in the MySql .fixtures.yml into your gemfile
3. Changing your puppet gem version from
   gem "puppet", ENV['PUPPET_VERSION'] || '~> 3.2.0'
   to
   gem "puppet", ENV['PUPPET_VERSION'] || '~> 5.2.0'

[devoncustard]

Any movement on this? I tried using the mongodb example for translating to a hiera3_backend config but no joy.

[devoncustard]

Small update.

I got the hiera3_backend approach working after noticing that the path value should existing in the options hash.

I went this route because i wanted to use 4.10 with the new embedded eyaml feature at the global level which meant a version 5 hiera config format

---
version: 5

defaults:
  datadir: /etc/puppetlabs/code/hieradata
  data_hash: yaml_data


hierarchy:
  - name: Consul
    hiera3_backend: consul
    options:
      host: consul.cha.rbxd.ds
      port: 8079
      use_ssl: false
      token: "these are not the tokens you are looking for"
      paths:
        - "/v1/kv/hiera/%{::team}/%{::environment}/%{::product}_%{::version}"
  - name: Yaml
    paths:
      - "%{::team}/%{::environment}/%{::role}.yaml"
      - "%{::team}/%{::environment}/%{::product}_%{::version}.yaml"
      - "%{::team}/%{::environment}/%{::product}.yaml"
      - "%{::team}/%{::environment}/config.yaml"
      - "secure/%{::environment}/%{::product}.yaml"
      - "secure/%{::environment}/config.yaml"
      - "common/%{::environment}/config.yaml"
      - "common/all/%{facts.domain}.yaml"
      - "common/all/%{facts.network}.yaml"
      - "common/all/config.yaml"

[lynxman]

This looks like a good workaround, mind if I update the docs with this?

[devoncustard]

Sure go for it.

[devoncustard]

This still needs some testing, im getting weird results, puppet lookup works on the master. the catalog compiles on the master, but when i try and run puppet agent -t on the node i get a message saying lookup cannot find the value. And there are lookup errors in the puppet logs....

[LMacchi]

I can help a bit with that @devoncustard.

puppet lookup and puppet apply use ruby to run, so you need to install the proper gems with system/vendor ruby (gem install). Puppet agent runs use jruby, since the puppetserver runs in jruby. In order to get the gems to the puppetserver, you'll do puppetserver gem install and restart (pe-)puppetserver.

Ruby and jruby are quite different creatures, so what works for one might not work for the other. Puppetserver irb might help you with that part.

Finally, you might run into needing different libraries for ruby and jruby, that's what happens with mssql. You can create code blocks for each library and then call the relevant one like this: https://github.com/LMacchi/hiera_mssql/blob/master/lib/puppet/functions/mssql_lookup_key.rb#L4

L.

[devoncustard]

@LMacchi ok ill have a look at that but given that it works with a standard Hiera 3 config like the one below im confused why the hiera 5 config doesnt work. I didnt think i needed to install any gems for hiera-consul ......

Like i said if i run puppet lookup in the node context on the master all is fine. Your comments about ruby and jruby make a lot of sense.

My original issue was trying to get hiera 5/eyaml and consul all working together in the global tier. Looks like ive managed to do that with the version 3 file. Probably the full reinstall of puppet cleared out some funky config and its all behaving now.

Ill probably try and solve this again after i take a timeout, bashed my head against a brick wall for hours over this today and its working with the version 3 config.

---

:backends:

• consul
• yaml
• eyaml

:hierarchy:

• "%{::team}/%{::environment}/%{::role}"
• "%{::team}/%{::environment}/%{::product}_%{::version}"
• "%{::team}/%{::environment}/%{::product}"
• "%{::team}/%{::environment}/config"
• "secure/%{::environment}/%{::product}"
• "secure/%{::environment}/config"
• "common/%{::environment}/config"
• "common/all/%{facts.domain}"
• "common/all/%{facts.network}"
• "common/all/config"

:yaml:
:datadir: "/etc/puppetlabs/code/hieradata"

:consul:
:host : consul.cha.rbxd.ds
:port : 8079
:use_ssl : false
:token : "nothing to see here, move along"
:paths :
- /v1/kv/hiera/%{::team}/%{::environment}/%{::product}_%{::version}

:eyaml:
:datadir: '/etc/puppetlabs/code/hieradata'
:pkcs7_private_key: /etc/puppetlabs/eyaml/private_key.pkcs7.pem
:pkcs7_public_key: /etc/puppetlabs/eyaml/public_key.pkcs7.pem

[iwagnerclgx]

I've been working on this a bit. I'd like to split up the lookup's up to support lookup_key (i.e == string), and data_hash (i.e. return consul json as hash).

I also think the services and nodes should be separated into a separate lookup function to make it easier to prevent collisions. This would require a major version bump since the API would change.

hierarchy:
  - name: "Consul Node"
    lookup_key: consul_lookup_node  
    options:
      host: 192.168.32.134
      port:  8080
      key_prefix : ::consul_node::

lookup('::consul_node::localhost') would query /v1/catalog/node/localhost, and return the hash for that node. Similarly for services, while still being compatible with the consul_info helper.

Thoughts?

[lynxman]

Hey @iwagnerclgx could you show some examples of that? I like it but it might break current implementations as well

[petems]

There's a vault Hiera 5 function for Vault by @davealden, I assume the code would be fairly similar: https://github.com/davealden/hiera-vault

[iwagnerclgx]

I have consul_lookup_key and consul_datahash implemented in https://github.com/iwagnerclgx/hiera-consul/tree/develop/lib/puppet/functions. I haven't done a lot of testing, so ymmv. I made these two only compatible with kv.

For the consul_lookup_service example, the function would only trigger if key_prefix was in the lookup string. So for instance, doing a lookup('::consul_service::webserver::') would lookup http://consul/v1/catalog/service/webserver. Similar for the other catalog items.

[davehewy]

Is anybody still looking at this? I'd like to get the docs updated to include a hiera 5 example. I can have a stab if its not already done?

[petems]

I've been using https://github.com/davealden/hiera-vault for Vault Hiera 5 integrations, and the implmentation would be fairly similar to Consul from looking if that code if it helps 😄

[lynxman]

Sorry, work has been crazy these last months, will look into it ASAP, if someone wants to have a go at it as well please feel free to.
Might as well transfer the code base to voxpopuli to ensure rolling support

[lynxman]

I've started working on this today :)

[devoncustard]

much as i love the idea of a dedicated consul backend for hiera 5, the lack of movement (and my own inability to write ruby) has led me to an alternative solution.

Using crayfishx/http_hiera works like charm. Kinda ironic as its gone full circle. took me about an hour to get this all working (and i can be quite dumb n slow at times)

I used this sample hiera.yaml to get it working. The KEY interpolation is really nice, and i havent looked too hard at the DIG functionality but it might make bulk config a lot easier.....

This will definitely get you working with consul and hiera 5.

---

version: 5

hierarchy:

• name: "Consul"
  lookup_key: hiera_http
  uris:
  • "http://consul.benifold.local:8500/v1/kv/__KEY__?raw"
    options:
    output: plain
    failure: graceful
    headers:
    X-Consul-Token: secretsauce

[negast]

Sorry to advertise here but as there have been few updates I made a consul lookup compatible with hiera 5 hiera_consul. The workings are a bit different I wrote about it in my readme. I did use some code from hiera-http though.