Concat 8.0.0 / Puppet 8.0 - Deferred function calls incompatibility

[fetzerms]

Describe the Bug

Concat v8 (and also prior) does not play well with Deferred function calls in Puppet 8. They do not get properly evaluated when using them as a content parameter. I first encountered this issue at voxpupuli/puppet-gitlab_ci_runner#167 - but tracked it down to concat / puppet v8.

Expected Behavior

Concat should be able to properly use Deferred function calls.

Steps to Reproduce

With docker.

Filename: Dockerfile-8

FROM debian:bullseye
RUN apt-get update && apt-get install -y wget
RUN wget https://apt.puppetlabs.com/puppet8-release-bullseye.deb
RUN dpkg -i puppet8-release-bullseye.deb
RUN apt-get update && apt-get install -y puppet-agent
RUN /opt/puppetlabs/bin/puppet module install puppetlabs-concat --version 8.0.1
RUN /opt/puppetlabs/bin/puppet apply -e "concat { '/tmp/file': ensure => present, } concat::fragment { 'tmpfile': target  => '/tmp/file', content => Deferred('md5', ['Hello']), order   => '01'}"

Commands:
docker build -f Dockerfile-8 -t concat8_puppet8 . && docker run -it concat8_puppet8 cat /tmp/file

Output:

[+] Building 13.1s (11/11) FINISHED                                                                                                                                                                                                                                                                                                                                        
 => [internal] load build definition from Dockerfile-8                                                                                                                                                                                                                                                                                                                0.0s
 => => transferring dockerfile: 543B                                                                                                                                                                                                                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/debian:bullseye                                                                                                                                                                                                                                                                                                    0.5s
 => [1/7] FROM docker.io/library/debian:bullseye@sha256:63d62ae233b588d6b426b7b072d79d1306bfd02a72bff1fc045b8511cc89ee09                                                                                                                                                                                                                                              0.0s
 => CACHED [2/7] RUN apt-get update && apt-get install -y wget                                                                                                                                                                                                                                                                                                        0.0s
 => CACHED [3/7] RUN wget https://apt.puppetlabs.com/puppet8-release-bullseye.deb                                                                                                                                                                                                                                                                                     0.0s
 => CACHED [4/7] RUN dpkg -i puppet8-release-bullseye.deb                                                                                                                                                                                                                                                                                                             0.0s
 => CACHED [5/7] RUN apt-get update && apt-get install -y puppet-agent                                                                                                                                                                                                                                                                                                0.0s
 => [6/7] RUN /opt/puppetlabs/bin/puppet module install puppetlabs-concat --version 8.0.1                                                                                                                                                                                                                                                                             5.7s
 => [7/7] RUN /opt/puppetlabs/bin/puppet apply -e "concat { '/tmp/file': ensure => present, } concat::fragment { 'tmpfile': target  => '/tmp/file', content => Deferred('md5', ['Hello']), order   => '01'}"                                                                                                                                                          5.9s
 => exporting to image                                                                                                                                                                                                                                                                                                                                                1.0s 
 => => exporting layers                                                                                                                                                                                                                                                                                                                                               1.0s 
 => => writing image sha256:0b0da3d09eb189bc665bcdcb5b56b5a6f23945a64201a4c216a6d5a78e5beed8                                                                                                                                                                                                                                                                          0.0s 
 => => naming to docker.io/library/concat8_puppet8                                                                                                                                                                                                                                                                                                                    0.0s 
#<Puppet::Pops::Evaluator::DeferredValue:0x00007fbc00e1c300>

Environment

• Version: 8.0.0
• Puppet Agent 8.0 on Debian Bullseye

Additional Context

I tried the same with puppet 7, which yields the correct result:

Dockerfile-7:

FROM debian:bullseye
RUN apt-get update && apt-get install -y wget
RUN wget https://apt.puppetlabs.com/puppet7-release-bullseye.deb
RUN dpkg -i puppet7-release-bullseye.deb
RUN apt-get update && apt-get install -y puppet-agent
RUN /opt/puppetlabs/bin/puppet module install puppetlabs-concat --version 8.0.1
RUN /opt/puppetlabs/bin/puppet apply -e "concat { '/tmp/file': ensure => present, } concat::fragment { 'tmpfile': target  => '/tmp/file', content => Deferred('md5', ['Hello']), order   => '01'}"

docker build -f Dockerfile-7 -t concat8_puppet7 . && docker run -it concat8_puppet7 cat /tmp/file
Output:

[+] Building 0.5s (11/11) FINISHED                                                                                                                                                                                                                                                                                                                                         
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                       0.0s
 => [internal] load build definition from Dockerfile-7                                                                                                                                                                                                                                                                                                                0.0s
 => => transferring dockerfile: 543B                                                                                                                                                                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/debian:bullseye                                                                                                                                                                                                                                                                                                    0.5s
 => [1/7] FROM docker.io/library/debian:bullseye@sha256:63d62ae233b588d6b426b7b072d79d1306bfd02a72bff1fc045b8511cc89ee09                                                                                                                                                                                                                                              0.0s
 => CACHED [2/7] RUN apt-get update && apt-get install -y wget                                                                                                                                                                                                                                                                                                        0.0s
 => CACHED [3/7] RUN wget https://apt.puppetlabs.com/puppet7-release-bullseye.deb                                                                                                                                                                                                                                                                                     0.0s
 => CACHED [4/7] RUN dpkg -i puppet7-release-bullseye.deb                                                                                                                                                                                                                                                                                                             0.0s
 => CACHED [5/7] RUN apt-get update && apt-get install -y puppet-agent                                                                                                                                                                                                                                                                                                0.0s
 => CACHED [6/7] RUN /opt/puppetlabs/bin/puppet module install puppetlabs-concat --version 8.0.1                                                                                                                                                                                                                                                                      0.0s
 => CACHED [7/7] RUN /opt/puppetlabs/bin/puppet apply -e "concat { '/tmp/file': ensure => present, } concat::fragment { 'tmpfile': target  => '/tmp/file', content => Deferred('md5', ['Hello']), order   => '01'}"                                                                                                                                                   0.0s
 => exporting to image                                                                                                                                                                                                                                                                                                                                                0.0s
 => => exporting layers                                                                                                                                                                                                                                                                                                                                               0.0s
 => => writing image sha256:7cc5bc2cd4ddf414f17111559facd0b8d8c4a54c923475af4a30812a27c73a6e                                                                                                                                                                                                                                                                          0.0s
 => => naming to docker.io/library/concat8_puppet7                                                                                                                                                                                                                                                                                                                    0.0s
8b1a9953c4611296a827abf8c47804d7

[fetzerms]

I'm not sure if checking here ( https://github.com/puppetlabs/puppetlabs-concat/blob/main/lib/puppet/type/concat_file.rb#L299 ) whether the content is a deferred function makes sense...

[fetzerms]

Update: The problem still persists with 8.0.1

[LukasAud]

Will need further investigation. We are, however, a bit overloaded currently trying to cut releases for modules. Next concat release (v8.1.0) should be cut very soon and will contain compatibility for Stdlib 9.x.x. I hope that will have some sort of impact on this issue. If not, I will come back to investigate further on the root of the problem.

[fetzerms]

@LukasAud issue persists with concat 9.0.0 / stdlib 9.1.0 and puppet-agent 8.1.0

[smoeding]

This bug also causes failures with the official apache module on Puppet 8.
Line 2629 in manifests/vhost.pp contains a Concat fragment to include Kerberos configuration using a deferred variable that also stopped working after updating to Puppet 8.

[fetzerms]

@smoeding can you check whether it works again with the changes that I proposed in my pull request?

[smoeding]

@fetzerms I manually applied your commit c7ffb95 and used Puppet 8 to run our otherwise unmodified code. With that commit there are no changes to the Apache config compared to what Puppet 7 has deployed.

So I can confirm that your commit fixes the problem I have observed.

[fetzerms]

@LukasAud can you look into my PR? I think I found the issue, but wonder if I'm now just working around some puppet bug, as it used to work with Puppet 7.

[smoeding]

@LukasAud This bug currently prevents me from upgrading my Apache nodes to Puppet 8.x. Is there any progress in finding the root of the problem here?

[fetzerms]

With 9.0.1 (which includes my workaround) this is fixed for me.