Persistent files in file_cache_path

[StephenKing]

I just figured out why one of my cookbooks didn't work well in CI (detecting that a cookbook_file wasn't changed).

This file was written to Chef::Config[:file_cache_path], which ends up on the host system in ~/.dokken/kitchen_sandbox/<unique_prefix>-<suitename>/cache and stays there between Chef runs - which was unexpected by my cookbook's execution.

Was it a bad idea to put a file there, which should not reflect the state from the previous chef run and I should put it instead to somewhere else? Or is this a result of a non-ideal implementation in kitchen-dokken? I understand that caching temp files there has a positive effect..

(to give you a bit of context: I'm writing the plugins + version of Jenkins plugin into that file, which allows me to decide, whether Jenkins needs to restart (code) - result is that Jenkins won't be restarted after plugin installations).

[StephenKing]

Having fixed that one issue resulted in the next one.. a template (for a Jenkins job) which still existed in the file_cache_path didn't notify the jenkins_job resource to trigger job creation.

Over the years, I got use take Chef::Config[:file_cache_path] as the default path for storing temp files. I never had the impression that I'm doing it wrong, so I more and more think that kitchen-dokken's approach here is is "bad".

[someara]

This is a consequence of bind mounting the sandbox... on a failed kitchen converge, test-kitchen never gets a chance to clean up like it would during kitchen-destroy.

I'll investigate the best way to handle this

[StephenKing]

Thanks for your feedback. My CI system runs kitchen test --destroy always and the data still persists.

[someara]

is this still an issue for you in 2.6.5?

[joerg]

Hi,

I am still seeing this in 2.6.5. Issues #120 pretty sure is related to this.

joerg@server ..s/chef-repo/cookbooks/some-jenkins (git)-[master] % chef --version
Chef Development Kit Version: 2.4.17
chef-client version: 13.6.4
delivery version: master (73ebb72a6c42b3d2ff5370c476be800fee7e5427)
berks version: 6.3.1
kitchen version: 1.19.2
inspec version: 1.45.13
joerg@server ..s/chef-repo/cookbooks/some-jenkins (git)-[master] % chef exec gem list --local | grep dokken
kitchen-dokken (2.6.5)

joerg@server ..s/chef-repo/cookbooks/some-jenkins (git)-[master] % kitchen destroy
-----> Starting Kitchen (v1.19.2)
-----> Destroying <default-rhel-69>...
       Deleting kitchen sandbox at /home/joerg/.dokken/kitchen_sandbox/9e416333ef-default-rhel-69
       Deleting verifier sandbox at /home/joerg/.dokken/verifier_sandbox/9e416333ef-default-rhel-69
       Finished destroying <default-rhel-69> (0m1.24s).
-----> Destroying <default-rhel-74>...
       Deleting kitchen sandbox at /home/joerg/.dokken/kitchen_sandbox/9e416333ef-default-rhel-74
       Deleting verifier sandbox at /home/joerg/.dokken/verifier_sandbox/9e416333ef-default-rhel-74
       Finished destroying <default-rhel-74> (0m10.79s).
-----> Kitchen is finished. (0m13.85s)
joerg@server ..s/chef-repo/cookbooks/some-jenkins (git)-[master] % du -sch ~/.dokken/kitchen_sandbox/*
51M     /home/joerg/.dokken/kitchen_sandbox/9e416333ef-default-rhel-69
53M     /home/joerg/.dokken/kitchen_sandbox/9e416333ef-default-rhel-74
104M    total

Edit: I am pretty sure this line is the problem here: https://github.com/someara/kitchen-dokken/blob/2ed012ecceb114db1da71b29622bc996a77ab1f6/lib/kitchen/helpers.rb#L135
Files in the sanbox are partially owned by root and not the user which is why they can't just be deleted. Maybe try deleting the contents in the container with root privileges too. This line will then delete the sandbox directory itself.

[joerg]

@someara I was going over the code during the weekend and thinking about this issue and fixing it would not be too much effort but I fail at understanding why the sandbox is a bind mount from the host. Removing the bind mound and keeping the sandbox inside the container would remove a lot of complexity and manual work of cleaning up and such. Is there a reason why the sandbox is a bind mount?

[jsirex]

I see exactly opposite case: Each time I do kitchen converge few times, without destroying, I see that all my cached files being re-downloaded (looks like file_cache_path destroys on each converge).
Probably we need here just more control over this, like an options kitchen dokken driver. So everybody gets what he wants.

[joerg]

@jsirex Yes, thats why I propose having the cache inside the container and not taking care of it at all. Removing the docker container will automatically remove the cache and no cleanup or anything can intervene in a negative way.

[joerg]

I've been digging deeper into test-kitchen and dokken and found that the central difference here is that for test-kitchen (vagrant) the kitchen-sandbox and verifier-sandbox are created locally with user permissions and then uploaded (usually via SCP or rsync) to the VM while dokken mounts these sanboxes directly into the containers. This direclty leads to two issues:

• The Sandboxes are used by chef inside the containers with root permissions, hence these files can't be deleted by the user calling kitchen.
• The upload defined by kitchen is kind of misused by dokken to upload the sandboxes to docker remote hosts so these two are conceptually not doing the same things.

So even if we change the behavior from using the mounted sandbox to transferring the sandbox files via "docker cp" [1] or some similar mechnism there is still this upload to the remote docker host which has to be solved.

@someara Any thoughts on this. I can prepare a pull request but I would like to know which path to follow before I invest time in this.

[1] https://docs.docker.com/engine/reference/commandline/cp/#extended-description

[jsirex]

Any "call" ends up with sandbox_cleanup. This is bad idea. Whenever I developing cookbook I want to run converge multiple times. And each time it re-downloads all huge stuff.
https://github.com/someara/kitchen-dokken/blob/master/lib/kitchen/provisioner/dokken.rb#L61

Probably this should only call cleanup_dokken_sandbox at destroy time (quick fix) or removed at all.
I had to monkey patch it temporary:

      def cleanup_dokken_sandbox
        return if sandbox_path.nil? || ENV['KITCHEN_DONT_CLEANUP']

        debug("Cleaning up local sandbox in #{sandbox_path}")
        FileUtils.rmtree(Dir.glob("#{sandbox_path}/*"))
      end

[djessich]

I found this problem on my CI too. When the CI server calls kitchen destroy at the end of a CI build, the file(s) from kitchen do not get deleted properly. In my case, the next build is affected, as the "starting point" has changed. I can also confirm what @jsirex and @joerg said by manual testing: Multiple runs redownload all the huge artifacts, so it is exactly the opposite. @someara Any thoughts on this?

[alice-sawatzky]

@jsirex a workaround for your issue is to use a named volume, and download all your huge stuff to that volume's mountpoint, and use chef features like remote_file's checksum property to prevent re-downloading:

suites:
- name: my_suite
  driver:
    volumes:
    - "my_vol:/var/chef-downloads"
...

although if you're using multiple suites you'll have to watch out for this bug: #152

[djessich]

Is there any progress on this? Does the problem persist with newest version of kitchen-dokken?