Potential race condition.

[daniel-at-matt3r]

We have vault cluster - 3 nodes, behind LB for 8200 traffic. Port 8201 is NOT part of the traffic balancing, servers are talking directly to each-other. We have the following consul-template scenario:
<------------------------------------>
{{- $tls_dir := "" }}
{{- $tls_ca_file := "" }}
{{- $tls_cert_file := "" }}
{{- $tls_key_file := "" }}

tls {
{{- with secret "kv_hyades/data/nomad/config/main/tls" }}
{{- $tls_http := .Data.data.http | sprig_default "false" }}
{{- $tls_rpc := .Data.data.rpc | sprig_default "false" }}
{{- $tls_dir = .Data.data.dir | sprig_default "/etc/nomad.d/tls" }}
{{- $tls_ca_file = sprig_list $tls_dir (sprig_default "ca.pem" .Data.data.ca_file) | sprig_join "/"}}
{{- $tls_cert_file = sprig_list $tls_dir (sprig_default "cert.pem" .Data.data.cert_file) | sprig_join "/"}}
{{- $tls_key_file = sprig_list $tls_dir (sprig_default "key.pem" .Data.data.key_file) | sprig_join "/"}}
http = {{ $tls_http }}
rpc = {{ $tls_rpc }}
{{ if or (eq $tls_http "true") (eq $tls_rpc "true") }}
ca_file = {{$tls_ca_file | sprig_quote }}
cert_file = {{$tls_cert_file | sprig_quote}}
key_file = {{$tls_key_file | sprig_quote}}
{{ end -}}
{{ end }}
}

{{- with secret "kv_hyades/data/nomad/config/main/tls/pem"}}
{{- $ca_pem := .Data.data.ca_pem | sprig_default "CA is not defined"}}
{{- $cert_pem := .Data.data.cert_pem | sprig_default "Certificate is not defined"}}
{{- $key_pem := .Data.data.key_pem | sprig_default "Private key is not defined"}}
{{- $ca_pem | writeToFile $tls_ca_file "root" "root" "0644" }}
{{- $cert_pem | writeToFile $tls_cert_file "root" "root" "0644" }}
{{- $key_pem | writeToFile $tls_key_file "root" "root" "0600" }}
{{- end }}
<------------------------------------->

periodically rendering is failing with the following error:

2025-01-23T18:06:28.185Z [DEBUG] (runner) was not watching 2 dependencies
2025-01-23T18:06:28.185Z [DEBUG] (watcher) adding vault.read(kv_hyades/data/nomad/config/main/tls)
2025-01-23T18:06:28.185Z [DEBUG] (watcher) adding vault.read(kv_hyades/data/nomad/config/main/tls/pem)
2025-01-23T18:06:28.185Z [DEBUG] (runner) diffing and updating dependencies
2025-01-23T18:06:28.185Z [DEBUG] (runner) watching 2 dependencies
2025-01-23T18:06:28.249Z [DEBUG] (runner) receiving dependency vault.read(kv_hyades/data/nomad/config/main/tls/pem)
2025-01-23T18:06:28.250Z [DEBUG] (runner) initiating run
2025-01-23T18:06:28.250Z [DEBUG] (runner) checking template 24b2bbee594237e7a1701138b392c14e
2025-01-23T18:06:28.251Z [ERR] (cli) 111.hcl.tpl: execute: template: :33:14: executing "" at <writeToFile $tls_ca_file "root" "root" "0644">: error calling writeToFile: open : no such file or directory

but scenario:
<--------------------------------------->
{{- $tls_dir := "" }}
{{- $tls_ca_file := "" }}
{{- $tls_cert_file := "" }}
{{- $tls_key_file := "" }}

tls {
{{- with secret "kv_hyades/data/nomad/config/main/tls" }}
{{- $tls_http := .Data.data.http | sprig_default "false" }}
{{- $tls_rpc := .Data.data.rpc | sprig_default "false" }}
{{- $tls_dir = .Data.data.dir | sprig_default "/etc/nomad.d/tls" }}
{{- $tls_ca_file = sprig_list $tls_dir (sprig_default "ca.pem" .Data.data.ca_file) | sprig_join "/"}}
{{- $tls_cert_file = sprig_list $tls_dir (sprig_default "cert.pem" .Data.data.cert_file) | sprig_join "/"}}
{{- $tls_key_file = sprig_list $tls_dir (sprig_default "key.pem" .Data.data.key_file) | sprig_join "/"}}
http = {{ $tls_http }}
rpc = {{ $tls_rpc }}
{{ if or (eq $tls_http "true") (eq $tls_rpc "true") }}
ca_file = {{$tls_ca_file | sprig_quote }}
cert_file = {{$tls_cert_file | sprig_quote}}
key_file = {{$tls_key_file | sprig_quote}}
{{ end -}}
{{ end }}
}

{{ if and $tls_ca_file $tls_cert_file $tls_key_file }}
"Proceed with writing to files only after validation"
{{- with secret "kv_hyades/data/nomad/config/main/tls/pem"}}
{{- $ca_pem := .Data.data.ca_pem | sprig_default "CA is not defined"}}
{{- $cert_pem := .Data.data.cert_pem | sprig_default "Certificate is not defined"}}
{{- $key_pem := .Data.data.key_pem | sprig_default "Private key is not defined"}}

{{- $ca_pem | writeToFile $tls_ca_file "root" "root" "0644" }}
{{- $cert_pem | writeToFile $tls_cert_file "root" "root" "0644" }}
{{- $key_pem | writeToFile $tls_key_file "root" "root" "0600" }}

{{ else }}
"Required TLS variables are missing or incomplete."
{{ end }}
{{- end }}
<--------------------------->

works all the time, and it creates files with certs/keys.

Looks like introducing IF block surrounding second WITH secret block is stopping race condition, and consul-template is getting properly defined vars - $tls_ca_file, $tls_cert_file and $tls_key_file