Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to properly handle failed SSM parameter fetching #44

Closed
thenano opened this issue Dec 7, 2023 · 5 comments
Closed

How to properly handle failed SSM parameter fetching #44

thenano opened this issue Dec 7, 2023 · 5 comments

Comments

@thenano
Copy link

thenano commented Dec 7, 2023

Hi,

First of all thank you very much for the work put into this, it was a crucial component of moving some of my workloads into lambda, and it works like a charm!
I've recently started experiencing intermittent lambda executions where my crypteia env variables don't get swapped out for the SSM secrets. I haven't dug into the root cause yet, but I assume something is failing when fetching the secrets from SSM, so I was wondering if there is some agreed upon way to handle these scenarios.
My use case is running lamby rails jobs that get triggered via an sqs queue (via lambdakiq). Sometimes the jobs will execute and I'll get errors because I'll just have x-crypteia-ssm:.. in my envs instead of the secrets. To make matters worse, I'm pretty sure that when facing high loads lambda copies over my jobs and I get more failures because they don't seem to startup and try to load the vars again.
I'm wondering if there's some way I can just kill the execution of the container straight away, and then my queue retry mechanisms will kick in and execute the job again?
Any help here is greatly appreciated, and apologies if this isn't the right place to post this question.

Thanks!

@jeremiahlukus
Copy link
Contributor

I think this should do the same as ssm and fail on deploy if unable to fetch a param.

@thenano
Copy link
Author

thenano commented Jun 12, 2024

@jeremiahlukus wow thanks so much for the fix!
is there change needed to our dockerfiles to get this update?

@metaskills
Copy link
Member

There will be but the build systems have to get fixed and we will need to do a preview release with testing to make sure everything works.

@jeremiahlukus
Copy link
Contributor

@thenano you can follow
#48

for updates. You can expect it to be tested and released this week though

@thenano
Copy link
Author

thenano commented Jun 13, 2024

Thank you so much @jeremiahlukus @metaskills

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants