[PENG-2342] Jobbergate Agent continually resubmits the same job if the job status update fails

[fschuch]

What

Create a local cache of job submissions for the jobbergate agent.:

• Check the local cache to see if a job submission was already dispatched to Slurm
• If the job was not already dispatched:
  • submit the job to Slurm
  • Create an entry in the cache to indicate that the job was submitted
• Attempt to update the Job Submission in the Jobbergate API
• If the update was successful delete the entry in the cache

Why

We've found a situation where the jobbergate-agent resubmits a job constantly to slurm despite the job having been successfully submitted.
After the Jobbergate Agent successfully submits a pending job submission to slurm, it then attempts to update the status of the job in the Jobbergate API to indicate that the job was submitted. However, if the call to update the job submission in the Jobbergate API fails for any reason, the job_submission will be left in the pending status. This means that in the next cycle of the Jobbergate Agent, it will still see the job as if it had never been submitted and will try to submit the job again.
Task: https://app.clickup.com/t/18022949/PENG-2342

---

Peer Review

Please follow the upstream omnivector documentation concerning
peer-review guidelines.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.17%. Comparing base (1223c89) to head (e205c72).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #607      +/-   ##
==========================================
+ Coverage   92.16%   92.17%   +0.01%     
==========================================
  Files          83       83              
  Lines        4417     4423       +6     
==========================================
+ Hits         4071     4077       +6     
  Misses        346      346              

Flag	Coverage Δ
agent	93.48% <ø> (+0.06%)	⬆️
api	95.09% <ø> (ø)
cli	87.70% <ø> (ø)
core	96.32% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dusktreader]

Nice work! I added one very minor comment that you may choose to ignore if you wish.