retryDeleteJob causes restart job to be unscheduled

[GhangZh]

Description

Steps to reproduce the issue

1. first delete Podgroup and delete Pod triggered retryDeleteJob called a few times and was penalized with a delay of a few seconds
2. at this point the deleteJob was called directly out of the queue, removing the job from the jobcache
3. At this point, the job retry or re-create a job with the same UID.
4. Then after a few seconds the old job is queued up from the ratelimit queue, and at this time to determine the JobTerminated = true, it will be deleted from the job cache again!

Then the job will always exist and never be scheduled.

Describe the results you received and expected

Expect not to have to delete the cache repeatedly for this scenario

What version of Volcano are you using?

volcano 1.9

Any other relevant information

No response

[aryasoni98]

@GhangZh I would love to work on this issue. Could you please assign it to me?