Fixes 2474: Recover from tasks panics

[jlsherrill]

Summary

Previously if a task paniced for some reason, the worker routine would get 'stuck' and refuse to process anything else. Now the task is marked as failed and the worker can continue to pick up another task.

Testing steps

modify the pkg/taskss/introspection.go to panic sometimes or all the time:

	if rand.Int()%2 == 0 {
		panic(fmt.Errorf("You are unlucky!!"))
	}

Create some repository with with snapshotting turned off (for simplicity)

curl -X POST --location "http://localhost:8000/api/content-sources/v1.0/repositories/" \
    -H "x-rh-identity: eyJpZGVudGl0eSI6eyJ0eXBlIjoiVXNlciIsInVzZXIiOnsidXNlcm5hbWUiOiJqdXN0aW4ifSwiYWNjb3VudF9udW1iZXIiOiIxMTQ4NjU4OSIsImludGVybmFsIjp7Im9yZ19pZCI6IjE2Nzg4NjE2In19fQo=" \
    -H "Content-Type: application/json" \
    -d "{
          \"name\": \"errata\",
          \"url\": \"https://jlsherrill.fedorapeople.org/fake-repos/needed-errata/\",
          \"snapshot\": false
        }"

Now trigger introspection:

 OPTIONS_ALWAYS_RUN_CRON_TASKS=true go run cmd/external-repos/main.go nightly-jobs

Run this a few times. eventually your workers will stop processing tasks. With this PR, they should always recover and mark the task as failed. You can check the tasks table' status column to confirm they are marked as failed.

Checklist

• Tested with snapshotting feature disabled and pulp server URL not configured if appropriate

[jlsherrill]

https://issues.redhat.com/browse/HMS-2474

[jlsherrill]

/retest

[jlsherrill]

/retest

[rverdile]

nice!

[jlsherrill]

merging as this does not require qe and the failing backend test is unrelated