What determines when a new scan job should be run?

[mvahlberg]

Hey,

We've implemented starboard operator v0.9.0 in our Kubernetes cluster, running with default settings except for OPERATOR_TARGET_NAMESPACES which we've set to empty string to scan all our namespaces. However, we're only receiving scans for 3 pods and nothing more.

Is this expected? Tried to edit the setting OPERATOR_CONCURRENT_SCAN_JOBS_LIMIT which default is set to the value of 3. I edited this and added a new value of 35. This resulted in that we got 35 scan jobs completed but after those jobs finished, nothing new started. My thought about starboard was that if we have the default value for OPERATOR_CONCURRENT_SCAN_JOBS_LIMIT (which is 3) - starboard-operator should initiate new jobs once the old once are finished?

I also have a question regarding when the next scan should occur? There is no point if we only scan once. Is there any kind of setting for this?

Hope someone can help me. If I solve this I will make PR to update documentation.

Best regards
🇸🇪

Edit: It seems like starboard-operator never deletes the completed jobs, therefore it is unable to initiate new jobs. Any suggestions?

[danielpacak]

👋 @mvahlberg To help you out I have couple of questions:

• What is the status of scan jobs that are created in the operator's namespace? (k get job -n )
• Do you see any errors in the operator logs?
• Do you see any VulnerabilityReport and / or ConfigAuditReport created in any namespace? (k get vuln,configaudit -A)

The operator scans existing workloads that do not have vulnerability reports. It also rescans a workloads whenever its deployment descriptor has changed. We haven't implemented rescanning based on regular intervals yet, but this is on our roadmap.

The operator should delete successfully completed scan jobs, but it may not delete failed ones, which would be a bug. I'm double checking that, but please confirm on your end.

[ilyassikai]

Hello,

"The operator scans existing workloads that do not have vulnerability reports. It also rescans a workloads whenever its deployment descriptor has changed. We haven't implemented rescanning based on regular intervals yet, ### but this is on our roadmap."

There is any news about this features.
Thanks

[chen-keinan]

@ilyassikai starboard project is in maintenance mode only , all activity has been moved to trivy-opetator project

[ilyassikai]

Thank you @chen-keinan for you reply,
this featue is available on trivy-opetator ?
I mean : scanning based on regular intervals because if I've a deployment who don't chnage for 1 year I want scan it.
In general I would like to scan deployment periodically and not wait the next deployement

Thanks

[chen-keinan]

@ilyassikai in trivy-operator we introduced TTL (configurable) to each report (24h by default) , meaning your workload will be scanned again every day.

I hope this is what you need

[ilyassikai]

exactly thank's

[danielpacak]

I played a bit and I was able to reproduce similar scenario when Starboard scans so called static pods, i.e. pods created by kubelet, which are owned by a Kubernetes Nodes rather than a replication controller (ReplicaSet, ReplicationController), DaemonSet or StatefulSet.

For example, in the listing below you can see the scan job that failed due to #234

$ kubectl get job -A --show-labels
NAMESPACE            NAME                                  COMPLETIONS   DURATION   AGE   LABELS
starboard-operator   scan-vulnerabilityreport-5f8cc7d8d8   1/1           11s        26m   app.kubernetes.io/managed-by=starboard-operator,pod-spec-hash=6779cdb7c7,starboard.resource.kind=DaemonSet,starboard.resource.name=kindnet,starboard.resource.namespace=kube-system,vulnerabilityReport.scanner=true
starboard-operator   scan-vulnerabilityreport-78fcf7d8b    1/1           6s         26m   app.kubernetes.io/managed-by=starboard-operator,pod-spec-hash=7c57456746,starboard.resource.kind=Node,starboard.resource.name=kind-control-plane,starboard.resource.namespace=kube-system,vulnerabilityReport.scanner=true

I'm currently working on the fix for #234 but in the meantime could you check what's the kind of resources for which you saw failed scan jobs? You could do that by displaying labels that we add to each scan job, e.g.:

$ kubectl get job -n starboard-operator -L starboard.resource.kind -L starboard.resource.namespace

[danielpacak]

Interesting. I have a clue why the scan job owned by the Workflow (Kind) cannot be processed by the operator. It's probably due to this code where we resolve the owner reference. Starboard is not aware of the Workload kind / CRD and we should probably handle this case in a more generic way.

Can you see in the logs of the operator error messages similar to unknown workload kind: Workflow ?
Also if you could share a snippet of logs that refer to the other two jobs that are associated with ReplicaSet that would help me a lot.

[mvahlberg]

Yes, in the logs I can find the error message unknown workload kind: Workflow but the job is still listed as completed. As with the other jobs, they are also listed as Completed. I understand that if the jobs were to be listed as 0/1 completed, starboard-operator would not delete these jobs but since all our jobs is listed as 1/1 Completed and we can view the vulnerability report - should the operator not delete these jobs?

[danielpacak]

Normally completed and failed jobs are deleted by the operator programmatically. However, due to the bug in release v0.9.0, for some completed jobs it might not be the case. For example, when we scan pods controlled by Argo Workflow the operator cannot resolve the Workload kind and fails: https://github.com/aquasecurity/starboard/blob/main/pkg/operator/controller/owner_resolver.go#L40

I created #373 to follow up on Argo / Workflow, but I'm still wondering why two other jobs in your example, which are owned by ReplicaSet, where not deleted.

[mvahlberg]

I see. The funny thing though is that the scan job for Argo Workflow did not fail, it was completed.

[danielpacak]

The job completed, but Operator failed persisting VulnerabilityReport owned by Argo Workflow due to unknown kind error. Scan jobs and Starboard Operator run as independent processes in their own containers. Therefore, to troubleshoot issues like that we need logs / status of scan jobs as well as logs of the operator itself.

That said, do you see any errors related to jobs scan-vulnerabilityreport-74c7f4c5c9 and scan-vulnerabilityreport-9fbb5cfdf in the operator logs, i.e. k logs -n starboard-operator deploy/starboard-operator?