Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support private container registries for Pods which do not specifying image pull secrets #120

Open
avdhoot opened this issue Aug 14, 2020 · 20 comments · May be fixed by #1199
Open

Support private container registries for Pods which do not specifying image pull secrets #120

avdhoot opened this issue Aug 14, 2020 · 20 comments · May be fixed by #1199
Assignees
Labels
🎨 design More about design and architecture than writing Go code 🙏 help wanted Extra attention is needed 🚀 enhancement New feature or request

Comments

@avdhoot
Copy link

avdhoot commented Aug 14, 2020

What steps did you take and what happened:

Followed install instructions. Tried to run scan using below on command.

$ kubectl starboard  find vulnerabilities deployment/XXX-qa-web -n XXXX-qa -v 3
I0814 12:52:38.530373   28112 scanner.go:56] Getting Pod template for workload: {Deployment XXXX-qa-web XXXX-qa}
I0814 12:52:40.842053   28112 scanner.go:71] Scanning with options: {ScanJobTimeout:0s DeleteScanJob:true}
I0814 12:52:41.183767   28112 runner.go:79] Running task and waiting forever
I0814 12:52:41.183840   28112 runnable_job.go:47] Creating runnable job: starboard/b75ba5e8-82c9-4915-ad35-4b35c37987ab
I0814 12:52:41.535929   28112 reflector.go:207] Starting reflector *v1.Job (30m0s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156
I0814 12:52:41.535978   28112 reflector.go:243] Listing and watching *v1.Job from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156
I0814 12:52:50.479003   28112 runnable_job.go:73] Stopping runnable job on task failure with status: Failed
I0814 12:52:50.479115   28112 runner.go:83] Stopping runner on task completion with error: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit
E0814 12:52:52.784556   28112 manager.go:177] Container default terminated with Error: 2020-08-14T07:22:49.629Z FATAL   unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:
        * unable to inspect the image (us.gcr.io/XXXX-1/XXXX:116579-23d73da-release-2019-10): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
        * GET https://us.gcr.io/v2/token?scope=repository%3xxxl-1%2FXXXX%3Apull&service=us.gcr.io: UNKNOWN: Unable to parse json key.


error: running scan job: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit

What did you expect to happen:
Scan should completed without error

Anything else you would like to add:
trivy support gcr. But I am not to able find way to pass custom ENV to trivy using starboard.

Environment:

  • Starboard version (use starboard version): Starboard Version: {Version:0.2.6 Commit:d43faefc56021ae55d4574054ce7de13175ca206 Date:2020-07-09T20:30:45Z}
  • Kubernetes version (use kubectl version): client:v1.17.10, server: v1.17.2
  • OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Ubuntu 18.04
@avdhoot
Copy link
Author

avdhoot commented Aug 19, 2020

@danielpacak In case you missed it ^^

@danielpacak
Copy link
Contributor

Hi @avdhoot ! Thank you for reporting this issue. I'm going to follow up on that and link it to a parent story or epic about configuring scanners. In your case the root cause is that we currently cannot pass TRIVY_USERNAME or TRIVY_PASSWORD envs, but there're more config options that users might want to pass to a scanner.

@danielpacak danielpacak self-assigned this Aug 20, 2020
@danielpacak danielpacak added the 🚀 enhancement New feature or request label Aug 20, 2020
@danielpacak
Copy link
Contributor

danielpacak commented Aug 31, 2020

@avdhoot Actually starboard can use image pull secret attached to the Pod template or the service account. In your case did you set the secret on the Pod template level or at the service account level?

@danielpacak danielpacak added this to the Release v0.3.0 milestone Aug 31, 2020
@danielpacak danielpacak added ⏳ additional info required Additional information required to close an issue and removed 🚀 enhancement New feature or request labels Aug 31, 2020
@avdhoot
Copy link
Author

avdhoot commented Aug 31, 2020

Our cluster created using kops. We seeded it with registry secrets using below command. We do not have image_pull secret attached to pod. @danielpacak
kops create secret dockerconfig -f ~/.docker/config.json

@danielpacak
Copy link
Contributor

danielpacak commented Sep 1, 2020

Thank you for providing all the details @avdhoot So just to rephrase and confirm my understanding:

You neither use image pull Secret set at Pod template not Service Account. You're using a feature1 of kops to store the config.json in /root/.docker/config.json on all nodes (including masters) so that both Kubernetes and system containers may use registries defined in it.

@avdhoot
Copy link
Author

avdhoot commented Sep 1, 2020

Thanks @danielpacak you are understanding is correct.

@danielpacak danielpacak removed this from the Release v0.3.0 milestone Sep 1, 2020
@danielpacak danielpacak added 🚀 enhancement New feature or request and removed ⏳ additional info required Additional information required to close an issue labels Sep 2, 2020
@danielpacak danielpacak changed the title private container registry support Support private container registries for Pods which do not specifying image pull secrets Sep 2, 2020
@danielpacak
Copy link
Contributor

@avdhoot I renamed the title of this issue to reflect the specific use case of kops and authorizing Kubernetes nodes to pull images from private registry and repositories.

As already mentioned Starboard only supports recommended way of handling private registries with image pull secrets.
We need to think more whether / how to support such cases as you described. /cc @lizrice

@n3rd253
Copy link

n3rd253 commented Nov 9, 2020

@danielpacak

I am trying to make use of the above discussed functionality within AKS. I am not finding that the operator pulls the imagePullSecret of the deployment. In my case I am exposing the secret as imagePullSecret on the deployment level.

Secret generation command as detailed here: ACR Auth Docs

kubectl create secret docker-registry acrcred --namespace platform --docker-server=aagacr.azurecr.io --docker-username=$(AagacrUser) --docker-password=$(AagacrKey)

ACR Cred Secret:

apiVersion: v1
data:
.dockerconfigjson: eyJhdXRocyI6eyJhYWdh[...]HSm1PR016WkRkak1UY3dPa2t5T0M1K09XRkZYMHhLZFM1UUxsY3ROMkptT1RZd1J6ZDJZa2QrVW5GM1RYRT0ifX19
kind: Secret
metadata:
creationTimestamp: "2020-11-09T18:02:46Z"
managedFields:
apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:.dockerconfigjson: {}
f:type: {}
manager: kubectl-create
operation: Update
time: "2020-11-09T18:02:46Z"
name: acrcred
namespace: platform
resourceVersion: "14763349"
selfLink: /api/v1/namespaces/platform/secrets/acrcred
uid: 886f8fca-4ae6-4a9a-a6b0-133a75d606af
type: kubernetes.io/dockerconfigjson`

Deployment Spec:

spec:
containers:
envFrom:
configMapRef:
name: pnrhelperservice-configmap
image: aagacr.azurecr.io/platform/pnrhelperservice:20201109.3
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
httpGet:
path: /api/v1/ping
port: 8080
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: platform-pnrhelperservice
ports:
containerPort: 8080
name: pnrhelpersvc
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /api/v1/ping
port: 8080
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 200m
memory: 64Mi
requests:
cpu: 100m
memory: 32Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
ALL
privileged: false
runAsNonRoot: true
runAsUser: 10000
startupProbe:
failureThreshold: 30
httpGet:
path: /api/v1/ping
port: 8080
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
mountPath: /tmp
name: tmp
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: svc-platform-pnrhelperservice-token-zjpdx
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
imagePullSecrets:
name: acrcred
nodeName: aks-apppool-15092464-vmss00000y
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 10000
runAsNonRoot: true
runAsUser: 10000
serviceAccount: svc-platform-pnrhelperservice
serviceAccountName: svc-platform-pnrhelperservice
terminationGracePeriodSeconds: 30
tolerations:
effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists
volumes:
emptyDir: {}
name: tmp
name: tls-platform-pnrhelperservice
secret:
defaultMode: 420
secretName: tls-platform-pnrhelperservice
name: acrcred
secret:
defaultMode: 420
secretName: acrcred
name: svc-platform-pnrhelperservice-token-zjpdx
secret:
defaultMode: 420
secretName: svc-platform-pnrhelperservice-token-zjpdx

Operator Logs:

Loading...
{"level":"info","ts":1604949460.8825052,"logger":"main","msg":"Resolving install mode","install mode":"SingleNamespace","operator namespace":"starboard-operator","target namespaces":["platform"]}
{"level":"info","ts":1604949460.8826096,"logger":"main","msg":"Constructing multi-namespaced cache","namespaces":["platform","starboard-operator"]}
{"level":"info","ts":1604949461.888167,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1604949461.888932,"logger":"main","msg":"Using Trivy as vulnerability scanner","image":"aquasec/trivy:0.11.0"}
{"level":"info","ts":1604949461.8890698,"logger":"main","msg":"Starting controllers manager"}
{"level":"info","ts":1604949461.8893642,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1604949461.8895195,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","source":"kind source: /, Kind="}
{"level":"info","ts":1604949461.8897321,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","source":"kind source: /, Kind="}
{"level":"info","ts":1604949462.0902774,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod"}
{"level":"info","ts":1604949462.0903811,"logger":"controller","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","worker count":1}
{"level":"info","ts":1604949462.0904198,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job"}
{"level":"info","ts":1604949462.090432,"logger":"controller","msg":"Starting workers","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","worker count":1}
{"level":"error","ts":1604949471.7839265,"logger":"controller.job","msg":"Scan job container","job":"starboard-operator/2b1b65b9-e131-4e13-9601-68fa1eb3cab6","container":"platform-pnrhelperservice","status.reason":"Error","status.message":"2020-11-09T19:17:51.225Z\t\u001b[31mFATAL\u001b[0m\tunable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t* unable to inspect the image (aagacr.azurecr.io/platform/pnrhelperservice:20201109.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* GET https://aagacr.azurecr.io/oauth2/token?scope=repository%3Aplatform%2Fpnrhelperservice%3Apull&service=aagacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\n\n\n","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).processFailedScanJob\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:163\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).Reconcile\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:69\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"}
{"level":"info","ts":1604949460.8825052,"logger":"main","msg":"Resolving install mode","install mode":"SingleNamespace","operator namespace":"starboard-operator","target namespaces":["platform"]}
{"level":"info","ts":1604949460.8826096,"logger":"main","msg":"Constructing multi-namespaced cache","namespaces":["platform","starboard-operator"]}
{"level":"info","ts":1604949461.888167,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1604949461.888932,"logger":"main","msg":"Using Trivy as vulnerability scanner","image":"aquasec/trivy:0.11.0"}
{"level":"info","ts":1604949461.8890698,"logger":"main","msg":"Starting controllers manager"}
{"level":"info","ts":1604949461.8893642,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1604949461.8895195,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","source":"kind source: /, Kind="}
{"level":"info","ts":1604949461.8897321,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","source":"kind source: /, Kind="}
{"level":"info","ts":1604949462.0902774,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod"}
{"level":"info","ts":1604949462.0903811,"logger":"controller","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","worker count":1}
{"level":"info","ts":1604949462.0904198,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job"}
{"level":"info","ts":1604949462.090432,"logger":"controller","msg":"Starting workers","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","worker count":1}
{"level":"error","ts":1604949471.7839265,"logger":"controller.job","msg":"Scan job container","job":"starboard-operator/2b1b65b9-e131-4e13-9601-68fa1eb3cab6","container":"platform-pnrhelperservice","status.reason":"Error","status.message":"2020-11-09T19:17:51.225Z\t\u001b[31mFATAL\u001b[0m\tunable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t
unable to inspect the image (aagacr.azurecr.io/platform/pnrhelperservice:20201109.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t
GET https://aagacr.azurecr.io/oauth2/token?scope=repository%3Aplatform%2Fpnrhelperservice%3Apull&service=aagacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\n\n\n","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).processFailedScanJob\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:163\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).Reconcile\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:69\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"}

Looking forward to your response, as it would save my day.

@danielpacak
Copy link
Contributor

@danielpacak

I am trying to make use of the above discussed functionality within AKS. I am not finding that the operator pulls the imagePullSecret of the deployment. In my case I am exposing the secret as imagePullSecret on the deployment level.

Secret generation command as detailed here: ACR Auth Docs

kubectl create secret docker-registry acrcred --namespace platform --docker-server=aagacr.azurecr.io --docker-username=$(AagacrUser) --docker-password=$(AagacrKey)

ACR Cred Secret:

apiVersion: v1
data:
.dockerconfigjson: eyJhdXRocyI6eyJhYWdh[...]HSm1PR016WkRkak1UY3dPa2t5T0M1K09XRkZYMHhLZFM1UUxsY3ROMkptT1RZd1J6ZDJZa2QrVW5GM1RYRT0ifX19
kind: Secret
metadata:
creationTimestamp: "2020-11-09T18:02:46Z"
managedFields:
apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:.dockerconfigjson: {}
f:type: {}
manager: kubectl-create
operation: Update
time: "2020-11-09T18:02:46Z"
name: acrcred
namespace: platform
resourceVersion: "14763349"
selfLink: /api/v1/namespaces/platform/secrets/acrcred
uid: 886f8fca-4ae6-4a9a-a6b0-133a75d606af
type: kubernetes.io/dockerconfigjson`

Deployment Spec:

spec:
containers:
envFrom:
configMapRef:
name: pnrhelperservice-configmap
image: aagacr.azurecr.io/platform/pnrhelperservice:20201109.3
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
httpGet:
path: /api/v1/ping
port: 8080
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: platform-pnrhelperservice
ports:
containerPort: 8080
name: pnrhelpersvc
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /api/v1/ping
port: 8080
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 200m
memory: 64Mi
requests:
cpu: 100m
memory: 32Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
ALL
privileged: false
runAsNonRoot: true
runAsUser: 10000
startupProbe:
failureThreshold: 30
httpGet:
path: /api/v1/ping
port: 8080
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
mountPath: /tmp
name: tmp
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: svc-platform-pnrhelperservice-token-zjpdx
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
imagePullSecrets:
name: acrcred
nodeName: aks-apppool-15092464-vmss00000y
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 10000
runAsNonRoot: true
runAsUser: 10000
serviceAccount: svc-platform-pnrhelperservice
serviceAccountName: svc-platform-pnrhelperservice
terminationGracePeriodSeconds: 30
tolerations:
effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists
volumes:
emptyDir: {}
name: tmp
name: tls-platform-pnrhelperservice
secret:
defaultMode: 420
secretName: tls-platform-pnrhelperservice
name: acrcred
secret:
defaultMode: 420
secretName: acrcred
name: svc-platform-pnrhelperservice-token-zjpdx
secret:
defaultMode: 420
secretName: svc-platform-pnrhelperservice-token-zjpdx

Operator Logs:

Loading...
{"level":"info","ts":1604949460.8825052,"logger":"main","msg":"Resolving install mode","install mode":"SingleNamespace","operator namespace":"starboard-operator","target namespaces":["platform"]}
{"level":"info","ts":1604949460.8826096,"logger":"main","msg":"Constructing multi-namespaced cache","namespaces":["platform","starboard-operator"]}
{"level":"info","ts":1604949461.888167,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1604949461.888932,"logger":"main","msg":"Using Trivy as vulnerability scanner","image":"aquasec/trivy:0.11.0"}
{"level":"info","ts":1604949461.8890698,"logger":"main","msg":"Starting controllers manager"}
{"level":"info","ts":1604949461.8893642,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1604949461.8895195,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","source":"kind source: /, Kind="}
{"level":"info","ts":1604949461.8897321,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","source":"kind source: /, Kind="}
{"level":"info","ts":1604949462.0902774,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod"}
{"level":"info","ts":1604949462.0903811,"logger":"controller","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","worker count":1}
{"level":"info","ts":1604949462.0904198,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job"}
{"level":"info","ts":1604949462.090432,"logger":"controller","msg":"Starting workers","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","worker count":1}
{"level":"error","ts":1604949471.7839265,"logger":"controller.job","msg":"Scan job container","job":"starboard-operator/2b1b65b9-e131-4e13-9601-68fa1eb3cab6","container":"platform-pnrhelperservice","status.reason":"Error","status.message":"2020-11-09T19:17:51.225Z\t\u001b[31mFATAL\u001b[0m\tunable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t* unable to inspect the image (aagacr.azurecr.io/platform/pnrhelperservice:20201109.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* GET https://aagacr.azurecr.io/oauth2/token?scope=repository%3Aplatform%2Fpnrhelperservice%3Apull&service=aagacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\n\n\n","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).processFailedScanJob\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:163\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).Reconcile\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:69\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"}
{"level":"info","ts":1604949460.8825052,"logger":"main","msg":"Resolving install mode","install mode":"SingleNamespace","operator namespace":"starboard-operator","target namespaces":["platform"]}
{"level":"info","ts":1604949460.8826096,"logger":"main","msg":"Constructing multi-namespaced cache","namespaces":["platform","starboard-operator"]}
{"level":"info","ts":1604949461.888167,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1604949461.888932,"logger":"main","msg":"Using Trivy as vulnerability scanner","image":"aquasec/trivy:0.11.0"}
{"level":"info","ts":1604949461.8890698,"logger":"main","msg":"Starting controllers manager"}
{"level":"info","ts":1604949461.8893642,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1604949461.8895195,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","source":"kind source: /, Kind="}
{"level":"info","ts":1604949461.8897321,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","source":"kind source: /, Kind="}
{"level":"info","ts":1604949462.0902774,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod"}
{"level":"info","ts":1604949462.0903811,"logger":"controller","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","worker count":1}
{"level":"info","ts":1604949462.0904198,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job"}
{"level":"info","ts":1604949462.090432,"logger":"controller","msg":"Starting workers","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","worker count":1}
{"level":"error","ts":1604949471.7839265,"logger":"controller.job","msg":"Scan job container","job":"starboard-operator/2b1b65b9-e131-4e13-9601-68fa1eb3cab6","container":"platform-pnrhelperservice","status.reason":"Error","status.message":"2020-11-09T19:17:51.225Z\t\u001b[31mFATAL\u001b[0m\tunable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t
unable to inspect the image (aagacr.azurecr.io/platform/pnrhelperservice:20201109.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t
GET https://aagacr.azurecr.io/oauth2/token?scope=repository%3Aplatform%2Fpnrhelperservice%3Apull&service=aagacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\n\n\n","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).processFailedScanJob\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:163\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).Reconcile\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:69\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"}

Looking forward to your response, as it would save my day.

Hi @n3rd253 Currently we only support private registries with Starboard CLI. I hasn't been implemented for the operator yet, but we're working on that

@mathieudebrito
Copy link

mathieudebrito commented Feb 8, 2021

Hi guys !
Just discovered this awesome project and I was wondering why Starboard could not fetch the images from my private repo :p
Note : the imagePullSecrets is given at the deployment level :

kind: Deployment
apiVersion: apps/v1
metadata:
  name: my-deployment
spec:
  template:
    spec:
      containers:
        - name: my-container
          image: 'private-repo/image-name:version'
      imagePullSecrets:
        - name: regcred

Note 2 : Even if I try it using the CLI, it does not work :

kubectl starboard scan vulnerabilityreports deployment/my-deployment
->
E0208 12:30:00.426504   59282 runnable_job.go:164] Container my-deployment terminated with Error: 2021-02-08T11:29:59.619Z      FATAL   unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:
        * unable to inspect the image (<private-repo/my-deployment:version>): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
        * GET https://private-repo/blabla : unsupported status code 401

error: running scan job: warning event received: Job has reached the specified backoff limit (BackoffLimitExceeded)

Is there any news / ETA on this or is this something you don't consider to add on a short time ?

Thank you very much anyway, it's a great job !

@danielpacak
Copy link
Contributor

Hi @mathieudebrito Starboard CLI and Starboard Operator v0.9 should read image pull Secret from pod template as explained here. If that's not the case it may be regression. I'll double check on our end.

Could you provide more details regarding the private registry used in your setup? Is it Docker Hub or a managed registry such as ECR or GCR? Which authentication method are you using? Basic vs auth token?

@mathieudebrito
Copy link

Hi @danielpacak Thank you for your quick answer !

If it helps, the secret is created as follow :

kubectl create secret docker-registry regcred --docker-server=$REGISTRY_URL --docker-username=$REGISTRY_USER --docker-password=$REGISTRY_PASSWORD --dry-run -o yaml | kubectl apply -f -

Thank you again !

@danielpacak
Copy link
Contributor

danielpacak commented Feb 9, 2021

Hey @mathieudebrito I can reproduce this error so it seems to be regression or a corner case that's not handled properly by Starboard (both operator and CLI as they share this functions). I'll track it down and try to fix it in the upcoming bug fix release v0.9.1.

Note: For some reason we cannot map registry server to container that refers to private image around these lines https://github.com/aquasecurity/starboard/blob/main/pkg/kube/secrets.go#L49

@danielpacak
Copy link
Contributor

danielpacak commented Feb 10, 2021

I found what was causing this problem. Essentially you can create image pull secret which is actually concatenation of registry server (rg.pl-waw.scw.cloud) and repository name (private). If that was the case Starboard could not parse registry server properly and match it with registry server taken from image reference. BTW, docker login strips down the repository part when it updates ~/.docker/config.json file

This was causing problem in v0.9.0

$ kubectl create secret docker-registry regcred \
  --docker-server=rg.pl-waw.scw.cloud/private \
  --docker-username=nologin \
  --docker-password=$SCW_SECRET_TOKEN --dry-run -o yaml | kubectl apply -f -

This would work in v0.9.0

$ kubectl create secret docker-registry regcred \
  --docker-server=rg.pl-waw.scw.cloud \
  --docker-username=nologin \
  --docker-password=$SCW_SECRET_TOKEN --dry-run -o yaml | kubectl apply -f -
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      imagePullSecrets:
      - name: regcred
      containers:
      - image: rg.pl-waw.scw.cloud/private/nginx:1.16
        imagePullPolicy: Always
        name: nginx
EOF

This particular problem is fixed in #379. However the original issue was created to support private registries without specifying image pull secrets on a pod. For example, by configuring nodes to authenticate with private registry.

@mathieudebrito
Copy link

mathieudebrito commented Feb 10, 2021

@danielpacak awesome thank you !
Indeed, I understand the issue is not the same, I should have created a new one, sorry for the inconvenience :(

Edit : Just tested the 0.9.1, it fixes well the issue I was having. I can now see the vulnerability reports, thank you so much for you so quick fix !

@maati55
Copy link

maati55 commented Feb 16, 2021

Hi, any chance to add supporting the private registry to the Helm, please?
I'm getting this error:
unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t* unable to inspect the image (****.azurecr.io/app:v1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* GET https://****.azurecr.io/oauth2/token?scope=repository%3Aapp%3Apull&service=****.azurecr.io: UNAUTHORIZED: authentication required

@danielpacak
Copy link
Contributor

Hi, any chance to add supporting the private registry to the Helm, please?
I'm getting this error:
unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t* unable to inspect the image (****.azurecr.io/app:v1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* GET https://****.azurecr.io/oauth2/token?scope=repository%3Aapp%3Apull&service=****.azurecr.io: UNAUTHORIZED: authentication required

Please create a separate issue for that @maati55 . This issue has nothing to do with Helm charts. We'll never close it if we report this way.

@danielpacak danielpacak added 🙏 help wanted Extra attention is needed 🎨 design More about design and architecture than writing Go code labels Mar 25, 2021
@ryanbrainard
Copy link

Thank you for providing all the details @avdhoot So just to rephrase and confirm my understanding:

You neither use image pull Secret set at Pod template not Service Account. You're using a feature1 of kops to store the config.json in /root/.docker/config.json on all nodes (including masters) so that both Kubernetes and system containers may use registries defined in it.

@avdhoot I renamed the title of this issue to reflect the specific use case of kops and authorizing Kubernetes nodes to pull images from private registry and repositories.

As already mentioned Starboard only supports recommended way of handling private registries with image pull secrets. We need to think more whether / how to support such cases as you described. /cc @lizrice

We are running into this same issue. Could it perhaps be as simple as adding a configuration to mount the Docker config file from the host as a volume onto the Starboard pods doing the scanning?

@danielpacak
Copy link
Contributor

danielpacak commented Jan 19, 2022

👋 @ryanbrainard I think we have already mentioned somewhere that mounting container runtime sockets is suboptimal for these reasons:

  • Starboard must figure out the Container Runtime used on each node. (Not sure Starboard should be aware of such infrastructure details.)
  • This would probably require scheduling scan jobs on a particular node where a scanned container image is cached. What if that node does not have enough resources or is tainted, etc.?
  • Additional permissions are usually required to run scan pods that use hostPath volume mounts (e.g. OpenShift, PSP and its successors, built-in and custom K8s admission controllers)

If you want to contribute to designing such functionality we'd appreciate reviewing design docs under the docs/design directory that are related to private and managed registries. You can also submit your own design proposal by following the template from existing docs.

@VF-mbrauer
Copy link
Contributor

@avdhoot, @maati55, @mathieudebrito, @ryanbrainard, you may have a look at the PR I created: #1103

@nekwar nekwar linked a pull request May 23, 2022 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🎨 design More about design and architecture than writing Go code 🙏 help wanted Extra attention is needed 🚀 enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants