[🐛] Having trouble pulling private images during workload scans

[phyzical]

• 1.99.2
• EKS

Expected behaviour

Should get aws_auth to pull private ecr images

Actual behaviour

{
    "name": "kubernetes-monitor",
    "hostname": "snyk-monitor-66f5c46dd6-wkc7h",
    "pid": 6,
    "level": 50,
    "error": {
        "message": "Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1",
        "name": "CredentialsError",
        "stack": "CredentialsError: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1\n    at Timeout.connectTimeout [as _onTimeout] (/srv/app/node_modules/aws-sdk/lib/http/node.js:69:15)\n    at listOnTimeout (node:internal/timers:559:17)\n    at processTimers (node:internal/timers:502:7)",
        "code": "CredentialsError"
    },
    "image": "XXXXX.dkr.ecr.ap-southeast-2.amazonaws.com/some-repo@sha256:43d0eeb5047449b7aaa443a5ca7bbe933fad1e04197090ab800e530def5e9f79",
    "msg": "failed to pull image docker/oci archive image",
    "time": "2022-11-18T01:54:20.228Z",
    "v": 0
}

Steps to reproduce

Hey we use AWS so eks and ecr and we run everything on fargate including snyk

Our ecrs are in a separate account to where the eks is hosted, we use the principle org approach to allow access so every account in our org should be able to see it.

ive confirmed that the role we have created is being added to the snyk monitor pod via the service account but no matter which role i provide to the pod i get the same error above.

ive also confirmed if i assume the role being provided on my machine it can describe the images in this cross account ecr

Do you know if there is any debug steps i could try on the pod to further diagnose the issue?

Thanks!

[Jimimaku]

• 1.99.2
• EKS

Erwartetes Verhalten

Sollte aws_auth dazu bringen, private ECR-Bilder zu ziehen

Tatsächliches Verhalten

{
    "name": "kubernetes-monitor",
    "hostname": "snyk-monitor-66f5c46dd6-wkc7h",
    "pid": 6,
    "level": 50,
    "error": {
        "message": "Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1",
        "name": "CredentialsError",
        "stack": "CredentialsError: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1\n    at Timeout.connectTimeout [as _onTimeout] (/srv/app/node_modules/aws-sdk/lib/http/node.js:69:15)\n    at listOnTimeout (node:internal/timers:559:17)\n    at processTimers (node:internal/timers:502:7)",
        "code": "CredentialsError"
    },
    "image": "XXXXX.dkr.ecr.ap-southeast-2.amazonaws.com/some-repo@sha256:43d0eeb5047449b7aaa443a5ca7bbe933fad1e04197090ab800e530def5e9f79",
    "msg": "failed to pull image docker/oci archive image",
    "time": "2022-11-18T01:54:20.228Z",
    "v": 0
}

Schritte zur Reproduktion

Hey, wir verwenden AWS so eks und ecr und wir betreiben alles auf fargate einschließlich snyk

Unsere ECRS befinden sich in einem separaten Konto, auf dem das eks gehostet wird, wir verwenden den prinzipiellen Org-Ansatz, um den Zugriff zu ermöglichen, so dass jedes Konto in unserer Organisation es sehen können sollte.

Ich habe bestätigt, dass die von uns erstellte Rolle über das Dienstkonto zum SNYK-Monitor-Pod hinzugefügt wird, aber unabhängig davon, welche Rolle ich dem Pod zur Verfügung stelle, erhalte ich den gleichen Fehler wie oben.

Ich habe auch bestätigt, wenn ich die Rolle übernehme, die auf meinem Computer bereitgestellt wird, kann es die Bilder in diesem kontoübergreifenden ECR beschreiben

Wissen Sie, ob es irgendwelche Debug-Schritte gibt, die ich auf dem Pod ausprobieren könnte, um das Problem weiter zu diagnostizieren?

Danke!

[Jimimaku]

• 1.99.2
• EKS

Expected behaviour

Should get aws_auth to pull private ecr images

Actual behaviour

{
    "name": "kubernetes-monitor",
    "hostname": "snyk-monitor-66f5c46dd6-wkc7h",
    "pid": 6,
    "level": 50,
    "error": {
        "message": "Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1",
        "name": "CredentialsError",
        "stack": "CredentialsError: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1\n    at Timeout.connectTimeout [as _onTimeout] (/srv/app/node_modules/aws-sdk/lib/http/node.js:69:15)\n    at listOnTimeout (node:internal/timers:559:17)\n    at processTimers (node:internal/timers:502:7)",
        "code": "CredentialsError"
    },
    "image": "XXXXX.dkr.ecr.ap-southeast-2.amazonaws.com/some-repo@sha256:43d0eeb5047449b7aaa443a5ca7bbe933fad1e04197090ab800e530def5e9f79",
    "msg": "failed to pull image docker/oci archive image",
    "time": "2022-11-18T01:54:20.228Z",
    "v": 0
}

Steps to reproduce

Hey we use AWS so eks and ecr and we run everything on fargate including snyk

Our ecrs are in a separate account to where the eks is hosted, we use the principle org approach to allow access so every account in our org should be able to see it.

ive confirmed that the role we have created is being added to the snyk monitor pod via the service account but no matter which role i provide to the pod i get the same error above.

ive also confirmed if i assume the role being provided on my machine it can describe the images in this cross account ecr

Do you know if there is any debug steps i could try on the pod to further diagnose the issue?

Thanks!

[ivanstanev]

Hey @phyzical, have you also ensured to set the fsGroup and "projected service account token` as described in https://github.com/snyk/kubernetes-monitor/tree/staging/snyk-monitor#using-eks-without-assigning-an-iam-role-to-a-node-group?

[grv231]

@phyzical Did you ever find solution to this issue? We have the same exact use-case and are not able to pull images from ECR setup in another account (but can pull same images in cluster itself, just not in snaky-monitor)

@ivanstanev I tried following the steps you gave above, using fsGroup and projected service account token, still doesn't resolve the problem.

[phyzical]

Ah sorry i missed this reply somehow,
@grv231 we moved away from synk for the image scanning as we couldn't get it working and found that what it would have offered based on public images was almost the same as what other tools we use would have provided i.e ecrs integrated scanner or local cli scanners