[Bug] Issue using aws-otel-nodejs in Lambda

[inception-cto]

Context

I am trying to configure ADOT and use it with the current implementation of Lambda through AWS CDK. AWS CDK offers a ready-to-use NodeJS lambda constructs @aws-cdk/aws-lambda-nodejs.

Unfortunately, it seems that there is an error/conflict between the otel-handler and the way @aws-cdk/aws-lambda-nodejs builds the lambda function. I am not sure if it is related to esbuild or not.

Update: It seems that the issue is similar to #97 .

Configuration

I am using the following code to create a Lambda function.

this.adotLayer = lambda.LayerVersion.fromLayerVersionArn(
      this,
      "adot-layer",
      "arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-ver-0-19-0:1"
    );

const lambdaFn = new NodejsFunction(this, `sample-fn`, {
      runtime: lambda.Runtime.NODEJS_14_X,
      description: props.description,
      handler: "handler",
      entry: "function/sample.ts",
      tracing: lambda.Tracing.ACTIVE,
      environment: {
        AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-handler", 
      },
      layers: [this.adotLayer],
    });

Error

The Lambda function is deployed properly. However, when running the function, I receive the following error:

{
    "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module '/var/task/package.json'\nRequire stack:\n- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/node/instrumentation.js\n- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/node/index.js\n- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/index.js\n- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/index.js\n- /opt/nodejs/node_modules/@opentelemetry/instrumentation-aws-lambda/build/src/aws-lambda.js\n- /opt/nodejs/node_modules/@opentelemetry/instrumentation-aws-lambda/build/src/index.js\n- /opt/wrapper.js\n- internal/preload",
    "stack": [
        "Runtime.ImportModuleError: Error: Cannot find module '/var/task/package.json'",
        "Require stack:",
        "- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/node/instrumentation.js",
        "- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/node/index.js",
        "- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/index.js",
        "- /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/index.js",
        "- /opt/nodejs/node_modules/@opentelemetry/instrumentation-aws-lambda/build/src/aws-lambda.js",
        "- /opt/nodejs/node_modules/@opentelemetry/instrumentation-aws-lambda/build/src/index.js",
        "- /opt/wrapper.js",
        "- internal/preload",
        "    at _loadUserApp (/var/runtime/UserFunction.js:100:13)",
        "    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)",
        "    at Object.<anonymous> (/var/runtime/index.js:43:30)",
        "    at Module._compile (internal/modules/cjs/loader.js:1068:30)",
        "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)",
        "    at Module.load (internal/modules/cjs/loader.js:933:32)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:774:14)",
        "    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)",
        "    at internal/main/run_main_module.js:17:47"
    ]
}

⚠️ Update

I added a dummy package.json to the project, and now I receive the same error in #97 .

"TypeError: Cannot redefine property: handler",
        "    at Function.defineProperty (<anonymous>)",
        "    at defineProperty (/opt/nodejs/node_modules/shimmer/index.js:14:10)",
        "    at AwsLambdaInstrumentation.wrap [as _wrap] (/opt/nodejs/node_modules/shimmer/index.js:56:3)",
        "    at InstrumentationNodeModuleFile.patch (/opt/nodejs/node_modules/@opentelemetry/instrumentation-aws-lambda/build/src/aws-lambda.js:67:26)",
        "    at AwsLambdaInstrumentation._onRequire (/opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/node/instrumentation.js:75:33)",
        "    at /opt/nodejs/node_modules/@opentelemetry/instrumentation/build/src/platform/node/instrumentation.js:102:29",
        "    at Module.Hook._require.Module.require (/opt/nodejs/node_modules/require-in-the-middle/index.js:154:32)",
        "    at Module.Hook._require.Module.require (/opt/nodejs/node_modules/require-in-the-middle/index.js:80:39)",
        "    at Module.Hook._require.Module.require (/opt/nodejs/node_modules/require-in-the-middle/index.js:80:39)",
        "    at Module.Hook._require.Module.require (/opt/nodejs/node_modules/require-in-the-middle/index.js:80:39)"

[srprash]

Seems like it could be related to aws/aws-xray-sdk-node#433 which is an esbuild issue with CDK aws-lambda-nodejs bundling.

[anuraaga]

Thanks @melek-inception - I have filed an issue upstream on whether it's possible to monkey patch libraries created with esbuild or CDK (not sure which is causing it to be non-redefinable).

open-telemetry/opentelemetry-js#2260

[willarmiros]

w.r.t. the original ImportModuleError, this is discussed in open-telemetry/opentelemetry-js#2193, and we've reached an agreement with the maintainers of the OTel JS SDK for an approach that won't require an empty package.json. It is not yet implemented, but should be a quick fix once out of the backlog.

For the newer TypeError issue, sounds like that might be fixable with a patch to the shimmer library used for monkey-patching. We'll continue discussion in the opentelemetry-js repo most likely.

[longility]

For anyone facing the issue related to TypeError: Cannot redefine property: handler, @melek-inception found a workaround for the esbuild gotcha.

For the lambda handler, instead of

export const handler = () => {
  // removed for brevity
};

Do this

const handler = () => {
  // removed for brevity
};
module.exports = { handler }

[inception-cto]

@longility , you beat me to update the GitHub issue 💯

The error seems to be linked to the esbuild immutability of the exported objects. Enforcing a mutable export of the handler fixes the problem.

Hope this help!

[inception-cto]

@willarmiros , I will close the ticket unless you think of a different workaround.

[colinmcdonald22]

Just wanted to add a note for anyone else dealing with this issue, we've also noticed other factors that cause exports to be immutable (and trigger the Cannot redefine property from shimmer)

In our case, it was an ESBuild plugin: @sentry/esbuild-plugin (opened an issue against the plugin here to track the issue)

Anything in the build process that causes exports to be immutable (like using export const handler = ..., or in our case, an ESBuild plugin wrapping our code) will break shimmer's wrapper.