What is the idea behind filterSensitiveLog?

[carlnordenfelt]

There is a function exposed called filterSensitiveLog.
It isn't entirely clear if this method is intended for use by SDK users or not.

I really like the general idea of having access to a function that will automatically filter out Command parameters that are deemed sensitive as it takes away one thing to consider in our services but it appears clunky at the moment.

An example:

const { ChangePasswordCommand, ChangePasswordRequest } = require('@aws-sdk/client-iam');
const cmd = new ChangePasswordCommand({
  NewPassword: 'foo',
  OldPassword: 'bar',
});
log.info('Command Params', ChangePasswordRequest.filterSensitiveLog(cmd.input));

>  [INFO] Command Params {
  NewPassword: '***SensitiveInformation***',
  OldPassword: '***SensitiveInformation***'
}

If it is intended to be used like this I would personally prefer to not have to require the Request object but rather to use it on the command directly:

log.info('Command Params', cmd.filterSensitiveLog());

It might be that I am getting this all wrong and I shouldn't even be using this method at all.
Some clarification on this would be much appreciated.

[trivikr]

What is the idea behind filterSensitiveLog?

The method is intended to filter sensitive data from the shapes before logging.
It ensures that sensitive data is not logged by the logger.

It is used by the loggerMiddleware while logging input/output

aws-sdk-js-v3/packages/middleware-logger/src/loggerMiddleware.ts

Lines 33 to 34 in 4f9e56f

	input: inputFilterSensitiveLog(args.input),
	output: outputFilterSensitiveLog(outputWithoutMetadata),

The method is not intended for usage by users directly, unless they're writing their custom logger.

[carlnordenfelt]

Thanks for the clarification.

Given what you wrote it does make it clear that it is not intended to be used the way we are.

I do want to ask/suggest if it could be a feature for the future though? It'd be a very helpful function to be able to easily and secure log request parameters for debugging and the like.

[trivikr]

The method is not intended for usage by users directly, unless they're writing their custom logger.

You can use the feature now if you're writing your custom logger or middleware for logging.

[carlnordenfelt]

Hi,

I think it might be a bit over the top for us to add a middleware.
Is there some sample code or similar that shows how that would work?

Really, all we would like is a way to get the request/response object with sensitive data masked without having to think twice about it. I just figured that since it is already implemented to some extent it should be pretty easy to expose the functionality but given that the current implementation wasn't intended to be used externally it would be a feature request.

[trivikr]

I think it might be a bit over the top for us to add a middleware.

Middleware are easier to understand as compared to event listeners. We wrote a blog post on introducing middleware.

all we would like is a way to get the request/response object with sensitive data masked without having to think twice about it.

You can refer the example in loggerMiddleware to write your custom middleware.

aws-sdk-js-v3/packages/middleware-logger/src/loggerMiddleware.ts

Lines 33 to 34 in d4cb977

	input: inputFilterSensitiveLog(args.input),
	output: outputFilterSensitiveLog(outputWithoutMetadata),

Your custom middleware can be with the initialize step and pass response.output to outputFilterSensitiveLog as follows:

client.middlewareStack.add(
  (next, context) => async (args) => {
    const result = next(args);
    const { outputFilterSensitiveLog } = context;
    const { $metadata, ...outputWithoutMetadata } = response.output;
    return {
      ...response,
      output: {
        $metadata,
        ...outputFilterSensitiveLog(outputWithoutMetadata)
      }
    };
  },
  {
    step: "initialize",
    name: "filterSensitiveLogFromOutut",
  }
);

[carlnordenfelt]

I think the middleware approach in general is a lot better than the old event listeners even if I barely used those.

Still, for this particular use case, writing a full middleware is a lot of excess code when really, all I would like to be able to do is:

const cmd = new ChangePasswordCommand({ ... });
log.info('Params', cmd.filterSensitive());

[ziljoc]

Hello,

I was following this discussion for some time and one of the several questions I have is: is it true that aws middleware can be executed only once the command is sent (order and priority of the stack does not matter for this question)? I am asking because I have the use case where I am interested of having two microservices, where 1st generates aws commands based on AI and the 2nd microservice sends them to aws. I do not want get into details of the use case but I am interested in logging commands in the 1st microservice. So question is: is it possible to filter sensitive data on the command using convenient way like @carlnordenfelt suggested

log.info('Command Params', cmd.filterSensitiveLog());

Or something similar?

Or I must add the middleware which will be executed only once the command is sent? If not, can any middleware be explicitly be chosen and executed without calling others? What I am asking is totally not responsibility of the aws sdk to expose, but out of curiosity I want to have a bigger picture here.

If the middleware can be executed only once the command is sent, then I need completely different approach. Also, I assume my use case is only one of the many when logging is needed but middleware is too big overhead, or won't help at all.