Inject header into S3 requests flagging them as coming from the PUDL Portal

[zaneselvans]

Overview

Ideally, we'd be able to differentiate between S3 traffic originating from users out there on the internet at large, vs. traffic coming from the PUDL Portal (🐍🕳️). However, the request is coming from some random user's web browser and IP address, so that seems hard. But we control the software originating the request.

Is there an easy way that we can inject information into the requests being sent to S3 from the PUDL Portal so that that information appears in the S3 logs, allowing us to differentiate between Portal usage and other usage of the open data? What information (if any) can we only get access to via the S3 logs vs. whatever we're able to glean from within the web app? Does the web app know how many requests have been made? How much data has been downloaded?

Success Criteria

How will we know that we're done?

• We are able to parse out which S3 usage comes from the PUDL Portal vs. other sources when processing our usage metrics.

Next steps

Preview Give feedback

1. ...