Skip to content

Preload List Processes

Jump to bottom
Nick Harper edited this page Dec 9, 2022 · 66 revisions

This document assumes familiarity with the Chromium HSTS preload list. For background, see:

Updating the Chromium preload list

This requires editing net/http/transport_security_state_static.json in a Chromium checkout. See this gist for how to do an efficient partial checkout of just the folder that contains the preload list.

The list should be updated with entries from https://hstspreload.org/api/v2/pending at least once once per Chromium release. This change should include bulk entries as well as any manual additions or edits.

See https://www.chromium.org/developers/contributing-code for instructions on how to contribute to Chromium.

Updating Bulk HSTS entries

For more detailed instructions, see go/hsts-playbook-internal (Google-only link).

roll_preload_list.py handles pending bulk removals and pending bulk additions:

# Get the latest tools.
git clone https://github.com/chromium/hstspreload.org $HOME/hstspreload.org

# Modify folder to taste.
SCAN_FOLDER_TODAY="$HOME/hstspreload-scans/"$(date "+%Y-%m-%d")
mkdir -p "$SCAN_FOLDER_TODAY"
hstspreload scan-pending | tee "$SCAN_FOLDER_TODAY/pending.json"

cd chromium/src
# Replace $HOME/hstspreload.org with the path to your checkout.
python3 \
  "$HOME/hstspreload.org/scripts/roll_preload_list.py" \
  "net/http/transport_security_state_static.json" \
  "$SCAN_FOLDER_TODAY/pending.json" \
  "$SCAN_FOLDER_TODAY/rejections.json"

Ideally, the domains in rejections.json should be set to rejected at hstspreload.org, but this is not automated yet.

We hope to automate this whole process some day: crbug.com/736188

Manual HSTS entries

We strongly encourage site operators to preload using the automated form at https://hstspreload.org/ – it instantly checks for requirements and best-practices, and allows submitting entries directly. At a rate approaching thousands of submissions per month, this also saves a lot of email for the preload list maintainer.

There are various reasons site operators might not be able to meet the automated requirements. Usually, the site should be updated to meet them, but sometimes we need to accommodate exceptions.

New manual entries (or manual edits to existing bulk entries) should be placed in the appropriate subsection of the section with the end comment END OF MANUAL CUSTOM ENTRIES.

Requirements for manual HSTS entries

Each entry in the preload list takes up space in the browser binary, and each manual entry requires manual maintenance work (must be added by hand, and can't be pruned automatically). Therefore, we need to be conservative about allowing exceptions:

  • Preloading the site should have some benefit to users. Good examples of this are:
    • Sites with many users.
    • Security- or privacy-sensitive sites (e.g. banks, health sites, shopping sites, social networks).
  • There should be a technical reason the site can't meet the automated requirements.
  • Unless there is high user benefit, we should not preload a large number of subdomains for a given domain. (Keep in mind: each byte added to Chrome generates gigabytes of global download bandwidth for every future release).

A site should also pass as many requirements as possible, with a specific reason for each requirement that cannot be met. To check requirements, use the form at hstspreload.org, or use the commandline tool:

hstspreload preloadabledomain example.com

Examples of valid reasons:

  • A large, old site can't move their top domain to includeSubDomains at once, and temporarily wants to preload sensitive subdomains like www.example.com or login.example.com. (In this case, it's also fine to add an entry for the eTLD+1 without includeSubDomains.)
  • The preload list scanner has a bug that prevents connecting to a domain. (For example, the scanner was incompatible with some HTTP/2 sites for half a year.) In this case, if Chrome and SSL Labs can connect to the site and verify that it has reasonable modern TLS support, the requirements can be checked manually. Sometimes running hstspreload +d locally on Mac/Linux allows checking domains that the live hstspreload.org site fails on.
  • The public suffix list data used by the requirements scanner is stale, and we need to waive a spurious internal.domain.name.cannot_compute_etld1 error.
  • A university department wants to preload their subdomain even if the university domain is not an eTLD.
  • An eTLD operator wants to preload an entire eTLD. We like this! (It's just not automated.)

The following are NOT valid reasons:

  • Skipping any requirements for performance reasons. (It is possible to write a performant site that meets all requirements).
    • In particular, we will not support waiving a redirects.http.first_redirect.insecure error for any new entries until Chromium issue #626180 has a solution.
  • A site operator wants to preload a domain without includeSubDomains, and cannot provide an example of a subdomain that is not ready for preloading (along with a good technical reason for why supporting HTTPS for it is difficult).
  • The site is not publicly accessible to most browser end-users for the same reason the scanner can't access it (e.g. internal corporate sites, sites without public DNS records).
  • A site operator doesn't want to meet certain requirements, and cannot articulate why.

Removals

The recommended removal process is to use the form at hstspreload.org/removal/. (Note: this currently only works for bulk entries preloaded starting Feb. 29, 2016.)

For cases not served by the form, removal requests are sent to the contact address at the bottom of hstspreload.org. People with @chromium.org accounts can access these in the corresponding Google Group. The preload list maintainer should subscribe and respond to these emails regularly.

If a domain passes the removal requirements, it can be removed from the preload list using the Updating the Chromium preload list steps above.

We have also historically merged removals into the beta branch fairly close to stable release, but we no longer do this unless a special situation calls for it. (For a historical example, see the merge from M58 to M57.)

Removal Requirements

Removal requirements are documented at: https://hstspreload.org/removal/

The removal requirements can also be checked using the hstspreload commandline tool as follows:

hstspreload removabledomain example.com

If a site does not support HTTPS at all anymore, there are alternative methods of validation. These should be avoided when possible, especially if the site still responds over HTTPS or if it appears to be a high-profile domain.

  • DNS validation (see below).
  • An email sent to the whois technical contact email address (and no one else) asking the site owner to verify the request.

Preloading other security mechanisms

Chromium is no longer accepting preloads of expect_ct and expect_staple.

Chromium is not accepting new pinsets for HPKP preloading.

DNS Validation

To validate the request for a domain using DNS:

  1. Generate a random token and ask the requester to add a temporary DNS TXT record for the domain in question, with the value Preload List Validation for [domain name]: [token goes here]

Command to generate a TXT value:

echo -n "Enter domain: " ; \
  read DOMAIN ; \
  echo "Preload List Validation for ${DOMAIN}: $(openssl rand -hex 16)"

Note that since DNS responses are generally not authenticated using a strong security mechanism, verifying headers over HTTPS is preferred whenever possible.

Clone this wiki locally