feat: implement the solidity compilation cache #6129

galargh · 2025-01-09T19:05:36Z

Because this PR includes a bug fix, relevant tests have been included.
Because this PR includes a new feature, the change was previously discussed on an Issue or with someone from the team.
I didn't do anything of this.

In this PR, I implement the solidity compilation cache as outlined in the associated design doc.

The implementation diverges from the design doc in 1 areas:

when the compiler output cache is cleaned up

changeset-bot · 2025-01-09T19:05:40Z

⚠️ No Changeset found

Latest commit: bb1d79f

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

💥 An error occurred when fetching the changed packages and changesets in this PR

Some errors occurred when validating the changesets config:
The package "@nomicfoundation/ignition-core" depends on the ignored package "hardhat", but "@nomicfoundation/ignition-core" is not being ignored. Please add "@nomicfoundation/ignition-core" to the `ignore` option.
The package "@nomicfoundation/hardhat-ignition-ethers" depends on the ignored package "hardhat", but "@nomicfoundation/hardhat-ignition-ethers" is not being ignored. Please add "@nomicfoundation/hardhat-ignition-ethers" to the `ignore` option.
The package "@nomicfoundation/hardhat-ignition-viem" depends on the ignored package "hardhat", but "@nomicfoundation/hardhat-ignition-viem" is not being ignored. Please add "@nomicfoundation/hardhat-ignition-viem" to the `ignore` option.
The package "@nomicfoundation/hardhat-ignition" depends on the ignored package "hardhat", but "@nomicfoundation/hardhat-ignition" is not being ignored. Please add "@nomicfoundation/hardhat-ignition" to the `ignore` option.
The package "@nomicfoundation/hardhat-ignition-ethers" depends on the ignored package "@nomicfoundation/hardhat-ethers", but "@nomicfoundation/hardhat-ignition-ethers" is not being ignored. Please add "@nomicfoundation/hardhat-ignition-ethers" to the `ignore` option.
The package "@nomicfoundation/hardhat-ignition" depends on the ignored package "@nomicfoundation/hardhat-network-helpers", but "@nomicfoundation/hardhat-ignition" is not being ignored. Please add "@nomicfoundation/hardhat-ignition" to the `ignore` option.
The package "@nomicfoundation/hardhat-ignition" depends on the ignored package "@nomicfoundation/hardhat-verify", but "@nomicfoundation/hardhat-ignition" is not being ignored. Please add "@nomicfoundation/hardhat-ignition" to the `ignore` option.
The package "@nomicfoundation/hardhat-ignition-viem" depends on the ignored package "@nomicfoundation/hardhat-viem", but "@nomicfoundation/hardhat-ignition-viem" is not being ignored. Please add "@nomicfoundation/hardhat-ignition-viem" to the `ignore` option.
The package "@nomicfoundation/ignition-core" depends on the ignored package "@nomicfoundation/eslint-plugin-hardhat-internal-rules", but "@nomicfoundation/ignition-core" is not being ignored. Please add "@nomicfoundation/ignition-core" to the `ignore` option.
The package "@nomicfoundation/hardhat-ignition-ethers" depends on the ignored package "@nomicfoundation/eslint-plugin-hardhat-internal-rules", but "@nomicfoundation/hardhat-ignition-ethers" is not being ignored. Please add "@nomicfoundation/hardhat-ignition-ethers" to the `ignore` option.
The package "@nomicfoundation/hardhat-ignition-viem" depends on the ignored package "@nomicfoundation/eslint-plugin-hardhat-internal-rules", but "@nomicfoundation/hardhat-ignition-viem" is not being ignored. Please add "@nomicfoundation/hardhat-ignition-viem" to the `ignore` option.
The package "@nomicfoundation/hardhat-ignition" depends on the ignored package "@nomicfoundation/eslint-plugin-hardhat-internal-rules", but "@nomicfoundation/hardhat-ignition" is not being ignored. Please add "@nomicfoundation/hardhat-ignition" to the `ignore` option.

vercel · 2025-01-09T19:05:41Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
hardhat	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 3, 2025 9:12pm

github-actions · 2025-01-10T14:25:46Z

hardhat

Total size of the bundle: 213M
Total number of dependencies (including transitive): 53

List of dependencies (sorted by size)

208M	total
29M	@ignored/edr-optimism-linux-x64-musl
29M	@ignored/edr-optimism-linux-x64-gnu
26M	@ignored/edr-optimism-linux-arm64-musl
26M	@ignored/edr-optimism-linux-arm64-gnu
22M	@ignored/edr-optimism-win32-x64-msvc
20M	esbuild
20M	@ignored/edr-optimism-darwin-x64
19M	@ignored/edr-optimism-darwin-arm64
2.8M	@sentry/tracing
2.5M	micro-eth-signer
1.9M	@noble/curves
1.7M	undici
1.2M	@sentry/types
1.2M	@noble/hashes
932K	@sentry/node
920K	@sentry/utils
856K	zod
856K	@ignored/hardhat-vnext-utils
624K	micro-packed
576K	tsx
548K	@sentry/core
544K	fast-equals
492K	@scure/bip39
460K	@ignored/edr
368K	ethereum-cryptography
344K	@sentry/hub
324K	@ignored/hardhat-vnext-errors
320K	enquirer
284K	semver
264K	@ignored/edr-optimism
192K	ws
180K	adm-zip
168K	@scure/base
136K	get-tsconfig
96K	@scure/bip32
92K	chalk
88K	tslib
88K	@sentry/minimal
76K	agent-base
72K	@nomicfoundation/solidity-analyzer
68K	debug
64K	lru_map
64K	https-proxy-agent
60K	@ignored/hardhat-vnext-zod-utils
56K	rfdc
48K	ansi-colors
40K	resolve-pkg-maps
36K	p-map
32K	cookie
24K	strip-ansi
24K	env-paths
24K	ansi-regex
20K	ms

galargh · 2025-01-30T12:31:51Z

v-next/hardhat/src/internal/builtin-plugins/solidity/build-system/build-system.ts

+    await Promise.all(
+      compilationJobs.map(async (compilationJob) =>
+        compilationJob.getBuildId(),
+      ),
+    );


Computing build IDs is time consuming as it involves hashing all the compiler input sources. That's why we do it asynchronously. Once a compilation job build ID is computed, it will be returned straight away on the subsequent requests.

galargh · 2025-01-30T12:36:15Z

v-next/hardhat/src/internal/builtin-plugins/solidity/build-system/build-system.ts

+    // NOTE: We're not waiting for the writes and clean to finish because we
+    // will only care about the result of these operations in subsequent runs
+    void Promise.all(
+      uncachedSuccessfulResults.map(async (result) => {
+        return this.#compilerOutputCache.set(
+          await result.compilationJob.getBuildId(),
+          result.compilerOutput,
+        );
+      }),
+    ).then(() => {
+      return this.#compilerOutputCache.clean();
+    });


When should we clean up the cache was one of the open questions stated in the design doc.

To do so from the cleanArtifacts method was suggested. I do like this suggestion as then all the "clean" operations would be grouped together. However, when working on the final implementation, I realised it might be more beneficial to call the cache clean from build after all.

As you can see in the snippet, this allows us to ensure that clean is only ever called after all the writes have finished. That way we can avoid (or at least postpone) implementing synchronisation mechanisms between writes and cleans in the cache component.

galargh · 2025-01-30T12:38:08Z

v-next/hardhat/src/internal/builtin-plugins/solidity/build-system/cache.ts

Context

This is the implementation of the caching component.

It is expected to support caching compiler outputs to begin with. We can extend/change it as needed in the future.

galargh · 2025-01-30T12:39:34Z

v-next/hardhat/src/internal/builtin-plugins/solidity/build-system/cache.ts

+    defaultMaxAgeMs: number = 7 * 24 * 60 * 60 * 1000, // 1 week
+    defaultMaxSize: number = 2 * 1024 * 1024 * 1024, // 2 GB


These are the default values we agreed upon here. They are not exposed to the user at the moment.

galargh · 2025-01-30T12:40:10Z

v-next/hardhat/src/internal/builtin-plugins/solidity/build-system/cache.ts

+    return (await exists(filePath)) ? readJsonFile<T>(filePath) : undefined;
+  }
+
+  public async clean(maxAgeMs?: number, maxSize?: number): Promise<void> {


Being able to override the default max age and size is really useful for testing purposes.

galargh · 2025-01-30T12:43:01Z

v-next/hardhat/src/internal/builtin-plugins/solidity/build-system/cache.ts

+    for (const fileInfo of sortedFileInfos) {
+      if (fileInfo.atimeMs < minAtimeMs || size > maxSize) {
+        filesToRemove.push(fileInfo.file);
+        size -= fileInfo.size;
+      } else {
+        break;
+      }
+    }


We remove the files starting from the front of the list (the "oldest" entries) until the total size of the cache is smaller than the maximum cache size and the oldest entry is newer than the maximum age of a cache entry.

galargh · 2025-01-30T12:45:17Z

v-next/hardhat/src/internal/builtin-plugins/solidity/build-system/compilation-job.ts

Context

The changes in this file are concerned with the getBuildId optimisations. They are related to the effort to make the getBuildId asynchronous and to cache resolved file content hashes.

galargh · 2025-01-30T12:46:20Z

v-next/hardhat/src/internal/builtin-plugins/solidity/build-system/compilation-job.ts

+  #solcInputWithoutSources: Omit<CompilerInput, "sources"> | undefined;
+  #resolvedFiles: ResolvedFile[] | undefined;


By adding these two new properties on the compilation job, we ensure they are only ever computed once per compilation job.

galargh · 2025-01-30T12:52:44Z

v-next/hardhat/src/internal/builtin-plugins/solidity/build-system/compilation-job.ts

+    const sortedSources = Object.fromEntries(
+      Object.entries(sources).sort((a, b) => a[0].localeCompare(b[0])),
+    );


We need to sort the sources because the sources map might be populated out of order which does affect serialisation.

galargh · 2025-01-30T12:53:33Z

v-next/hardhat/src/types/solidity/build-system.ts

  buildId: string;
  contractArtifactsGenerated: string[];
+  warnings: CompilerOutputError[];


Since we're storing the entire compiler outputs in the cache, we're able to extra warnings out of them even when they're retrieved from cache.

galargh · 2025-01-30T12:59:26Z

v-next/hardhat/test/internal/builtin-plugins/solidity/build-system/build-system.ts

+      // eslint-disable-next-line @typescript-eslint/consistent-type-assertions -- We don't care about hooks in this context
+      hooks.setContext({} as HookContext);


When we call build() it is asserted that the hook context is not undefined. We don't actually use hooks in this tests.

A proper HookContext contains hre. I didn't want to create a full hre for this test. Hence, the type cast.

galargh · 2025-01-30T19:55:36Z

v-next/hardhat-utils/src/crypto.ts

+export async function createNonCryptographicHashId(
+  data: string,
+): Promise<string> {
+  const message = new TextEncoder().encode(data);
+  const buffer = await crypto.subtle.digest("SHA-1", message);
+  const array = Array.from(new Uint8Array(buffer));
+  return array.map((b) => b.toString(16).padStart(2, "0")).join("");
 }


Context

The changes to this file are related to the getBuildId optimisations.

After the changes, createNonCryptographicHashId is an asynchronous function. I also changed the hashing algorithm from MD5 to SHA-1. This is OK because the only place where we currently use this function is the getBuildId method of the CompilationJobImplementation.

galargh · 2025-01-30T19:56:14Z

v-next/hardhat-utils/src/fs.ts

+export async function getAccessTime(absolutePath: string): Promise<Date> {
+  try {
+    const stats = await fsPromises.stat(absolutePath);
+    return stats.atime;
+  } catch (e) {
+    ensureError<NodeJS.ErrnoException>(e);
+    if (e.code === "ENOENT") {
+      throw new FileNotFoundError(absolutePath, e);
+    }
+
+    throw new FileSystemAccessError(e.message, e);
+  }
+}


Context

Here, I add two new helper functions, getAccessTime and getSize which return last access time and size of a file respectively.

They are needed to implement the clean part of the caching component which removes cache entries based on file size and "age".

galargh · 2025-01-30T19:56:37Z

v-next/hardhat/src/internal/builtin-plugins/solidity/build-system/artifacts.ts

@@ -104,9 +104,9 @@ declare module "@ignored/hardhat-vnext/types/artifacts" {
 }`;
 }

-export function getBuildInfo(
+export async function getBuildInfo(


Context

It is necessary to change the signatures of getBuildInfo and getBuildInfoOutput because they both use getBuildId which turned asynchronous as part of the getBuildId optimisations.

galargh · 2025-01-30T20:11:18Z

v-next/hardhat/src/internal/builtin-plugins/solidity/build-system/resolved-file.ts

+
+import { ResolvedFileType } from "../../../../types/solidity.js";
+
+export class ProjectResolvedFileImplementation implements ProjectResolvedFile {


Context

I add implementations of ProjectResolvedFile and NpmPackageResolvedFile interfaces here. These are necessary in order to lazily store content hashes on them, as proposed as part of the getBuildId optimisations.

galargh · 2025-01-30T20:11:43Z

v-next/hardhat/src/types/solidity/resolved-file.ts

@@ -35,14 +35,15 @@ export interface ResolvedNpmPackage {
 */
 export enum ResolvedFileType {
  PROJECT_FILE = "PROJECT_FILE",
-  NPM_PACKGE_FILE = "NPM_PACKAGE_FILE",
+  NPM_PACKAGE_FILE = "NPM_PACKAGE_FILE",
 }

 /**
 * A file that's part of the Hardhat project (i.e. not installed through npm).
 */
 export interface ProjectResolvedFile {


Context

I add getContentHash() methods on the ProjectResolvedFile and NpmPackageResolvedFile interfaces here. These functions are needed to implement the getBuildId optimisations.

github-actions bot added the status:ready This issue is ready to be worked on label Jan 9, 2025

vercel bot deployed to Preview January 9, 2025 19:06 View deployment

galargh force-pushed the build-system-cache branch from 1807993 to a1d4dca Compare January 10, 2025 14:24

vercel bot deployed to Preview January 10, 2025 14:25 View deployment

galargh force-pushed the build-system-cache branch from a1d4dca to ae3401d Compare January 10, 2025 14:26

vercel bot deployed to Preview January 10, 2025 14:27 View deployment

galargh force-pushed the build-system-cache branch from ae3401d to db3ac09 Compare January 10, 2025 15:20

vercel bot deployed to Preview January 10, 2025 15:21 View deployment

alcuadrado added the v-next A Hardhat v3 development task label Jan 12, 2025

galargh added 12 commits January 28, 2025 14:51

wip: implement the compilation job and artifacts caches

94a0a3c

wip: try moving the artifacts higher

8635878

chore: add version parameter to the cache constructor

6e89b99

chore: remove the setFiles/getFiles methods from the cache interface

1a755bc

chore: remove the artifacts cache

a622510

chore: remove compiler output cache from run compilation job

7a20c12

chore: add saving to cache to the build function

7c4f198

feat: add cache cleanup function

100579e

chore: clean cache only after all the writes are complete

0787fcc

chore: remove unused has method from cache

ed3328a

chore: move cache value type declaration to the constructor

cd0adef

chore: remove unused function from build system implementation

73ba8e0

galargh force-pushed the build-system-cache branch from db3ac09 to 73ba8e0 Compare January 28, 2025 15:01

vercel bot deployed to Preview January 28, 2025 15:03 View deployment

chore: simplify successful result check

b224e1a

vercel bot deployed to Preview January 28, 2025 15:14 View deployment

galargh added 2 commits January 28, 2025 16:23

feat: make getBuildId and getSolcInput on CompilationJob async

41bddca

chore: add helper functions for getting part of solc inputs

96096b2

vercel bot deployed to Preview January 29, 2025 17:37 View deployment

chore: remove async from a method that doesn't need it

7647d54

vercel bot deployed to Preview January 29, 2025 17:41 View deployment

chore: remove forgotten describe.only

5ce2a3d

vercel bot deployed to Preview January 29, 2025 17:46 View deployment

galargh commented Jan 30, 2025

View reviewed changes

galargh changed the title ~~[WIP: not ready for review] build system cache exploration~~ feat: implement the solidity compilation cache Jan 30, 2025

galargh commented Jan 30, 2025

View reviewed changes

galargh requested a review from alcuadrado January 30, 2025 12:59

galargh marked this pull request as ready for review January 30, 2025 12:59

galargh added 2 commits January 30, 2025 16:21

chore: turn resolved files into classes

c14cf21

chore: replace static source content hash with per resolved file one

222a6c8

vercel bot deployed to Preview January 30, 2025 15:38 View deployment

kanej linked an issue Jan 30, 2025 that may be closed by this pull request

Compilation caching in the local file system #5991

Open

1 task

chore: move resolved file interface implementations out of types

b1377d3

galargh commented Jan 30, 2025

View reviewed changes

vercel bot deployed to Preview January 30, 2025 19:55 View deployment

galargh commented Jan 30, 2025

View reviewed changes

test: add a 1ms sleep before cleaning cache

b1d6cad

vercel bot deployed to Preview January 30, 2025 20:11 View deployment

galargh commented Jan 30, 2025

View reviewed changes

chore: fix lint

0dbc016

vercel bot deployed to Preview January 30, 2025 20:16 View deployment

Merge branch 'v-next' into build-system-cache

d8eeaa6

vercel bot deployed to Preview January 30, 2025 20:27 View deployment

galargh mentioned this pull request Feb 3, 2025

Enable stack traces in the EDR provider #6214

Open

Merge remote-tracking branch 'origin/v-next' into build-system-cache

bb1d79f

vercel bot deployed to Preview February 3, 2025 21:12 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement the solidity compilation cache #6129

feat: implement the solidity compilation cache #6129

galargh commented Jan 9, 2025 •

edited

Loading

changeset-bot bot commented Jan 9, 2025 •

edited

Loading

vercel bot commented Jan 9, 2025 •

edited

Loading

github-actions bot commented Jan 10, 2025 •

edited

Loading

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

galargh Jan 30, 2025

		defaultMaxAgeMs: number = 7 * 24 * 60 * 60 * 1000, // 1 week
		defaultMaxSize: number = 2 * 1024 * 1024 * 1024, // 2 GB

		#solcInputWithoutSources: Omit<CompilerInput, "sources"> \| undefined;
		#resolvedFiles: ResolvedFile[] \| undefined;

		// eslint-disable-next-line @typescript-eslint/consistent-type-assertions -- We don't care about hooks in this context
		hooks.setContext({} as HookContext);


		import { ResolvedFileType } from "../../../../types/solidity.js";

		export class ProjectResolvedFileImplementation implements ProjectResolvedFile {

feat: implement the solidity compilation cache #6129

Are you sure you want to change the base?

feat: implement the solidity compilation cache #6129

Conversation

galargh commented Jan 9, 2025 • edited Loading

changeset-bot bot commented Jan 9, 2025 • edited Loading

⚠️ No Changeset found

vercel bot commented Jan 9, 2025 • edited Loading

github-actions bot commented Jan 10, 2025 • edited Loading

hardhat

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Context

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Context

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Context

Choose a reason for hiding this comment

Context

Choose a reason for hiding this comment

Context

Choose a reason for hiding this comment

Context

Choose a reason for hiding this comment

Context

galargh commented Jan 9, 2025 •

edited

Loading

changeset-bot bot commented Jan 9, 2025 •

edited

Loading

vercel bot commented Jan 9, 2025 •

edited

Loading

github-actions bot commented Jan 10, 2025 •

edited

Loading