Merge branch 'main' into fil/tailwind

README.md

@@ -2,10 +2,12 @@
 
 [**Observable Framework**](https://observablehq.com/framework/) is a free, [open-source](./LICENSE), static site generator for data apps, dashboards, reports, and more. Framework combines JavaScript on the front-end for interactive graphics with any language on the back-end for data analysis. Framework features [data loaders](https://observablehq.com/framework/loaders) that precompute static snapshots of data at build time for dashboards that load instantly.
 
-<picture>
-  <source media="(prefers-color-scheme: dark)" srcset="https://observablehq.observablehq.cloud/oss-analytics/@observablehq/framework/downloads-dark.svg">
-  <img alt="Daily downloads of Observable Framework" src="https://observablehq.observablehq.cloud/oss-analytics/@observablehq/framework/downloads.svg">
-</picture>
+<a href="https://observablehq.observablehq.cloud/oss-analytics/@observablehq/framework">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://observablehq.observablehq.cloud/oss-analytics/@observablehq/framework/downloads-dark.svg">
+    <img alt="Daily downloads of Observable Framework" src="https://observablehq.observablehq.cloud/oss-analytics/@observablehq/framework/downloads.svg">
+  </picture>
+</a>
 
 <sub>Daily downloads of Observable Framework · [oss-analytics](https://observablehq.observablehq.cloud/oss-analytics/)</sub>
 

docs/config.md

@@ -197,9 +197,13 @@ footer: ({path}) => `<a href="https://github.com/example/test/blob/main/src${pat
 
 The base path when serving the site. Currently this only affects the custom 404 page, if any.
 
-## cleanUrls <a href="https://github.com/observablehq/framework/releases/tag/v1.3.0" class="observablehq-version-badge" data-version="^1.3.0" title="Added in 1.3.0"></a>
+## preserveIndex <a href="https://github.com/observablehq/framework/pulls/1784" class="observablehq-version-badge" data-version="prerelease" title="Added in #1784"></a>
 
-Whether page links should be “clean”, _i.e._, formatted without a `.html` extension. Defaults to true. If true, a link to `config.html` will be formatted as `config`. Regardless of this setting, a link to an index page will drop the implied `index.html`; for example `foo/index.html` will be formatted as `foo/`.
+Whether page links should preserve `/index` for directories. Defaults to false. If true, a link to `/` will be formatted as `/index` if the **preserveExtension** option is false or `/index.html` if the **preserveExtension** option is true.
+
+## preserveExtension <a href="https://github.com/observablehq/framework/pulls/1784" class="observablehq-version-badge" data-version="prerelease" title="Added in #1784"></a>
+
+Whether page links should preserve the `.html` extension. Defaults to false. If true, a link to `/foo` will be formatted as `/foo.html`.
 
 ## toc
 
@@ -297,6 +301,45 @@ export default {
 };
 ```
 
+## duckdb <a href="https://github.com/observablehq/framework/pull/1734" class="observablehq-version-badge" data-version="prerelease" title="Added in #1734"></a>
+
+The **duckdb** option configures [self-hosting](./lib/duckdb#self-hosting-of-extensions) and loading of [DuckDB extensions](./lib/duckdb#extensions) for use in [SQL code blocks](./sql) and the `sql` and `DuckDBClient` built-ins. For example, a geospatial data app might enable the [`spatial`](https://duckdb.org/docs/extensions/spatial/overview.html) and [`h3`](https://duckdb.org/community_extensions/extensions/h3.html) extensions like so:
+
+```js run=false
+export default {
+  duckdb: {
+    extensions: ["spatial", "h3"]
+  }
+};
+```
+
+The **extensions** option can either be an array of extension names, or an object whose keys are extension names and whose values are configuration options for the given extension, including its **source** repository (defaulting to the keyword _core_ for core extensions, and otherwise _community_; can also be a custom repository URL), whether to **load** it immediately (defaulting to true, except for known extensions that support autoloading), and whether to **install** it (_i.e._ to self-host, defaulting to true). As additional shorthand, you can specify `[name]: true` to install and load the named extension from the default (_core_ or _community_) source repository, or `[name]: string` to install and load the named extension from the given source repository.
+
+The configuration above is equivalent to:
+
+```js run=false
+export default {
+  duckdb: {
+    extensions: {
+      spatial: {
+        source: "https://extensions.duckdb.org/",
+        install: true,
+        load: true
+      },
+      h3: {
+        source: "https://community-extensions.duckdb.org/",
+        install: true,
+        load: true
+      }
+    }
+  }
+};
+```
+
+The `json` and `parquet` are configured (and therefore self-hosted) by default. To expressly disable self-hosting of extension, you can set its **install** property to false, or equivalently pass null as the extension configuration object.
+
+For more, see [DuckDB extensions](./lib/duckdb#extensions).
+
 ## markdownIt <a href="https://github.com/observablehq/framework/releases/tag/v1.1.0" class="observablehq-version-badge" data-version="^1.1.0" title="Added in v1.1.0"></a>
 
 A hook for registering additional [markdown-it](https://github.com/markdown-it/markdown-it) plugins. For example, to use [markdown-it-footnote](https://github.com/markdown-it/markdown-it-footnote), first install the plugin with either `npm add markdown-it-footnote` or `yarn add markdown-it-footnote`, then register it like so:

docs/getting-started.md

@@ -535,7 +535,7 @@ The <code>build</code> command generates the `dist` directory; you can then copy
 
 <pre data-copy>npx http-server dist</pre>
 
-<div class="tip">By default, Framework generates “clean” URLs by dropping the `.html` extension from page links. Not all webhosts support this; some need the <a href="./config#clean-urls"><b>cleanUrls</b> config option</a> set to false.</div>
+<div class="tip">By default, Framework generates “clean” URLs by dropping the `.html` extension from page links. Not all webhosts support this; some need the <a href="./config#preserve-extension"><b>preserveExtension</b> config option</a> set to true.</div>
 
 <div class="tip">When deploying to GitHub Pages without using GitHub’s related actions (<a href="https://github.com/actions/configure-pages">configure-pages</a>,
 <a href="https://github.com/actions/deploy-pages">deploy-pages</a>, and

docs/lib/duckdb.md

@@ -65,7 +65,7 @@ const db2 = await DuckDBClient.of({base: FileAttachment("quakes.db")});
 db2.queryRow(`SELECT COUNT() FROM base.events`)
 ```
 
-For externally-hosted data, you can create an empty `DuckDBClient` and load a table from a SQL query, say using [`read_parquet`](https://duckdb.org/docs/guides/import/parquet_import) or [`read_csv`](https://duckdb.org/docs/guides/import/csv_import). DuckDB offers many affordances to make this easier (in many cases it detects the file format and uses the correct loader automatically).
+For externally-hosted data, you can create an empty `DuckDBClient` and load a table from a SQL query, say using [`read_parquet`](https://duckdb.org/docs/guides/import/parquet_import) or [`read_csv`](https://duckdb.org/docs/guides/import/csv_import). DuckDB offers many affordances to make this easier. (In many cases it detects the file format and uses the correct loader automatically.)
 
 ```js run=false
 const db = await DuckDBClient.of();
@@ -105,3 +105,100 @@ const sql = DuckDBClient.sql({quakes: `https://earthquake.usgs.gov/earthquakes/f
 ```sql echo
 SELECT * FROM quakes ORDER BY updated DESC;
 ```
+
+## Extensions <a href="https://github.com/observablehq/framework/pull/1734" class="observablehq-version-badge" data-version="prerelease" title="Added in #1734"></a>
+
+[DuckDB extensions](https://duckdb.org/docs/extensions/overview.html) extend DuckDB’s functionality, adding support for additional file formats, new types, and domain-specific functions. For example, the [`json` extension](https://duckdb.org/docs/data/json/overview.html) provides a `read_json` method for reading JSON files:
+
+```sql echo
+SELECT bbox FROM read_json('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson');
+```
+
+To read a local file (or data loader), use `FileAttachment` and interpolation `${…}`:
+
+```sql echo
+SELECT bbox FROM read_json(${FileAttachment("../quakes.json").href});
+```
+
+For convenience, Framework configures the `json` and `parquet` extensions by default. Some other [core extensions](https://duckdb.org/docs/extensions/core_extensions.html) also autoload, meaning that you don’t need to explicitly enable them; however, Framework will only [self-host extensions](#self-hosting-of-extensions) if you explicitly configure them, and therefore we recommend that you always use the [**duckdb** config option](../config#duckdb) to configure DuckDB extensions. Any configured extensions will be automatically [installed and loaded](https://duckdb.org/docs/extensions/overview#explicit-install-and-load), making them available in SQL code blocks as well as the `sql` and `DuckDBClient` built-ins.
+
+For example, to configure the [`spatial` extension](https://duckdb.org/docs/extensions/spatial/overview.html):
+
+```js run=false
+export default {
+  duckdb: {
+    extensions: ["spatial"]
+  }
+};
+```
+
+You can then use the `ST_Area` function to compute the area of a polygon:
+
+```sql echo run=false
+SELECT ST_Area('POLYGON((0 0, 0 1, 1 1, 1 0, 0 0))'::GEOMETRY) as area;
+```
+
+To tell which extensions have been loaded, you can run the following query:
+
+```sql
+FROM duckdb_extensions() WHERE loaded AND JSON '1';
+```
+
+```sql run=false
+FROM duckdb_extensions() WHERE loaded;
+```
+
+<div class="warning">
+
+If the `duckdb_extensions()` function runs before DuckDB autoloads a core extension (such as `json`), it might not be included in the returned set.
+
+</div>
+
+### Self-hosting of extensions
+
+As with [npm imports](../imports#self-hosting-of-npm-imports), configured DuckDB extensions are self-hosted, improving performance, stability, & security, and allowing you to develop offline. Extensions are downloaded to the DuckDB cache folder, which lives in <code>.observablehq/<wbr>cache/<wbr>\_duckdb</code> within the source root (typically `src`). You can clear the cache and restart the preview server to re-fetch the latest versions of any DuckDB extensions. If you use an [autoloading core extension](https://duckdb.org/docs/extensions/core_extensions.html#list-of-core-extensions) that is not configured, DuckDB-Wasm [will load it](https://duckdb.org/docs/api/wasm/extensions.html#fetching-duckdb-wasm-extensions) from the default extension repository, `extensions.duckdb.org`, at runtime.
+
+## Configuring
+
+The second argument to `DuckDBClient.of` and `DuckDBClient.sql` is a [`DuckDBConfig`](https://shell.duckdb.org/docs/interfaces/index.DuckDBConfig.html) object which configures the behavior of DuckDB-Wasm. By default, Framework sets the `castBigIntToDouble` and `castTimestampToDate` query options to true. To instead use [`BigInt`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt):
+
+```js run=false
+const bigdb = DuckDBClient.of({}, {query: {castBigIntToDouble: false}});
+```
+
+By default, `DuckDBClient.of` and `DuckDBClient.sql` automatically load all [configured extensions](#extensions). To change the loaded extensions for a particular `DuckDBClient`, use the **extensions** config option. For example, pass an empty array to instantiate a DuckDBClient with no loaded extensions (even if your configuration lists several):
+
+```js echo run=false
+const simpledb = DuckDBClient.of({}, {extensions: []});
+```
+
+Alternatively, you can configure extensions to be self-hosted but not load by default using the **duckdb** config option and the `load: false` shorthand:
+
+```js run=false
+export default {
+  duckdb: {
+    extensions: {
+      spatial: false,
+      h3: false
+    }
+  }
+};
+```
+
+You can then selectively load extensions as needed like so:
+
+```js echo run=false
+const geosql = DuckDBClient.sql({}, {extensions: ["spatial", "h3"]});
+```
+
+In the future, we’d like to allow DuckDB to be configured globally (beyond just [extensions](#extensions)) via the [**duckdb** config option](../config#duckdb); please upvote [#1791](https://github.com/observablehq/framework/issues/1791) if you are interested in this feature.
+
+## Versioning
+
+Framework currently uses [DuckDB-Wasm 1.29.0](https://github.com/duckdb/duckdb-wasm/releases/tag/v1.29.0), which aligns with [DuckDB 1.1.1](https://github.com/duckdb/duckdb/releases/tag/v1.1.1). You can load a different version of DuckDB-Wasm by importing `npm:@duckdb/duckdb-wasm` directly, for example:
+
+```js run=false
+import * as duckdb from "npm:@duckdb/[email protected]";
+```
+
+However, you will not be able to change the version of DuckDB-Wasm used by SQL code blocks or the `sql` or `DuckDBClient` built-ins, nor can you use Framework’s support for self-hosting extensions with a different version of DuckDB-Wasm.

docs/lib/sqlite.md

@@ -6,6 +6,14 @@
 import SQLite from "npm:@observablehq/sqlite";
 ```
 
+If you prefer to use sql.js directly, you can import and initialize it like so:
+
+```js run=false
+import initSqlJs from "npm:sql.js";
+
+const SQLite = await initSqlJs({locateFile: (name) => import.meta.resolve("npm:sql.js/dist/") + name});
+```
+
 We also provide `SQLiteDatabaseClient`, a [`DatabaseClient`](https://observablehq.com/@observablehq/database-client-specification) implementation.
 
 ```js run=false

docs/project-structure.md

@@ -99,7 +99,7 @@ For this site, routes map to files as:
 /hello → dist/hello.html → src/hello.md
 ```
 
-This assumes [“clean URLs”](./config#clean-urls) as supported by most static site servers; `/hello` can also be accessed as `/hello.html`, and `/` can be accessed as `/index` and `/index.html`. (Some static site servers automatically redirect to clean URLs, but we recommend being consistent when linking to your site.)
+This assumes [“clean URLs”](./config#preserve-extension) as supported by most static site servers; `/hello` can also be accessed as `/hello.html`, and `/` can be accessed as `/index` and `/index.html`. (Some static site servers automatically redirect to clean URLs, but we recommend being consistent when linking to your site.)
 
 Apps should always have a top-level `index.md` in the source root; this is your app’s home page, and it’s what people visit by default.
 

docs/sql.md

@@ -29,7 +29,7 @@ sql:
 
 <div class="tip">For performance and reliability, we recommend using local files rather than loading data from external servers at runtime. You can use a <a href="./data-loaders">data loader</a> to take a snapshot of a remote data during build if needed.</div>
 
-You can also register tables via code (say to have sources that are defined dynamically via user input) by defining the `sql` symbol with [DuckDBClient.sql](./lib/duckdb).
+You can also register tables via code (say to have sources that are defined dynamically via user input) by defining the `sql` symbol with [DuckDBClient.sql](./lib/duckdb). To register [DuckDB extensions](./lib/duckdb#extensions), use the [**duckdb** config option](./config#duckdb).
 
 ## SQL code blocks
 

docs/telemetry.md.js

@@ -0,0 +1,32 @@
+import {readFile} from "node:fs/promises";
+
+process.stdout.write(`# Telemetry
+
+Observable Framework collects anonymous usage data to help us improve the product. This data is sent to Observable and is not shared with third parties. Telemetry data is covered by [Observable’s privacy policy](https://observablehq.com/privacy-policy).
+
+You can [opt-out of telemetry](#disabling-telemetry) by setting the \`OBSERVABLE_TELEMETRY_DISABLE\` environment variable to \`true\`.
+
+## What is collected?
+
+The following data is collected:
+
+~~~ts run=false
+${(await readFile("./src/telemetryData.d.ts", "utf-8")).trim()}
+~~~
+
+To inspect telemetry data, set the \`OBSERVABLE_TELEMETRY_DEBUG\` environment variable to \`true\`. This will print the telemetry data to stderr instead of sending it to Observable. See [\`telemetry.ts\`](https://github.com/observablehq/framework/blob/main/src/telemetry.ts) for source code.
+
+## What is not collected?
+
+We never collect identifying or sensitive information, such as environment variables, file names or paths, or file contents.
+
+## Disabling telemetry
+
+Setting the \`OBSERVABLE_TELEMETRY_DISABLE\` environment variable to \`true\` disables telemetry collection entirely. For example:
+
+~~~sh
+OBSERVABLE_TELEMETRY_DISABLE=true npm run build
+~~~
+
+Setting the \`OBSERVABLE_TELEMETRY_DEBUG\` environment variable to \`true\` also disables telemetry collection, instead printing telemetry data to stderr. Use this to inspect what telemetry data would be collected.
+`);