HTML-render format-spec docs

.github/workflows/quarto-render.yml

@@ -0,0 +1,33 @@
+# Cloned from https://github.com/TileDB-Inc/tiledb-quarto-template
+
+name: Render and deploy Quarto files
+on: 
+  push:
+  pull_request:
+
+jobs:
+  quarto-render-and-deploy:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: "Install Quarto"
+      uses: quarto-dev/quarto-actions/install-quarto@main
+      with:
+        version: 0.9.141
+
+    - name: "Quarto render"
+      shell: bash
+      run: |
+        quarto render --execute-debug      
+
+    - name: "Deploy to gh-pages"
+      uses: peaceiris/actions-gh-pages@v3
+      # Change to the name of your repo's primary branch name:
+      if: github.ref == 'refs/heads/main'
+      with:
+        # This is GitHub Actions magic; no secrets for us to manage; and this works first-time
+        # without any extra configs other than visiting Settings -> Pages in your GitHub repo.
+        github_token: ${{ secrets.GITHUB_TOKEN }}
+        publish_dir: docs
+        destination_dir: docs

.gitignore

@@ -1,5 +1,5 @@
 .vscode*
-*.swp
+*.sw?
 build/*
 dist/*
 cmake-build-*/*
@@ -31,3 +31,9 @@ doc/source/_sidebar.rst.inc
 doc/source/gensidebar.pyc
 examples/cmake_project/build
 CMakeUserPresets.json
+
+# Quarto dev-cache
+/.quarto/
+docs/.quarto
+# Let CI build this
+/docs

_quarto.yml

@@ -0,0 +1,62 @@
+# Cloned from https://github.com/TileDB-Inc/tiledb-quarto-template
+project:
+  type: website
+  output-dir: docs
+
+format:
+  html:
+    toc: true
+    theme: 
+      light: [flatly, "quarto-materials/tiledb.scss"]
+    mainfont: Helvetica
+    fontsize: 1rem
+    linkcolor: "#4d9fff"
+    code-copy: true
+    code-overflow: wrap
+    css: "quarto-materials/tiledb.css"
+
+website:
+  favicon: "images/favicon.ico"
+  repo-url: https://github.com/TileDB-Inc/TileDB
+  page-navigation: true
+  navbar:
+    background: light
+    logo: "quarto-materials/tiledb-logo.png"
+    collapse-below: lg
+    left:
+      - text: "Home page"
+        href: "https://tiledb.com"
+      - text: "Contact us"
+        href: "https://tiledb.com/contact"
+      - text: "Repo"
+        href: "https://github.com/TileDB-Inc/TileDB"
+
+  sidebar:
+    - id: placeholder
+      title: ""
+      style: "floating"
+      collapse-level: 2
+      align: left
+
+      contents:
+        - href: 'index.md'
+
+        - section: 'Format Specification'
+          href: "format_spec/FORMAT_SPEC.md"
+          contents:
+            - section: 'Array'
+              contents:
+                - href: "format_spec/array_file_hierarchy.md"
+                - href: "format_spec/array_schema.md"
+                - href: "format_spec/fragment.md"
+                - href: "format_spec/array_metadata.md"
+                - href: "format_spec/tile.md"
+                - href: "format_spec/generic_tile.md"
+            - section: 'Group'
+              contents:
+                - href: "format_spec/group_file_hierarchy.md"
+            - section: 'Other'
+              contents:
+                - href: "format_spec/consolidated_fragment_metadata_file.md"
+                - href: "format_spec/filter_pipeline.md"
+                - href: "format_spec/vacuum_file.md"

format_spec/FORMAT_SPEC.md

@@ -1,9 +1,9 @@
 # Format Specification
 
+**Notes:**  
 
-:information_source: **Notes:**  
-- The current TileDB format version number is **12** (`uint32_t`).
-- All data written by TileDB and referenced in this document is **little-endian**. 
+* The current TileDB format version number is **12** (`uint32_t`).
+* All data written by TileDB and referenced in this document is **little-endian**. 
 
 ## Table of Contents
 
@@ -16,3 +16,7 @@
    * [Generic Tile](./generic_tile.md)
 * **Group**
    * [File hierarchy](./group_file_hierarchy.md)
+* **Other**
+   * [Consolidated Fragment Metadata File](./consolidated_fragment_metadata_file.md)
+   * [Filter Pipeline](./filter_pipeline.md)
+   * [Vacuum Pipeline](./vacuum_file.md)

format_spec/array_file_hierarchy.md

@@ -24,6 +24,7 @@ my_array                                # array folder
 ```
 
 A `<timestamped_name>` above has format `__t1_t2_uuid_v`, where 
+
 * `t1` and `t2` are timestamps in milliseconds elapsed since 1970-01-01 00:00:00 +0000 (UTC)
 * `uuid` is a unique identifier
 * `v` is the format version
@@ -36,4 +37,4 @@ Inside the array folder, you can find the following:
 * Inside the same commit folder, any number of [consolidated commits files](./consolidated_commits_file.md) of the form `<timestamped_name>.con`.
 * Inside the same commit folder, any number of [ignore files](./ignore_file.md) of the form `<timestamped_name>.ign`.
 * Inside of a fragment metadata folder, any number of [consolidated fragment metadata files](./consolidated_fragment_metadata_file.md) of the form `<timestamped_name>.meta`.
-* [Array metadata](./array_metadata.md) folder `__meta`.
+* [Array metadata](./array_metadata.md) folder `__meta`.

format_spec/array_metadata.md

@@ -15,11 +15,13 @@ my_array                            # array folder
 ```
 
 `<timestamped_name>` has format `__t1_t2_uuid_v`, where:
+
 * `t1` and `t2` are timestamps in milliseconds elapsed since 1970-01-01 00:00:00 +0000 (UTC)
 * `uuid` is a unique identifier
 * `v` is the format version
 
 The array metadata folder can contain:
+
 * Any number of [array metadata files](#array-metadata-file)
 * Any number of [vacuum files](./vacuum_file.md)
 

format_spec/array_schema.md

@@ -18,8 +18,8 @@ my_array                            # array folder
 
 
 The array schema folder can contain:
-* Any number of [array schema files](#array-schema-file)
 
+* Any number of [array schema files](#array-schema-file)
 
 ## Previous Array Schema Version
 

format_spec/consolidated_commits_file.md

@@ -11,6 +11,7 @@ my_array                              # array folder
 ```
 
 `<timestamped_name>` has format `__t1_t2_uuid_v`, where:
+
 * `t1` and `t2` are timestamps in milliseconds elapsed since 1970-01-01 00:00:00 +0000 (UTC)
 * `uuid` is a unique identifier
 * `v` is the format version
@@ -21,4 +22,4 @@ There may be multiple such files in the array commits folder. Each consolidated
 | :--- | :--- | :--- |
 | URI 1 followed by a new line character | `uint8_t[]` | URI 1 |
 | … | … | … |
-| URI N followed by a new line character | `uint8_t[]` | URI N |
+| URI N followed by a new line character | `uint8_t[]` | URI N |

format_spec/consolidated_fragment_metadata_file.md

@@ -11,6 +11,7 @@ my_array                              # array folder
 ```
 
 `<timestamped_name>` has format `__t1_t2_uuid_v`, where:
+
 * `t1` and `t2` are timestamps in milliseconds elapsed since 1970-01-01 00:00:00 +0000 (UTC)
 * `uuid` is a unique identifier
 * `v` is the format version
@@ -28,4 +29,4 @@ There may be multiple such files in the array folder. Each consolidated fragment
 | URI N offset | `uint64_t` | The offset in the file where the URI N footer begins |
 | URI 1 footer | [Footer](./fragment.md#footer) | Serialized footer of URI (fragment) 1 |
 | … | … | … |
-| URI N footer | [Footer](./fragment.md#footer) | Serialized footer of URI (fragment) N |
+| URI N footer | [Footer](./fragment.md#footer) | Serialized footer of URI (fragment) N |

format_spec/filter_pipeline.md

@@ -13,9 +13,10 @@ The filter pipeline has internal format:
 | Filter N | [Filter](#filter) | Nth filter |
 
 For var size data, the filter pipeline tries to fit integral cells in a chunk. It uses the following heuristic if the cell doesn't fit:
-  - If the chunk is not yet at 50% capacity, add the cell to the current chunk.
-  - If the chunk is over 50% capacity and adding the cell would make it less than 150% of the maximum chunk size, add it to this chunk.
-  - Else, start a new chunk.
+
+* If the chunk is not yet at 50% capacity, add the cell to the current chunk.
+* If the chunk is over 50% capacity and adding the cell would make it less than 150% of the maximum chunk size, add it to this chunk.
+* Else, start a new chunk.
 
 ## Filter
 
@@ -58,4 +59,4 @@ The filter options for `TILEDB_FILTER_POSITIVE_DELTA` has internal format:
 
 ### Other Filter Options
 
-The remaining filters \(`TILEDB_FILTER_{BITSHUFFLE,BYTESHUFFLE,CHECKSUM_MD5,CHECKSUM_256}` do not serialize any options.
+The remaining filters \(`TILEDB_FILTER_{BITSHUFFLE,BYTESHUFFLE,CHECKSUM_MD5,CHECKSUM_256}` do not serialize any options.

format_spec/fragment.md

@@ -29,6 +29,7 @@ my_array                                    # array folder
 * `v` is the format version
 
 There can be any number of fragments in an array. The fragment folder contains:
+
 * A single [fragment metadata file](#fragment-metadata-file) named `__fragment_metadata.tdb`. 
 * Any number of [data files](#data-file). For each fixed-sized attribute `foo1` (or dimension `bar1`), there is a single data file `a0.tdb` (`d0.tdb`) containing the values along this attribute (dimension). For every var-sized attribute `foo2` (or dimensions `bar2`), there are two data files; `a1_var.tdb` (`d1_var.tdb`) containing the var-sized values of the attribute (dimension) and `a1.tdb` (`d1.tdb`) containing the starting offsets of each value in `a1_var.tdb` (`d1_var.rdb`). Both fixed-sized and var-sized attributes can be nullable. A nullable attribute, `foo3`, will have an additional file `a2_validity.tdb` that contains its validity vector.
 * The names of the data files are not dependent on the names of the attributes/dimensions. The file names are determined by the order of the attributes and dimensions in the array schema.

format_spec/ignore_file.md

@@ -12,6 +12,7 @@ my_array                           # array folder
 or in the array metadata folder:
 
 In the file name:
+
 * `t1` and `t2` are timestamps in milliseconds elapsed since 1970-01-01 00:00:00 +0000 (UTC)
 * `uuid` is a unique identifier
 * `v` is the format version

format_spec/tile.md

@@ -246,8 +246,3 @@ The data output of the encryption filter is:
 | Metadata part N | `uint8_t[]` |  The encrypted bytes of data part N |
 
 Note that the original input metadata in **not** part of the output.
-
-
-
-
-

format_spec/vacuum_file.md

@@ -21,6 +21,7 @@ my_array                        # array folder
 ```
 
 In the file name:
+
 * `t1` and `t2` are timestamps in milliseconds elapsed since 1970-01-01 00:00:00 +0000 (UTC)
 * `uuid` is a unique identifier
 * `v` is the format version

index.md

@@ -0,0 +1,3 @@
+# Public code docs
+
+This is public documentation of the code within this repository.

quarto-materials/tiledb.css

@@ -0,0 +1,55 @@
+/*
+Cloned from https://github.com/TileDB-Inc/tiledb-quarto-template
+
+tiledb light blue #4d9fff
+tiledb dark  blue #0a2580
+*/
+
+.navbar-nav:hover .nav-link:hover {
+  color: #0a2580;
+}
+
+.sidebar-navigation .text-start {
+  font-weight: bold;
+}
+
+.sidebar.sidebar-navigation .active {
+  /*
+  color: #800000;
+  background-color: #e0e0e0;
+  */
+}
+
+.sidebar.sidebar-navigation .active, .sidebar.sidebar-navigation .show>.nav-link {
+  /*color: #0a2580;*/
+  color: #4d9fff;
+  background-color: #e0e0e0;
+  padding-left: 4px;
+  padding-right: 4px;
+}
+
+a {
+  color: #4d9fff;
+}
+a:before, a:focus, a:hover, a:link, a:visited {
+  color: #4d9fff;
+}
+
+code, p code:not(.sourceCode), li code:not(.sourceCode), kbd, pre {
+  color: #000000;
+  background-color: #f0f0f0;
+  font-size: 12px;
+  direction: ltr;
+  border-radius: 3px;
+}
+
+pre {
+  font-size: 12px;
+  padding: 10px;
+  text-decoration: none;
+
+  white-space: pre-wrap;       /* css-3 */
+  white-space: -moz-pre-wrap;  /* Mozilla, since 1999 */
+  white-space: -pre-wrap;      /* Opera 4-6 */
+  white-space: -o-pre-wrap;    /* Opera 7 */
+}

quarto-materials/tiledb.scss

@@ -0,0 +1,26 @@
+/*
+Cloned from https://github.com/TileDB-Inc/tiledb-quarto-template
+
+tiledb light blue #4d9fff
+tiledb dark  blue #0a2580
+*/
+
+/*-- scss:defaults --*/
+/*
+$navbar-bg: #800000;
+$navbar-fg: #eeeeee;
+$sidebar-fg: #800000;
+$footer-bg: #800000;
+$footer-fg: #eeeeee;
+*/
+
+/*-- scss:rules --*/
+h1, h2, h3 {
+  font-weight: 600; /* semibold */
+  /*
+  color: #4d9fff;
+  background-color: #e0e0e0;
+  */
+  border-radius: 6px;
+  padding: 6px;
+}