"Golden spike" PR

.pre-commit-config.yaml

@@ -2,7 +2,7 @@ default_language_version:
   python: python3
 # Skip the pre-commit check for below directories to have
 # a consistency with the official tfrecord preprocessing scripts
-exclude: "^(streaming/text/convert/enwiki/)"
+exclude: "^(examples/text/enwiki_tok/)"
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.

STYLE_GUIDE.md

@@ -142,10 +142,10 @@ so other contributors will know why this error was silenced.
 A public API, generally speaking, can be invoked by a user without a leading underscore in any portion of the path.
 The following are examples of public APIs:
 
-* Standalone functions in public modules (e.g. `streaming.base.distributed.get_world_size`)
-* Classes in public modules (e.g. `streaming.base.format.MDSWriter`)
-* Public methods in public classes (e.g. `streaming.base.format.MDSWriter.write`)
-* Public modules (e.g. `streaming.base.dataset`)
+* Standalone functions in public modules (e.g. `streaming.distributed.get_world_size`)
+* Classes in public modules (e.g. `streaming.format.MDSWriter`)
+* Public methods in public classes (e.g. `streaming.format.MDSWriter.write`)
+* Public modules (e.g. `streaming.dataset`)
 
 The following rules apply to public APIs:
 1. All public APIs must have a docstring (see the Documentation section below)
@@ -201,14 +201,14 @@ All public modules must define `__all__` to be the list of members that should b
 The variable is necessary to 1) limit what `from XXX import *` imports, and 2) ensure that the documentation only
 includes exported members, not unrelated re-imports.
 
-For example, from [streaming/base/dataset.py](streaming/base/dataset.py)
+For example, from [streaming/dataset.py](streaming/dataset.py)
 
 ```python
 """The :class:`Dataset` class, used for building streaming iterable datasets."""
 from torch.utils.data import IterableDataset
 
-from streaming.base.format import reader_from_json
-from streaming.base.spanner import Spanner
+from streaming.format import reader_from_json
+from streaming.spanner import Spanner
 
 __all__ = ["Dataset"]  # export only the Dataset, not other imports like `Spanner` or `reader_from_json`
 

streaming/multimodal/convert/laion/__init__.py → benchmarks/__init__.py

@@ -1,4 +1,4 @@
 # Copyright 2023 MosaicML Streaming authors
 # SPDX-License-Identifier: Apache-2.0
 
-"""LAION dataset creation."""
+"""Streaming benchmarks."""

benchmarks/backends/__init__.py

@@ -0,0 +1,4 @@
+# Copyright 2023 MosaicML Streaming authors
+# SPDX-License-Identifier: Apache-2.0
+
+"""Benchmarking generating/iterating datasets of different backends and formats."""