Allow disk-based persistence to be completely disabled

.vscode/launch.json

@@ -17,6 +17,19 @@
       "skipFiles": ["node_modules/*/**", "<node_internals>/*/**"],
       "outputCapture": "std"
     },
+    {
+      "type": "node",
+      "request": "launch",
+      "name": "Azurite Service - Loki, in-memory",
+      "cwd": "${workspaceFolder}",
+      "runtimeArgs": ["-r", "ts-node/register"],
+      "args": ["${workspaceFolder}/src/azurite.ts", "-d", "debug.log", "--inMemoryPersistence"],
+      "env": {
+        "AZURITE_ACCOUNTS": ""
+      },
+      "skipFiles": ["node_modules/*/**", "<node_internals>/*/**"],
+      "outputCapture": "std"
+    },
     {
       "type": "node",
       "request": "launch",
@@ -204,6 +217,34 @@
       "internalConsoleOptions": "openOnSessionStart",
       "outputCapture": "std"
     },
+    {
+      "type": "node",
+      "request": "launch",
+      "name": "Current Mocha TS File - Loki, in-memory",
+      "cwd": "${workspaceFolder}",
+      "runtimeArgs": ["-r", "ts-node/register"],
+      "args": [
+        "${workspaceFolder}/node_modules/mocha/bin/_mocha",
+        "-u",
+        "tdd",
+        "--timeout",
+        "999999",
+        "--colors",
+        "${workspaceFolder}/${relativeFile}"
+      ],
+      "env": {
+        "AZURITE_ACCOUNTS": "",
+        "AZURE_TABLE_STORAGE": "",
+        "DATATABLES_ACCOUNT_NAME": "<name of your storage account>",
+        "DATATABLES_ACCOUNT_KEY" : "<account key for your storage account>",
+        "AZURE_DATATABLES_STORAGE_STRING": "https://<your account name>.table.core.windows.net",
+        "AZURE_DATATABLES_SAS": "?<sas query string>",
+        "NODE_TLS_REJECT_UNAUTHORIZED": "0",
+        "AZURITE_TEST_INMEMORYPERSISTENCE": "true"
+      },
+      "internalConsoleOptions": "openOnSessionStart",
+      "outputCapture": "std"
+    },
     {
       "name": "VSC Extension",
       "type": "extensionHost",

.vscodeignore

@@ -30,4 +30,5 @@ temp
 .github
 .prettierrc.json
 README.mcr.md
-release
+release
+docs

ChangeLog.md

@@ -4,6 +4,10 @@
 
 ## Upcoming Release
 
+General:
+
+- Add `--inMemoryPersistence`  and `--extentMemoryLimit` options and related configs to store all data in-memory without disk persistence. (issue #2227)
+
 ## 2023.10 Version 3.27.0
 
 General:

README.md

@@ -33,6 +33,8 @@
     - [Certificate Configuration (HTTPS)](#certificate-configuration-https)
     - [OAuth Configuration](#oauth-configuration)
     - [Skip API Version Check](#skip-api-version-check)
+    - [Disable Product Style Url](#disable-product-style-url)
+    - [Use in-memory storage](#use-in-memory-storage)
     - [Command Line Options Differences between Azurite V2](#command-line-options-differences-between-azurite-v2)
   - [Supported Environment Variable Options](#supported-environment-variable-options)
     - [Customized Storage Accounts & Keys](#customized-storage-accounts--keys)
@@ -198,6 +200,8 @@ Following extension configurations are supported:
 - `azurite.oauth` OAuth oauthentication level. Candidate level values: `basic`.
 - `azurite.skipApiVersionCheck` Skip the request API version check, by default false.
 - `azurite.disableProductStyleUrl` Force parsing storage account name from request Uri path, instead of from request Uri host.
+- `azurite.inMemoryPersistence` Disable persisting any data to disk. If the Azurite process is terminated, all data is lost.
+- `azurite.extentMemoryLimit` When using in-memory persistence, limit the total size of extents (blob and queue content) to a specific number of bytes. This does not limit blob, queue, or table metadata. Defaults to 50% of total memory.
 
 ### [DockerHub](https://hub.docker.com/_/microsoft-azure-storage-azurite)
 
@@ -430,6 +434,43 @@ Optional. When using FQDN instead of IP in request Uri host, by default Azurite
 --disableProductStyleUrl
 ```
 
+### Use in-memory storage
+
+Optional. Disable persisting any data to disk and only store data in-memory. If the Azurite process is terminated, all
+data is lost. By default, LokiJS persists blob and queue metadata to disk and content to extent files. Table storage
+persists all data to disk. This behavior can be disabled using this option. This setting is rejected when the SQL based
+metadata implementation is enabled (via `AZURITE_DB`). This setting is rejected when the `--location` option is
+specified.
+
+```cmd
+--inMemoryPersistence
+```
+
+By default, the in-memory extent store (for blob and queue content) is limited to 50% of the total memory on the host
+machine. This is evaluated to using [`os.totalmem()`](https://nodejs.org/api/os.html#ostotalmem). This limit can be
+overridden using the `--extentMemoryLimit <bytes>` option. There is no restriction on the value specified for this
+option but virtual memory may be used if the limit exceeds the amount of available physical memory as provided by the
+operating system. A high limit may eventually lead to out of memory errors or reduced performance.
+
+The queue and blob extent storage count towards the same limit. The `--extentMemoryLimit` setting is rejected when
+`--inMemoryPersistence` is not specified. LokiJS storage (blob and queue metadata and table metadata and content) does
+not contribute to this limit and is unbounded which is the same as without the `--inMemoryPersistence` option.
+
+```cmd
+--extentMemoryLimit <bytes>
+```
+
+This option is rejected when `--inMemoryPersistence` is not specified.
+
+When the limit is reached, write operations to the blob or queue endpoints which carry content will fail with an `HTTP
+409` status code, a custom storage error code of `MemoryExtentStoreAtSizeLimit`, and a helpful error message.
+Well-behaved storage SDKs and tools will not a retry on this failure and will return a related error message. If this
+error is met, consider deleting some in-memory content (blobs or queues), raising the limit, or restarting the Azurite
+server thus resetting the storage completely.
+
+Note that if many hundreds of megabytes of content (queue message or blob content) are stored in-memory, it can take
+noticeably longer than usual for the process to terminate since all the consumed memory needs to be released.
+
 ### Command Line Options Differences between Azurite V2
 
 Azurite V3 supports SharedKey, Account Shared Access Signature (SAS), Service SAS, OAuth, and Public Container Access authentications, you can use any Azure Storage SDKs or tools like Storage Explorer to connect Azurite V3 with any authentication strategy.

azure-pipelines.yml

@@ -38,6 +38,12 @@ jobs:
         displayName: "npm run test:blob"
         env: {}
 
+      - script: |
+          npm run test:blob:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:blob:in-memory"
+        env: {}
+
   - job: blobtestubuntu22_04
     displayName: Blob Test Linux Ubuntu 22.04
     pool:
@@ -65,6 +71,12 @@ jobs:
         displayName: "npm run test:blob"
         env: {}
 
+      - script: |
+          npm run test:blob:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:blob:in-memory"
+        env: {}
+
   - job: blobtestwin
     displayName: Blob Test Windows
     pool:
@@ -92,6 +104,12 @@ jobs:
         displayName: "npm run test:blob"
         env: {}
 
+      - script: |
+          npm run test:blob:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:blob:in-memory"
+        env: {}
+
   - job: blobtestmac
     displayName: Blob Test Mac
     pool:
@@ -119,6 +137,12 @@ jobs:
         displayName: "npm run test:blob"
         env: {}
 
+      - script: |
+          npm run test:blob:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:blob:in-memory"
+        env: {}
+
   - job: blobtestmysql
     displayName: Blob Test Mysql
     pool:
@@ -181,6 +205,12 @@ jobs:
         displayName: "npm run test:queue"
         env: {}
 
+      - script: |
+          npm run test:queue:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:queue:in-memory"
+        env: {}
+
   - job: queuetestwin
     displayName: Queue Test Windows
     pool:
@@ -208,6 +238,12 @@ jobs:
         displayName: "npm run test:queue"
         env: {}
 
+      - script: |
+          npm run test:queue:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:queue:in-memory"
+        env: {}
+
   - job: queuetestmac
     displayName: Queue Test Mac
     pool:
@@ -235,6 +271,12 @@ jobs:
         displayName: "npm run test:queue"
         env: {}
 
+      - script: |
+          npm run test:queue:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:queue:in-memory"
+        env: {}
+
   - job: tabletestlinux
     displayName: Table Test Linux
     pool:
@@ -263,6 +305,12 @@ jobs:
         displayName: "npm run test:table"
         env: {}
 
+      - script: |
+          npm run test:table:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:table:in-memory"
+        env: {}
+
   - job: tabletestwin
     displayName: Table Test Windows
     pool:
@@ -291,6 +339,12 @@ jobs:
         displayName: "npm run test:table"
         env: {}
 
+      - script: |
+          npm run test:table:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:table:in-memory"
+        env: {}
+
   - job: tabletestmac
     displayName: Table Test Mac
     pool:
@@ -319,6 +373,12 @@ jobs:
         displayName: "npm run test:table"
         env: {}
 
+      - script: |
+          npm run test:table:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:table:in-memory"
+        env: {}
+
   - job: azuritenodejslinux
     displayName: Azurite Linux
     pool: