-
Notifications
You must be signed in to change notification settings - Fork 67
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add support for mounting
.tgz files with filesystem metadata append…
…ed (#477) * Bump dev version * Correct VFS image file extension in lzfs.R script * Remove `.data` assumption in `webr::mount()` * Add support for v2.0 VFS filesystem image format * Avoid Emscripten WORKERFS `mount()` under Node Instead, use our own `mountImageData()` function to create VFS nodes for each file in the VFS metadata package. TODO: This currently handles only metadata given in the form of the `packages` property. Emscripten supports additional `files` and `blobs` properties, and in the future we should also support those here. Fixes #328. * Update webr::mount() to default to v2.0 VFS images * Update documentation for VFS v2.0 * Update NEWS.md * Reorganise VFS mounting into TS module `mount.ts` * Mount as URL under Node if source begins http[s] * Add unit tests for mounting WORKERFS and NODEFS * Export types from webr-chan.ts * Fallback to mounting `.data` before using archive Also improves warning messaging during fallback(s). * Interpret metadata values as signed integers * Read metadata from tar contents if hint is missing * Add unit test for .tgz with no metadata hint
- Loading branch information
1 parent
d455321
commit 4655e96
Showing
21 changed files
with
455 additions
and
163 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| Type: Package | ||
| Package: webr | ||
| Title: WebR Support Package | ||
| Version: 0.4.1 | ||
| Version: 0.4.1.9000 | ||
| Authors@R: c( | ||
| person("George", "Stagg", , "[email protected]", role = c("aut", "cre")), | ||
| person("Lionel", "Henry", , "[email protected]", role = "aut"), | ||
|
|
@@ -17,4 +17,4 @@ Imports: | |
| Encoding: UTF-8 | ||
| LazyData: true | ||
| Roxygen: list(markdown = TRUE) | ||
| RoxygenNote: 7.3.1 | ||
| RoxygenNote: 7.3.2 | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| a, b, c | ||
| 9, 8, 7 | ||
| 4, 5, 6 | ||
| x, y, z | ||
| 1, 2, 3 | ||
| 7, 8, 9 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"files":[{"filename":"/abc/bar.csv","start":0,"end":24},{"filename":"/abc/foo.csv","start":24,"end":48}],"remote_package_size":48} |
Binary file not shown.
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| x, y, z | ||
| 1, 2, 3 | ||
| 7, 8, 9 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| import { FSMetaData, WebR } from '../../webR/webr-main'; | ||
| import fs from 'fs'; | ||
|
|
||
| const webR = new WebR({ | ||
| baseUrl: '../dist/', | ||
| RArgs: ['--quiet'], | ||
| }); | ||
|
|
||
| beforeAll(async () => { | ||
| await webR.init(); | ||
| await webR.evalRVoid('dir.create("/mnt")'); | ||
| }); | ||
|
|
||
| async function cleanupMnt() { | ||
| try { | ||
| await webR.FS.unmount("/mnt"); | ||
| } catch (e) { | ||
| const err = e as Error; | ||
| if (err.message !== "FS error") throw err; | ||
| } | ||
| } | ||
|
|
||
| describe('Mount filesystem using R API', () => { | ||
| test('Mount v1.0 filesystem image', async () => { | ||
| await expect(webR.evalRVoid( | ||
| 'webr::mount("/mnt", "tests/webR/data/test_image.data", "workerfs")' | ||
| )).resolves.not.toThrow(); | ||
| expect(await webR.evalRString("list.files('/mnt/abc')[2]")).toEqual("foo.csv"); | ||
| expect(await webR.evalRString("readLines('/mnt/abc/bar.csv')[1]")).toEqual("a, b, c"); | ||
| await cleanupMnt(); | ||
| }); | ||
|
|
||
| test('Mount v2.0 filesystem image', async () => { | ||
| await expect(webR.evalRVoid( | ||
| 'webr::mount("/mnt", "tests/webR/data/test_image.tar.gz", "workerfs")' | ||
| )).resolves.not.toThrow(); | ||
| expect(await webR.evalRString("list.files('/mnt/abc')[2]")).toEqual("foo.csv"); | ||
| expect(await webR.evalRString("readLines('/mnt/abc/bar.csv')[1]")).toEqual("a, b, c"); | ||
| await cleanupMnt(); | ||
| }); | ||
|
|
||
| test('Mount v2.0 filesystem image - no metadata hint', async () => { | ||
| await expect(webR.evalRVoid( | ||
| 'webr::mount("/mnt", "tests/webR/data/test_image_no_hint.tgz", "workerfs")' | ||
| )).resolves.not.toThrow(); | ||
| expect(await webR.evalRString("list.files('/mnt/abc')[2]")).toEqual("foo.csv"); | ||
| expect(await webR.evalRString("readLines('/mnt/abc/bar.csv')[1]")).toEqual("a, b, c"); | ||
| await cleanupMnt(); | ||
| }); | ||
|
|
||
| test('Mount filesystem image from URL', async () => { | ||
| const url = "https://repo.r-wasm.org/bin/emscripten/contrib/4.4/cli_3.6.3.js.metadata"; | ||
| await expect(webR.evalRVoid(` | ||
| webr::mount("/mnt", "${url}", "workerfs") | ||
| `)).resolves.not.toThrow(); | ||
| expect(await webR.evalRString("readLines('/mnt/DESCRIPTION')[1]")).toEqual("Package: cli"); | ||
| await cleanupMnt(); | ||
| }); | ||
|
|
||
| test('Mount NODEFS filesystem type', async () => { | ||
| await expect(webR.evalRVoid(` | ||
| webr::mount("/mnt", "tests/webR/data/testing", "nodefs") | ||
| `)).resolves.not.toThrow(); | ||
| expect(await webR.evalRString("readLines('/mnt/foo.csv')[2]")).toEqual("1, 2, 3"); | ||
| await cleanupMnt(); | ||
| }); | ||
| }); | ||
|
|
||
| describe('Mount filesystem using JS API', () => { | ||
| test('Mount filesystem image using Buffer', async () => { | ||
| const data = fs.readFileSync("tests/webR/data/test_image.data"); | ||
| const buf = fs.readFileSync("tests/webR/data/test_image.js.metadata"); | ||
| const metadata = JSON.parse(new TextDecoder().decode(buf)) as FSMetaData; | ||
| await expect( | ||
| webR.FS.mount("WORKERFS", { packages: [{ blob: data, metadata: metadata }] }, '/mnt') | ||
| ).resolves.not.toThrow(); | ||
| expect(await webR.evalRString("list.files('/mnt/abc')[2]")).toEqual("foo.csv"); | ||
| expect(await webR.evalRString("readLines('/mnt/abc/bar.csv')[1]")).toEqual("a, b, c"); | ||
| await cleanupMnt(); | ||
| }); | ||
|
|
||
| test('Mount filesystem image using Blob', async () => { | ||
| const data = new Blob([fs.readFileSync("tests/webR/data/test_image.data")]); | ||
| const buf = fs.readFileSync("tests/webR/data/test_image.js.metadata"); | ||
| const metadata = JSON.parse(new TextDecoder().decode(buf)) as FSMetaData; | ||
| await expect( | ||
| webR.FS.mount("WORKERFS", { packages: [{ blob: data, metadata: metadata }] }, '/mnt') | ||
| ).resolves.not.toThrow(); | ||
| expect(await webR.evalRString("list.files('/mnt/abc')[2]")).toEqual("foo.csv"); | ||
| expect(await webR.evalRString("readLines('/mnt/abc/bar.csv')[1]")).toEqual("a, b, c"); | ||
| await cleanupMnt(); | ||
| }); | ||
|
|
||
| test('Mount NODEFS filesystem type', async () => { | ||
| await expect( | ||
| webR.FS.mount("NODEFS", { root: 'tests/webR/data/testing' }, '/mnt') | ||
| ).resolves.not.toThrow(); | ||
| expect(await webR.evalRString("readLines('/mnt/foo.csv')[2]")).toEqual("1, 2, 3"); | ||
| await cleanupMnt(); | ||
| }); | ||
| }); |
Oops, something went wrong.