diff --git a/docs/nats/devlog.md b/docs/nats/devlog.md new file mode 100644 index 0000000000..f1d89fe55e --- /dev/null +++ b/docs/nats/devlog.md @@ -0,0 +1,299 @@ +# NATS Development and Integration Log + +## [x] Goal: nats from nodejs + +- start a nats server in cocalc\-docker +- connect from nats cli outside docker +- connect to it from the nodejs client over a websocket + +```sh +nats-server -p 5004 + +nats context save --select --server nats://localhost:5004 nats + +nats sub '>' +``` + +Millions of messages a second works \-\- and you can run like 5x of these at once without saturating nats\-server. + +```js +import { connect, StringCodec } from "nats"; +const nc = await connect({ port: 5004 }); +console.log(`connected to ${nc.getServer()}`); +const sc = StringCodec(); + +const t0 = Date.now(); +for (let i = 0; i < 1000000; i++) { + nc.publish("hello", sc.encode("world")); +} +await nc.drain(); +console.log(Date.now() - t0); +``` + +That was connecting over TCP. Now can we connect via websocket? + +## [x] Goal: Websocket from browser + +First need to start a nats **websocket** server instead on port 5004: + +[https://nats.io/blog/getting\-started\-nats\-ws/](https://nats.io/blog/getting-started-nats-ws/) + +```sh +nats context save --select --server ws://localhost:5004 ws +~/nats/nats.js/lib$ nats context select ws +NATS Configuration Context "ws" + + Server URLs: ws://localhost:5004 + Path: /projects/3fa218e5-7196-4020-8b30-e2127847cc4f/.config/nats/context/ws.json + +~/nats/nats.js/lib$ nats pub foo bar +21:24:53 Published 3 bytes to "foo" +~/nats/nats.js/lib$ +``` + +## + +- their no\-framework html example DOES work for me! +- [https://localhost:4043/projects/3fa218e5\-7196\-4020\-8b30\-e2127847cc4f/files/nats/nats.js/lib/ws.html](https://localhost:4043/projects/3fa218e5-7196-4020-8b30-e2127847cc4f/files/nats/nats.js/lib/ws.html) +- It takes about 1\-2 seconds to send **one million messages** from browser outside docker to what is running inside there! + +## [x] Goal: actually do something useful + +- nats server +- browser connects via websocket port 5004 +- nodejs hub connects via tcp +- hub answers a ping or something else from the browser... + +This worked perfectly with no difficulty. It's very fast and flexible and robust. + +Reconnects work, etc. + +## [x] Goal: proxying + +- nats server with websocket listening on localhost:5004 +- proxy it via node\-proxy in the hub to localhost:4043/nats +- as above + +This totally worked! + +Everything is working that I try?! + +Maybe NATS totally kicks ass. + +## [x] Goal: do something actually useful. + +- authentication: is there a way to too who the user who made the websocket connection is? + - worry about this **later** \- obviously possible and not needed for a POC +- let's try to make `write_text_file_to_project` also be possible via nats. +- OK, made some of api/v2 usable. Obviously this is really minimal POC. + +## [x] GOAL: do something involving the project + +The most interesting use case for nats/jetsteam is timetravel collab editing, where this is all a VERY natural fit. + +But for now, let's just do _something_ at all. + +This worked - I did project exec with subject projects.{project_id}.api + +## [x] Goal: Queue group for hub api + +- change this to be a queue group and test by starting a few servers at once + +## [x] Goal: Auth Strategy that is meaningful + +Creating a creds file that encodes a JWT that says what you can publish and subscribe to, then authenticating with that works. + +- make it so user with account_id can publish to hub.api.{account_id} makes it so we know the account_id automatically by virtue of what was published to. This works. + +## [x] Goal: Solve Critical Auth Problems + +Now need to solve two problems: + +- [x] GOAL: set the creds for a browser client in a secure http cookie, so the browser can't directly access it + +I finally figured this out after WASTING a lot of time with stupid AI misleading me and trying actively to get me to write very stupid insecure code as a lazy workaround. AI really is very, very dangerous... The trick was to read the docs repeatedly, increase logging a lot, and \-\- most imporantly \-\- read the relevant Go source code of NATS itself. The answer is to modify the JWT so that it explicitly has bearer set: `nsc edit user wstein --bearer` + +This makes it so the server doesn't check the signature of the JWT against the _user_ . Putting exactly the JWT token string in the cookie then works because "bearer" literally tells the backend server not to do the signature check. I think this is secure and the right approach because the server checks that the JWT is valid using the account and operator signatures. + +**WAIT!** Using signing keys [https://docs.nats.io/using\-nats/nats\-tools/nsc/signing_keys](https://docs.nats.io/using-nats/nats-tools/nsc/signing_keys) \(and https://youtu.be/KmGtnFxHnVA?si=0uvLMBTJ5TUpem4O \) is VASTLY superior. There's just one JWT issued to each user, and we make a server\-side\-only JWT for their account that has everything. The user never has to reconnect or change their JWT. We can adjust the subject on the fly to account for running projects \(or collaboration changes\) at any time server side. Also the size limits go away, so we don't have to compress project_id's \(probably\). + +## Goal: Implement Auth Solution for Browsers + +- [x] automate creation of creds for browser clients, i.e., what we just did with the nsc tool manually +- + +--- + +This is my top priority goal for NOW! + +What's the plan? + +Need to figure out how to do all the nsc stuff from javascript, storing results in the database? + +- Question: how do we manage creating signing keys and users from nodejs? Answer: clear from many sources that we must use the nsc CLI tool via subprocess calls. Seems fine to me. +- [x] When a user signs in, we check for their JWT in the database. If it is there, set the cookie. If not, create the signing key and JWT for them, save in database, and set the cookie. +- [x] update nats\-server resolver state after modifying signing cookie's subjects configuration. + +``` +nsc edit operator --account-jwt-server-url nats://localhost:4222 +``` + +Now I can do `nsc push` and it just works. + +[x] TODO: when signing out, need to delete the jwt cookie or dangerous private info leaks... and also new info not set properly. + +- [x] similar creds for projects, I.e., access to a project means you can publish to `projects.{project_id}.>` Also, projects should have access to something under hub. + +## [x] Goal: Auth for Projects + +Using an env variable I got a basic useful thing up and running. + +--- + +Some thoughts about project auth security: + +- [ ] when collaborators on a project leave maybe we change JWT? Otherwise, in theory any user of a project can probably somehow get access to the project's JWT \(it's in memory at least\) and still act as the project. Changing JWT requires reconnect. This could be "for later", since even now we don't have this level of security! +- [ ] restarting project could change JWT. That's like the current project's secret token being changed. + +## [ ] Goal: nats-server automation of creation and configuration of system account, operator, etc. + +- This looks helpful: https://www.synadia.com/newsletter/nats-weekly-27/ +- NOT DONE YET + +## [x] Goal: Terminal! Something complicated involving the project which is NOT just request/response + +- Implementing terminals goes beyond request/response. +- It could also leverage jetstream if we want for state (?). +- Multiple connected client + +Project/compute server sends terminal output to + + project.{project_id}.terminal.{sha1(path)} + +Anyone who can read project gets to see this. + +Browser sends terminal input to + + project.{project_id}.{group}.{account_id}.terminal.{sha1(path)} + +API calls: + + - to start terminal + - to get history (move to jetstream?) + +If I can get this to work, then collaborative editing and everything else is basically the same (just more details). + +## [x] Goal: Terminal! #now + +Make it so an actual terminal works, i.e., UI integration. + +## [x] Goal: Terminal JetStream state + +Use Jetstream to store messages from terminal, so user can reconnect without loss. !? This is very interesting... + +First problem -- we used the system account SYS for all our users; however, +SYS can't use jetstreams, as explained here https://github.com/nats-io/nats-server/discussions/6033 + +Let's redo *everything* with a new account called "cocalc". + +```sh +~/nats$ nsc create account --name=cocalc +[ OK ] generated and stored account key "AD4G6R62BDDQUSCJVLZNA7ES7R3A6DWXLYUWGZV74EJ2S6VBC7DQVM3I" +[ OK ] added account "cocalc" +~/nats$ nats context save admin --creds=/projects/3fa218e5-7196-4020-8b30-e2127847cc4f/.local/share/nats/nsc/keys/creds/MyOperator/cocalc/admin.creds +~/nats$ nsc edit account cocalc --js-enable 1 +~/nats$ nsc push -a cocalc +``` + +```js +// making the stream for ALL terminal activity +await jsm.streams.add({ name: 'project-81e0c408-ac65-4114-bad5-5f4b6539bd0e-terminal', subjects: ['project.81e0c408-ac65-4114-bad5-5f4b6539bd0e.terminal.>'] }); + +// making a consumer for just one subject (e.g., one terminal frame) +z = await jsm.consumers.add('project-81e0c408-ac65-4114-bad5-5f4b6539bd0e-terminal',{name:'9149af7632942a94ea13877188153bd8bf2ace57',filter:['project.81e0c408-ac65-4114-bad5-5f4b6539bd0e.terminal.9149af7632942a94ea13877188153bd8bf2ace57']}) +c = await js.consumers.get('project-81e0c408-ac65-4114-bad5-5f4b6539bd0e-terminal', '9149af7632942a94ea13877188153bd8bf2ace57') +for await (const m of await c.consume()) { console.log(cc.client.nats_client.jc.decode(m.data))} +``` + +NOTE!!! The above consumer is ephemeral -- it disappears if we don't grab it via c within a few seconds!!!! https://docs.nats.io/using-nats/developer/develop_jetstream/consumers + +## [ ] Goal: Jetstream permissions + +- [x] project should set up the stream for capturing terminal outputs. +- [x] delete old messages with a given subject. `nats stream purge project-81e0c408-ac65-4114-bad5-5f4b6539bd0e-terminal --seq=7000` + - there is a setting max\_msgs\_per\_subject on a stream, so **we just set that and are done!** Gees. It is too easy. +- [x] handle the other messages like resize +- [x] need to move those other messages to a different subject that isn't part of the stream!! +- [ ] permissions for jetstream usage and access +- [ ] use non\-json for the data.... +- [ ] refactor code so basic parameters \(e.g., subject names, etc.\) are defined in one place that can be imported in both the frontend and backend. +- [ ] font size keyboard shortcut +- [ ] need a better algorithm for sizing since we don't know when a user disconnects! + - when one user proposes a size, all other clients get asked their current size and only those that respond matter. how to do this? + +## [ ] Goal: Basic Collab Document Editing + +Plan. + +- [x] Use a kv store hosted on nats to trac syncstring objects as before. This means anybody can participate \(browser, compute server, project\) without any need to contact the database, hence eliminating all proxying! + +[x] Next Goal \- collaborative file editing \-\- some sort of "proof of concept"! This requires implementing the "ordered patches list" but on jetstream. Similar to the nats SyncTable I wrote yesterday, except will use jetstream directly, since it is an event stream, after all. + +- [x] synctable\-stream: change to one big stream for the whole project but **consume** a specific subject in that stream? + +[ ] cursors \- an ephemeral table + +--- + +- [ ] Subject For Particular File: `project.${project_id}.patches.${sha1(path)}` +- [ ] Stream: Records everything with this subject `project.${project_id}.patches` +- [ ] It would be very nice if we can use the server assigned timestamps.... but probably not + - [ ] For transitioning and de\-archiving, there must be a way to do this, since they have a backup/restore process + +## [ ] Goal: PostgreSQL Changefeed Synctable + +This is critical to solve. This sucks now. This is key to eliminating "hub\-websocket". This might be very easy. Here's the plan: + +- [x] make a request/response listener that listens on hub.account.{account\_id} and hub.db.project.{project\_id} for a db query. +- [x] if changes is false, just responds with the result of the query. +- [ ] if changes is true, get kv store k named `account-{account_id}` or `project-{project_id}` \(which can be used by project or compute server\). + - let id be the sha1 hash of the query \(and options\) + - k.id.update is less than X seconds ago, do nothing... it's already being updated by another server. + - do the query to the database \(with changes true\) + - write the results into k under k.id.data.key = value. + - keep watching for changes so long as k.id.interest is at most n\*X seconds ago. + - Also set k.id.update to now. + - return id +- [ ] another message to `hub.db.{account_id}` which contains a list of id's. + - When get this one, update k.id.interest to now for each of the id's. + +With the above algorithm, it should be very easy to reimplement the client side of SyncTable. Moreover, there are many advantages: + +- For a fixed account\_id or project\-id, there's no extra work at all for 1 versus 100 of them. I.e., this is great for opening a bunch of distinct browser windows. +- If you refresh your browser, everything stays stable \-\- nothing changes at all and you instantly have your data. Same if the network drops and resumes. +- When implementing our new synctable, we can immediately start with the possibly stale data from the last time it was active, then update it to the correct data. Thus even if everything but NATS is done/unavailable, the experience would be much better. It's like "local first", but somehow "network mesh first". With a leaf node it would literally be local first. + +--- + +This is working well! + +TODO: + +- [x] build full proof of concept SyncTable on top of my current implementation of synctablekvatomic, to _make sure it is sufficient_ + - this worked and wasn't too difficult + +THEN do the following to make it robust and scalable + +- [ ] store in nats which servers are actively managing which synctables +- [ ] store in nats the client interest data, instead of storing it in memory in a server? i.e., instead of client making an api call, they could instead just update a kv and say "i am interested in this changefeed". This approach would make everything just keep working easily even as servers scale up/down/restart. + +--- + +## [ ] Goal: Terminal and **compute server** + +Another thing to do for compute servers: + +- use jetstream and KV to agree on _who_ is running the terminal? + +This is critical to see how easily we can support compute servers using nats + jetstream. + diff --git a/src/README.md b/src/README.md index d8af265355..c26fe8e275 100644 --- a/src/README.md +++ b/src/README.md @@ -1,12 +1,12 @@ # How to Build and Run CoCalc -Updated: **Jan 2023** +**Updated: Feb 2025** -CoCalc is a pretty large and complicated project, and it will only work with the current standard LTS release of node.js \( at least 16.8.x\) and a recent version of [pnpm](https://pnpm.io/). +CoCalc is a pretty large and complicated project, and it will only work with the current standard LTS release of node.js \( at least 18.17.1\) and a recent version of [pnpm](https://pnpm.io/). Also, you will need a LOT of RAM, a minimum of 16 GB. **It's very painful to do development with less than 32 GB of RAM.** **Node.js and NPM Version Requirements:** -- You must be using Node version 16.8.x or newer. **CoCalc will definitely NOT work with any older version!** In a [CoCalc.com](http://CoCalc.com) project, you can put this in `~/.bashrc` to get a valid node version: +- You must be using Node version 18.17.1 or newer. **CoCalc will definitely NOT work with any older version!** In a [CoCalc.com](http://CoCalc.com) project, you can put this in `~/.bashrc` to get a valid node version: ```sh . /cocalc/nvm/nvm.sh @@ -56,22 +56,25 @@ To install required dependencies, run hand, you prefer that development packages be installed globally, you can jump directly to the above `pip install` command outside the context of a virtual environment. -## Initial Build +## Build and Start -Launch the install and build **for doing development:** +Launch the install and build **for doing development.** + +If you export the PORT environment variable, that determines what port everything listens on. This determines subtle things about configuration, so do this once and for all in a consistent way. **Note**: If you installed `pnpm` locally (instead of globally), simply run `npm run` in place of `pnpm` to execute these commands via [NPM run scripts](https://docs.npmjs.com/cli/v10/using-npm/scripts). ```sh -~/cocalc/src$ pnpm make-dev +~/cocalc/src$ pnpm build-dev ``` -This will do `pnpm install` for all packages, and also build the typescript/coffeescript, and anything else into a dist directory for each module. Once `pnpm make` finishes successfully, you can start using CoCalc by starting the database and the backend hub in two separate terminals. +This will do `pnpm install` for all packages, and also build the typescript/coffeescript, and anything else into a dist directory for each module. Once `pnpm build-dev` finishes successfully, you can start using CoCalc by starting the database, nats server and the backend hub in three terminals. \(Note that 'pnpm nats\-server' will download, install and configure NATS automatically.\) You can start the database, nats\-server and hub in any order. ```sh -~/cocalc/src$ pnpm database # in one terminal -~/cocalc/src$ pnpm hub # in another terminal +~/cocalc/src$ pnpm database # in one terminal +~/cocalc/src$ pnpm nats-server # in one terminal +~/cocalc/src$ pnpm hub # in another terminal ``` The hub will send minimal logging to stdout, and the rest to `data/logs/log`. @@ -95,7 +98,7 @@ The main \(only?\) difference is that static and next webpack builds are created If necessary, you can delete all the `node_modules` and `dist` directories in all packages and start over as follows: ```sh -~/cocalc/src$ pnpm clean && pnpm make-dev +~/cocalc/src$ pnpm clean && pnpm build-dev ``` ## Doing Development @@ -218,3 +221,4 @@ Regarding VS Code, the relevant settings can be found by searching for "autosave There's some `@cocalc/` packages at [NPMJS.com](http://NPMJS.com). However, _**we're no longer using**_ _**them in any way**_, and don't plan to publish anything new unless there is a compelling use case. + diff --git a/src/package.json b/src/package.json index b3600a4773..aa566602ce 100644 --- a/src/package.json +++ b/src/package.json @@ -17,7 +17,10 @@ "version-check": "pip3 install typing_extensions mypy || pip3 install --break-system-packages typing_extensions mypy && ./workspaces.py version-check && mypy scripts/check_npm_packages.py", "test-parallel": "unset DEBUG && pnpm run version-check && cd packages && pnpm run -r --parallel test", "test": "unset DEBUG && pnpm run version-check && cd packages && pnpm run -r test", - "prettier-all": "cd packages/" + "prettier-all": "cd packages/", + "nats-server": "cd ${COCALC_ROOT:=$INIT_CWD}/packages/backend && node -e \"require('@cocalc/backend/nats/conf').main()\" && cd ${COCALC_ROOT:=$INIT_CWD}/data/nats && ./bin/nats-server -c server.conf", + "nats-server-verbose": "cd ${COCALC_ROOT:=$INIT_CWD}/packages/backend && node -e \"require('@cocalc/backend/nats/conf').main()\" && cd ${COCALC_ROOT:=$INIT_CWD}/data/nats && ./bin/nats-server -DV -c server.conf", + "nats-cli": "echo; echo '# Use CoCalc config of NATS (nats and nsc) via this subshell:'; echo; NATS_URL=nats://${COCALC_NATS_SERVER:=localhost}:${COCALC_NATS_PORT:=4222} XDG_DATA_HOME=${COCALC_ROOT:=$INIT_CWD}/data XDG_CONFIG_HOME=${COCALC_ROOT:=$INIT_CWD}/data PATH=${COCALC_ROOT:=$INIT_CWD}/data/nats/bin:$PATH bash" }, "repository": { "type": "git", diff --git a/src/packages/backend/auth/cookie-names.ts b/src/packages/backend/auth/cookie-names.ts index 8569ad4e04..a192989504 100644 --- a/src/packages/backend/auth/cookie-names.ts +++ b/src/packages/backend/auth/cookie-names.ts @@ -13,6 +13,7 @@ setting the following environment variable: import basePath from "@cocalc/backend/base-path"; import getLogger from "@cocalc/backend/logger"; +import { basePathCookieName } from "@cocalc/util/misc"; const log = getLogger("cookie-names"); @@ -20,15 +21,26 @@ const log = getLogger("cookie-names"); // when the user is signed in. export const REMEMBER_ME_COOKIE_NAME = process.env.COCALC_REMEMBER_ME_COOKIE_NAME ?? - `${basePath.length <= 1 ? "" : encodeURIComponent(basePath)}remember_me`; + basePathCookieName({ basePath, name: "remember_me" }); log.debug("REMEMBER_ME_COOKIE_NAME", REMEMBER_ME_COOKIE_NAME); // Name of user provided api key cookie, with appropriate base path. // This is set by the user when using the api from node.js, especially // via a websocket. -export const API_COOKIE_NAME = - process.env.COCALC_API_COOKIE_NAME ?? - `${basePath.length <= 1 ? "" : encodeURIComponent(basePath)}api_key`; +export const API_COOKIE_NAME = basePathCookieName({ + basePath, + name: "api_key", +}); log.debug("API_COOKIE_NAME", API_COOKIE_NAME); + +export const NATS_JWT_COOKIE_NAME = basePathCookieName({ + basePath, + name: "nats_jwt_cookie", +}); + +export const ACCOUNT_ID_COOKIE_NAME = basePathCookieName({ + basePath, + name: "account_id", +}); diff --git a/src/packages/backend/data.ts b/src/packages/backend/data.ts index df8548e628..abef0389ed 100644 --- a/src/packages/backend/data.ts +++ b/src/packages/backend/data.ts @@ -47,7 +47,7 @@ interface CoCalcSSLEnvConfig extends Dict { SMC_DB_SSL_CA_FILE?: string; SMC_DB_SSL_CLIENT_CERT_FILE?: string; SMC_DB_SSL_CLIENT_KEY_FILE?: string; - SMC_DB_SSL_CLIENT_KEY_PASSPHRASE?:string; + SMC_DB_SSL_CLIENT_KEY_PASSPHRASE?: string; } // This interface is used to specify environment variables to be passed to the "psql" command for @@ -75,11 +75,14 @@ export interface PsqlSSLEnvConfig { // We extend the existing ConnectionOptions interface to include certificate file paths, since these // are used when connecting to Postgres outside of Node (e.g., for raw psql queries). // -export type SSLConfig = ConnectionOptions & { - caFile?: string; - clientCertFile?: string; - clientKeyFile?: string; -} | boolean | undefined; +export type SSLConfig = + | (ConnectionOptions & { + caFile?: string; + clientCertFile?: string; + clientKeyFile?: string; + }) + | boolean + | undefined; /** * Converts an environment-variable-driven SSLEnvConfig into a superset of the SSL context expected @@ -87,7 +90,9 @@ export type SSLConfig = ConnectionOptions & { * * @param env */ -export function sslConfigFromCoCalcEnv(env: CoCalcSSLEnvConfig = process.env): SSLConfig { +export function sslConfigFromCoCalcEnv( + env: CoCalcSSLEnvConfig = process.env, +): SSLConfig { const sslConfig: SSLConfig = {}; if (env.SMC_DB_SSL_CA_FILE) { @@ -101,7 +106,7 @@ export function sslConfigFromCoCalcEnv(env: CoCalcSSLEnvConfig = process.env): S } if (env.SMC_DB_SSL_CLIENT_KEY_FILE) { - sslConfig.clientKeyFile = env.SMC_DB_SSL_CLIENT_KEY_FILE + sslConfig.clientKeyFile = env.SMC_DB_SSL_CLIENT_KEY_FILE; sslConfig.key = readFileSync(env.SMC_DB_SSL_CLIENT_KEY_FILE); } @@ -109,7 +114,9 @@ export function sslConfigFromCoCalcEnv(env: CoCalcSSLEnvConfig = process.env): S sslConfig.passphrase = env.SMC_DB_SSL_CLIENT_KEY_PASSPHRASE; } - return isEmpty(sslConfig) ? (env.SMC_DB_SSL?.toLowerCase() === "true") : sslConfig; + return isEmpty(sslConfig) + ? env.SMC_DB_SSL?.toLowerCase() === "true" + : sslConfig; } /** @@ -174,6 +181,14 @@ export const secrets: string = process.env.SECRETS ?? join(data, "secrets"); export const logs: string = process.env.LOGS ?? join(data, "logs"); export const blobstore: "disk" | "sqlite" = (process.env.COCALC_JUPYTER_BLOBSTORE_IMPL as any) ?? "sqlite"; +export const nats: string = process.env.COCALC_NATS ?? join(data, "nats"); + +export const natsPorts = { + server : parseInt(process.env.COCALC_NATS_PORT ?? '4222'), + ws : parseInt(process.env.COCALC_NATS_WS_PORT ?? '8443') +} +export const natsServer = process.env.COCALC_NATS_SERVER ?? 'localhost'; +export const natsWebsocketServer = `ws://${natsServer}:${natsPorts.ws}`; export let apiKey: string = process.env.API_KEY ?? ""; export let apiServer: string = process.env.API_SERVER ?? ""; diff --git a/src/packages/backend/execute-code.test.ts b/src/packages/backend/execute-code.test.ts index 4a3a43209f..d8a3b1db4f 100644 --- a/src/packages/backend/execute-code.test.ts +++ b/src/packages/backend/execute-code.test.ts @@ -3,6 +3,14 @@ * License: MS-RSL – see LICENSE.md for details */ +/* + +DEVELOPMENT: + +pnpm exec jest --watch --forceExit --detectOpenHandles "execute-code.test.ts" + +*/ + process.env.COCALC_PROJECT_MONITOR_INTERVAL_S = "1"; // default is much lower, might fail if you have more procs than the default process.env.COCALC_PROJECT_INFO_PROC_LIMIT = "10000"; @@ -101,12 +109,12 @@ describe("test timeout", () => { describe("test longer execution", () => { it( - "runs 5 seconds", + "runs 1 seconds", async () => { const t0 = Date.now(); const { stdout, stderr, exit_code } = await executeCode({ command: "sh", - args: ["-c", "echo foo; sleep 5; echo bar"], + args: ["-c", "echo foo; sleep 1; echo bar"], err_on_exit: false, bash: false, }); @@ -114,7 +122,7 @@ describe("test longer execution", () => { expect(stderr).toBe(""); expect(exit_code).toBe(0); const t1 = Date.now(); - expect((t1 - t0) / 1000).toBeGreaterThan(4.9); + expect((t1 - t0) / 1000).toBeGreaterThan(0.9); }, 10 * 1000, ); @@ -256,6 +264,10 @@ describe("async", () => { expect(s.exit_code).toEqual(1); }); + // TODO: I really don't like these tests, which waste a lot of my time. + // Instead of taking 5+ seconds to test some polling implementation, + // they should have a parameter to change the polling interval, so the + // test can be much quicker. -- WS it( "long running async job", async () => { diff --git a/src/packages/backend/execute-code.ts b/src/packages/backend/execute-code.ts index f1208eee1e..bf7acc5cd9 100644 --- a/src/packages/backend/execute-code.ts +++ b/src/packages/backend/execute-code.ts @@ -17,7 +17,6 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { EventEmitter } from "node:stream"; import shellEscape from "shell-escape"; - import getLogger from "@cocalc/backend/logger"; import { envToInt } from "@cocalc/backend/misc/env-to-number"; import { aggregate } from "@cocalc/util/aggregate"; @@ -149,7 +148,7 @@ async function executeCodeNoAggregate( opts.timeout ??= PROJECT_EXEC_DEFAULT_TIMEOUT_S; opts.ulimit_timeout ??= true; opts.err_on_exit ??= true; - opts.verbose ??= true; + opts.verbose ??= false; if (opts.verbose) { log.debug(`input: ${opts.command} ${opts.args?.join(" ")}`); diff --git a/src/packages/backend/get-listing.ts b/src/packages/backend/get-listing.ts index fd3dcdae52..0854ea82b1 100644 --- a/src/packages/backend/get-listing.ts +++ b/src/packages/backend/get-listing.ts @@ -19,27 +19,30 @@ Browser client code only uses this through the websocket anyways. import { reuseInFlight } from "@cocalc/util/reuse-in-flight"; import type { Dirent, Stats } from "node:fs"; -import { lstat, readdir, readlink, stat } from "node:fs/promises"; +import { lstat, opendir, readdir, readlink, stat } from "node:fs/promises"; import { getLogger } from "./logger"; import { DirectoryListingEntry } from "@cocalc/util/types"; import { join } from "path"; -const logger = getLogger("directory-listing"); +const logger = getLogger("backend:directory-listing"); // SMC_LOCAL_HUB_HOME is used for developing cocalc inside cocalc... const HOME = process.env.SMC_LOCAL_HUB_HOME ?? process.env.HOME ?? ""; const getListing = reuseInFlight( - async( + async ( path: string, // assumed in home directory! hidden: boolean = false, - home = HOME, + { home = HOME, limit }: { home?: string; limit?: number } = {}, ): Promise => { const dir = join(home, path); logger.debug(dir); const files: DirectoryListingEntry[] = []; let file: Dirent; - for (file of await readdir(dir, { withFileTypes: true })) { + for await (file of await opendir(dir)) { + if (limit && files.length >= limit) { + break; + } if (!hidden && file.name[0] === ".") { continue; } diff --git a/src/packages/backend/jest.config.js b/src/packages/backend/jest.config.js index 140b9467f2..fdf02d2738 100644 --- a/src/packages/backend/jest.config.js +++ b/src/packages/backend/jest.config.js @@ -1,6 +1,7 @@ /** @type {import('ts-jest').JestConfigWithTsJest} */ module.exports = { - preset: 'ts-jest', - testEnvironment: 'node', - testMatch: ['**/?(*.)+(spec|test).ts?(x)'], + preset: "ts-jest", + testEnvironment: "node", + setupFiles: ["./test/setup.js"], + testMatch: ["**/?(*.)+(spec|test).ts?(x)"], }; diff --git a/src/packages/backend/misc/async-utils-node.ts b/src/packages/backend/misc/async-utils-node.ts index 641f42095a..b908568cb3 100644 --- a/src/packages/backend/misc/async-utils-node.ts +++ b/src/packages/backend/misc/async-utils-node.ts @@ -3,16 +3,7 @@ * License: MS-RSL – see LICENSE.md for details */ -import { access, readFile, unlink } from "node:fs/promises"; +import { readFile, unlink } from "node:fs/promises"; +import { pathExists } from "fs-extra"; -export async function exists(path: string): Promise { - // fs.exists is deprecated - try { - await access(path); - return true; - } catch { - return false; - } -} - -export { readFile, unlink }; +export { readFile, unlink, pathExists as exists }; diff --git a/src/packages/backend/nats/conf.ts b/src/packages/backend/nats/conf.ts new file mode 100644 index 0000000000..35f5e04fe2 --- /dev/null +++ b/src/packages/backend/nats/conf.ts @@ -0,0 +1,121 @@ +/* +Configure nats-server, i.e., generate configuration files. + +node -e "require('@cocalc/backend/nats/conf').main()" + +*/ + +import { pathExists } from "fs-extra"; +import { data, nats, natsPorts, natsServer } from "@cocalc/backend/data"; +import { join } from "path"; +import getLogger from "@cocalc/backend/logger"; +import { writeFile } from "fs/promises"; +import { NATS_JWT_COOKIE_NAME } from "@cocalc/backend/auth/cookie-names"; +import nsc from "./nsc"; +import { executeCode } from "@cocalc/backend/execute-code"; +import { startServer } from "./server"; +import { kill } from "node:process"; +import { delay } from "awaiting"; + +const logger = getLogger("backend:nats:install"); + +// this is assumed in cocalc/src/package.json: +const confPath = join(nats, "server.conf"); + +// for now for local dev: +export const natsServerUrl = `nats://${natsServer}:${natsPorts.server}`; +export const natsAccountName = "cocalc"; + +export async function configureNatsServer() { + logger.debug("configureNatsServer", { confPath }); + if (await pathExists(confPath)) { + logger.debug( + `configureNatsServer: target conf file '${confPath}' already exists so not doing anything`, + ); + return; + } + + await writeFile( + confPath, + ` +listen: ${natsServer}:${natsPorts.server} + +jetstream: enabled + +jetstream { + store_dir: data/nats/jetstream +} + +websocket { + listen: "${natsServer}:${natsPorts.ws}" + no_tls: true + jwt_cookie: "${NATS_JWT_COOKIE_NAME}" +} + +resolver { + type: full + dir: 'data/nats/jwt' + allow_delete: true + interval: "1m" + timeout: "3s" +} + +${await configureNsc()} +`, + ); + + const pid = startServer(); + let d = 1000; + while (true) { + try { + // push initial operator/account/user configuration so its possible + // to configure other accounts + await nsc(["push", "-u", natsServerUrl]); + break; + } catch (err) { + console.log(err); + await delay(d); + d = Math.min(15000, d * 1.3); + } + } + kill(pid); +} + +export async function configureNsc() { + // initialize the local nsc account config + await nsc(["init", "--name", natsAccountName]); + // set the url for the operat + await nsc(["edit", "operator", "--account-jwt-server-url", natsServerUrl]); + // make cocalc user able to pub and sub to everything + await nsc(["edit", "user", "--name", "cocalc", "--allow-pubsub", ">"]); + // enable jetstream for the cocalc account + await nsc(["edit", "account", "--js-mem-storage=-1", "--js-disk-storage=-1"]); + // set nats default context to cocalc user, so using the nats cli works. + await executeCode({ + command: join(nats, "bin", "nats"), + args: [ + "context", + "save", + "--select", + "--nsc=nsc://cocalc/cocalc/cocalc", + "cocalc", + ], + env: { + XDG_DATA_HOME: data, + XDG_CONFIG_HOME: data, + PATH: `${join(nats, "bin")}:${process.env.PATH}`, + }, + verbose: true, + }); + + // return the operator and system_account for inclusion in server config + const { stdout } = await nsc(["generate", "config", "--nats-resolver"]); + const i = stdout.indexOf("system_account"); + const j = stdout.indexOf("\n", i + 1); + return stdout.slice(0, j); +} + +export async function main() { + await configureNatsServer(); + process.exit(0); +} diff --git a/src/packages/backend/nats/env.ts b/src/packages/backend/nats/env.ts new file mode 100644 index 0000000000..598aac94b3 --- /dev/null +++ b/src/packages/backend/nats/env.ts @@ -0,0 +1,9 @@ +import { sha1 } from "@cocalc/backend/sha1"; +import { JSONCodec } from "nats"; +import { getConnection } from "./index"; + +export async function getEnv() { + const jc = JSONCodec(); + const nc = await getConnection(); + return { nc, jc, sha1 }; +} diff --git a/src/packages/backend/nats/index.ts b/src/packages/backend/nats/index.ts new file mode 100644 index 0000000000..fc21010f17 --- /dev/null +++ b/src/packages/backend/nats/index.ts @@ -0,0 +1,56 @@ +import { join } from "path"; +import { nats, natsPorts, natsServer } from "@cocalc/backend/data"; +import { readFile } from "node:fs/promises"; +import getLogger from "@cocalc/backend/logger"; +import { connect, credsAuthenticator } from "nats"; +import { getEnv } from "./env"; +export { getEnv }; +import { delay } from "awaiting"; +import { reuseInFlight } from "@cocalc/util/reuse-in-flight"; +import { CONNECT_OPTIONS } from "@cocalc/util/nats"; +import { inboxPrefix } from "@cocalc/nats/names"; +import { setNatsClient } from "@cocalc/nats/client"; + +export function init() { + setNatsClient({ getNatsEnv: getEnv }); +} +init(); + +const logger = getLogger("backend:nats"); + +export async function getCreds(): Promise { + const filename = join(nats, "nsc/keys/creds/cocalc/cocalc/cocalc.creds"); + try { + return (await readFile(filename)).toString().trim(); + } catch { + logger.debug( + `getCreds -- please create ${filename}, which is missing. Nothing will work.`, + ); + return undefined; + } +} + +let wait = 2000; +let nc: Awaited> | null = null; +export const getConnection = reuseInFlight(async () => { + logger.debug("connecting to nats"); + + while (nc == null) { + try { + const creds = await getCreds(); + nc = await connect({ + ...CONNECT_OPTIONS, + authenticator: credsAuthenticator(new TextEncoder().encode(creds)), + inboxPrefix: inboxPrefix({}), + servers: `${natsServer}:${natsPorts.server}` + }); + logger.debug(`connected to ${nc.getServer()}`); + } catch (err) { + logger.debug(`WARNING/ERROR: FAILED TO CONNECT TO nats-server: ${err}`); + logger.debug(`will retry in ${wait} ms`); + await delay(wait); + wait = Math.min(7500, 1.25 * wait); + } + } + return nc; +}); diff --git a/src/packages/backend/nats/install.ts b/src/packages/backend/nats/install.ts new file mode 100644 index 0000000000..5e6c678cda --- /dev/null +++ b/src/packages/backend/nats/install.ts @@ -0,0 +1,148 @@ +/* +Ensure installed specific correct versions of the following +three GO programs in {data}/nats/bin on this server, correct +for this architecture: + + - nats + - nats-server + - nsc + +We assume curl and python3 are installed. + +DEVELOPMENT: + +Installation happens automatically, e.g,. when you do 'pnpm nats-server' or +start the hub via 'pnpm hub'. However, you can explicitly do +an install as follows: + +~/cocalc/src/packages/backend/nats$ DEBUG=cocalc:* DEBUG_CONSOLE=yes node +Welcome to Node.js v18.17.1. +Type ".help" for more information. +> await require('@cocalc/backend/nats/install').install() + +Installing just the server: + +> await require('@cocalc/backend/nats/install').installNatsServer() +*/ + +import { nats } from "@cocalc/backend/data"; +import { join } from "path"; +import { pathExists } from "fs-extra"; +import { executeCode } from "@cocalc/backend/execute-code"; +import getLogger from "@cocalc/backend/logger"; + +const VERSIONS = { + // https://github.com/nats-io/nats-server/releases + "nats-server": "v2.10.26", + // https://github.com/nats-io/natscli/releases + nats: "v0.1.6", +}; + +export const bin = join(nats, "bin"); +const logger = getLogger("backend:nats:install"); + +export async function install(noUpgrade = false) { + logger.debug("ensure nats binaries installed in ", bin); + + if (!(await pathExists(bin))) { + await executeCode({ command: "mkdir", args: ["-p", bin] }); + } + + await Promise.all([ + installNatsServer(noUpgrade), + installNsc(noUpgrade), + installNatsCli(noUpgrade), + ]); +} + +// call often, but runs at most once and ONLY does something if +// there is no binary i.e., it doesn't upgrade. +let installed = false; +export async function ensureInstalled() { + if (installed) { + return; + } + installed = true; + await install(true); +} + +async function getVersion(name: string) { + try { + const { stdout } = await executeCode({ + command: join(bin, name), + args: ["--version"], + }); + const v = stdout.trim().split(/\s/g); + return v[v.length - 1]; + } catch { + return ""; + } +} + +export async function installNatsServer(noUpgrade) { + if (noUpgrade && (await pathExists(join(bin, "nats-server")))) { + return; + } + if ((await getVersion("nats-server")) == VERSIONS["nats-server"]) { + logger.debug( + `nats-server version ${VERSIONS["nats-server"]} already installed`, + ); + return; + } + const command = `curl -sf https://binaries.nats.dev/nats-io/nats-server/v2@${VERSIONS["nats-server"]} | sh`; + logger.debug("installing nats-server: ", command); + await executeCode({ + command, + path: bin, + verbose: true, + }); +} + +export async function installNsc(noUpgrade) { + const nsc = join(bin, "nsc"); + if (noUpgrade && (await pathExists(nsc))) { + return; + } + + if (!(await pathExists(nsc))) { + await executeCode({ + command: `curl -LO https://raw.githubusercontent.com/nats-io/nsc/main/install.py`, + path: bin, + verbose: true, + }); + const { stdout } = await executeCode({ + path: bin, + env: { PYTHONDONTWRITEBYTECODE: 1 }, + command: + "python3 -c 'import os, sys; sys.path.insert(0,\".\"); import install; print(install.release_url(sys.platform, os.uname()[4], sys.argv[1] if len(sys.argv) > 1 else None))'", + }); + await executeCode({ + command: `curl -sL ${stdout.trim()} -o nsc.zip && unzip nsc.zip -d . && rm nsc.zip install.py`, + path: bin, + verbose: true, + }); + } else { + await executeCode({ + command: nsc, + args: ["update"], + path: bin, + verbose: true, + }); + } +} + +export async function installNatsCli(noUpgrade) { + if (noUpgrade && (await pathExists(join(bin, "nats")))) { + return; + } + if ((await getVersion("nats")) == VERSIONS["nats"]) { + logger.debug(`nats version ${VERSIONS["nats"]} already installed`); + return; + } + logger.debug("installing nats cli"); + await executeCode({ + command: `curl -sf https://binaries.nats.dev/nats-io/natscli/nats@${VERSIONS["nats"]} | sh`, + path: bin, + verbose: true, + }); +} diff --git a/src/packages/backend/nats/nsc.ts b/src/packages/backend/nats/nsc.ts new file mode 100644 index 0000000000..03e5d0510a --- /dev/null +++ b/src/packages/backend/nats/nsc.ts @@ -0,0 +1,33 @@ +/* +Run the Nats nsc command line tool with appropriate environment. + +https://docs.nats.io/using-nats/nats-tools/nsc + +If you want to run nsc in a terminal, do this: + +# DATA=your data/ directory, with data/nats, etc. in it, e.g., +# in a dev install this is cocalc/src/data: + +export DATA=$HOME/cocalc/src/data +export PATH=$DATA/nats/bin:$PATH +export XDG_DATA_HOME=$DATA +export XDG_CONFIG_HOME=$DATA +*/ + +import { bin, ensureInstalled } from "./install"; +import { data } from "@cocalc/backend/data"; +import { join } from "path"; +import { executeCode } from "@cocalc/backend/execute-code"; + +export default async function nsc(args: string[]) { + await ensureInstalled(); // make sure (once) that nsc is installed + return await executeCode({ + command: join(bin, "nsc"), + args, + env: { XDG_DATA_HOME: data, XDG_CONFIG_HOME: data }, + // It is important to set this to false except maybe for temporary debugging! + // Reason is that this command is used to get JWT's, which are basically private keys, + // and it is very bad to log those. + verbose: false, + }); +} diff --git a/src/packages/backend/nats/server.ts b/src/packages/backend/nats/server.ts new file mode 100644 index 0000000000..7e8c3dedc0 --- /dev/null +++ b/src/packages/backend/nats/server.ts @@ -0,0 +1,15 @@ +import { nats } from "@cocalc/backend/data"; +import { join } from "path"; +import { spawn } from "node:child_process"; + +export function startServer(): number { + const { pid } = spawn( + join(nats, "bin", "nats-server"), + ["-c", join(nats, "server.conf")], + { cwd: nats }, + ); + if (pid == null) { + throw Error("issue spawning nats-server"); + } + return pid; +} diff --git a/src/packages/backend/nats/service.ts b/src/packages/backend/nats/service.ts new file mode 100644 index 0000000000..7fca5653c6 --- /dev/null +++ b/src/packages/backend/nats/service.ts @@ -0,0 +1,16 @@ +import { + callNatsService as call, + createNatsService as create, +} from "@cocalc/nats/service"; +import type { + CallNatsServiceFunction, + CreateNatsServiceFunction, +} from "@cocalc/nats/service"; + +import { getEnv } from "@cocalc/backend/nats/env"; + +export const callNatsService: CallNatsServiceFunction = async (opts) => + await call({ ...opts, env: await getEnv() }); + +export const createNatsService: CreateNatsServiceFunction = async (opts) => + await create({ ...opts, env: await getEnv() }); diff --git a/src/packages/backend/nats/sync.ts b/src/packages/backend/nats/sync.ts new file mode 100644 index 0000000000..ef0d9aac36 --- /dev/null +++ b/src/packages/backend/nats/sync.ts @@ -0,0 +1,36 @@ +import { stream as createStream, type Stream } from "@cocalc/nats/sync/stream"; +import { + dstream as createDstream, + type DStream, +} from "@cocalc/nats/sync/dstream"; +import { kv as createKV, type KV } from "@cocalc/nats/sync/kv"; +import { dkv as createDKV, type DKV } from "@cocalc/nats/sync/dkv"; +import { dko as createDKO, type DKO } from "@cocalc/nats/sync/dko"; +import { getEnv } from "@cocalc/backend/nats/env"; +import { createOpenFiles, type OpenFiles } from "@cocalc/nats/sync/open-files"; + +export type { Stream, DStream, KV, DKV, DKO }; + +export async function stream(opts): Promise> { + return await createStream({ env: await getEnv(), ...opts }); +} + +export async function dstream(opts): Promise> { + return await createDstream({ env: await getEnv(), ...opts }); +} + +export async function kv(opts): Promise> { + return await createKV({ env: await getEnv(), ...opts }); +} + +export async function dkv(opts): Promise> { + return await createDKV({ env: await getEnv(), ...opts }); +} + +export async function dko(opts): Promise> { + return await createDKO({ env: await getEnv(), ...opts }); +} + +export async function openFiles(project_id: string, opts?): Promise { + return await createOpenFiles({ env: await getEnv(), project_id, ...opts }); +} diff --git a/src/packages/backend/nats/test/files/read.test.ts b/src/packages/backend/nats/test/files/read.test.ts new file mode 100644 index 0000000000..48bb2c860c --- /dev/null +++ b/src/packages/backend/nats/test/files/read.test.ts @@ -0,0 +1,105 @@ +/* +Test async streaming read of files from a compute servers using NATS. + + +DEVELOPMENT: + +pnpm exec jest --watch --forceExit --detectOpenHandles "read.test.ts" + +*/ + +import "@cocalc/backend/nats"; +import { close, createServer, readFile } from "@cocalc/nats/files/read"; +import { createReadStream } from "fs"; +import { file as tempFile } from "tmp-promise"; +import { writeFile as fsWriteFile } from "fs/promises"; +import { sha1 } from "@cocalc/backend/sha1"; + +describe("do a basic test that the file read service works", () => { + const project_id = "00000000-0000-4000-8000-000000000000"; + const compute_server_id = 0; + it("create the read server", async () => { + await createServer({ + project_id, + compute_server_id, + createReadStream, + }); + }); + + let cleanups: any[] = []; + const CONTENT = "cocalc"; + let source; + it("creates the file we will read", async () => { + const { path, cleanup } = await tempFile(); + source = path; + await fsWriteFile(path, CONTENT); + cleanups.push(cleanup); + }); + + it("reads the file into memory", async () => { + const r = await readFile({ project_id, compute_server_id, path: source }); + // will get just one chunk + for await (const chunk of r) { + expect(chunk.toString()).toEqual(CONTENT); + } + }); + + it("closes the write server", async () => { + close({ project_id, compute_server_id }); + for (const f of cleanups) { + f(); + } + }); +}); + +describe("do a larger test that involves multiple chunks and a different name", () => { + const project_id = "00000000-0000-4000-8000-000000000000"; + const compute_server_id = 0; + const name = "b"; + it("create the read server", async () => { + await createServer({ + project_id, + compute_server_id, + createReadStream, + name, + }); + }); + + let cleanups: any[] = []; + let CONTENT = ""; + for (let i = 0; i < 1000000; i++) { + CONTENT += `${i}`; + } + let source; + it("creates the file we will read", async () => { + const { path, cleanup } = await tempFile(); + source = path; + await fsWriteFile(path, CONTENT); + cleanups.push(cleanup); + }); + + it("reads the file into memory", async () => { + const r = await readFile({ + project_id, + compute_server_id, + path: source, + name, + }); + // will get many chunks. + let chunks: Buffer[] = []; + for await (const chunk of r) { + chunks.push(chunk); + } + expect(chunks.length).toBeGreaterThan(1); + const s = Buffer.concat(chunks).toString(); + expect(s.length).toBe(CONTENT.length); + expect(sha1(s)).toEqual(sha1(CONTENT)); + }); + + it("closes the write server", async () => { + close({ project_id, compute_server_id, name }); + for (const f of cleanups) { + f(); + } + }); +}); diff --git a/src/packages/backend/nats/test/files/write.test.ts b/src/packages/backend/nats/test/files/write.test.ts new file mode 100644 index 0000000000..f0711f8ac9 --- /dev/null +++ b/src/packages/backend/nats/test/files/write.test.ts @@ -0,0 +1,125 @@ +/* +Test async streaming writing of files to compute servers using NATS. + + +DEVELOPMENT: + + pnpm exec jest --watch --forceExit --detectOpenHandles "write.test.ts" + +*/ + +import "@cocalc/backend/nats"; +import { close, createServer, writeFile } from "@cocalc/nats/files/write"; +import { createWriteStream, createReadStream } from "fs"; +import { file as tempFile } from "tmp-promise"; +import { writeFile as fsWriteFile, readFile } from "fs/promises"; +import { sha1 } from "@cocalc/backend/sha1"; + +describe("do a basic test that the file writing service works", () => { + const project_id = "00000000-0000-4000-8000-000000000000"; + const compute_server_id = 0; + it("create the write server", async () => { + await createServer({ + project_id, + compute_server_id, + createWriteStream, + }); + }); + + let cleanups: any[] = []; + const CONTENT = "cocalc"; + let source; + it("creates the file we will read", async () => { + const { path, cleanup } = await tempFile(); + source = path; + await fsWriteFile(path, CONTENT); + cleanups.push(cleanup); + }); + + let dest; + it("write to a new file", async () => { + const { path, cleanup } = await tempFile(); + dest = path; + cleanups.push(cleanup); + + const stream = createReadStream(source); + const { bytes, chunks } = await writeFile({ + stream, + project_id, + compute_server_id, + path, + }); + expect(chunks).toBe(1); + expect(bytes).toBe(CONTENT.length); + }); + + it("confirm that the dest file is correct", async () => { + const d = (await readFile(dest)).toString(); + expect(d).toEqual(CONTENT); + }); + + it("closes the write server", async () => { + close({ project_id, compute_server_id }); + for (const f of cleanups) { + f(); + } + }); +}); + +describe("do a more challenging test that involves a larger file that has to be broken into many chunks", () => { + const project_id = "00000000-0000-4000-8000-000000000000"; + const compute_server_id = 1; + + it("create the write server", async () => { + await createServer({ + project_id, + compute_server_id, + createWriteStream, + }); + }); + + let cleanups: any[] = []; + let CONTENT = ""; + for (let i = 0; i < 1000000; i++) { + CONTENT += `${i}`; + } + let source; + it("creates the file we will read", async () => { + const { path, cleanup } = await tempFile(); + source = path; + await fsWriteFile(path, CONTENT); + cleanups.push(cleanup); + }); + + let dest; + it("write to a new file", async () => { + const { path, cleanup } = await tempFile(); + dest = path; + cleanups.push(cleanup); + + const stream = createReadStream(source); + const { bytes, chunks } = await writeFile({ + stream, + project_id, + compute_server_id, + path, + }); + expect(chunks).toBeGreaterThan(1); + expect(bytes).toBe(CONTENT.length); + }); + + it("confirm that the dest file is correct", async () => { + const d = (await readFile(dest)).toString(); + expect(d.length).toEqual(CONTENT.length); + // not directly comparing, since huge and if something goes wrong the output + // saying the test failed is huge. + expect(sha1(d)).toEqual(sha1(CONTENT)); + }); + + it("closes the write server", async () => { + close({ project_id, compute_server_id }); + for (const f of cleanups) { + f(); + } + }); +}); diff --git a/src/packages/backend/nats/test/llm.test.ts b/src/packages/backend/nats/test/llm.test.ts new file mode 100644 index 0000000000..6514279cf0 --- /dev/null +++ b/src/packages/backend/nats/test/llm.test.ts @@ -0,0 +1,84 @@ +/* +Test LLM NATS streaming. + +DEVELOPMENT: + + pnpm exec jest --watch --forceExit --detectOpenHandles "llm.test.ts" + +*/ + +// this sets client +import "@cocalc/backend/nats"; + +import { init, close } from "@cocalc/nats/llm/server"; +import { llm } from "@cocalc/nats/llm/client"; + +describe("create an llm server, client, and stub evaluator, and run an evaluation", () => { + // define trivial evaluate + const OUTPUT = "Thanks for asing about "; + async function evaluate({ input, stream }) { + stream(OUTPUT); + stream(input); + stream(); + } + + it("creates the server", async () => { + await init(evaluate); + }); + + it("calls the llm", async () => { + const v: string[] = []; + const input = "cocalc"; + const all = await llm({ + account_id: "00000000-0000-4000-8000-000000000000", + system: "in cocalc", + input, + stream: (text) => { + v.push(text); + }, + }); + expect(all).toBe(OUTPUT + input); + expect(v[0]).toBe(OUTPUT); + expect(v[1]).toBe(input); + }); + + it("closes the server", async () => { + await close(); + }); +}); + +describe("test an evaluate that throws an error half way through", () => { + // define trivial evaluate + const OUTPUT = "Thanks for asing about "; + const ERROR = "I give up"; + async function evaluate({ stream }) { + stream(OUTPUT); + throw Error(ERROR); + } + + it("creates the server", async () => { + await init(evaluate); + }); + + it("calls the llm", async () => { + const v: string[] = []; + const input = "cocalc"; + await expect( + async () => + await llm({ + account_id: "00000000-0000-4000-8000-000000000000", + system: "in cocalc", + input, + stream: (text) => { + v.push(text); + }, + }), + ).rejects.toThrow(ERROR); + expect(v[0]).toBe(OUTPUT); + expect(v.length).toBe(1); + }); + + it("closes the server", async () => { + await close(); + }); +}); diff --git a/src/packages/backend/nats/test/service.test.ts b/src/packages/backend/nats/test/service.test.ts new file mode 100644 index 0000000000..4a1f9d45e8 --- /dev/null +++ b/src/packages/backend/nats/test/service.test.ts @@ -0,0 +1,35 @@ +/* + +DEVELOPMENT: + +pnpm exec jest --watch --forceExit --detectOpenHandles "service.test.ts" + +*/ + +import { + callNatsService, + createNatsService, +} from "@cocalc/backend/nats/service"; + +describe("create a service and test it out", () => { + let s; + it("creates a service", async () => { + s = await createNatsService({ service: "echo", handler: (mesg) => mesg }); + expect(await callNatsService({ service: "echo", mesg: "hello" })).toBe( + "hello", + ); + }); + it("closes the services", async () => { + s.close(); + + let t = ""; + // expect( ...).toThrow doesn't seem to work with this: + try { + await callNatsService({ service: "echo", mesg: "hi" }); + } catch (err) { + t = `${err}`; + } + expect(t).toContain("Not Available"); + }); +}); + diff --git a/src/packages/backend/nats/test/sync/dko.test.ts b/src/packages/backend/nats/test/sync/dko.test.ts new file mode 100644 index 0000000000..4a5a414330 --- /dev/null +++ b/src/packages/backend/nats/test/sync/dko.test.ts @@ -0,0 +1,107 @@ +/* +Testing basic ops with kv + +DEVELOPMENT: + +pnpm exec jest --watch --forceExit --detectOpenHandles "dko.test.ts" + +*/ + +import { dko as createDko } from "@cocalc/backend/nats/sync"; +import { once } from "@cocalc/util/async-utils"; + +describe("create a public kv and do basic operations", () => { + let kv; + const name = `test-${Math.random()}`; + + it("creates the kv", async () => { + kv = await createDko({ name }); + expect(kv.getAll()).toEqual({}); + }); + + it("adds a key to the kv", () => { + kv.a = { x: 10 }; + expect(kv.a).toEqual({ x: 10 }); + }); + + it("complains if value is not an object", () => { + expect(() => { + kv.x = 5; + }).toThrow("object"); + }); + + it("waits for the kv to be longterm saved, then closing and recreates the kv and verifies that the key is there.", async () => { + await kv.save(); + kv.close(); + kv = await createDko({ name }); + expect(kv.a).toEqual({ x: 10 }); + }); + + it("closes the kv", async () => { + kv.close(); + expect(kv.getAll).toThrow("closed"); + }); +}); + +describe("opens a kv twice and verifies the cached works and is reference counted", () => { + let kv1; + let kv2; + const name = `test-${Math.random()}`; + + it("creates the same kv twice", async () => { + kv1 = await createDko({ name }); + kv2 = await createDko({ name }); + expect(kv1.getAll()).toEqual({}); + expect(kv1 === kv2).toBe(true); + }); + + it("closes kv1 (one reference)", async () => { + kv1.close(); + expect(kv2.getAll).not.toThrow(); + }); + + it("closes kv2 (another reference)", async () => { + kv2.close(); + // really closed! + expect(kv2.getAll).toThrow("closed"); + }); + + it("create and see it is new now", async () => { + kv1 = await createDko({ name }); + expect(kv1 === kv2).toBe(false); + }); +}); + +describe("opens a kv twice at once and observe sync", () => { + let kv1; + let kv2; + const name = `test-${Math.random()}`; + + it("creates the kv twice", async () => { + kv1 = await createDko({ name, noCache: true }); + kv2 = await createDko({ name, noCache: true }); + expect(kv1.getAll()).toEqual({}); + expect(kv2.getAll()).toEqual({}); + expect(kv1 === kv2).toBe(false); + }); + + it("sets a value in one and sees that it is NOT instantly set in the other", () => { + kv1.a = { x: 25 }; + expect(kv2.a).toBe(undefined); + }); + + it("awaits save and then sees the value *eventually* appears in the other", async () => { + kv1.save(); + // initially not there. + while (kv2.a?.x === undefined) { + await once(kv2, "change"); + } + expect(kv2.a).toEqual(kv1.a); + }); + + it("close up", () => { + kv1.close(); + kv2.close(); + }); +}); + diff --git a/src/packages/backend/nats/test/sync/dkv-merge.test.ts b/src/packages/backend/nats/test/sync/dkv-merge.test.ts new file mode 100644 index 0000000000..25f7b37b46 --- /dev/null +++ b/src/packages/backend/nats/test/sync/dkv-merge.test.ts @@ -0,0 +1,175 @@ +/* +Testing merge conflicts with dkv + +DEVELOPMENT: + +pnpm exec jest --watch --forceExit --detectOpenHandles "dkv-merge.test.ts" + +*/ + +import { dkv as createDkv } from "@cocalc/backend/nats/sync"; +import { once } from "@cocalc/util/async-utils"; +import { diff_match_patch } from "@cocalc/util/dmp"; + +async function getKvs(opts?) { + const name = `test-${Math.random()}`; + // We disable autosave so that we have more precise control of how conflicts + // get resolved, etc. for testing purposes. + const kv1 = await createDkv( + { name, noAutosave: true, ...opts, noCache: true }, + ); + const kv2 = await createDkv( + { name, noAutosave: true, ...opts, noCache: true }, + ); + return { kv1, kv2 }; +} + +describe("test the default 'local first' merge conflict resolution function", () => { + it("sets up and resolves a merge conflict", async () => { + const { kv1, kv2 } = await getKvs(); + kv1.set("x", 5); + kv2.set("x", 10); + expect(kv1["x"]).toEqual(5); + expect(kv2["x"]).toEqual(10); + + // now make kv2 save, which makes kv1 detect a conflict: + await kv2.save(); + // kv1 just resolves it in its own favor. + expect(kv1["x"]).toEqual(5); + await kv1.save(); + + // wait until kv2 gets a change, which will be learning about + // how the merge conflict got resolved. + if (kv2.get("x") != 5) { + // might have to wait + await once(kv2, "change"); + } + expect(kv2["x"]).toEqual(5); + }); +}); + +describe("test the default 'local first' merge conflict resolution function, but where we do the sets in the opposite order", () => { + it("sets up and resolves a merge conflict", async () => { + const { kv1, kv2 } = await getKvs(); + kv2.set("x", 10); + kv1.set("x", 5); + expect(kv1["x"]).toEqual(5); + expect(kv2["x"]).toEqual(10); + + // now make kv2 save, which makes kv1 detect a conflict: + await kv2.save(); + // kv1 just resolves it in its own favor. + expect(kv1["x"]).toEqual(5); + await kv1.save(); + + // wait until kv2 gets a change, which will be learning about + // how the merge conflict got resolved. + if (kv2.get("x") != 5) { + // might have to wait + await once(kv2, "change"); + } + expect(kv2["x"]).toEqual(5); + }); +}); + +describe("test a trivial merge conflict resolution function", () => { + it("sets up and resolves a merge conflict", async () => { + const { kv1, kv2 } = await getKvs({ + merge: () => { + // our merge strategy is to replace the value by 'conflict' + return "conflict"; + }, + }); + kv1.set("x", 5); + kv2.set("x", 10); + expect(kv1["x"]).toEqual(5); + expect(kv2["x"]).toEqual(10); + + // now make kv2 save, which makes kv1 detect a conflict: + await kv2.save(); + if (kv1["x"] != "conflict") { + // might have to wait + await once(kv1, "change"); + } + expect(kv1["x"]).toEqual("conflict"); + + await kv1.save(); + // wait until kv2 gets a change, which will be learning about + // how the merge conflict got resolved. + if (kv2["x"] != "conflict") { + // might have to wait + await once(kv2, "change"); + } + expect(kv2["x"]).toEqual("conflict"); + }); +}); + +describe("test a 3-way merge of strings conflict resolution function", () => { + const dmp = new diff_match_patch(); + const threeWayMerge = (opts: { + prev: string; + local: string; + remote: string; + }) => { + return dmp.patch_apply( + dmp.patch_make(opts.prev, opts.local), + opts.remote, + )[0]; + }; + it("sets up and resolves a merge conflict", async () => { + const { kv1, kv2 } = await getKvs({ + merge: ({ local, remote, prev = "" }) => { + // our merge strategy is to replace the value by 'conflict' + return threeWayMerge({ local, remote, prev }); + }, + }); + kv1.set("x", "cocalc"); + await kv1.save(); + if (kv2["x"] != "cocalc") { + // might have to wait + await once(kv2, "change"); + } + expect(kv2["x"]).toEqual("cocalc"); + await kv2.save(); + + kv2.set("x", "cocalc!"); + kv1.set("x", "LOVE cocalc"); + await kv2.save(); + if (kv1.get("x") != "LOVE cocalc!") { + await once(kv1, "change"); + } + expect(kv1.get("x")).toEqual("LOVE cocalc!"); + await kv1.save(); + if (kv2.get("x") != "LOVE cocalc!") { + await once(kv2, "change"); + } + expect(kv2.get("x")).toEqual("LOVE cocalc!"); + }); +}); + +describe("test a 3-way merge of that merges objects", () => { + it("sets up and resolves a merge conflict", async () => { + const { kv1, kv2 } = await getKvs({ + merge: ({ local, remote }) => { + return { ...remote, ...local }; + }, + }); + kv1.set("x", { a: 5 }); + await kv1.save(); + await once(kv2, "change"); + expect(kv2["x"]).toEqual({ a: 5 }); + + kv1.set("x", { a: 5, b: 15, c: 12 }); + kv2.set("x", { a: 5, b: 7, d: 3 }); + await kv2.save(); + if (kv1.get("x").d != 3) { + await once(kv1, "change"); + } + expect(kv1.get("x")).toEqual({ a: 5, b: 15, c: 12, d: 3 }); + await kv1.save(); + if (kv2.get("x").b != 15) { + await once(kv2, "change"); + } + expect(kv2.get("x")).toEqual({ a: 5, b: 15, c: 12, d: 3 }); + }); +}); diff --git a/src/packages/backend/nats/test/sync/dkv.test.ts b/src/packages/backend/nats/test/sync/dkv.test.ts new file mode 100644 index 0000000000..eca64ad709 --- /dev/null +++ b/src/packages/backend/nats/test/sync/dkv.test.ts @@ -0,0 +1,353 @@ +/* +Testing basic ops with dkv + +DEVELOPMENT: + +pnpm exec jest --watch --forceExit --detectOpenHandles "dkv.test.ts" + +*/ + +import { dkv as createDkv } from "@cocalc/backend/nats/sync"; +import { once } from "@cocalc/util/async-utils"; +import { delay } from "awaiting"; + +describe("create a public dkv and do basic operations", () => { + let kv; + const name = `test-${Math.random()}`; + + it("creates the dkv", async () => { + kv = await createDkv({ name }); + expect(kv.getAll()).toEqual({}); + }); + + it("adds a key to the dkv", () => { + kv.a = 10; + expect(kv.a).toEqual(10); + }); + + it("waits for the dkv to be longterm saved, then closing and recreates the kv and verifies that the key is there.", async () => { + await kv.save(); + kv.close(); + kv = await createDkv({ name }); + expect(kv.a).toEqual(10); + }); + + it("closes the kv", async () => { + kv.close(); + expect(kv.getAll).toThrow("closed"); + }); +}); + +describe("opens a dkv twice and verifies the cache works and is reference counted", () => { + let kv1; + let kv2; + const name = `test-${Math.random()}`; + + it("creates the same dkv twice", async () => { + kv1 = await createDkv({ name }); + kv2 = await createDkv({ name }); + expect(kv1.getAll()).toEqual({}); + expect(kv1 === kv2).toBe(true); + }); + + it("closes kv1 (one reference)", async () => { + kv1.close(); + expect(kv2.getAll).not.toThrow(); + }); + + it("closes kv2 (another reference)", async () => { + kv2.close(); + // really closed! + expect(kv2.getAll).toThrow("closed"); + }); + + it("create and see it is new now", async () => { + kv1 = await createDkv({ name }); + expect(kv1 === kv2).toBe(false); + }); +}); + +describe("opens a dkv twice at once and observe sync", () => { + let kv1; + let kv2; + const name = `test-${Math.random()}`; + + it("creates the dkv twice", async () => { + kv1 = await createDkv({ name, noCache: true }); + kv2 = await createDkv({ name, noCache: true }); + expect(kv1.getAll()).toEqual({}); + expect(kv2.getAll()).toEqual({}); + expect(kv1 === kv2).toBe(false); + }); + + it("sets a value in one and sees that it is NOT instantly set in the other", () => { + kv1.a = 25; + expect(kv2.a).toBe(undefined); + }); + + it("awaits save and then sees the value *eventually* appears in the other", async () => { + await kv1.save(); + // initially not there. + expect(kv2.a).toBe(undefined); + await once(kv2, "change"); + expect(kv2.a).toBe(kv1.a); + }); + + it("close up", () => { + kv1.close(); + kv2.close(); + }); +}); + +describe("check server assigned times", () => { + let kv; + const name = `test-${Math.random()}`; + + it("create a kv", async () => { + kv = await createDkv({ name }); + expect(kv.getAll()).toEqual({}); + expect(kv.time()).toEqual({}); + }); + + it("set a key, then get the time and confirm it is reasonable", async () => { + kv.a = { b: 7 }; + // not serve assigned yet + expect(kv.time("a")).toEqual(undefined); + kv.save(); + await once(kv, "change"); + // now we must have it. + // sanity check: within a second + expect(kv.time("a").getTime()).toBeCloseTo(Date.now(), -3); + // all the times + expect(Object.keys(kv.time()).length).toBe(1); + }); + + it("setting again with a *different* value changes the time", async () => { + kv.a = { b: 8 }; + const t0 = kv.time("a"); + await once(kv, "change"); + expect(kv.time("a").getTime()).toBeCloseTo(Date.now(), -3); + expect(t0).not.toEqual(kv.time("a")); + }); + + it("close", () => { + kv.close(); + }); +}); + +describe("test deleting and clearing a dkv", () => { + let kv1; + let kv2; + + const reset = async () => { + const name = `test-${Math.random()}`; + kv1 = await createDkv({ name, noCache: true }); + kv2 = await createDkv({ name, noCache: true }); + }; + + it("creates the dkv twice without caching so can make sure sync works", async () => { + await reset(); + expect(kv1.getAll()).toEqual({}); + expect(kv2.getAll()).toEqual({}); + expect(kv1 === kv2).toBe(false); + }); + + it("adds an entry, deletes it and confirms", async () => { + kv1.foo = "bar"; + expect(kv1.has("foo")).toBe(true); + expect(kv2.has("foo")).toBe(false); + await once(kv2, "change"); + expect(kv2.foo).toBe(kv1.foo); + expect(kv2.has("foo")).toBe(true); + delete kv1.foo; + await once(kv2, "change"); + expect(kv2.foo).toBe(undefined); + expect(kv2.has("foo")).toBe(false); + }); + + it("adds an entry, clears it and confirms", async () => { + await reset(); + + kv1.foo10 = "bar"; + await once(kv2, "change"); + expect(kv2.foo10).toBe(kv1.foo10); + kv2.clear(); + expect(kv2.has("foo10")).toBe(false); + await once(kv1, "change"); + expect(kv1.has("foo10")).toBe(false); + }); + + it("adds an entry, syncs, adds another local entry (not sync'd), clears in sync and confirms NOT everything was cleared", async () => { + await reset(); + kv1["foo"] = Math.random(); + await kv1.save(); + if (kv2["foo"] != kv1["foo"]) { + await once(kv2, "change"); + } + expect(kv2["foo"]).toBe(kv1["foo"]); + kv1["bar"] = "yyy"; + expect(kv2["bar"]).toBe(undefined); + // this ONLY clears 'foo', not 'bar' + kv2.clear(); + await once(kv1, "change"); + expect(kv1.has("bar")).toBe(true); + }); + + it("adds an entry, syncs, adds another local entry (not sync'd), clears in first one, and confirms everything was cleared", async () => { + await reset(); + + const key = Math.random(); + kv1[key] = Math.random(); + await kv1.save(); + if (kv2[key] != kv1[key]) { + await once(kv2, "change"); + } + const key2 = Math.random(); + kv1[key2] = "yyy"; + expect(kv2[key2]).toBe(undefined); + // this ONLY clears foo, not xxx + kv1.clear(); + expect(kv1.has(key2)).toBe(false); + }); +}); + +describe("set several items, confirm write worked, save, and confirm they are still there after save", () => { + const name = `test-${Math.random()}`; + const count = 50; + // the time thresholds should be "trivial" + it(`adds ${count} entries`, async () => { + const kv = await createDkv({ name }); + expect(kv.getAll()).toEqual({}); + const obj: any = {}; + const t0 = Date.now(); + for (let i = 0; i < count; i++) { + obj[`${i}`] = i; + kv.set(`${i}`, i); + } + expect(Date.now() - t0).toBeLessThan(50); + expect(Object.keys(kv.getAll()).length).toEqual(count); + expect(kv.getAll()).toEqual(obj); + await kv.save(); + expect(Date.now() - t0).toBeLessThan(1000); + expect(Object.keys(kv.getAll()).length).toEqual(count); + // // the local state maps should also get cleared quickly, + // // but there is no event for this, so we loop: + // @ts-ignore: saved is private + while (Object.keys(kv.generalDKV.local).length > 0) { + await delay(10); + } + // @ts-ignore: local is private + expect(kv.generalDKV.local).toEqual({}); + // @ts-ignore: saved is private + expect(kv.generalDKV.saved).toEqual({}); + }); +}); + +describe("do an insert and clear test", () => { + const name = `test-${Math.random()}`; + const count = 100; + it(`adds ${count} entries, saves, clears, and confirms empty`, async () => { + const kv = await createDkv({ name }); + expect(kv.getAll()).toEqual({}); + for (let i = 0; i < count; i++) { + kv[`${i}`] = i; + } + expect(Object.keys(kv.getAll()).length).toEqual(count); + await kv.save(); + expect(Object.keys(kv.getAll()).length).toEqual(count); + kv.clear(); + expect(kv.getAll()).toEqual({}); + await kv.save(); + expect(kv.getAll()).toEqual({}); + }); +}); + +describe("create many distinct clients at once, write to all of them, and see that that results are merged", () => { + const name = `test-${Math.random()}`; + const count = 5; + const clients: any[] = []; + + it(`creates the ${count} clients`, async () => { + for (let i = 0; i < count; i++) { + clients[i] = await createDkv({ name, noCache: true }); + } + }); + + // what the combination should be + let combined: any = {}; + it("writes a separate key/value for each client", () => { + for (let i = 0; i < count; i++) { + clients[i].set(`${i}`, i); + combined[`${i}`] = i; + expect(clients[i].get(`${i}`)).toEqual(i); + } + }); + + it("saves and checks that everybody has the combined values", async () => { + for (const kv of clients) { + await kv.save(); + } + let done = false; + let i = 0; + while (!done && i < 50) { + done = true; + i += 1; + for (const client of clients) { + if (client.length != count) { + done = false; + await delay(10); + break; + } + } + } + for (const client of clients) { + expect(client.length).toEqual(count); + expect(client.getAll()).toEqual(combined); + } + }); +}); + +describe("tests involving null/undefined values", () => { + let kv1; + let kv2; + const name = `test-${Math.random()}`; + + it("creates the dkv twice", async () => { + kv1 = await createDkv({ name, noAutosave: true, noCache: true }); + kv2 = await createDkv({ name, noAutosave: true, noCache: true }); + expect(kv1.getAll()).toEqual({}); + expect(kv1 === kv2).toBe(false); + }); + + it("sets a value to null, which is fully supported like any other value", () => { + kv1.a = null; + expect(kv1.a).toBe(null); + expect(kv1.a === null).toBe(true); + expect(kv1.length).toBe(1); + }); + + it("make sure null value sync's as expected", async () => { + kv1.save(); + await once(kv2, "change"); + expect(kv2.a).toBe(null); + expect(kv2.a === null).toBe(true); + expect(kv2.length).toBe(1); + }); + + it("sets a value to undefined, which is the same as deleting a value", () => { + kv1.a = undefined; + expect(kv1.a).toBe(undefined); + expect(kv1.a === undefined).toBe(true); + expect(kv1.length).toBe(0); + expect(kv1.getAll()).toEqual({}); + }); + + it("make sure undefined (i.e., delete) sync's as expected", async () => { + kv1.save(); + await once(kv2, "change"); + expect(kv2.a).toBe(undefined); + expect(kv2.a === undefined).toBe(true); + expect(kv2.length).toBe(0); + expect(kv1.getAll()).toEqual({}); + }); +}); diff --git a/src/packages/backend/nats/test/sync/dstream.test.ts b/src/packages/backend/nats/test/sync/dstream.test.ts new file mode 100644 index 0000000000..d660674190 --- /dev/null +++ b/src/packages/backend/nats/test/sync/dstream.test.ts @@ -0,0 +1,253 @@ +/* +Testing basic ops with dsteam (distributed streams) + +DEVELOPMENT: + +pnpm exec jest --watch --forceExit --detectOpenHandles "dstream.test.ts" + +*/ + +import { dstream as createDstream } from "@cocalc/backend/nats/sync"; +import { once } from "@cocalc/util/async-utils"; + +async function create() { + const name = `test-${Math.random()}`; + return await createDstream({ name, noAutosave: true }); +} + +describe("create a dstream and do some basic operations", () => { + let s; + + it("creates stream", async () => { + s = await create(); + }); + + it("starts out empty", () => { + expect(s.getAll()).toEqual([]); + expect(s.length).toEqual(0); + }); + + const mesg = { stdout: "hello" }; + it("publishes a message to the stream and confirms it is there", () => { + s.push(mesg); + expect(s.getAll()).toEqual([mesg]); + expect(s.length).toEqual(1); + expect(s[0]).toEqual(mesg); + }); + + it("verifies that unsaved changes works properly", async () => { + expect(s.hasUnsavedChanges()).toBe(true); + expect(s.unsavedChanges()).toEqual([mesg]); + await s.save(); + expect(s.hasUnsavedChanges()).toBe(false); + expect(s.unsavedChanges()).toEqual([]); + }); + + it("confirm persistence: closes and re-opens stream and confirms message is still there", async () => { + const name = s.name; + await s.save(); + // close s: + await s.close(); + // using s fails + expect(s.getAll).toThrow("closed"); + // create new stream with same name + const t = await createDstream({ name }); + // ensure it is NOT just from the cache + expect(s === t).toBe(false); + // make sure it has our message + expect(t.getAll()).toEqual([mesg]); + }); +}); + +describe("create two dstreams and observe sync between them", () => { + const name = `test-${Math.random()}`; + let s1, s2; + it("creates two distinct dstream objects s1 and s2 with the same name", async () => { + s1 = await createDstream({ name, noAutosave: true, noCache: true }); + s2 = await createDstream({ name, noAutosave: true, noCache: true }); + // definitely distinct + expect(s1 === s2).toBe(false); + }); + + it("writes to s1 and observes s2 doesn't see anything until we save", async () => { + s1.push("hello"); + expect(s1[0]).toEqual("hello"); + expect(s2.length).toEqual(0); + s1.save(); + await once(s2, "change"); + expect(s2[0]).toEqual("hello"); + expect(s2.getAll()).toEqual(["hello"]); + }); + + it("now write to s2 and save and see that reflected in s1", async () => { + s2.push("hi from s2"); + s2.save(); + await once(s1, "change"); + expect(s1[1]).toEqual("hi from s2"); + }); + + it("write to s1 and s2 and save at the same time and see some 'random choice' of order gets imposed by the server", async () => { + s1.push("s1"); + s2.push("s2"); + // our changes are reflected locally + expect(s1.getAll()).toEqual(["hello", "hi from s2", "s1"]); + expect(s2.getAll()).toEqual(["hello", "hi from s2", "s2"]); + // now kick off the two saves *in parallel* + s1.save(); + s2.save(); + await once(s1, "change"); + if (s2.length != s1.length) { + await once(s2, "change"); + } + expect(s1.getAll()).toEqual(s2.getAll()); + // in fact s1,s2 is the order since we called s1.save first: + expect(s1.getAll()).toEqual(["hello", "hi from s2", "s1", "s2"]); + }); +}); + +describe("get sequence number and time of message", () => { + let s; + + it("creates stream and write message", async () => { + s = await create(); + s.push("hello"); + }); + + it("sequence number is initialized undefined because it is server assigned ", async () => { + const n = s.seq(0); + expect(n).toBe(undefined); + }); + + it("time also undefined because it is server assigned ", async () => { + const t = s.time(0); + expect(t).toBe(undefined); + }); + + it("save and get server assigned sequence number", async () => { + s.save(); + await once(s, "change"); + const n = s.seq(0); + expect(n).toBeGreaterThan(0); + }); + + it("get server assigned time", async () => { + const t = s.time(0); + // since testing on the same machine as server, these times should be close: + expect(t.getTime() - Date.now()).toBeLessThan(5000); + }); + + it("publish another message and get next server number is bigger", async () => { + const n = s.seq(0); + s.push("there"); + await s.save(); + const m = s.seq(1); + expect(m).toBeGreaterThan(n); + }); + + it("and time is bigger", async () => { + if (s.time(1) == null) { + await once(s, "change"); + } + expect(s.time(0).getTime()).toBeLessThan(s.time(1).getTime()); + }); +}); + +describe("closing also saves by default, but not if autosave is off", () => { + let s; + const name = `test-${Math.random()}`; + + it("creates stream and write a message", async () => { + s = await createDstream({ name, noAutosave: false /* the default */ }); + s.push(389); + }); + + it("closes then opens and message is there, since autosave is on", async () => { + await s.close(); + const t = await createDstream({ name }); + expect(t[0]).toEqual(389); + }); + + it("make another stream with autosave off, and close which causes LOSS OF DATA", async () => { + const name = `test-${Math.random()}`; + const s = await createDstream({ name, noAutosave: true }); + s.push(389); + s.close(); + const t = await createDstream({ name, noAutosave: true }); + // data is gone forever! + expect(t.length).toBe(0); + }); +}); + +describe("testing start_seq", () => { + const name = `test-${Math.random()}`; + let seq; + it("creates a stream and adds 3 messages, noting their assigned sequence numbers", async () => { + const s = await createDstream({ name, noAutosave: true }); + s.push(1, 2, 3); + expect(s.getAll()).toEqual([1, 2, 3]); + // save, thus getting sequence numbers + s.save(); + while (s.seq(2) == null) { + s.save(); + await once(s, "change"); + } + seq = [s.seq(0), s.seq(1), s.seq(2)]; + // tests partly that these are integers... + const n = seq.reduce((a, b) => a + b, 0); + expect(typeof n).toBe("number"); + expect(n).toBeGreaterThan(2); + await s.close(); + }); + + let s; + it("it opens the stream but starting with the last sequence number, so only one message", async () => { + s = await createDstream({ + name, + noAutosave: true, + start_seq: seq[2], + }); + expect(s.length).toBe(1); + expect(s.getAll()).toEqual([3]); + }); + + it("it then pulls in the previous message, so now two messages are loaded", async () => { + await s.load({ start_seq: seq[1] }); + expect(s.length).toBe(2); + expect(s.getAll()).toEqual([2, 3]); + }); +}); + +describe("a little bit of a stress test", () => { + const name = `test-${Math.random()}`; + const count = 100; + let s; + it(`creates a stream and pushes ${count} messages`, async () => { + s = await createDstream({ + name, + noAutosave: true, + }); + for (let i = 0; i < count; i++) { + s.push({ i }); + } + expect(s.length).toBe(count); + // NOTE: warning -- this is **MUCH SLOWER**, e.g., 10x slower, + // running under jest, hence why count is small. + await s.save(); + expect(s.length).toBe(count); + }); +}); + +describe("dstream typescript test", () => { + it("creates stream", async () => { + const name = `test-${Math.random()}`; + const s = await createDstream({ name }); + + // write a message with the correct type + s.push("foo"); + + // wrong type -- no way to test this, but if you uncomment + // this you should get a typescript error: + + // s.push({ foo: "bar" }); + }); +}); diff --git a/src/packages/backend/nats/test/sync/open-files.test.ts b/src/packages/backend/nats/test/sync/open-files.test.ts new file mode 100644 index 0000000000..12a00a6af3 --- /dev/null +++ b/src/packages/backend/nats/test/sync/open-files.test.ts @@ -0,0 +1,117 @@ +/* +Unit test basic functionality of the openFiles distributed key:value +store. Projects and compute servers use this to know what files +to open so they can fulfill their backend responsibilities: + - computation + - save to disk + - load from disk when file changes + +DEVELOPMENT: + +pnpm exec jest --watch --forceExit --detectOpenHandles "open-files.test.ts" + +*/ + +import { openFiles as createOpenFiles } from "@cocalc/backend/nats/sync"; +import { once } from "@cocalc/util/async-utils"; +import { delay } from "awaiting"; + +const project_id = "00000000-0000-4000-8000-000000000000"; +async function create() { + return await createOpenFiles(project_id, { noAutosave: true, noCache: true }); +} + +describe("create open file tracker and do some basic operations", () => { + let o1, o2; + let file1 = `${Math.random()}.txt`; + let file2 = `${Math.random()}.txt`; + + it("creates two open files tracker (tracking same project) and clear them", async () => { + o1 = await create(); + o2 = await create(); + // ensure caching disable so our sync tests are real + expect(o1.getDkv() === o2.getDkv()).toBe(false); + o1.clear(); + await o1.save(); + expect(o1.hasUnsavedChanges()).toBe(false); + o2.clear(); + while (o2.hasUnsavedChanges()) { + try { + // expected due to merge conflict and autosave being disabled. + await o2.save(); + } catch { + await delay(10); + } + } + }); + + it("confirm they are cleared", async () => { + expect(o1.getAll()).toEqual([]); + expect(o2.getAll()).toEqual([]); + }); + + it("touch file in one and observe change and timestamp getting assigned by server", async () => { + o1.touch(file1); + expect(o1.get(file1).time).toBeCloseTo(Date.now(), -3); + }); + + it("touches file in one and observes change by OTHER", async () => { + o1.touch(file2); + expect(o1.get(file2)?.path).toBe(file2); + expect(o2.get(file2)).toBe(undefined); + o1.save(); + if (o2.get(file2) == null) { + await once(o2, "change", 250); + expect(o2.get(file2).path).toBe(file2); + expect(o2.get(file2).time == null).toBe(false); + } + }); + + it("get all in o2 sees both file1 and file2", async () => { + const v = o2.getAll(); + expect(v[0].path).toBe(file1); + expect(v[1].path).toBe(file2); + expect(v.length).toBe(2); + }); + + it("delete file1", async () => { + o1.delete(file1); + expect(o1.get(file1)).toBe(undefined); + expect(o1.getAll().length).toBe(1); + o1.save(); + await delay(1000); + if (o2.get(file1) != null) { + await once(o2, "change", 250); + } + expect(o2.get(file1)).toBe(undefined); + // should be 1 due to file2 still being there: + expect(o2.getAll().length).toBe(1); + }); + + it("sets an error", async () => { + o2.setError(file2, Error("test error")); + expect(o2.get(file2).error.error).toBe("Error: test error"); + expect(typeof o2.get(file2).error.time == "number").toBe(true); + expect(Math.abs(Date.now() - o2.get(file2).error.time)).toBeLessThan(10000); + try { + // get a conflict due to above so resolve it... + await o2.save(); + } catch { + o2.save(); + } + if (!o1.get(file2).error) { + await once(o1, "change", 250); + } + expect(o1.get(file2).error.error).toBe("Error: test error"); + }); + + it("clears the error", async () => { + o1.setError(file2); + expect(o1.get(file2).error).toBe(undefined); + o1.save(); + if (o2.get(file2).error) { + await once(o2, "change", 250); + } + expect(o2.get(file2).error).toBe(undefined); + }); +}); diff --git a/src/packages/backend/nats/test/time.test.ts b/src/packages/backend/nats/test/time.test.ts new file mode 100644 index 0000000000..d15fd3bf3f --- /dev/null +++ b/src/packages/backend/nats/test/time.test.ts @@ -0,0 +1,30 @@ +/* +DEVELOPMENT: + +pnpm exec jest --watch --forceExit --detectOpenHandles "time.test.ts" +*/ + +// this sets client +import "@cocalc/backend/nats"; + +import time, { getSkew } from "@cocalc/nats/time"; + +describe("get time from nats", () => { + it("tries to get the time before the skew, so it is not initialized yet", () => { + expect(time).toThrow("clock skew not known"); + }); + + it("gets the skew, so that time is initialized", async () => { + const skew = await getSkew(); + expect(Math.abs(skew)).toBeLessThan(1000); + }); + + it("gets the time, which should be close to our time on a test system", () => { + // times in ms, so divide by 1000 so expecting to be within a second + expect(time() / 1000).toBeCloseTo(Date.now() / 1000, 0); + }); + + it("time is a number", () => { + expect(typeof time()).toBe("number"); + }); +}); diff --git a/src/packages/backend/package.json b/src/packages/backend/package.json index 0cb3d239ef..75d44684ad 100644 --- a/src/packages/backend/package.json +++ b/src/packages/backend/package.json @@ -5,6 +5,7 @@ "exports": { "./*": "./dist/*.js", "./database": "./dist/database/index.js", + "./nats": "./dist/nats/index.js", "./server-settings": "./dist/server-settings/index.js", "./auth/*": "./dist/auth/*.js", "./auth/tokens/*": "./dist/auth/tokens/*.js" @@ -18,7 +19,7 @@ "clean": "rm -rf dist node_modules", "build": "pnpm exec tsc --build", "tsc": "pnpm exec tsc --watch --pretty --preserveWatchOutput", - "test": "pnpm exec jest --detectOpenHandles", + "test": "pnpm exec jest --forceExit --detectOpenHandles", "prepublishOnly": "pnpm test" }, "files": [ @@ -31,6 +32,7 @@ "license": "SEE LICENSE.md", "dependencies": { "@cocalc/backend": "workspace:*", + "@cocalc/nats": "workspace:*", "@cocalc/util": "workspace:*", "@types/debug": "^4.1.12", "@types/watchpack": "^2.4.4", @@ -40,11 +42,13 @@ "fs-extra": "^11.2.0", "lodash": "^4.17.21", "lru-cache": "^7.18.3", + "nats": "^2.29.1", "password-hash": "^1.2.2", "prom-client": "^13.0.0", "rimraf": "^5.0.5", "shell-escape": "^0.2.0", "supports-color": "^9.0.2", + "tmp-promise": "^3.0.3", "underscore": "^1.12.1" }, "repository": { diff --git a/src/packages/backend/path-watcher.ts b/src/packages/backend/path-watcher.ts index d7ec85413e..71e03c6958 100644 --- a/src/packages/backend/path-watcher.ts +++ b/src/packages/backend/path-watcher.ts @@ -4,7 +4,7 @@ */ /* -Watch A DIRECTORY for changes of the files in *that* directory only (not recursive). +Watch A DIRECTORY for changes of the files in *that* directory only (not recursive). Use ./watcher.ts for a single file. Slightly generalized fs.watch that works even when the directory doesn't exist, @@ -54,7 +54,7 @@ const logger = getLogger("backend:path-watcher"); const POLLING = true; const DEFAULT_POLL_MS = parseInt( - process.env.COCALC_FS_WATCHER_POLL_INTERVAL_MS ?? "3000", + process.env.COCALC_FS_WATCHER_POLL_INTERVAL_MS ?? "2000", ); const ChokidarOpts: WatchOptions = { @@ -165,3 +165,40 @@ export class Watcher extends EventEmitter { close(this); } } + +export class MultipathWatcher extends EventEmitter { + private paths: { [path: string]: Watcher } = {}; + private options; + + constructor(options?) { + super(); + this.options = options; + } + + has = (path: string) => { + return this.paths[path] != null; + }; + + add = (path: string) => { + if (this.has(path)) { + // already watching + return; + } + this.paths[path] = new Watcher(path, this.options); + this.paths[path].on("change", () => this.emit("change", path)); + }; + + delete = (path: string) => { + if (!this.has(path)) { + return; + } + this.paths[path].close(); + delete this.paths[path]; + }; + + close = () => { + for (const path in this.paths) { + this.delete(path); + } + }; +} diff --git a/src/packages/backend/sha1.test.ts b/src/packages/backend/sha1.test.ts index c4e680ad25..ab087c32f8 100644 --- a/src/packages/backend/sha1.test.ts +++ b/src/packages/backend/sha1.test.ts @@ -1,4 +1,5 @@ -import { sha1, uuidsha1 } from "./sha1"; +import { sha1, sha1base64, uuidsha1 } from "./sha1"; +import * as misc from "@cocalc/util/misc"; const cocalc = "CoCalc"; const hash = "c898c97dca68742a5a6331f9fa0ca02483cbfd25"; @@ -30,3 +31,27 @@ describe("UUIDs", () => { expect(uuidsha1(Buffer.from(cocalc))).toBe(uuid); }); }); + +describe("compare this nodejs implementation to the pure javascript implementation", () => { + it("CoCalc/string", () => { + expect(sha1(cocalc)).toBe(misc.sha1(cocalc)); + }); + it("hash", () => { + expect(sha1(hash)).toBe(misc.sha1(hash)); + }); + it("uuid", () => { + expect(sha1(uuid)).toBe(misc.sha1(uuid)); + }); +}); + +describe("base64: compare this nodejs implementation to the pure javascript implementation", () => { + it("CoCalc/string", () => { + expect(sha1base64(cocalc)).toBe(misc.sha1base64(cocalc)); + }); + it("hash", () => { + expect(sha1base64(hash)).toBe(misc.sha1base64(hash)); + }); + it("uuid", () => { + expect(sha1base64(uuid)).toBe(misc.sha1base64(uuid)); + }); +}); diff --git a/src/packages/backend/sha1.ts b/src/packages/backend/sha1.ts index c3863c1c78..bf96f2aff1 100644 --- a/src/packages/backend/sha1.ts +++ b/src/packages/backend/sha1.ts @@ -2,12 +2,14 @@ sha1 hash functionality */ -import { createHash } from "crypto"; +import { createHash, type BinaryToTextEncoding } from "crypto"; // compute sha1 hash of data in hex -export function sha1(data: Buffer | string): string { +export function sha1( + data: Buffer | string, + encoding: BinaryToTextEncoding = "hex", +): string { const sha1sum = createHash("sha1"); - if (typeof data === "string") { sha1sum.update(data, "utf8"); } else { @@ -20,11 +22,16 @@ export function sha1(data: Buffer | string): string { sha1sum.update(uint8Array); } - return sha1sum.digest("hex"); + return sha1sum.digest(encoding); +} + +export function sha1base64(data: Buffer | string): string { + return sha1(data, "base64"); } // Compute a uuid v4 from the Sha-1 hash of data. // Optionally, if knownSha1 is given, just uses that, rather than recomputing it. +// WARNING: try to avoid using this, since it discards information! export function uuidsha1(data: Buffer | string, knownSha1?: string): string { const s = knownSha1 ?? sha1(data); let i = -1; diff --git a/src/packages/backend/test/setup.js b/src/packages/backend/test/setup.js new file mode 100644 index 0000000000..6ed5b82df9 --- /dev/null +++ b/src/packages/backend/test/setup.js @@ -0,0 +1,4 @@ +// test/setup.js + +// checked for in some code to behave differently while running unit tests. +process.env.COCALC_TEST_MODE = true; diff --git a/src/packages/backend/tsconfig.json b/src/packages/backend/tsconfig.json index 43ebbc564b..8d855cf7c6 100644 --- a/src/packages/backend/tsconfig.json +++ b/src/packages/backend/tsconfig.json @@ -7,5 +7,5 @@ "outDir": "dist" }, "exclude": ["node_modules", "dist", "test"], - "references": [{ "path": "../util" }] + "references": [{ "path": "../util", "path": "../nats" }] } diff --git a/src/packages/comm/websocket/types.ts b/src/packages/comm/websocket/types.ts index 43c33bfe0b..4264656c5a 100644 --- a/src/packages/comm/websocket/types.ts +++ b/src/packages/comm/websocket/types.ts @@ -11,7 +11,7 @@ between the frontend app and the project. import type { NBGraderAPIOptions, RunNotebookOptions, -} from "@cocalc/jupyter/nbgrader/types"; +} from "@cocalc/util/jupyter/nbgrader-types"; import type { Channel } from "@cocalc/sync/client/types"; import type { Options } from "@cocalc/util/code-formatter"; export type { Channel }; diff --git a/src/packages/database/nats/changefeeds.ts b/src/packages/database/nats/changefeeds.ts new file mode 100644 index 0000000000..7a5a656426 --- /dev/null +++ b/src/packages/database/nats/changefeeds.ts @@ -0,0 +1,271 @@ +/* + +1. turn off nats-server handling for the hub by sending this message from a browser as an admin: + + await cc.client.nats_client.hub.system.terminate({service:'db'}) + +2. Run this + + require("@cocalc/database/nats/changefeeds").init() + + echo 'require("@cocalc/database/nats/changefeeds").init()' | node + +*/ + +import getLogger from "@cocalc/backend/logger"; +import { JSONCodec } from "nats"; +import userQuery from "@cocalc/database/user-query"; +import { getConnection } from "@cocalc/backend/nats"; +import { getUserId } from "@cocalc/nats/hub-api"; +import { callback } from "awaiting"; +import { db } from "@cocalc/database"; +import { + createSyncTable, + CHANGEFEED_INTEREST_PERIOD_MS, +} from "@cocalc/nats/sync/synctable"; +import { sha1 } from "@cocalc/backend/misc_node"; +import jsonStableStringify from "json-stable-stringify"; +import { reuseInFlight } from "@cocalc/util/reuse-in-flight"; +import { uuid } from "@cocalc/util/misc"; +import { delay } from "awaiting"; +import { Svcm } from "@nats-io/services"; + +const logger = getLogger("database:nats:changefeeds"); + +const jc = JSONCodec(); + +let api: any | null = null; +export async function init() { + const subject = "hub.*.*.db"; + logger.debug(`init -- subject='${subject}', options=`, { + queue: "0", + }); + const nc = await getConnection(); + + // @ts-ignore + const svcm = new Svcm(nc); + + const service = await svcm.add({ + name: "db-server", + version: "0.1.0", + description: "CoCalc Database Service (changefeeds)", + }); + + api = service.addEndpoint("api", { subject }); + + for await (const mesg of api) { + handleRequest(mesg, nc); + } +} + +export function terminate() { + logger.debug("terminating service"); + api?.stop(); + api = null; + // also, stop reporting data into the streams + cancelAllChangefeeds(); +} + +async function handleRequest(mesg, nc) { + let resp; + try { + const { account_id, project_id } = getUserId(mesg.subject); + const { name, args } = jc.decode(mesg.data) ?? ({} as any); + // logger.debug(`got request: "${JSON.stringify({ name, args })}"`); + if (!name) { + throw Error("api endpoint name must be given in message"); + } + // logger.debug("handling server='db' request:", { + // account_id, + // project_id, + // name, + // }); + resp = await getResponse({ name, args, account_id, project_id, nc }); + } catch (err) { + // logger.debug("ERROR", err); + resp = { error: `${err}` }; + } + // logger.debug(`Responding with "${JSON.stringify(resp)}"`); + mesg.respond(jc.encode(resp)); +} + +async function getResponse({ name, args, account_id, project_id, nc }) { + if (name == "userQuery") { + const opts = { ...args[0], account_id, project_id }; + if (!opts.changes) { + // a normal query + return await userQuery(opts); + } else { + return await createChangefeed(opts, nc); + } + } else { + throw Error(`name='${name}' not implemented`); + } +} + +function queryTable(query) { + return Object.keys(query)[0]; +} + +const changefeedHashes: { [id: string]: string } = {}; +const changefeedInterest: { [hash: string]: number } = {}; +const changefeedSynctables: { [hash: string]: any } = {}; + +function cancelChangefeed(id) { + logger.debug("cancelChangefeed", { id }); + const hash = changefeedHashes[id]; + if (!hash) { + // already canceled + return; + } + changefeedSynctables[hash]?.close(); + delete changefeedSynctables[hash]; + delete changefeedInterest[hash]; + delete changefeedHashes[id]; + db().user_query_cancel_changefeed({ id }); +} + +function cancelAllChangefeeds() { + logger.debug("cancelAllChangefeeds"); + for (const id in changefeedHashes) { + cancelChangefeed(id); + } +} + +// This is tricky. We return the first result as a normal +// async function, but then handle (and don't return) +// the subsequent calls to cb generated by the changefeed. +const createChangefeed = reuseInFlight( + async (opts, nc) => { + const query = opts.query; + // the query *AND* the user making it define the thing: + const user = { account_id: opts.account_id, project_id: opts.project_id }; + const hash = sha1( + jsonStableStringify({ + query, + ...user, + }), + ); + const now = Date.now(); + if (changefeedInterest[hash]) { + changefeedInterest[hash] = now; + logger.debug("using existing changefeed for", queryTable(query), user); + return; + } + logger.debug("creating new changefeed for", queryTable(query), user); + const changes = uuid(); + changefeedHashes[changes] = hash; + const env = { nc, jc, sha1 }; + // If you change any settings below, you might also have to change them in + // src/packages/sync/table/changefeed-nats.ts + const synctable = createSyncTable({ + query, + env, + account_id: opts.account_id, + project_id: opts.project_id, + // atomic = false is just way too slow due to the huge number of distinct + // messages, which NATS is not as good with. + atomic: true, + immutable: false, + }); + changefeedSynctables[hash] = synctable; + + try { + await synctable.init(); + } catch (err) { + cancelChangefeed(changes); + } + + // if (global.z == null) { + // global.z = {}; + // } + // global.z[synctable.table] = synctable; + + const handleFirst = ({ cb, err, rows }) => { + if (err || rows == null) { + cb(err ?? "missing result"); + return; + } + const current = synctable.get(); + const databaseKeys = new Set(); + for (const obj of rows) { + databaseKeys.add(synctable.getKey(obj)); + synctable.set(obj); + } + for (const key in current) { + if (!databaseKeys.has(key)) { + // console.log("remove from synctable", key); + synctable.delete(key); + } + } + cb(); + }; + + const handleUpdate = ({ action, new_val, old_val }) => { + // action = 'insert', 'update', 'delete', 'close' + // e.g., {"action":"insert","new_val":{"title":"testingxxxxx","project_id":"81e0c408-ac65-4114-bad5-5f4b6539bd0e"}} + const obj = new_val ?? old_val; + if (obj == null) { + // nothing we can do with this + return; + } + if (action == "insert" || action == "update") { + const cur = synctable.get(new_val); + // logger.debug({ table: queryTable(query), action, new_val, old_val }); + synctable.set({ ...cur, ...new_val }); + } else if (action == "delete") { + synctable.delete(old_val); + } else if (action == "close") { + cancelChangefeed(changes); + } + }; + + const f = (cb) => { + let first = true; + db().user_query({ + ...opts, + changes, + cb: (err, x) => { + if (first) { + first = false; + handleFirst({ cb, err, rows: x?.[synctable.table] }); + return; + } + handleUpdate(x as any); + }, + }); + }; + try { + await callback(f); + // it's running successfully + changefeedInterest[hash] = Date.now(); + + const watch = async () => { + // it's all setup and running. If there's no interest for a while, stop watching + while (true) { + await delay(CHANGEFEED_INTEREST_PERIOD_MS); + if ( + Date.now() - changefeedInterest[hash] > + CHANGEFEED_INTEREST_PERIOD_MS + ) { + logger.debug( + "insufficient interest in the changefeed, so we stop it.", + query, + ); + cancelChangefeed(changes); + return; + } + } + }; + + // do not block on this. + watch(); + return; + } catch (err) { + // if anything goes wrong, make sure we don't think the changefeed is working. + cancelChangefeed(changes); + throw err; + } + }, + { createKey: (args) => jsonStableStringify(args[0]) }, +); diff --git a/src/packages/database/package.json b/src/packages/database/package.json index 8055b6a727..0642d0a0ea 100644 --- a/src/packages/database/package.json +++ b/src/packages/database/package.json @@ -5,6 +5,7 @@ "exports": { ".": "./dist/index.js", "./accounts/*": "./dist/accounts/*.js", + "./nats/*": "./dist/nats/*.js", "./pool": "./dist/pool/index.js", "./pool/*": "./dist/pool/*.js", "./postgres/*": "./dist/postgres/*.js", @@ -19,7 +20,9 @@ "dependencies": { "@cocalc/backend": "workspace:*", "@cocalc/database": "workspace:*", + "@cocalc/nats": "workspace:*", "@cocalc/util": "workspace:*", + "@nats-io/services": "3.0.0-25", "@types/lodash": "^4.14.202", "@types/pg": "^8.6.1", "@types/uuid": "^8.3.1", @@ -31,6 +34,7 @@ "json-stable-stringify": "^1.0.1", "lodash": "^4.17.21", "lru-cache": "^7.18.3", + "nats": "^2.29.1", "node-fetch": "2.6.7", "pg": "^8.7.1", "random-key": "^0.3.2", diff --git a/src/packages/database/postgres-server-queries.coffee b/src/packages/database/postgres-server-queries.coffee index 96c826c8e7..e995ae09e7 100644 --- a/src/packages/database/postgres-server-queries.coffee +++ b/src/packages/database/postgres-server-queries.coffee @@ -62,19 +62,10 @@ read = require('read') passwordHash = require("@cocalc/backend/auth/password-hash").default; registrationTokens = require('./postgres/registration-tokens').default; {updateUnreadMessageCount} = require('./postgres/messages'); +centralLog = require('./postgres/central-log').default; stripe_name = require('@cocalc/util/stripe/name').default; -# log events, which contain personal information (email, account_id, ...) -PII_EVENTS = ['create_account', - 'change_password', - 'change_email_address', - 'webapp-add_passport', - 'get_user_auth_token', - 'successful_sign_in', - 'webapp-email_sign_up', - 'create_account_registration_token' - ] exports.extend_PostgreSQL = (ext) -> class PostgreSQL extends ext # write an event to the central_log table @@ -83,27 +74,11 @@ exports.extend_PostgreSQL = (ext) -> class PostgreSQL extends ext event : required # string value : required # object cb : undefined - - # always expire central_log entries after 1 year, unless … - expire = expire_time(365*24*60*60) - # exception events expire earlier - if opts.event == 'uncaught_exception' - expire = misc.expire_time(30 * 24 * 60 * 60) # del in 30 days - else - # and user-related events according to the PII time, although "never" falls back to 1 year - v = opts.value - if v.ip_address? or v.email_address? or opts.event in PII_EVENTS - expire = await pii_expire(@) ? expire - - @_query - query : 'INSERT INTO central_log' - values : - 'id::UUID' : misc.uuid() - 'event::TEXT' : opts.event - 'value::JSONB' : opts.value - 'time::TIMESTAMP' : 'NOW()' - 'expire::TIMESTAMP' : expire - cb : (err) => opts.cb?(err) + try + await centralLog(opts) + opts.cb?() + catch err + opts.cb?(err) uncaught_exception: (err) => # call when things go to hell in some unexpected way; at least @@ -1168,56 +1143,6 @@ exports.extend_PostgreSQL = (ext) -> class PostgreSQL extends ext opts.cb(err) ) - ### - User auth token - ### - # save an auth token in the database - save_auth_token: (opts) => - opts = defaults opts, - account_id : required - auth_token : required - ttl : 12*3600 # ttl in seconds (default: 12 hours) - cb : required - if not @_validate_opts(opts) then return - @_query - query : 'INSERT INTO auth_tokens' - values : - 'auth_token :: CHAR(24) ' : opts.auth_token - 'expire :: TIMESTAMP ' : expire_time(opts.ttl) - 'account_id :: UUID ' : opts.account_id - cb : opts.cb - - # Get account_id of account with given auth_token. If it - # is not defined, get back undefined instead. - get_auth_token_account_id: (opts) => - opts = defaults opts, - auth_token : required - cb : required # cb(err, account_id) - @_query - query : 'SELECT account_id, expire FROM auth_tokens' - where : - 'auth_token = $::CHAR(24)' : opts.auth_token - cb : one_result (err, x) => - if err - opts.cb(err) - else if not x? - opts.cb() # nothing - else if x.expire <= new Date() - opts.cb() - else - opts.cb(undefined, x.account_id) - - delete_auth_token: (opts) => - opts = defaults opts, - auth_token : required - cb : undefined # cb(err) - @_query - query : 'DELETE FROM auth_tokens' - where : - 'auth_token = $::CHAR(24)' : opts.auth_token - cb : opts.cb - - ### Password reset ### @@ -1305,7 +1230,7 @@ exports.extend_PostgreSQL = (ext) -> class PostgreSQL extends ext return # If expire no pii expiration is set, use 1 year as a fallback - expire = await pii_expire(@) ? expire_time(365*24*60*60) + expire = await pii_expire() ? expire_time(365*24*60*60) @_query query : 'INSERT INTO file_access_log' diff --git a/src/packages/database/postgres/central-log.ts b/src/packages/database/postgres/central-log.ts new file mode 100644 index 0000000000..ec14314308 --- /dev/null +++ b/src/packages/database/postgres/central-log.ts @@ -0,0 +1,43 @@ +import getPool from "@cocalc/database/pool"; +import { pii_expire } from "./pii"; +import { uuid } from "@cocalc/util/misc"; + +// log events, which contain personal information (email, account_id, ...) +const PII_EVENTS = new Set([ + "create_account", + "change_password", + "change_email_address", + "webapp-add_passport", + "get_user_auth_token", + "successful_sign_in", + "webapp-email_sign_up", + "create_account_registration_token", +]); + +export default async function centralLog({ + event, + value, +}: { + event: string; + value: object; +}) { + const pool = getPool(); + + let expire; + if (value["ip_address"] || value["email_address"] || PII_EVENTS.has(event)) { + const date = await pii_expire(); + if (date == null) { + expire = "NOW() + INTERVAL '6 MONTHS'"; + } else { + expire = `NOW() + INTERVAL '${(date.valueOf() - Date.now()) / 1000} seconds'`; + } + } else if (event == "uncaught_exception") { + expire = "NOW() + INTERVAL '1 MONTH'"; + } else { + expire = "NOW() + INTERVAL '1 YEAR'"; + } + await pool.query( + `INSERT INTO central_log(id,event,value,time,expire) VALUES($1,$2,$3,NOW(),${expire})`, + [uuid(), event, value], + ); +} diff --git a/src/packages/database/postgres/pii.ts b/src/packages/database/postgres/pii.ts index c260b73fde..7e09835d7a 100644 --- a/src/packages/database/postgres/pii.ts +++ b/src/packages/database/postgres/pii.ts @@ -4,13 +4,12 @@ */ import { expire_time } from "@cocalc/util/misc"; -import { PostgreSQL } from "./types"; import { get_server_settings } from "./server-settings"; // this converts what's in the pii_expired setting to a new Date in the future export function pii_retention_to_future( pii_retention: number | false, - data?: T & { expire?: Date } + data?: T & { expire?: Date }, ): Date | undefined { if (!pii_retention) return; const future: Date = expire_time(pii_retention); @@ -25,9 +24,8 @@ export function pii_retention_to_future( // if data is set, it's expire field will be set. in any case, it returns the "Date" // in the future. export async function pii_expire( - db: PostgreSQL, - data?: T & { expire?: Date } + data?: T & { expire?: Date }, ): Promise { - const settings = await get_server_settings(db); + const settings = await get_server_settings(); return pii_retention_to_future(settings.pii_retention, data); } diff --git a/src/packages/database/postgres/server-settings.ts b/src/packages/database/postgres/server-settings.ts index d1da1a89fe..1cd45ccd32 100644 --- a/src/packages/database/postgres/server-settings.ts +++ b/src/packages/database/postgres/server-settings.ts @@ -1,10 +1,8 @@ -import { PostgreSQL } from "./types"; import { AllSiteSettings } from "@cocalc/util/db-schema/types"; import { callback2 } from "@cocalc/util/async-utils"; +import { db } from "@cocalc/database"; // just to make this async friendly, that's all -export async function get_server_settings( - db: PostgreSQL -): Promise { - return await callback2(db.get_server_settings_cached); +export async function get_server_settings(): Promise { + return await callback2(db().get_server_settings_cached); } diff --git a/src/packages/database/postgres/types.ts b/src/packages/database/postgres/types.ts index b5c4c622e0..37bfcb2ebd 100644 --- a/src/packages/database/postgres/types.ts +++ b/src/packages/database/postgres/types.ts @@ -346,7 +346,13 @@ export interface PostgreSQL extends EventEmitter { set_project_status(opts: { project_id: string; status: ProjectStatus }): void; - touch(opts: { project_id: string; account_id: string; cb: CB }); + touch(opts: { + project_id?: string; + account_id: string; + action?: string; + path?: string; + cb: CB; + }); get_project_extra_env(opts: { project_id: string; cb: CB }): void; diff --git a/src/packages/database/settings/auth-sso-types.ts b/src/packages/database/settings/auth-sso-types.ts index 63f938092d..65da3f916b 100644 --- a/src/packages/database/settings/auth-sso-types.ts +++ b/src/packages/database/settings/auth-sso-types.ts @@ -5,21 +5,10 @@ import { PostgreSQL } from "@cocalc/database/postgres/types"; -// see @hub/sign-in -interface RecordSignInOpts { - ip_address: string; - successful: boolean; - database: PostgreSQL; - email_address?: string; - account_id?: string; - remember_me: boolean; -} - export interface PassportLoginOpts { passports: { [k: string]: PassportStrategyDB }; database: PostgreSQL; strategyName: string; - record_sign_in: (opts: RecordSignInOpts) => void; // a function of that old "hub/sign-in" module profile: any; // complex object id: string; // id is required. e.g. take the email address – see create_passport in postgres-server-queries.coffee first_name?: string; diff --git a/src/packages/database/settings/customize.ts b/src/packages/database/settings/customize.ts index 0eca745720..549431f49b 100644 --- a/src/packages/database/settings/customize.ts +++ b/src/packages/database/settings/customize.ts @@ -4,64 +4,13 @@ */ import getStrategies from "@cocalc/database/settings/get-sso-strategies"; -import { - KUCALC_COCALC_COM, - KucalcValues, -} from "@cocalc/util/db-schema/site-defaults"; -import { Strategy } from "@cocalc/util/types/sso"; +import { KUCALC_COCALC_COM } from "@cocalc/util/db-schema/site-defaults"; +import type { Strategy } from "@cocalc/util/types/sso"; import { ServerSettings, getServerSettings } from "./server-settings"; import siteURL from "./site-url"; - -export interface Customize { - siteName?: string; - siteDescription?: string; - organizationName?: string; - organizationEmail?: string; - organizationURL?: string; - termsOfServiceURL?: string; - helpEmail?: string; - contactEmail?: string; - isCommercial?: boolean; - kucalc?: KucalcValues; - sshGateway?: boolean; - sshGatewayDNS?: string; - logoSquareURL?: string; - logoRectangularURL?: string; - splashImage?: string; - indexInfo?: string; - indexTagline?: string; - imprint?: string; - policies?: string; - shareServer?: boolean; - landingPages?: boolean; - dns?: string; - siteURL?: string; - googleAnalytics?: string; - anonymousSignup?: boolean; - anonymousSignupLicensedShares?: boolean; - emailSignup?: boolean; - accountCreationInstructions?: string; - zendesk?: boolean; // true if zendesk support is configured. - stripePublishableKey?: string; - imprint_html?: string; - policies_html?: string; - reCaptchaKey?: string; - sandboxProjectsEnabled?: boolean; - sandboxProjectId?: string; - verifyEmailAddresses?: boolean; - strategies?: Strategy[]; - openaiEnabled?: boolean; - googleVertexaiEnabled?: boolean; - mistralEnabled?: boolean; - anthropicEnabled?: boolean; - ollamaEnabled?: boolean; - neuralSearchEnabled?: boolean; - jupyterApiEnabled?: boolean; - computeServersEnabled?: boolean; - cloudFilesystemsEnabled?: boolean; - githubProjectId?: string; - support?: string; -} +import { copy_with } from "@cocalc/util/misc"; +import type { Customize } from "@cocalc/util/db-schema/server-settings"; +export type { Customize }; const fallback = (a?: string, b?: string): string => typeof a == "string" && a.length > 0 ? a : `${b}`; @@ -77,107 +26,107 @@ for a few seconds. let cachedSettings: ServerSettings | undefined = undefined; let cachedCustomize: Customize | undefined = undefined; -export default async function getCustomize(): Promise { +export default async function getCustomize( + fields?: string[], +): Promise { const [settings, strategies]: [ServerSettings, Strategy[]] = await Promise.all([getServerSettings(), getStrategies()]); - if (settings === cachedSettings && cachedCustomize != null) { - return cachedCustomize; + if (!(settings === cachedSettings && cachedCustomize != null)) { + cachedSettings = settings; + cachedCustomize = { + siteName: fallback(settings.site_name, "On Premises CoCalc"), + siteDescription: fallback( + settings.site_description, + "Collaborative Calculation using Python, Sage, R, Julia, and more.", + ), + + organizationName: settings.organization_name, + organizationEmail: settings.organization_email, + organizationURL: settings.organization_url, + termsOfServiceURL: settings.terms_of_service_url, + + helpEmail: settings.help_email, + contactEmail: fallback(settings.organization_email, settings.help_email), + + isCommercial: settings.commercial, + + kucalc: settings.kucalc, + sshGateway: settings.ssh_gateway, + sshGatewayDNS: settings.ssh_gateway_dns, + + anonymousSignup: settings.anonymous_signup, + anonymousSignupLicensedShares: settings.anonymous_signup_licensed_shares, + emailSignup: settings.email_signup, + accountCreationInstructions: settings.account_creation_email_instructions, + + logoSquareURL: settings.logo_square, + logoRectangularURL: settings.logo_rectangular, + splashImage: settings.splash_image, + + shareServer: !!settings.share_server, + + // additionally restrict showing landing pages only in cocalc.com-mode + landingPages: + !!settings.landing_pages && settings.kucalc === KUCALC_COCALC_COM, + + googleAnalytics: settings.google_analytics, + + indexInfo: settings.index_info_html, + indexTagline: settings.index_tagline, + imprint: settings.imprint, + policies: settings.policies, + support: settings.support, + + // Is important for invite emails, password reset, etc. (e.g., so we can construct a url to our site). + // This *can* start with http:// to explicitly use http instead of https, and can end + // in something like :3594 to indicate a port. + dns: settings.dns, + // siteURL is derived from settings.dns and the basePath -- it combines the dns, https:// + // and the basePath. It never ends in a slash. This is used in practice for + // things like invite emails, password reset, etc. + siteURL: await siteURL(settings.dns), + + zendesk: + settings.zendesk_token && + settings.zendesk_username && + settings.zendesk_uri, + + // obviously only the public key here! + stripePublishableKey: settings.stripe_publishable_key, + + // obviously only the public key here too! + reCaptchaKey: settings.re_captcha_v3_publishable_key, + + // a sandbox project + sandboxProjectId: settings.sandbox_project_id, + sandboxProjectsEnabled: settings.sandbox_projects_enabled, + + // true if openai integration is enabled -- this impacts the UI only, and can be + // turned on and off independently of whether there is an api key set. + openaiEnabled: settings.openai_enabled, + // same for google vertex (exposed as gemini), and others + googleVertexaiEnabled: settings.google_vertexai_enabled, + mistralEnabled: settings.mistral_enabled, + anthropicEnabled: settings.anthropic_enabled, + ollamaEnabled: settings.ollama_enabled, + + neuralSearchEnabled: settings.neural_search_enabled, + + // if extra Jupyter API functionality for sandboxed ephemeral code execution is available. + jupyterApiEnabled: settings.jupyter_api_enabled, + + computeServersEnabled: settings.compute_servers_enabled, + cloudFilesystemsEnabled: settings.cloud_filesystems_enabled, + + // GitHub proxy project + githubProjectId: settings.github_project_id, + + // public info about SSO strategies + strategies, + + verifyEmailAddresses: settings.verify_emails && settings.email_enabled, + }; } - cachedSettings = settings; - cachedCustomize = { - siteName: fallback(settings.site_name, "On Premises CoCalc"), - siteDescription: fallback( - settings.site_description, - "Collaborative Calculation using Python, Sage, R, Julia, and more.", - ), - - organizationName: settings.organization_name, - organizationEmail: settings.organization_email, - organizationURL: settings.organization_url, - termsOfServiceURL: settings.terms_of_service_url, - - helpEmail: settings.help_email, - contactEmail: fallback(settings.organization_email, settings.help_email), - - isCommercial: settings.commercial, - - kucalc: settings.kucalc, - sshGateway: settings.ssh_gateway, - sshGatewayDNS: settings.ssh_gateway_dns, - - anonymousSignup: settings.anonymous_signup, - anonymousSignupLicensedShares: settings.anonymous_signup_licensed_shares, - emailSignup: settings.email_signup, - accountCreationInstructions: settings.account_creation_email_instructions, - - logoSquareURL: settings.logo_square, - logoRectangularURL: settings.logo_rectangular, - splashImage: settings.splash_image, - - shareServer: !!settings.share_server, - - // additionally restrict showing landing pages only in cocalc.com-mode - landingPages: - !!settings.landing_pages && settings.kucalc === KUCALC_COCALC_COM, - - googleAnalytics: settings.google_analytics, - - indexInfo: settings.index_info_html, - indexTagline: settings.index_tagline, - imprint: settings.imprint, - policies: settings.policies, - support: settings.support, - - // Is important for invite emails, password reset, etc. (e.g., so we can construct a url to our site). - // This *can* start with http:// to explicitly use http instead of https, and can end - // in something like :3594 to indicate a port. - dns: settings.dns, - // siteURL is derived from settings.dns and the basePath -- it combines the dns, https:// - // and the basePath. It never ends in a slash. This is used in practice for - // things like invite emails, password reset, etc. - siteURL: await siteURL(settings.dns), - - zendesk: - settings.zendesk_token && - settings.zendesk_username && - settings.zendesk_uri, - - // obviously only the public key here! - stripePublishableKey: settings.stripe_publishable_key, - - // obviously only the public key here too! - reCaptchaKey: settings.re_captcha_v3_publishable_key, - - // a sandbox project - sandboxProjectId: settings.sandbox_project_id, - sandboxProjectsEnabled: settings.sandbox_projects_enabled, - - // true if openai integration is enabled -- this impacts the UI only, and can be - // turned on and off independently of whether there is an api key set. - openaiEnabled: settings.openai_enabled, - // same for google vertex (exposed as gemini), and others - googleVertexaiEnabled: settings.google_vertexai_enabled, - mistralEnabled: settings.mistral_enabled, - anthropicEnabled: settings.anthropic_enabled, - ollamaEnabled: settings.ollama_enabled, - - neuralSearchEnabled: settings.neural_search_enabled, - - // if extra Jupyter API functionality for sandboxed ephemeral code execution is available. - jupyterApiEnabled: settings.jupyter_api_enabled, - - computeServersEnabled: settings.compute_servers_enabled, - cloudFilesystemsEnabled: settings.cloud_filesystems_enabled, - - // GitHub proxy project - githubProjectId: settings.github_project_id, - - // public info about SSO strategies - strategies, - - verifyEmailAddresses: settings.verify_emails && settings.email_enabled, - }; - - return cachedCustomize; + return fields ? copy_with(cachedCustomize, fields) : cachedCustomize; } diff --git a/src/packages/database/tsconfig.json b/src/packages/database/tsconfig.json index 02282074ad..a9234ba729 100644 --- a/src/packages/database/tsconfig.json +++ b/src/packages/database/tsconfig.json @@ -7,7 +7,8 @@ }, "exclude": ["node_modules", "dist"], "references": [ - { "path": "../util" }, { "path": "../backend" }, + { "path": "../nats" }, + { "path": "../util" } ] } diff --git a/src/packages/file-server/README.md b/src/packages/file-server/README.md new file mode 100644 index 0000000000..574be09ddf --- /dev/null +++ b/src/packages/file-server/README.md @@ -0,0 +1,61 @@ +# CoCalcFS -- the Cocalc project file system + +We manage the following types of filesystems, which are exported via NFS: + +The file server support an unlimited number of namespaces. + +There will be a large number of the following filesystems, and they are tracked in a sqlite3 database. The file server hosts and manages potentially more than one filesystem owned by projects *and* multiple filesystems owned by cocalc accounts , and also by cocalc organizations (not fleshed out). After rewriting the code, it's basically the same work and just more flexible to support future development. + +- **project:** each CoCalc project owns usually exactly one project filesystem, which is its home directory and is named ''. This gets mounted by the Kubernetes pod \(the "home base"\), and ALSO by compute servers. It can either exist in some zpool or be archived as a ZFS replication stream. Projects can also create an unlimited number of other filesystems. + +- **user:** a user filesystem is owned by an _account_ \(i.e., a CoCalc user\), not by a project. It may optionally be mounted on any project that the user is a collaborator on. E.g., an instructor could make a user volume, then mount it read\-only on all of their student's projects. Or a user could mount a user volume read\-write on several of their projects. An account can own many user filesystems, each with a name. + +- **group:** group of users with one billing contact. Otherwise same as above. + +The name is a unicode string that must be less than 64 characters. + +There's also one of these in each pool: + +- **archives:** output of zfs replication streams. + +In addition, the following are stored in their own separate pool and get mounted read-only via Kubernetes. + +- **data:** misc data that is mounted and shared by all projects + +- **images:** project images, i.e., sage installs, version of ubuntu, etc. + +## ZFS + +### Archive + +We primarily archive projects in one big directory on the same pool as we +server the projects. The **ONLY** reason for this is because `"zpool import"` is +at least O\(n\) complexity, where n is the number of datasets in the pool, +and the constant in the O isn't great. If we have 4 million projects, we can +only realistically have up to 100K datasets in order to keep "zpool import" times +down to 1-2 minutes. We thus archive the other 3.9 million projects. The time +to dearchive is roughly 3 seconds / GB. + +We can of course mirror the contents of the archive to cloud storage for then fall +back to it in order to save space and reduce longterm storage costs, at scale, +if necessary. + +archive contents: + +- zfs stream with snapshots +- a tarball of the last snapshot of project contents +- dump of the NATS stream and kv for that project. + +### NOTE: Units + +By default both ZFS and `df -h` use GiB and write it G (e.g., "gibibyte"). + +```sh +root@prod-42:/cocalcfs/projects/default/00000000-0000-0000-0000-000000000002# zfs set refquota=2147483648 cocalcfs0/default/projects/00000000-0000-0000-0000-000000000002 +root@prod-42:/cocalcfs/projects/default/00000000-0000-0000-0000-000000000002# zfs get refquota cocalcfs0/default/projects/00000000-0000-0000-0000-000000000002 +NAME PROPERTY VALUE SOURCE +cocalcfs0/default/projects/00000000-0000-0000-0000-000000000002 refquota 2G local +root@prod-42:/cocalcfs/projects/default/00000000-0000-0000-0000-000000000002# df -h . +Filesystem Size Used Avail Use% Mounted on +cocalcfs0/default/projects/00000000-0000-0000-0000-000000000002 2.0G 568M 1.5G 28% /cocalcfs/projects/default/00000000-0000-0000-0000-000000000002 +``` diff --git a/src/packages/file-server/jest.config.js b/src/packages/file-server/jest.config.js new file mode 100644 index 0000000000..189f1544d5 --- /dev/null +++ b/src/packages/file-server/jest.config.js @@ -0,0 +1,8 @@ +/** @type {import('ts-jest').JestConfigWithTsJest} */ +module.exports = { + preset: "ts-jest", + testEnvironment: "node", + setupFiles: ["./test/setup.js"], + testMatch: ["**/?(*.)+(spec|test).ts?(x)"], + maxConcurrency: 1, +}; diff --git a/src/packages/file-server/package.json b/src/packages/file-server/package.json new file mode 100644 index 0000000000..b86e2abd1a --- /dev/null +++ b/src/packages/file-server/package.json @@ -0,0 +1,43 @@ +{ + "name": "@cocalc/file-server", + "version": "1.0.0", + "description": "CoCalc File Server", + "exports": { + "./zfs": "./dist/zfs/index.js", + "./zfs/*": "./dist/zfs/*.js" + }, + "scripts": { + "preinstall": "npx only-allow pnpm", + "build": "pnpm exec tsc --build", + "tsc": "pnpm exec tsc --watch --pretty --preserveWatchOutput", + "test": "pnpm exec jest --runInBand" + }, + "files": [ + "dist/**", + "README.md", + "package.json" + ], + "author": "SageMath, Inc.", + "keywords": [ + "utilities", + "nats", + "cocalc" + ], + "license": "SEE LICENSE.md", + "dependencies": { + "@cocalc/backend": "workspace:*", + "@cocalc/file-server": "workspace:*", + "@cocalc/nats": "workspace:*", + "@cocalc/util": "workspace:*", + "@types/better-sqlite3": "^7.6.4", + "@types/lodash": "^4.14.202", + "awaiting": "^3.0.0", + "better-sqlite3": "^8.3.0", + "lodash": "^4.17.21" + }, + "repository": { + "type": "git", + "url": "https://github.com/sagemathinc/cocalc" + }, + "homepage": "https://github.com/sagemathinc/cocalc/tree/master/src/packages/file-server" +} diff --git a/src/packages/file-server/test/setup.js b/src/packages/file-server/test/setup.js new file mode 100644 index 0000000000..6ed5b82df9 --- /dev/null +++ b/src/packages/file-server/test/setup.js @@ -0,0 +1,4 @@ +// test/setup.js + +// checked for in some code to behave differently while running unit tests. +process.env.COCALC_TEST_MODE = true; diff --git a/src/packages/file-server/tsconfig.json b/src/packages/file-server/tsconfig.json new file mode 100644 index 0000000000..9d5e2b1337 --- /dev/null +++ b/src/packages/file-server/tsconfig.json @@ -0,0 +1,9 @@ +{ + "extends": "../tsconfig.json", + "compilerOptions": { + "rootDir": "./", + "outDir": "dist" + }, + "exclude": ["node_modules", "dist", "test"], + "references": [{ "path": "../util", "path": "../nats", "path": "../backend" }] +} diff --git a/src/packages/file-server/zfs/archive.ts b/src/packages/file-server/zfs/archive.ts new file mode 100644 index 0000000000..70347e14c6 --- /dev/null +++ b/src/packages/file-server/zfs/archive.ts @@ -0,0 +1,236 @@ +/* +Archiving and restore filesystems +*/ + +import { get, set } from "./db"; +import { createSnapshot, zfsGetSnapshots } from "./snapshots"; +import { + filesystemDataset, + filesystemArchivePath, + filesystemArchiveFilename, + filesystemDatasetTemp, + filesystemMountpoint, +} from "./names"; +import { exec } from "./util"; +import { + mountFilesystem, + unmountFilesystem, + zfsGetProperties, +} from "./properties"; +import { delay } from "awaiting"; +import { primaryKey, type PrimaryKey } from "./types"; +import { isEqual } from "lodash"; + +export async function dearchiveFilesystem( + opts: PrimaryKey & { + // called during dearchive with status updates: + progress?: (status: { + // a number between 0 and 100 indicating progress + progress: number; + // estimated number of seconds remaining + seconds_remaining?: number; + // how much of the total data we have de-archived + read?: number; + // total amount of data to de-archive + total?: number; + }) => void; + }, +) { + const start = Date.now(); + opts.progress?.({ progress: 0 }); + const pk = primaryKey(opts); + const filesystem = get(pk); + if (!filesystem.archived) { + throw Error("filesystem is not archived"); + } + const { used_by_dataset, used_by_snapshots } = filesystem; + const total = (used_by_dataset ?? 0) + (used_by_snapshots ?? 0); + const dataset = filesystemDataset(filesystem); + let done = false; + let progress = 0; + if (opts.progress && total > 0) { + (async () => { + const t0 = Date.now(); + let lastProgress = 0; + while (!done) { + await delay(750); + let x; + try { + x = await zfsGetProperties(dataset); + } catch { + // this is expected to fail, e.g., if filesystem doesn't exist yet. + } + if (done) { + return; + } + const read = x.used_by_dataset + x.used_by_snapshots; + progress = Math.min(100, Math.round((read * 100) / total)); + if (progress == lastProgress) { + continue; + } + lastProgress = progress; + let seconds_remaining: number | undefined = undefined; + if (progress > 0) { + const rate = (Date.now() - t0) / progress; + seconds_remaining = Math.ceil((rate * (100 - progress)) / 1000); + } + opts.progress?.({ progress, seconds_remaining, total, read }); + if (progress >= 100) { + break; + } + } + })(); + } + + // now we de-archive it: + const stream = filesystemArchiveFilename(filesystem); + await exec({ + verbose: true, + // have to use sudo sh -c because zfs recv only supports reading from stdin: + command: `sudo sh -c 'cat ${stream} | zfs recv ${dataset}'`, + what: { + ...pk, + desc: "de-archive a filesystem via zfs recv", + }, + }); + done = true; + if (progress < 100) { + opts.progress?.({ + progress: 100, + seconds_remaining: 0, + total, + read: total, + }); + } + await mountFilesystem(filesystem); + // mounting worked so remove the archive + await exec({ + command: "sudo", + args: ["rm", stream], + what: { + ...pk, + desc: "removing the stream during de-archive", + }, + }); + set({ ...pk, archived: false }); + return { milliseconds: Date.now() - start }; +} + +export async function archiveFilesystem(fs: PrimaryKey) { + const start = Date.now(); + const pk = primaryKey(fs); + const filesystem = get(pk); + if (filesystem.archived) { + throw Error("filesystem is already archived"); + } + // create or get most recent snapshot + const snapshot = await createSnapshot({ ...filesystem, ifChanged: true }); + // where archive of this filesystem goes: + const archive = filesystemArchivePath(filesystem); + const stream = filesystemArchiveFilename(filesystem); + await exec({ + command: "sudo", + args: ["mkdir", "-p", archive], + what: { ...pk, desc: "make archive target directory" }, + }); + + await mountFilesystem(filesystem); + const find = await hashFileTree({ + verbose: true, + path: filesystemMountpoint(filesystem), + what: { ...pk, desc: "getting sha1sum of file listing" }, + }); + // mountpoint will be used for test below, and also no point in archiving + // if we can't even unmount filesystem + await unmountFilesystem(filesystem); + + // make *full* zfs send + await exec({ + verbose: true, + // have to use sudo sh -c because zfs send only supports writing to stdout: + command: `sudo sh -c 'zfs send -e -c -R ${filesystemDataset(filesystem)}@${snapshot} > ${stream}'`, + what: { + ...pk, + desc: "zfs send of full filesystem dataset to archive it", + }, + }); + + // verify that the entire send stream is valid + const temp = filesystemDatasetTemp(filesystem); + try { + await exec({ + verbose: true, + // have to use sudo sh -c because zfs send only supports writing to stdout: + command: `sudo sh -c 'cat ${stream} | zfs recv ${temp}'`, + what: { + ...pk, + desc: "verify the archive zfs send is valid", + }, + }); + // inspect the list of all files, and verify that it is identical (has same sha1sum). + // I think this should be not necessary because the above read didn't fail, and there + // are supposed to be checksums. But I also think there are some ways to corrupt a + // stream so it reads in as empty (say), so this will definitely catch that. + const findtest = await hashFileTree({ + verbose: true, + path: filesystemMountpoint(filesystem), // same mountpoint due to being part of recv data + what: { ...pk, desc: "getting sha1sum of file listing" }, + }); + if (findtest != find) { + throw Error( + "files in archived filesystem do not match. Refusing to archive!", + ); + } + // Inspect list of snapshots, and verify they are identical as well. This is another + // good consistency check that the stream works. + const snapshots = await zfsGetSnapshots(temp); + if (!isEqual(snapshots, filesystem.snapshots)) { + throw Error( + "snapshots in archived filesystem do not match. Refusing to archive!", + ); + } + } finally { + // destroy the temporary filesystem + await exec({ + verbose: true, + command: "sudo", + args: ["zfs", "destroy", "-r", temp], + what: { + ...pk, + desc: "destroying temporary filesystem dataset used for testing archive stream", + }, + }); + } + + // destroy dataset + await exec({ + verbose: true, + command: "sudo", + args: ["zfs", "destroy", "-r", filesystemDataset(filesystem)], + what: { ...pk, desc: "destroying filesystem dataset" }, + }); + + // set as archived in database + set({ ...pk, archived: true }); + + return { snapshot, milliseconds: Date.now() - start }; +} + +// Returns a hash of the file tree. This uses the find command to get path names, but +// doesn't actually read the *contents* of any files, so it's reasonbly fast. +async function hashFileTree({ + path, + what, + verbose, +}: { + path: string; + what?; + verbose?; +}): Promise { + const { stdout } = await exec({ + verbose, + command: `sudo sh -c 'cd "${path}" && find . -xdev -printf "%p %s %TY-%Tm-%Td %TH:%TM\n" | sha1sum'`, + what, + }); + return stdout; +} diff --git a/src/packages/file-server/zfs/backup.ts b/src/packages/file-server/zfs/backup.ts new file mode 100644 index 0000000000..5ded8ee4af --- /dev/null +++ b/src/packages/file-server/zfs/backup.ts @@ -0,0 +1,176 @@ +/* +Make backups using bup. +*/ + +import { bupFilesystemMountpoint, filesystemSnapshotMountpoint } from "./names"; +import { get, getRecent, set } from "./db"; +import { exec } from "./util"; +import getLogger from "@cocalc/backend/logger"; +import { exists } from "@cocalc/backend/misc/async-utils-node"; +import { join } from "path"; +import { mountFilesystem } from "./properties"; +import { split } from "@cocalc/util/misc"; +import { BUP_INTERVAL_MS } from "./config"; +import { primaryKey, type PrimaryKey } from "./types"; +import { createSnapshot } from "./snapshots"; + +const logger = getLogger("file-server:zfs:backup"); + +const EXCLUDES = [".conda", ".npm", "cache", ".julia", ".local/share/pnpm"]; + +export async function createBackup( + fs: PrimaryKey, +): Promise<{ BUP_DIR: string }> { + const pk = primaryKey(fs); + logger.debug("createBackup", pk); + const filesystem = get(pk); + await mountFilesystem(pk); + const snapshot = await createSnapshot({ ...filesystem, ifChanged: true }); + const mountpoint = filesystemSnapshotMountpoint({ ...filesystem, snapshot }); + const excludes: string[] = []; + for (const path of EXCLUDES) { + excludes.push("--exclude"); + excludes.push(join(mountpoint, path)); + } + logger.debug("createBackup: index", pk); + const BUP_DIR = bupFilesystemMountpoint(filesystem); + if (!(await exists(BUP_DIR))) { + await exec({ + verbose: true, + command: "sudo", + args: ["mkdir", "-p", BUP_DIR], + what: { ...pk, desc: "make bup repo" }, + }); + await exec({ + verbose: true, + command: "sudo", + args: ["bup", "-d", BUP_DIR, "init"], + what: { ...pk, desc: "bup init" }, + }); + } + await exec({ + verbose: true, + env: { BUP_DIR }, + command: "sudo", + args: [ + "--preserve-env", + "bup", + "index", + ...excludes, + "-x", + mountpoint, + "--no-check-device", + ], + what: { ...pk, desc: "creating bup index" }, + }); + logger.debug("createBackup: save", pk); + await exec({ + verbose: true, + env: { BUP_DIR }, + command: "sudo", + args: [ + "--preserve-env", + "bup", + "save", + "-q", + "--strip", + "-n", + "master", + mountpoint, + ], + what: { ...pk, desc: "save new bup snapshot" }, + }); + + const { stdout } = await exec({ + env: { BUP_DIR }, + command: "sudo", + args: ["--preserve-env", "bup", "ls", "master"], + what: { ...pk, desc: "getting name of backup" }, + }); + const v = split(stdout); + const last_bup_backup = v[v.length - 2]; + logger.debug("createBackup: created ", { last_bup_backup }); + set({ ...pk, last_bup_backup }); + + // prune-older --unsafe --keep-all-for 8d --keep-dailies-for 4w --keep-monthlies-for 6m --keep-yearlies-for 10y + logger.debug("createBackup: prune", pk); + await exec({ + verbose: true, + env: { BUP_DIR }, + command: "sudo", + args: [ + "--preserve-env", + "bup", + "prune-older", + "--unsafe", + "--keep-all-for", + "8d", + "--keep-dailies-for", + "4w", + "--keep-monthlies-for", + "6m", + "--keep-yearlies-for", + "5y", + ], + what: { ...pk, desc: "save new bup snapshot" }, + }); + + return { BUP_DIR }; +} + +// Go through ALL filesystems with last_edited >= cutoff and make a bup +// backup if they are due. +// cutoff = a Date (default = 1 week ago) +export async function maintainBackups(cutoff?: Date) { + logger.debug("backupActiveFilesystems: getting..."); + const v = getRecent({ cutoff }); + logger.debug( + `backupActiveFilesystems: considering ${v.length} filesystems`, + cutoff, + ); + let i = 0; + for (const { archived, last_edited, last_bup_backup, ...pk } of v) { + if (archived || !last_edited) { + continue; + } + const age = + new Date(last_edited).valueOf() - bupToDate(last_bup_backup).valueOf(); + if (age < BUP_INTERVAL_MS) { + // there's a new backup already + continue; + } + try { + await createBackup(pk); + } catch (err) { + logger.debug(`backupActiveFilesystems: error -- ${err}`); + } + i += 1; + if (i % 10 == 0) { + logger.debug(`backupActiveFilesystems: ${i}/${v.length}`); + } + } +} + +function bupToDate(dateString?: string): Date { + if (!dateString) { + return new Date(0); + } + // Extract components using regular expression + const match = dateString.match( + /^(\d{4})-(\d{2})-(\d{2})-(\d{2})(\d{2})(\d{2})$/, + ); + + if (match) { + const [_, year, month, day, hour, minute, second] = match; // Destructure components + return new Date( + parseInt(year), + parseInt(month) - 1, + parseInt(day), + parseInt(hour), + parseInt(minute), + parseInt(second), + ); + } else { + throw Error("Invalid bup date format"); + } +} diff --git a/src/packages/file-server/zfs/config.ts b/src/packages/file-server/zfs/config.ts new file mode 100644 index 0000000000..96b0df9d74 --- /dev/null +++ b/src/packages/file-server/zfs/config.ts @@ -0,0 +1,98 @@ +import { join } from "path"; +import { databaseFilename } from "./names"; + +// we ONLY put filesystems on pools whose name have this prefix. +// all other pools are ignored. We also mount everything in /{PREFIX} on the filesystem. +const PREFIX = process.env.COCALC_TEST_MODE ? "cocalcfs-test" : "cocalcfs"; + +const DATA = `/${PREFIX}`; + +const SQLITE3_DATABASE_FILE = databaseFilename(DATA); + +// Directory on server where filesystems get mounted (so NFS can serve them) +const FILESYSTEMS = join(DATA, "filesystems"); + +// Directory on server where zfs send streams (and tar?) are stored +const ARCHIVES = join(DATA, "archives"); + +// Directory to store data used in pulling as part of sync. +// E.g., this keeps around copies of the sqlite state database of each remote. +const PULL = join(DATA, "pull"); + +// Directory for bup +const BUP = join(DATA, "bup"); + +export const context = { + namespace: process.env.NAMESPACE ?? "default", + PREFIX, + DATA, + SQLITE3_DATABASE_FILE, + FILESYSTEMS, + ARCHIVES, + PULL, + BUP, +}; + +// WARNING: this "setContext" is global. It's very useful for **UNIT TESTING**, but +// for any other use, you want to set this at most once and never again!!! The reason +// is because with nodejs you could have async code running all over the place, and +// changing the context out from under it would lead to nonsense and corruption. +export function setContext({ + namespace, + prefix, +}: { + namespace?: string; + prefix?: string; +}) { + context.namespace = namespace ?? process.env.NAMESPACE ?? "default"; + context.PREFIX = prefix ?? PREFIX; + context.DATA = `/${context.PREFIX}`; + context.SQLITE3_DATABASE_FILE = databaseFilename(context.DATA); + context.FILESYSTEMS = join(context.DATA, "filesystems"); + context.ARCHIVES = join(context.DATA, "archives"); + context.PULL = join(context.DATA, "pull"); + context.BUP = join(context.DATA, "bup"); +} + +// Every filesystem has at least this much quota (?) +export const MIN_QUOTA = 1024 * 1024 * 1; // 1MB + +// We periodically do "zpool list" to find out what pools are available +// and how much space they have left. This info is cached for this long +// to avoid excessive calls: +export const POOLS_CACHE_MS = 15000; + +// two hour default for running any commands (e.g., zfs send/recv) +export const DEFAULT_EXEC_TIMEOUT_MS = 2 * 1000 * 60 * 60; + +// **all** user files for filesystems have this owner and group. +export const UID = 2001; +export const GID = 2001; + +// We make/update snapshots periodically, with this being the minimum interval. +export const SNAPSHOT_INTERVAL_MS = 60 * 30 * 1000; +//export const SNAPSHOT_INTERVAL_MS = 10 * 1000; + +// Lengths of time in minutes to keep these snapshots +export const SNAPSHOT_INTERVALS_MS = { + halfhourly: 30 * 1000 * 60, + daily: 60 * 24 * 1000 * 60, + weekly: 60 * 24 * 7 * 1000 * 60, + monthly: 60 * 24 * 7 * 4 * 1000 * 60, +}; + +// How many of each type of snapshot to retain +export const SNAPSHOT_COUNTS = { + halfhourly: 24, + daily: 14, + weekly: 7, + monthly: 4, +}; + +// Minimal interval for bup backups +export const BUP_INTERVAL_MS = 24 * 1000 * 60 * 60; + +// minimal interval for zfs streams +export const STREAM_INTERVAL_MS = 24 * 1000 * 60 * 60; +// when more than this many streams, we recompact down +export const MAX_STREAMS = 30; diff --git a/src/packages/file-server/zfs/copy.ts b/src/packages/file-server/zfs/copy.ts new file mode 100644 index 0000000000..8248d3061f --- /dev/null +++ b/src/packages/file-server/zfs/copy.ts @@ -0,0 +1,14 @@ +/* +Copy between projects on this server +*/ + +export async function copy(opts: { + source_project_id: string; + target_project_id?: string; + source_path: string; + target_path: string; + rsyncOptions?: string; +}) { + console.log("copy", opts); + throw Error("copy: not implemented"); +} diff --git a/src/packages/file-server/zfs/create.ts b/src/packages/file-server/zfs/create.ts new file mode 100644 index 0000000000..7be3005470 --- /dev/null +++ b/src/packages/file-server/zfs/create.ts @@ -0,0 +1,208 @@ +import { create, get, getDb, deleteFromDb, filesystemExists } from "./db"; +import { exec } from "./util"; +import { + filesystemArchivePath, + bupFilesystemMountpoint, + filesystemDataset, + filesystemMountpoint, +} from "./names"; +import { getPools, initializePool } from "./pools"; +import { dearchiveFilesystem } from "./archive"; +import { UID, GID } from "./config"; +import { createSnapshot } from "./snapshots"; +import { type Filesystem, primaryKey, type PrimaryKey } from "./types"; + +export async function createFilesystem( + opts: PrimaryKey & { + affinity?: string; + clone?: PrimaryKey; + }, +): Promise { + if (filesystemExists(opts)) { + return get(opts); + } + const pk = primaryKey(opts); + const { namespace } = pk; + const { affinity, clone } = opts; + const source = clone ? get(clone) : undefined; + + const db = getDb(); + // select a pool: + let pool: undefined | string = undefined; + + if (source != null) { + // use same pool as source filesystem. (we could use zfs send/recv but that's much slower and not a clone) + pool = source.pool; + } else { + if (affinity) { + // if affinity is set, have preference to use same pool as other filesystems with this affinity. + const x = db + .prepare( + "SELECT pool, COUNT(pool) AS cnt FROM filesystems WHERE namespace=? AND affinity=? ORDER by cnt DESC", + ) + .get(namespace, affinity) as { pool: string; cnt: number } | undefined; + pool = x?.pool; + } + if (!pool) { + // assign one with *least* filesystems + const x = db + .prepare( + "SELECT pool, COUNT(pool) AS cnt FROM filesystems GROUP BY pool ORDER by cnt ASC", + ) + .all() as any; + const pools = await getPools(); + if (Object.keys(pools).length > x.length) { + // rare case: there exists a pool that isn't used yet, so not + // represented in above query at all; use it + const v = new Set(); + for (const { pool } of x) { + v.add(pool); + } + for (const name in pools) { + if (!v.has(name)) { + pool = name; + break; + } + } + } else { + if (x.length == 0) { + throw Error("cannot create filesystem -- no available pools"); + } + // just use the least crowded + pool = x[0].pool; + } + } + } + if (!pool) { + throw Error("bug -- unable to select a pool"); + } + + const { cnt } = db + .prepare( + "SELECT COUNT(pool) AS cnt FROM filesystems WHERE pool=? AND namespace=?", + ) + .get(pool, namespace) as { cnt: number }; + + if (cnt == 0) { + // initialize pool for use in this namespace: + await initializePool({ pool, namespace }); + } + + if (source == null) { + // create filesystem on the selected pool + const mountpoint = filesystemMountpoint(pk); + const dataset = filesystemDataset({ ...pk, pool }); + await exec({ + verbose: true, + command: "sudo", + args: [ + "zfs", + "create", + "-o", + `mountpoint=${mountpoint}`, + "-o", + "compression=lz4", + "-o", + "dedup=on", + dataset, + ], + what: { + ...pk, + desc: `create filesystem ${dataset} for filesystem on the selected pool mounted at ${mountpoint}`, + }, + }); + await exec({ + verbose: true, + command: "sudo", + args: ["chown", "-R", `${UID}:${GID}`, mountpoint], + whate: { + ...pk, + desc: `setting permissions of filesystem mounted at ${mountpoint}`, + }, + }); + } else { + // clone source + // First ensure filesystem isn't archived + // (we might alternatively de-archive to make the clone...?) + if (source.archived) { + await dearchiveFilesystem(source); + } + // Get newest snapshot, or make one if there are none + const snapshot = await createSnapshot({ ...source, ifChanged: true }); + if (!snapshot) { + throw Error("bug -- source should have snapshot"); + } + const source_snapshot = `${filesystemDataset(source)}@${snapshot}`; + await exec({ + verbose: true, + command: "sudo", + args: [ + "zfs", + "clone", + "-o", + `mountpoint=${filesystemMountpoint(pk)}`, + "-o", + "compression=lz4", + "-o", + "dedup=on", + source_snapshot, + filesystemDataset({ ...pk, pool }), + ], + what: { + ...pk, + desc: `clone filesystem from ${source_snapshot}`, + }, + }); + } + + // update database + create({ ...pk, pool, affinity }); + return get(pk); +} + +// delete -- This is very dangerous -- it deletes the filesystem, +// the archive, and any backups and removes knowledge the filesystem from the db. + +export async function deleteFilesystem(fs: PrimaryKey) { + const filesystem = get(fs); + const dataset = filesystemDataset(filesystem); + if (!filesystem.archived) { + await exec({ + verbose: true, + command: "sudo", + args: ["zfs", "destroy", "-r", dataset], + what: { + ...filesystem, + desc: `destroy dataset ${dataset} containing the filesystem`, + }, + }); + } + await exec({ + verbose: true, + command: "sudo", + args: ["rm", "-rf", filesystemMountpoint(filesystem)], + what: { + ...filesystem, + desc: `delete directory '${filesystemMountpoint(filesystem)}' where filesystem was stored`, + }, + }); + await exec({ + verbose: true, + command: "sudo", + args: ["rm", "-rf", bupFilesystemMountpoint(filesystem)], + what: { + ...filesystem, + desc: `delete directory '${bupFilesystemMountpoint(filesystem)}' where backups were stored`, + }, + }); + await exec({ + verbose: true, + command: "sudo", + args: ["rm", "-rf", filesystemArchivePath(filesystem)], + what: { + ...filesystem, + desc: `delete directory '${filesystemArchivePath(filesystem)}' where archives were stored`, + }, + }); + deleteFromDb(filesystem); +} diff --git a/src/packages/file-server/zfs/db.ts b/src/packages/file-server/zfs/db.ts new file mode 100644 index 0000000000..9b33d202dc --- /dev/null +++ b/src/packages/file-server/zfs/db.ts @@ -0,0 +1,277 @@ +/* +Database +*/ + +import Database from "better-sqlite3"; +import { context } from "./config"; +import { + primaryKey, + type PrimaryKey, + type Filesystem, + type RawFilesystem, + type SetFilesystem, + OWNER_ID_FIELDS, +} from "./types"; +import { is_array, is_date } from "@cocalc/util/misc"; + +let db: { [file: string]: Database.Database } = {}; + +const tableName = "filesystems"; +const schema = { + // this uniquely defines the filesystem (it's the compound primary key) + owner_type: "TEXT", + owner_id: "TEXT", + namespace: "TEXT", + name: "TEXT", + + // data about the filesystem + pool: "TEXT", + archived: "INTEGER", + affinity: "TEXT", + nfs: "TEXT", + snapshots: "TEXT", + last_edited: "TEXT", + last_send_snapshot: "TEXT", + last_bup_backup: "TEXT", + error: "TEXT", + last_error: "TEXT", + used_by_dataset: "INTEGER", + used_by_snapshots: "INTEGER", + quota: "INTEGER", +}; + +const WHERE_PRIMARY_KEY = + "WHERE namespace=? AND owner_type=? AND owner_id=? AND name=?"; +function primaryKeyArgs(fs: PrimaryKey) { + const { namespace, owner_type, owner_id, name } = primaryKey(fs); + return [namespace, owner_type, owner_id, name]; +} + +export function getDb(databaseFile?): Database.Database { + const file = databaseFile ?? context.SQLITE3_DATABASE_FILE; + if (db[file] == null) { + db[file] = new Database(file); + initDb(db[file]); + } + return db[file]!; +} + +function initDb(db) { + const columnDefinitions = Object.entries(schema) + .map(([name, type]) => `${name} ${type}`) + .join(", "); + + // Create table if it doesn't exist + db.prepare( + `CREATE TABLE IF NOT EXISTS ${tableName} ( + ${columnDefinitions}, + PRIMARY KEY (namespace, owner_type, owner_id, name) + )`, + ).run(); + + // Check for missing columns and add them + const existingColumnsStmt = db.prepare(`PRAGMA table_info(${tableName})`); + const existingColumns = existingColumnsStmt.all().map((row) => row.name); + + for (const [name, type] of Object.entries(schema)) { + if (!existingColumns.includes(name)) { + db.prepare(`ALTER TABLE ${tableName} ADD COLUMN ${name} ${type}`).run(); + } + } +} + +// This is extremely dangerous and mainly used for unit testing: +export function resetDb() { + const db = new Database(context.SQLITE3_DATABASE_FILE); + db.prepare("DROP TABLE IF EXISTS filesystems").run(); + initDb(db); +} + +function convertToSqliteType({ value, getFilesystem }) { + if (is_array(value)) { + return value.join(","); + } else if (is_date(value)) { + return value.toISOString(); + } else if (typeof value == "boolean") { + return value ? 1 : 0; + } else if (typeof value == "function") { + const x = value(getFilesystem()); + if (typeof x == "function") { + throw Error("function must not return a function"); + } + // returned value needs to be converted + return convertToSqliteType({ value: x, getFilesystem }); + } + return value; +} + +export function set(obj: SetFilesystem) { + const pk = primaryKey(obj); + const fields: string[] = []; + const values: any[] = []; + let filesystem: null | Filesystem = null; + const getFilesystem = () => { + if (filesystem == null) { + filesystem = get(pk); + } + return filesystem; + }; + for (const field in obj) { + if (pk[field] !== undefined || OWNER_ID_FIELDS.includes(field)) { + continue; + } + fields.push(field); + values.push(convertToSqliteType({ value: obj[field], getFilesystem })); + } + let query = `UPDATE filesystems SET + ${fields.map((field) => `${field}=?`).join(", ")} + ${WHERE_PRIMARY_KEY} + `; + for (const x of primaryKeyArgs(pk)) { + values.push(x); + } + const db = getDb(); + db.prepare(query).run(...values); +} + +// Call this if something that should never happen, does in fact, happen. +// It will set the error state of the filesystem and throw the exception. +// Admins will be regularly notified of all filesystems in an error state. +export function fatalError( + obj: PrimaryKey & { + err: Error; + desc?: string; + }, +) { + set({ + ...primaryKey(obj), + error: `${obj.err}${obj.desc ? " - " + obj.desc : ""}`, + last_error: new Date(), + }); + throw obj.err; +} + +export function clearError(fs: PrimaryKey) { + set({ ...fs, error: null }); +} + +export function clearAllErrors() { + const db = getDb(); + db.prepare("UPDATE filesystems SET error=null").run(); +} + +export function getErrors() { + const db = getDb(); + return db + .prepare("SELECT * FROM filesystems WHERE error!=''") + .all() as RawFilesystem[]; +} + +export function touch(fs: PrimaryKey) { + set({ ...fs, last_edited: new Date() }); +} + +export function filesystemExists( + fs: PrimaryKey, + databaseFile?: string, +): boolean { + const db = getDb(databaseFile); + const x = db + .prepare("SELECT COUNT(*) AS count FROM filesystems " + WHERE_PRIMARY_KEY) + .get(...primaryKeyArgs(fs)); + return (x as any).count > 0; +} + +export function get(fs: PrimaryKey, databaseFile?: string): Filesystem { + const db = getDb(databaseFile); + const filesystem = db + .prepare("SELECT * FROM filesystems " + WHERE_PRIMARY_KEY) + .get(...primaryKeyArgs(fs)) as any; + if (filesystem == null) { + throw Error(`no filesystem ${JSON.stringify(fs)}`); + } + for (const key of ["nfs", "snapshots"]) { + filesystem[key] = sqliteStringToArray(filesystem[key]); + } + filesystem["archived"] = !!filesystem["archived"]; + if (filesystem.last_edited) { + filesystem.last_edited = new Date(filesystem.last_edited); + } + if (filesystem.last_error) { + filesystem.last_error = new Date(filesystem.last_error); + } + return filesystem as Filesystem; +} + +export function create( + obj: PrimaryKey & { + pool: string; + affinity?: string; + }, +) { + if (!obj.pool.startsWith(context.PREFIX)) { + throw Error(`pool must start with ${context.PREFIX} - ${obj.pool}`); + } + getDb() + .prepare( + "INSERT INTO filesystems(namespace, owner_type, owner_id, name, pool, affinity, last_edited) VALUES(?,?,?,?,?,?,?)", + ) + .run( + ...primaryKeyArgs(obj), + obj.pool, + obj.affinity, + new Date().toISOString(), + ); +} + +export function deleteFromDb(fs: PrimaryKey) { + getDb() + .prepare("DELETE FROM filesystems " + WHERE_PRIMARY_KEY) + .run(...primaryKeyArgs(fs)); +} + +export function getAll({ + namespace = context.namespace, +}: { namespace?: string } = {}): RawFilesystem[] { + const db = getDb(); + return db + .prepare("SELECT * FROM filesystems WHERE namespace=?") + .all(namespace) as RawFilesystem[]; +} + +export function getNamespacesAndPools(): { namespace: string; pool: string }[] { + const db = getDb(); + return db + .prepare("SELECT DISTINCT namespace, pool FROM filesystems") + .all() as any; +} + +export function getRecent({ + namespace, + cutoff, + databaseFile, +}: { + namespace?: string; + cutoff?: Date; + databaseFile?: string; +} = {}): RawFilesystem[] { + const db = getDb(databaseFile); + if (cutoff == null) { + cutoff = new Date(Date.now() - 1000 * 60 * 60 * 24 * 7); + } + const query = "SELECT * FROM filesystems WHERE last_edited>=?"; + if (namespace == null) { + return db.prepare(query).all(cutoff.toISOString()) as RawFilesystem[]; + } else { + return db + .prepare(`${query} AND namespace=?`) + .all(cutoff.toISOString(), namespace) as RawFilesystem[]; + } +} + +function sqliteStringToArray(s?: string): string[] { + if (!s) { + return []; + } + return s.split(","); +} diff --git a/src/packages/file-server/zfs/index.ts b/src/packages/file-server/zfs/index.ts new file mode 100644 index 0000000000..8729ea2d04 --- /dev/null +++ b/src/packages/file-server/zfs/index.ts @@ -0,0 +1,29 @@ +export { getPools, initializePool, initializeAllPools } from "./pools"; +export { + getModifiedFiles, + deleteSnapshot, + deleteExtraSnapshotsOfActiveFilesystems, + deleteExtraSnapshots, +} from "./snapshots"; +export { + getAll, + getRecent, + get, + set, + clearError, + getErrors, + clearAllErrors, +} from "./db"; +export { shareNFS, unshareNFS } from "./nfs"; +export { createFilesystem, deleteFilesystem } from "./create"; +export { createSnapshot, getSnapshots, maintainSnapshots } from "./snapshots"; +export { + mountFilesystem, + unmountFilesystem, + setQuota, + syncProperties, +} from "./properties"; +export { archiveFilesystem, dearchiveFilesystem } from "./archive"; +export { maintainBackups, createBackup } from "./backup"; +export { recv, send, recompact, maintainStreams } from "./streams"; +export { pull } from "./pull"; diff --git a/src/packages/file-server/zfs/names.ts b/src/packages/file-server/zfs/names.ts new file mode 100644 index 0000000000..aee7b41cd5 --- /dev/null +++ b/src/packages/file-server/zfs/names.ts @@ -0,0 +1,176 @@ +import { join } from "path"; +import { context } from "./config"; +import { primaryKey, type PrimaryKey } from "./types"; +import { randomId } from "@cocalc/nats/names"; + +export function databaseFilename(data: string) { + return join(data, "database.sqlite3"); +} + +export function namespaceDataset({ + pool, + namespace, +}: { + pool: string; + namespace: string; +}) { + return `${pool}/${namespace}`; +} + +// Archives +// There is one single dataset for each namespace/pool pair: All the different +// archives across filesystems are stored in the *same* dataset, since there is no +// point in separating them. +export function archivesDataset({ + pool, + namespace, +}: { + pool: string; + namespace: string; +}) { + return `${namespaceDataset({ pool, namespace })}/archives`; +} + +export function archivesMountpoint({ + pool, + namespace, +}: { + pool: string; + namespace: string; +}) { + return join(context.ARCHIVES, namespace, pool); +} + +export function filesystemArchivePath({ + pool, + ...fs +}: PrimaryKey & { pool: string }) { + const pk = primaryKey(fs); + return join( + archivesMountpoint({ pool, namespace: pk.namespace }), + pk.owner_type, + pk.owner_id, + pk.name, + ); +} + +export function filesystemArchiveFilename(opts: PrimaryKey & { pool: string }) { + const { owner_type, owner_id, name } = primaryKey(opts); + return join( + filesystemArchivePath(opts), + `full-${owner_type}-${owner_id}-${name}.zfs`, + ); +} + +export function filesystemStreamsPath(opts: PrimaryKey & { pool: string }) { + return join(filesystemArchivePath(opts), "streams"); +} + +export function filesystemStreamsFilename({ + snapshot1, + snapshot2, + ...opts +}: PrimaryKey & { snapshot1: string; snapshot2: string; pool: string }) { + return join(filesystemStreamsPath(opts), `${snapshot1}-${snapshot2}.zfs`); +} + +// Bup +export function bupDataset({ + pool, + namespace, +}: { + pool: string; + namespace: string; +}) { + return `${namespaceDataset({ pool, namespace })}/bup`; +} + +export function bupMountpoint({ + pool, + namespace, +}: { + pool: string; + namespace: string; +}) { + return join(context.BUP, namespace, pool); +} + +export function bupFilesystemMountpoint({ + pool, + ...fs +}: PrimaryKey & { pool: string }) { + const pk = primaryKey(fs); + return join( + bupMountpoint({ ...pk, pool }), + pk.owner_type, + pk.owner_id, + pk.name, + ); +} + +// Filesystems + +export function filesystemsPath({ namespace }) { + return join(context.FILESYSTEMS, namespace); +} + +export function filesystemMountpoint(fs: PrimaryKey) { + const pk = primaryKey(fs); + return join(filesystemsPath(pk), pk.owner_type, pk.owner_id, pk.name); +} + +export function filesystemSnapshotMountpoint( + opts: PrimaryKey & { snapshot: string }, +) { + return join(filesystemMountpoint(opts), ".zfs", "snapshot", opts.snapshot); +} + +export function filesystemsDataset({ + pool, + namespace, +}: { + pool: string; + namespace: string; +}) { + return `${namespaceDataset({ pool, namespace })}/filesystems`; +} + +// There is one single dataset for each project_id/namespace/pool tripple since it +// is critical to separate each project to properly support snapshots, clones, +// backups, etc. +export function filesystemDataset({ + pool, + ...fs +}: PrimaryKey & { pool: string }) { + const { namespace, owner_type, owner_id, name } = primaryKey(fs); + // NOTE: we *could* use a heirarchy of datasets like this: + // ${owner_type}/${owner_id}/${name} + // However, that greatly increases the raw number of datasets, and there's a huge performance + // penalty. Since the owner_type is a fixed small list, owner_id is a uuid and the name is + // more general, there's no possible overlaps just concating them as below, and this way there's + // only one dataset, rather than three. (We also don't need to worry about deleting parents + // when there are no children...) + return `${filesystemsDataset({ pool, namespace: namespace })}/${owner_type}-${owner_id}-${name}`; +} + +export function tempDataset({ + pool, + namespace, +}: { + pool: string; + namespace: string; +}) { + return `${namespaceDataset({ pool, namespace })}/temp`; +} + +export function filesystemDatasetTemp({ + pool, + ...fs +}: PrimaryKey & { pool: string }) { + const { namespace, owner_type, owner_id, name } = primaryKey(fs); + return `${tempDataset({ pool, namespace })}/${owner_type}-${owner_id}-${name}-${randomId()}`; +} + +// NOTE: We use "join" for actual file paths and explicit +// strings with / for ZFS filesystem names, since in some whacky +// futuristic world maybe this server is running on MS Windows. diff --git a/src/packages/file-server/zfs/nfs.ts b/src/packages/file-server/zfs/nfs.ts new file mode 100644 index 0000000000..a3c6f87603 --- /dev/null +++ b/src/packages/file-server/zfs/nfs.ts @@ -0,0 +1,111 @@ +import { get, set, touch } from "./db"; +import { exec } from "./util"; +import { filesystemDataset, filesystemMountpoint } from "./names"; +import { primaryKey, type PrimaryKey } from "./types"; + +// Ensure that this filesystem is mounted and setup so that export to the +// given client is allowed. +// Returns the remote that the client should use for NFS mounting, i.e., +// this return s, then type "mount s /mnt/..." to mount the filesystem. +// If client is not given, just sets the share at NFS level +// to what's specified in the database. +export async function shareNFS({ + client, + ...fs +}: PrimaryKey & { client?: string }): Promise { + client = client?.trim(); + const pk = primaryKey(fs); + const { pool, nfs } = get(pk); + let hostname; + if (client) { + hostname = await hostnameFor(client); + if (!nfs.includes(client)) { + nfs.push(client); + // update database which tracks what the share should be. + set({ ...pk, nfs: ({ nfs }) => [...nfs, client] }); + } + } + // actually ensure share is configured. + const name = filesystemDataset({ pool, ...pk }); + const sharenfs = + nfs.length > 0 + ? `${nfs.map((client) => `rw=${client}`).join(",")},no_root_squash,crossmnt,no_subtree_check` + : "off"; + await exec({ + verbose: true, + command: "sudo", + args: ["zfs", "set", `sharenfs=${sharenfs}`, name], + }); + if (nfs.length > 0) { + touch(pk); + } + if (client) { + return `${hostname}:${filesystemMountpoint(pk)}`; + } else { + return ""; + } +} + +// remove given client from nfs sharing +export async function unshareNFS({ + client, + ...fs +}: PrimaryKey & { client: string }) { + const pk = primaryKey(fs); + let { nfs } = get(pk); + if (!nfs.includes(client)) { + // nothing to do + return; + } + nfs = nfs.filter((x) => x != client); + // update database which tracks what the share should be. + set({ ...pk, nfs }); + // update zfs/nfs to no longer share to this client + await shareNFS(pk); +} + +let serverIps: null | string[] = null; +async function hostnameFor(client: string) { + if (serverIps == null) { + const ipRegex = /^(\d{1,3}\.){3}\d{1,3}$/; + const { stdout } = await exec({ + verbose: false, + command: "ifconfig", + }); + let i = stdout.indexOf("inet "); + const v: string[] = []; + while (i != -1) { + let j = stdout.indexOf("\n", i); + if (j == -1) { + break; + } + const x = stdout.slice(i, j).split(" "); + const ip = x[1]; + if (ipRegex.test(ip)) { + v.push(ip); + } + i = stdout.indexOf("inet ", j); + } + if (v.length == 0) { + throw Error("unable to determine server ip address"); + } + serverIps = v; + } + for (const ip of serverIps) { + if (subnetMatch(ip, client)) { + return ip; + } + } + throw Error("found no matching subdomain"); +} + +// a and b are ip addresses. Return true +// if the are on the same subnet, by which +// we mean that the first *TWO* segments match, +// since that's the size of our subnets usually. +// TODO: make configurable (?). +function subnetMatch(a, b) { + const v = a.split("."); + const w = b.split("."); + return v[0] == w[0] && v[1] == w[1]; +} diff --git a/src/packages/file-server/zfs/pools.ts b/src/packages/file-server/zfs/pools.ts new file mode 100644 index 0000000000..b71bdc45f2 --- /dev/null +++ b/src/packages/file-server/zfs/pools.ts @@ -0,0 +1,228 @@ +/* +This code sets things up for each pool and namespace, e.g., defining datasets, creating directories, +etc. as defined in config and names. + +WARNING: For efficientcy and sanity, it assumes that once something is setup, it stays setup. +If there is a chaos monkey running around breaking things (e.g., screwing up +file permissions, deleting datasets, etc.,) then this code won't help at all. + +OPERATIONS: + +- To add a new pool, just create it using zfs with a name sthat starts with context.PREFIX. + It should automatically start getting used within POOLS_CACHE_MS by newly created filesystems. + +*/ + +import { reuseInFlight } from "@cocalc/util/reuse-in-flight"; +import { context, POOLS_CACHE_MS } from "./config"; +import { exec } from "./util"; +import { + archivesDataset, + archivesMountpoint, + namespaceDataset, + filesystemsDataset, + filesystemsPath, + bupDataset, + bupMountpoint, + tempDataset, +} from "./names"; +import { exists } from "@cocalc/backend/misc/async-utils-node"; +import { getNamespacesAndPools } from "./db"; + +// Make sure all pools and namespaces are initialized for all existing filesystems. +// This should be needed after booting up the server and importing the pools. +export async function initializeAllPools() { + // TODO: maybe import all here? + + for (const { namespace, pool } of getNamespacesAndPools()) { + await initializePool({ namespace, pool }); + } +} + +interface Pool { + name: string; + state: "ONLINE" | "OFFLINE"; + size: number; + allocated: number; + free: number; +} + +type Pools = { [name: string]: Pool }; +let poolsCache: { [prefix: string]: Pools } = {}; + +export const getPools = reuseInFlight( + async ({ noCache }: { noCache?: boolean } = {}): Promise => { + if (!noCache && poolsCache[context.PREFIX]) { + return poolsCache[context.PREFIX]; + } + const { stdout } = await exec({ + verbose: true, + command: "zpool", + args: ["list", "-j", "--json-int", "-o", "size,allocated,free"], + }); + const { pools } = JSON.parse(stdout); + const v: { [name: string]: Pool } = {}; + for (const name in pools) { + if (!name.startsWith(context.PREFIX)) { + continue; + } + const pool = pools[name]; + for (const key in pool.properties) { + pool.properties[key] = pool.properties[key].value; + } + v[name] = { name, state: pool.state, ...pool.properties }; + } + poolsCache[context.PREFIX] = v; + if (!process.env.COCALC_TEST_MODE) { + // only clear cache in non-test mode + setTimeout(() => { + delete poolsCache[context.PREFIX]; + }, POOLS_CACHE_MS); + } + return v; + }, +); + +// OK to call this again even if initialized already. +export const initializePool = reuseInFlight( + async ({ + namespace = context.namespace, + pool, + }: { + namespace?: string; + pool: string; + }) => { + if (!pool.startsWith(context.PREFIX)) { + throw Error( + `pool (="${pool}") must start with the prefix '${context.PREFIX}'`, + ); + } + // archives and filesystems for each namespace are in this dataset + await ensureDatasetExists({ + name: namespaceDataset({ namespace, pool }), + }); + + // Initialize archives dataset, used for archiving filesystems. + await ensureDatasetExists({ + name: archivesDataset({ pool, namespace }), + mountpoint: archivesMountpoint({ pool, namespace }), + }); + // This sets up the parent filesystem for all filesystems + // and enable compression and dedup. + await ensureDatasetExists({ + name: filesystemsDataset({ namespace, pool }), + }); + await ensureDatasetExists({ + name: tempDataset({ namespace, pool }), + dedup: "off", + }); + // Initialize bup dataset, used for backups. + await ensureDatasetExists({ + name: bupDataset({ pool, namespace }), + mountpoint: bupMountpoint({ pool, namespace }), + compression: "off", + dedup: "off", + }); + + const filesystems = filesystemsPath({ namespace }); + if (!(await exists(filesystems))) { + await exec({ + verbose: true, + command: "sudo", + args: ["mkdir", "-p", filesystems], + }); + await exec({ + verbose: true, + command: "sudo", + args: ["chmod", "a+rx", context.FILESYSTEMS], + }); + await exec({ + verbose: true, + command: "sudo", + args: ["chmod", "a+rx", filesystems], + }); + } + }, +); + +// If a dataset exists, it is assumed to exist henceforth for the life of this process. +// That's fine for *this* application here of initializing pools, since we never delete +// anything here. +const datasetExistsCache = new Set(); +async function datasetExists(name: string): Promise { + if (datasetExistsCache.has(name)) { + return true; + } + try { + await exec({ + verbose: true, + command: "zfs", + args: ["list", name], + }); + datasetExistsCache.add(name); + return true; + } catch { + return false; + } +} + +async function isMounted(dataset): Promise { + const { stdout } = await exec({ + command: "zfs", + args: ["get", "mounted", dataset, "-j"], + }); + const x = JSON.parse(stdout); + return x.datasets[dataset].properties.mounted.value == "yes"; +} + +async function ensureDatasetExists({ + name, + mountpoint, + compression = "lz4", + dedup = "on", +}: { + name: string; + mountpoint?: string; + compression?: "lz4" | "off"; + dedup?: "on" | "off"; +}) { + if (await datasetExists(name)) { + if (mountpoint && !(await isMounted(name))) { + // ensure mounted + await exec({ + verbose: true, + command: "sudo", + args: ["zfs", "mount", name], + }); + } + return; + } + await exec({ + verbose: true, + command: "sudo", + args: [ + "zfs", + "create", + "-o", + `mountpoint=${mountpoint ? mountpoint : "none"}`, + "-o", + `compression=${compression}`, + "-o", + `dedup=${dedup}`, + name, + ], + }); + // make sure it is very hard to accidentally delete the entire dataset + // see https://github.com/openzfs/zfs/issues/4134#issuecomment-2565724994 + const safety = `${name}@safety`; + await exec({ + verbose: true, + command: "sudo", + args: ["zfs", "snapshot", safety], + }); + await exec({ + verbose: true, + command: "sudo", + args: ["zfs", "hold", "safety", safety], + }); +} diff --git a/src/packages/file-server/zfs/properties.ts b/src/packages/file-server/zfs/properties.ts new file mode 100644 index 0000000000..77c228a942 --- /dev/null +++ b/src/packages/file-server/zfs/properties.ts @@ -0,0 +1,127 @@ +import { exec } from "./util"; +import { filesystemDataset } from "./names"; +import { get, set } from "./db"; +import { MIN_QUOTA } from "./config"; +import { primaryKey, type PrimaryKey } from "./types"; + +export async function setQuota({ + // quota in **number of bytes**. + // If quota is smaller than actual dataset, then the quota is set to what is + // actually used (plus 10 MB), hopefully allowing user to delete some data. + // The quota is never less than MIN_QUOTA. + // The value stored in database is *also* then set to this amount. + // So this is not some magic fire and forget setting, but something + // that cocalc should regularly call when starting the filesystem. + quota, + noSync, + ...fs +}: { + quota: number; + noSync?: boolean; +} & PrimaryKey) { + const pk = primaryKey(fs); + // this will update current usage in the database + await syncProperties(pk); + const { pool, used_by_dataset } = get(pk); + const used = (used_by_dataset ?? 0) + 10 * 1024; + if (quota < used) { + quota = used!; + } + quota = Math.max(MIN_QUOTA, quota); + try { + await exec({ + verbose: true, + command: "sudo", + args: [ + "zfs", + "set", + // refquota so snapshots don't count against the user + `refquota=${quota}`, + filesystemDataset({ pool, ...pk }), + ], + }); + } finally { + // this sets quota in database in bytes to whatever was just set above. + await syncProperties(pk); + } +} + +// Sync with ZFS the properties for the given filesystem by +// setting the database to what is in ZFS: +// - total space used by snapshots +// - total space used by dataset +// - the quota +export async function syncProperties(fs: PrimaryKey) { + const pk = primaryKey(fs); + const { pool, archived } = get(pk); + if (archived) { + // they can't have changed + return; + } + set({ + ...pk, + ...(await zfsGetProperties(filesystemDataset({ pool, ...pk }))), + }); +} + +export async function zfsGetProperties(dataset: string): Promise<{ + used_by_snapshots: number; + used_by_dataset: number; + quota: number | null; +}> { + const { stdout } = await exec({ + command: "zfs", + args: [ + "list", + dataset, + "-j", + "--json-int", + "-o", + "usedsnap,usedds,refquota", + ], + }); + const x = JSON.parse(stdout); + const { properties } = x.datasets[dataset]; + return { + used_by_snapshots: properties.usedbysnapshots.value, + used_by_dataset: properties.usedbydataset.value, + quota: properties.refquota.value ? properties.refquota.value : null, + }; +} + +export async function mountFilesystem(fs: PrimaryKey) { + const pk = primaryKey(fs); + const { pool } = get(pk); + try { + await exec({ + command: "sudo", + args: ["zfs", "mount", filesystemDataset({ pool, ...pk })], + what: { ...pk, desc: "mount filesystem" }, + }); + } catch (err) { + if (`${err}`.includes("already mounted")) { + // fine + return; + } + throw err; + } +} + +export async function unmountFilesystem(fs: PrimaryKey) { + const pk = primaryKey(fs); + const { pool } = get(pk); + try { + await exec({ + verbose: true, + command: "sudo", + args: ["zfs", "unmount", filesystemDataset({ pool, ...pk })], + what: { ...pk, desc: "unmount filesystem" }, + }); + } catch (err) { + if (`${err}`.includes("not currently mounted")) { + // fine + } else { + throw err; + } + } +} diff --git a/src/packages/file-server/zfs/pull.ts b/src/packages/file-server/zfs/pull.ts new file mode 100644 index 0000000000..a0a0236814 --- /dev/null +++ b/src/packages/file-server/zfs/pull.ts @@ -0,0 +1,301 @@ +/* +Use zfs replication over ssh to pull recent filesystems from +one file-server to another one. + +This will be used for: + +- backup +- moving a filesystem from one region/cluster to another +*/ + +import { + type Filesystem, + type RawFilesystem, + primaryKey, + PrimaryKey, +} from "./types"; +import { exec } from "./util"; +import { + databaseFilename, + filesystemDataset, + filesystemMountpoint, +} from "./names"; +import { filesystemExists, getRecent, get, set } from "./db"; +import getLogger from "@cocalc/backend/logger"; +import { getSnapshots } from "./snapshots"; +import { createFilesystem, deleteFilesystem } from "./create"; +import { context } from "./config"; +import { archiveFilesystem, dearchiveFilesystem } from "./archive"; +import { deleteSnapshot } from "./snapshots"; +import { isEqual } from "lodash"; +import { join } from "path"; +import { readdir, unlink } from "fs/promises"; + +const logger = getLogger("file-server:zfs:pull"); + +// number of remote backups of db sqlite file to keep. +const NUM_DB_TO_KEEP = 10; + +// This is used for unit testing. It's what fields should match +// after doing a sync, except snapshots where local is a superset, +// unless you pull with deleteSnapshots set to true. +export const SYNCED_FIELDS = [ + // these four fields identify the filesystem, so they better get sync'd: + "namespace", + "owner_type", + "owner_id", + "name", + // snaphots -- reflects that we replicated properly. + "snapshots", + + // last_edited is useful for targetting sync work and making decisions, e.g.., should we delete + "last_edited", + // these just get directly sync'd. They aren't used unless somehow local were to actually server + // data directly. + "affinity", + "nfs", +]; + +interface Remote { + // remote = user@hostname that you can ssh to + remote: string; + // filesystem prefix of the remote server, so {prefix}/database.sqlite3 has the + // database that defines the state of the remote server. + prefix: string; +} + +// Copy from remote to here every filesystem that has changed since cutoff. +export async function pull({ + cutoff, + filesystem, + remote, + prefix, + deleteFilesystemCutoff, + deleteSnapshots, + dryRun, +}: Remote & { + // pulls everything that's changed with remote last_edited >= cutoff. + cutoff?: Date; + // alternatively -- if given, only pull this filesystem and nothing else: + filesystem?: PrimaryKey; + + // DANGER: if set, any local filesystem with + // cutoff <= last_edited <= deleteFilesystemCutoff + // gets actually deleted. This makes it possible, e.g., to delete every filesystem + // that was deleted on the main server in the last 6 months and deleted at least 1 + // month ago, so we have a bit of time before destroy backups. + deleteFilesystemCutoff?: Date; + // if true, delete local snapshots if they were deleted on the remote. + deleteSnapshots?: boolean; + // just say how much will happen, but don't do anything. + dryRun?: boolean; +}): Promise<{ + toUpdate: { remoteFs: Filesystem; localFs?: Filesystem }[]; + toDelete: RawFilesystem[]; +}> { + logger.debug("pull: from ", { remote, prefix, cutoff, filesystem }); + if (prefix.startsWith("/")) { + throw Error("prefix should not start with /"); + } + if (cutoff == null) { + cutoff = new Date(Date.now() - 1000 * 60 * 60 * 24 * 7); + } + logger.debug("pull: get the remote sqlite database"); + await exec({ command: "mkdir", args: ["-p", context.PULL] }); + const remoteDatabase = join( + context.PULL, + `${remote}:${prefix}---${new Date().toISOString()}.sqlite3`, + ); + // delete all but the most recent remote database files for this remote/prefix (?). + const oldDbFiles = (await readdir(context.PULL)) + .sort() + .filter((x) => x.startsWith(`${remote}:${prefix}---`)) + .slice(0, -NUM_DB_TO_KEEP); + for (const path of oldDbFiles) { + await unlink(join(context.PULL, path)); + } + + await exec({ + command: "scp", + args: [`${remote}:/${databaseFilename(prefix)}`, remoteDatabase], + }); + + logger.debug("pull: compare state"); + const recent = + filesystem != null + ? [get(filesystem, remoteDatabase)] + : getRecent({ cutoff, databaseFile: remoteDatabase }); + const toUpdate: { remoteFs: Filesystem; localFs?: Filesystem }[] = []; + for (const fs of recent) { + const remoteFs = get(fs, remoteDatabase); + if (!filesystemExists(fs)) { + toUpdate.push({ remoteFs }); + } else { + const localFs = get(fs); + if (remoteFs.archived != localFs.archived) { + // different archive state, so needs an update to resolve this (either way) + toUpdate.push({ remoteFs, localFs }); + continue; + } + if (deleteSnapshots) { + // sync if *any* snapshots differ + if (!isEqual(remoteFs.snapshots, localFs.snapshots)) { + toUpdate.push({ remoteFs, localFs }); + } + } else { + // only sync if newest snapshots are different + const newestRemoteSnapshot = + remoteFs.snapshots[remoteFs.snapshots.length - 1]; + if (!newestRemoteSnapshot) { + // no snapshots yet, so nothing to do. + continue; + } + const newestLocalSnapshot = + localFs.snapshots[localFs.snapshots.length - 1]; + if ( + !newestLocalSnapshot || + newestRemoteSnapshot > newestLocalSnapshot + ) { + toUpdate.push({ remoteFs, localFs }); + } + } + } + } + + logger.debug(`pull: toUpdate.length = ${toUpdate.length}`); + if (!dryRun) { + for (const x of toUpdate) { + logger.debug("pull: updating ", x); + await pullOne({ ...x, remote, deleteSnapshots }); + } + } + + const toDelete: RawFilesystem[] = []; + if (deleteFilesystemCutoff) { + for (const fs of getRecent({ cutoff })) { + if (!filesystemExists(fs, remoteDatabase)) { + if (new Date(fs.last_edited ?? 0) <= deleteFilesystemCutoff) { + // it's old enough to delete: + toDelete.push(fs); + } + } + } + } + logger.debug(`pull: toDelete.length = ${toDelete.length}`); + if (!dryRun) { + for (const fs of toDelete) { + logger.debug("pull: deleting", fs); + await deleteFilesystem(fs); + } + } + + return { toUpdate, toDelete }; +} + +async function pullOne({ + remoteFs, + localFs, + remote, + deleteSnapshots, +}: { + remoteFs: Filesystem; + localFs?: Filesystem; + remote?: string; + deleteSnapshots?: boolean; +}) { + logger.debug("pull:", { remoteFs, localFs, remote, deleteSnapshots }); + if (localFs == null) { + localFs = await createFilesystem(remoteFs); + } + + // sync last_edited, affinity and nfs fields in all cases + set({ + ...primaryKey(localFs), + last_edited: remoteFs.last_edited, + affinity: remoteFs.affinity, + nfs: remoteFs.nfs, + }); + + if (localFs.archived && !remoteFs.archived) { + // it's back in use: + await dearchiveFilesystem(localFs); + // don't return -- will then possibly sync more below, in case of new changes + } else if (!localFs.archived && remoteFs.archived) { + // we just archive ours. Note in theory there is a chance + // that our local version is not update-to-date with the remote + // version. However, the point of archiving is it should only happen + // many weeks after a filesystem stopped being used, and by that + // point we should have already pull'd the latest version. + // Don't bother worrying about deleting snapshots. + await archiveFilesystem(localFs); + return; + } + if (localFs.archived && remoteFs.archived) { + // nothing to do + // Also, don't bother worrying about deleting snapshots, since can't. + return; + } + const snapshot = newestCommonSnapshot(localFs.snapshots, remoteFs.snapshots); + const newest_snapshot = remoteFs.snapshots[remoteFs.snapshots.length - 1]; + if (!newest_snapshot || snapshot == newest_snapshot) { + logger.debug("pull: already have the newest snapshot locally"); + } else { + const mountpoint = filesystemMountpoint(localFs); + try { + if (!snapshot) { + // full replication with nothing local + await exec({ + verbose: true, + command: `ssh ${remote} "zfs send -e -c -R ${filesystemDataset(remoteFs)}@${newest_snapshot}" | sudo zfs recv -o mountpoint=${mountpoint} -F ${filesystemDataset(localFs)}`, + what: { + ...localFs, + desc: "pull: doing a full receive from remote", + }, + }); + } else { + // incremental based on the last common snapshot + const force = + localFs.snapshots[localFs.snapshots.length - 1] == snapshot + ? "" + : " -F "; + await exec({ + verbose: true, + command: `ssh ${remote} "zfs send -e -c -I @${snapshot} ${filesystemDataset(remoteFs)}@${newest_snapshot}" | sudo zfs recv -o mountpoint=${mountpoint} -F ${filesystemDataset(localFs)} ${force}`, + what: { + ...localFs, + desc: "pull: doing an incremental replication from remote", + }, + }); + } + } finally { + // even if there was an error, update local snapshots, since we likely have some new + // ones (e.g., even if there was a partial receive, interrupted by a network drop). + await getSnapshots(localFs); + } + } + + if (deleteSnapshots) { + // In general due to snapshot trimming, the + // list of snapshots on local might NOT match remote, but after replication + // local will always have a *supserset* of remote. We thus may have to + // trim some snapshots: + const remoteSnapshots = new Set(remoteFs.snapshots); + const localSnapshots = get(localFs).snapshots; + for (const snapshot of localSnapshots) { + if (!remoteSnapshots.has(snapshot)) { + await deleteSnapshot({ ...localFs, snapshot }); + } + } + } +} + +// s0 and s1 are sorted oldest-to-newest lists of names of snapshots. +// return largest that is in common between the two or undefined if nothing is in common +function newestCommonSnapshot(s0: string[], s1: string[]) { + const t1 = new Set(s1); + for (let i = s0.length - 1; i >= 0; i--) { + if (t1.has(s0[i])) { + return s0[i]; + } + } +} diff --git a/src/packages/file-server/zfs/snapshots.ts b/src/packages/file-server/zfs/snapshots.ts new file mode 100644 index 0000000000..c2d3ab4e7f --- /dev/null +++ b/src/packages/file-server/zfs/snapshots.ts @@ -0,0 +1,315 @@ +/* +Manage creating and deleting rolling snapshots of a filesystem. + +We keep track of all state in the sqlite database, so only have to touch +ZFS when we actually need to do something. Keep this in mind though since +if you try to mess with snapshots directly then the sqlite database won't +know you did that. +*/ + +import { exec } from "./util"; +import { get, getRecent, set } from "./db"; +import { filesystemDataset, filesystemMountpoint } from "./names"; +import { splitlines } from "@cocalc/util/misc"; +import getLogger from "@cocalc/backend/logger"; +import { + SNAPSHOT_INTERVAL_MS, + SNAPSHOT_INTERVALS_MS, + SNAPSHOT_COUNTS, +} from "./config"; +import { syncProperties } from "./properties"; +import { primaryKey, type PrimaryKey } from "./types"; +import { isEqual } from "lodash"; + +const logger = getLogger("file-server:zfs/snapshots"); + +export async function maintainSnapshots(cutoff?: Date) { + await deleteExtraSnapshotsOfActiveFilesystems(cutoff); + await snapshotActiveFilesystems(cutoff); +} + +// If there any changes to the filesystem since the last snapshot, +// and there are no snapshots since SNAPSHOT_INTERVAL_MS ms ago, +// make a new one. Always returns the most recent snapshot name. +// Error if filesystem is archived. +export async function createSnapshot({ + force, + ifChanged, + ...fs +}: PrimaryKey & { + force?: boolean; + // note -- ifChanged is VERY fast, but it's not instantaneous... + ifChanged?: boolean; +}): Promise { + logger.debug("createSnapshot: ", fs); + const pk = primaryKey(fs); + const { pool, archived, snapshots } = get(pk); + if (archived) { + throw Error("cannot snapshot an archived filesystem"); + } + if (!force && !ifChanged && snapshots.length > 0) { + // check for sufficiently recent snapshot + const last = new Date(snapshots[snapshots.length - 1]); + if (Date.now() - last.valueOf() < SNAPSHOT_INTERVAL_MS) { + // snapshot sufficiently recent + return snapshots[snapshots.length - 1]; + } + } + + // Check to see if nothing change on disk since last snapshot - if so, don't make a new one: + if (!force && snapshots.length > 0) { + const written = await getWritten(pk); + if (written == 0) { + // for sure definitely nothing written, so no possible + // need to make a snapshot + return snapshots[snapshots.length - 1]; + } + } + + const snapshot = new Date().toISOString(); + await exec({ + verbose: true, + command: "sudo", + args: [ + "zfs", + "snapshot", + `${filesystemDataset({ ...pk, pool })}@${snapshot}`, + ], + what: { ...pk, desc: "creating snapshot of project" }, + }); + set({ + ...pk, + snapshots: ({ snapshots }) => [...snapshots, snapshot], + }); + syncProperties(pk); + return snapshot; +} + +async function getWritten(fs: PrimaryKey) { + const pk = primaryKey(fs); + const { pool } = get(pk); + const { stdout } = await exec({ + verbose: true, + command: "zfs", + args: ["list", "-Hpo", "written", filesystemDataset({ ...pk, pool })], + what: { + ...pk, + desc: "getting amount of newly written data in project since last snapshot", + }, + }); + return parseInt(stdout); +} + +export async function zfsGetSnapshots(dataset: string) { + const { stdout } = await exec({ + command: "zfs", + args: ["list", "-j", "-o", "name", "-r", "-t", "snapshot", dataset], + }); + const snapshots = Object.keys(JSON.parse(stdout).datasets).map( + (name) => name.split("@")[1], + ); + return snapshots; +} + +// gets snapshots from disk via zfs *and* sets the list of snapshots +// in the database to match (and also updates sizes) +export async function getSnapshots(fs: PrimaryKey) { + const pk = primaryKey(fs); + const filesystem = get(fs); + const snapshots = await zfsGetSnapshots(filesystemDataset(filesystem)); + if (!isEqual(snapshots, filesystem.snapshots)) { + set({ ...pk, snapshots }); + syncProperties(fs); + } + return snapshots; +} + +export async function deleteSnapshot({ + snapshot, + ...fs +}: PrimaryKey & { snapshot: string }) { + const pk = primaryKey(fs); + logger.debug("deleteSnapshot: ", pk, snapshot); + const { pool, last_send_snapshot } = get(pk); + if (snapshot == last_send_snapshot) { + throw Error( + "can't delete snapshot since it is the last one used for a zfs send", + ); + } + await exec({ + verbose: true, + command: "sudo", + args: [ + "zfs", + "destroy", + `${filesystemDataset({ ...pk, pool })}@${snapshot}`, + ], + what: { ...pk, desc: "destroying a snapshot of a project" }, + }); + set({ + ...pk, + snapshots: ({ snapshots }) => snapshots.filter((x) => x != snapshot), + }); + syncProperties(pk); +} + +/* +Remove snapshots according to our retention policy, and +never delete last_stream if set. + +Returns names of deleted snapshots. +*/ +export async function deleteExtraSnapshots(fs: PrimaryKey): Promise { + const pk = primaryKey(fs); + logger.debug("deleteExtraSnapshots: ", pk); + const { last_send_snapshot, snapshots } = get(pk); + if (snapshots.length == 0) { + // nothing to do + return []; + } + + // sorted from BIGGEST to smallest + const times = snapshots.map((x) => new Date(x).valueOf()); + times.reverse(); + const save = new Set(); + if (last_send_snapshot) { + save.add(new Date(last_send_snapshot).valueOf()); + } + for (const type in SNAPSHOT_COUNTS) { + const count = SNAPSHOT_COUNTS[type]; + const length_ms = SNAPSHOT_INTERVALS_MS[type]; + + // Pick the first count newest snapshots at intervals of length + // length_ms milliseconds. + let n = 0, + i = 0, + last_tm = 0; + while (n < count && i < times.length) { + const tm = times[i]; + if (!last_tm || tm <= last_tm - length_ms) { + save.add(tm); + last_tm = tm; + n += 1; // found one more + } + i += 1; // move to next snapshot + } + } + const toDelete = snapshots.filter((x) => !save.has(new Date(x).valueOf())); + for (const snapshot of toDelete) { + await deleteSnapshot({ ...pk, snapshot }); + } + return toDelete; +} + +// Go through ALL projects with last_edited >= cutoff stored +// here and run trimActiveFilesystemSnapshots. +export async function deleteExtraSnapshotsOfActiveFilesystems(cutoff?: Date) { + const v = getRecent({ cutoff }); + logger.debug( + `deleteSnapshotsOfActiveFilesystems: considering ${v.length} filesystems`, + ); + let i = 0; + for (const fs of v) { + if (fs.archived) { + continue; + } + try { + await deleteExtraSnapshots(fs); + } catch (err) { + logger.debug(`deleteSnapshotsOfActiveFilesystems: error -- ${err}`); + } + i += 1; + if (i % 10 == 0) { + logger.debug(`deleteSnapshotsOfActiveFilesystems: ${i}/${v.length}`); + } + } +} + +// Go through ALL projects with last_edited >= cutoff and snapshot them +// if they are due a snapshot. +// cutoff = a Date (default = 1 week ago) +export async function snapshotActiveFilesystems(cutoff?: Date) { + logger.debug("snapshotActiveFilesystems: getting..."); + const v = getRecent({ cutoff }); + logger.debug( + `snapshotActiveFilesystems: considering ${v.length} projects`, + cutoff, + ); + let i = 0; + for (const fs of v) { + if (fs.archived) { + continue; + } + try { + await createSnapshot(fs); + } catch (err) { + // error is already logged in error field of database + logger.debug(`snapshotActiveFilesystems: error -- ${err}`); + } + i += 1; + if (i % 10 == 0) { + logger.debug(`snapshotActiveFilesystems: ${i}/${v.length}`); + } + } +} + +/* +Get list of files modified since given snapshot (or last snapshot if not given). + +**There's probably no good reason to ever use this code!** + +The reason is because it's really slow, e.g., I added the +cocalc src directory (5000) files and it takes about 6 seconds +to run this. In contrast. "time find .", which lists EVERYTHING +takes less than 0.074s. You could do that before and after, then +compare them, and it'll be a fraction of a second. +*/ +interface Mod { + time: number; + change: "-" | "+" | "M" | "R"; // remove/create/modify/rename + // see "man zfs diff": + type: "B" | "C" | "/" | ">" | "|" | "@" | "P" | "=" | "F"; + path: string; +} + +export async function getModifiedFiles({ + snapshot, + ...fs +}: PrimaryKey & { snapshot: string }) { + const pk = primaryKey(fs); + logger.debug(`getModifiedFiles: `, pk); + const { pool, snapshots } = get(pk); + if (snapshots.length == 0) { + return []; + } + if (snapshot == null) { + snapshot = snapshots[snapshots.length - 1]; + } + const { stdout } = await exec({ + verbose: true, + command: "sudo", + args: [ + "zfs", + "diff", + "-FHt", + `${filesystemDataset({ ...pk, pool })}@${snapshot}`, + ], + what: { ...pk, desc: "getting files modified since last snapshot" }, + }); + const mnt = filesystemMountpoint(pk) + "/"; + const files: Mod[] = []; + for (const line of splitlines(stdout)) { + const x = line.split(/\t/g); + let path = x[3]; + if (path.startsWith(mnt)) { + path = path.slice(mnt.length); + } + files.push({ + time: parseFloat(x[0]) * 1000, + change: x[1] as any, + type: x[2] as any, + path, + }); + } + return files; +} diff --git a/src/packages/file-server/zfs/streams.ts b/src/packages/file-server/zfs/streams.ts new file mode 100644 index 0000000000..6720c21798 --- /dev/null +++ b/src/packages/file-server/zfs/streams.ts @@ -0,0 +1,253 @@ +/* +Send/Receive incremental replication streams of a filesystem. +*/ + +import { type PrimaryKey } from "./types"; +import { get, getRecent, set } from "./db"; +import getLogger from "@cocalc/backend/logger"; +import { + filesystemStreamsPath, + filesystemStreamsFilename, + filesystemDataset, +} from "./names"; +import { exec } from "./util"; +import { split } from "@cocalc/util/misc"; +import { join } from "path"; +import { getSnapshots } from "./snapshots"; +import { STREAM_INTERVAL_MS, MAX_STREAMS } from "./config"; + +const logger = getLogger("file-server:zfs:send"); + +export async function send(fs: PrimaryKey) { + const filesystem = get(fs); + if (filesystem.archived) { + logger.debug("filesystem is archived, so nothing to do", fs); + return; + } + const { snapshots } = filesystem; + const newest_snapshot = snapshots[snapshots.length - 1]; + if (!newest_snapshot) { + logger.debug("no snapshots yet"); + return; + } + if (newest_snapshot == filesystem.last_send_snapshot) { + logger.debug("no new snapshots", fs); + // the most recent snapshot is the same as the last one we used to make + // an archive, so nothing to do. + return; + } + await exec({ + command: "sudo", + args: ["mkdir", "-p", filesystemStreamsPath(filesystem)], + what: { ...filesystem, desc: "make send target directory" }, + }); + + let stream; + if (!filesystem.last_send_snapshot) { + logger.debug("doing first ever send -- a full send"); + stream = filesystemStreamsFilename({ + ...filesystem, + snapshot1: new Date(0).toISOString(), + snapshot2: newest_snapshot, + }); + try { + await exec({ + verbose: true, + command: `sudo sh -c 'zfs send -e -c -R ${filesystemDataset(filesystem)}@${newest_snapshot} > ${stream}.temp'`, + what: { + ...filesystem, + desc: "send: zfs send of full filesystem dataset (first full send)", + }, + }); + } catch (err) { + await exec({ + verbose: true, + command: "sudo", + args: ["rm", `${stream}.temp`], + }); + throw err; + } + } else { + logger.debug("doing incremental send"); + const snapshot1 = filesystem.last_send_snapshot; + const snapshot2 = newest_snapshot; + stream = filesystemStreamsFilename({ + ...filesystem, + snapshot1, + snapshot2, + }); + try { + await exec({ + verbose: true, + command: `sudo sh -c 'zfs send -e -c -I @${snapshot1} ${filesystemDataset(filesystem)}@${snapshot2} > ${stream}.temp'`, + what: { + ...filesystem, + desc: "send: zfs incremental send", + }, + }); + } catch (err) { + await exec({ + verbose: true, + command: "sudo", + args: ["rm", `${stream}.temp`], + }); + throw err; + } + } + await exec({ + verbose: true, + command: "sudo", + args: ["mv", `${stream}.temp`, stream], + }); + set({ ...fs, last_send_snapshot: newest_snapshot }); +} + +async function getStreams(fs: PrimaryKey) { + const filesystem = get(fs); + const streamsPath = filesystemStreamsPath(filesystem); + const { stdout } = await exec({ + command: "sudo", + args: ["ls", streamsPath], + what: { ...filesystem, desc: "getting list of streams" }, + }); + return split(stdout.trim()).filter((path) => path.endsWith(".zfs")); +} + +export async function recv(fs: PrimaryKey) { + const filesystem = get(fs); + if (filesystem.archived) { + throw Error("filesystem must not be archived"); + } + const streams = await getStreams(filesystem); + if (streams.length == 0) { + logger.debug("no streams"); + return; + } + const { snapshots } = filesystem; + const newest_snapshot = snapshots[snapshots.length - 1] ?? ""; + const toRead = streams.filter((snapshot) => snapshot >= newest_snapshot); + if (toRead.length == 0) { + return; + } + const streamsPath = filesystemStreamsPath(filesystem); + try { + for (const stream of toRead) { + await exec({ + verbose: true, + command: `sudo sh -c 'cat ${join(streamsPath, stream)} | zfs recv ${filesystemDataset(filesystem)}'`, + what: { + ...filesystem, + desc: `send: zfs incremental receive`, + }, + }); + } + } finally { + // ensure snapshots and size info in our database is up to date: + await getSnapshots(fs); + } +} + +function getRange(streamName) { + const v = streamName.split("Z-"); + return { snapshot1: v + "Z", snapshot2: v[1].slice(0, -".zfs".length) }; +} + +// Replace older streams so that there are at most maxStreams total streams. +export async function recompact({ + maxStreams, + ...fs +}: PrimaryKey & { maxStreams: number }) { + const filesystem = get(fs); + const { snapshots } = filesystem; + const streams = await getStreams(filesystem); + if (streams.length <= maxStreams) { + // nothing to do + return; + } + if (maxStreams < 1) { + throw Error("maxStreams must be at least 1"); + } + // replace first n streams by one full replication stream + let n = streams.length - maxStreams + 1; + let snapshot2 = getRange(streams[n - 1]).snapshot2; + while (!snapshots.includes(snapshot2) && n < streams.length) { + snapshot2 = getRange(streams[n]).snapshot2; + if (snapshots.includes(snapshot2)) { + break; + } + n += 1; + } + if (!snapshots.includes(snapshot2)) { + throw Error( + "bug -- this can't happen because we never delete the last snapshot used for send", + ); + } + + const stream = filesystemStreamsFilename({ + ...filesystem, + snapshot1: new Date(0).toISOString(), + snapshot2, + }); + try { + await exec({ + verbose: true, + command: `sudo sh -c 'zfs send -e -c -R ${filesystemDataset(filesystem)}@${snapshot2} > ${stream}.temp'`, + what: { + ...filesystem, + desc: "send: zfs send of full filesystem dataset (first full send)", + }, + }); + // if this rm were to fail, then things would be left in a broken state, + // since ${stream}.temp also gets deleted in the catch. But it seems + // highly unlikely this rm of the old streams would ever fail. + const path = filesystemStreamsPath(filesystem); + await exec({ + verbose: true, + command: "sudo", + // full paths to the first n streams: + args: ["rm", "-f", ...streams.slice(0, n).map((x) => join(path, x))], + }); + await exec({ + verbose: true, + command: "sudo", + args: ["mv", `${stream}.temp`, stream], + }); + } catch (err) { + await exec({ + verbose: true, + command: "sudo", + args: ["rm", "-f", `${stream}.temp`], + }); + throw err; + } +} + +// Go through ALL filesystems with last_edited >= cutoff and send a stream if due, +// and also ensure number of streams isn't too large. +export async function maintainStreams(cutoff?: Date) { + logger.debug("backupActiveFilesystems: getting..."); + const v = getRecent({ cutoff }); + logger.debug(`maintainStreams: considering ${v.length} filesystems`, cutoff); + let i = 0; + for (const { archived, last_edited, last_send_snapshot, ...pk } of v) { + if (archived || !last_edited) { + continue; + } + const age = + new Date(last_edited).valueOf() - new Date(last_send_snapshot ?? 0).valueOf(); + if (age < STREAM_INTERVAL_MS) { + // there's a new enough stream already + continue; + } + try { + await send(pk); + await recompact({ ...pk, maxStreams: MAX_STREAMS }); + } catch (err) { + logger.debug(`maintainStreams: error -- ${err}`); + } + i += 1; + if (i % 10 == 0) { + logger.debug(`maintainStreams: ${i}/${v.length}`); + } + } +} diff --git a/src/packages/file-server/zfs/test/archive.test.ts b/src/packages/file-server/zfs/test/archive.test.ts new file mode 100644 index 0000000000..dfbb003bdb --- /dev/null +++ b/src/packages/file-server/zfs/test/archive.test.ts @@ -0,0 +1,105 @@ +/* +DEVELOPMENT: + +pnpm exec jest --watch archive.test.ts +*/ + +import { executeCode } from "@cocalc/backend/execute-code"; +import { createTestPools, deleteTestPools, init, describe } from "./util"; +import { + archiveFilesystem, + dearchiveFilesystem, + createFilesystem, + createSnapshot, + getSnapshots, + get, +} from "@cocalc/file-server/zfs"; +import { filesystemMountpoint } from "@cocalc/file-server/zfs/names"; +import { readFile, writeFile } from "fs/promises"; +import { join } from "path"; + +describe("create a project, put in some files/snapshot, archive the project, confirm gone, de-archive it, and confirm files are back as expected", () => { + jest.setTimeout(10000); + let x: any = null; + + beforeAll(async () => { + x = await createTestPools({ count: 1, size: "1G" }); + await init(); + }); + + afterAll(async () => { + if (x != null) { + await deleteTestPools(x); + } + }); + + const project_id = "00000000-0000-0000-0000-000000000001"; + const mnt = filesystemMountpoint({ project_id, namespace: "default" }); + const FILE_CONTENT = "hello"; + const FILENAME = "cocalc.txt"; + it("creates a project and write a file", async () => { + const filesystem = await createFilesystem({ + project_id, + }); + expect(filesystem.owner_type).toBe("project"); + expect(filesystem.owner_id).toBe(project_id); + const path = join(mnt, FILENAME); + await writeFile(path, FILE_CONTENT); + }); + + let snapshot1, snapshot2; + const FILE_CONTENT2 = "hello2"; + const FILENAME2 = "cocalc2.txt"; + + it("create a snapshot and write another file, so there is a nontrivial snapshot to be archived", async () => { + snapshot1 = await createSnapshot({ project_id }); + expect(!!snapshot1).toBe(true); + const path = join(mnt, FILENAME2); + await writeFile(path, FILE_CONTENT2); + snapshot2 = await createSnapshot({ project_id, force: true }); + expect(snapshot2).not.toEqual(snapshot1); + }); + + it("archive the project and checks project is no longer in zfs", async () => { + expect(get({ project_id }).archived).toBe(false); + await archiveFilesystem({ project_id }); + const { stdout } = await executeCode({ + command: "zfs", + args: ["list", x.pools[0]], + }); + expect(stdout).not.toContain(project_id); + expect(get({ project_id }).archived).toBe(true); + }); + + it("archiving an already archived project is an error", async () => { + await expect( + async () => await archiveFilesystem({ project_id }), + ).rejects.toThrow(); + }); + + it("dearchive project and verify zfs filesystem is back, along with files and snapshots", async () => { + let called = false; + await dearchiveFilesystem({ + project_id, + progress: () => { + called = true; + }, + }); + expect(called).toBe(true); + expect(get({ project_id }).archived).toBe(false); + + expect((await readFile(join(mnt, FILENAME))).toString()).toEqual( + FILE_CONTENT, + ); + expect((await readFile(join(mnt, FILENAME2))).toString()).toEqual( + FILE_CONTENT2, + ); + expect(await getSnapshots({ project_id })).toEqual([snapshot1, snapshot2]); + }); + + it("dearchiving an already de-archived project is an error", async () => { + await expect( + async () => await dearchiveFilesystem({ project_id }), + ).rejects.toThrow(); + }); +}); diff --git a/src/packages/file-server/zfs/test/create-types.test.ts b/src/packages/file-server/zfs/test/create-types.test.ts new file mode 100644 index 0000000000..150adda7b5 --- /dev/null +++ b/src/packages/file-server/zfs/test/create-types.test.ts @@ -0,0 +1,78 @@ +/* +DEVELOPMENT: + +pnpm exec jest --watch create-types.test.ts +*/ + +import { createTestPools, deleteTestPools, init, describe } from "./util"; +import { + createFilesystem, +} from "@cocalc/file-server/zfs"; +import type { Filesystem } from "../types"; + +describe("create some account and organization filesystems", () => { + let x: any = null; + + beforeAll(async () => { + x = await createTestPools({ count: 1, size: "1G" }); + await init(); + }); + + afterAll(async () => { + if (x != null) { + await deleteTestPools(x); + } + }); + + // Making these the same intentionally to ensure the filesystem properly + // does not distinguish types based on the owner_id. + const project_id = "00000000-0000-0000-0000-000000000001"; + const account_id = "00000000-0000-0000-0000-000000000001"; + const group_id = "00000000-0000-0000-0000-000000000001"; + const filesystems: Filesystem[] = []; + it("creates filesystems associated to the project, account and group", async () => { + const fs = await createFilesystem({ project_id }); + expect(fs.owner_id).toBe(project_id); + filesystems.push(fs); + const fs2 = await createFilesystem({ account_id, name: "cocalc" }); + expect(fs2.owner_id).toBe(account_id); + filesystems.push(fs2); + const fs3 = await createFilesystem({ group_id, name: "data" }); + expect(fs3.owner_id).toBe(group_id); + filesystems.push(fs3); + }); + + it("tries to create an account and group filesystem with empty name and gets an error", async () => { + expect(async () => { + await createFilesystem({ account_id }); + }).rejects.toThrow("name must be nonempty"); + expect(async () => { + await createFilesystem({ group_id }); + }).rejects.toThrow("name must be nonempty"); + }); + + it('for projects the name defaults to "home"', async () => { + expect(async () => { + await createFilesystem({ project_id, name: "" }); + }).rejects.toThrow("must be nonempty"); + expect(filesystems[0].name).toBe("home"); + }); + + it("name must be less than 64 characters", async () => { + let name = ""; + for (let i = 0; i < 63; i++) { + name += "x"; + } + await createFilesystem({ account_id, name }); + name += 1; + expect(async () => { + await createFilesystem({ account_id, name }); + }).rejects.toThrow("name must be at most 63 characters"); + }); + + it("name must not have 'funny characters'", async () => { + expect(async () => { + await createFilesystem({ account_id, name: "$%@!" }); + }).rejects.toThrow("name must only contain"); + }); +}); diff --git a/src/packages/file-server/zfs/test/create.test.ts b/src/packages/file-server/zfs/test/create.test.ts new file mode 100644 index 0000000000..3dd55339ea --- /dev/null +++ b/src/packages/file-server/zfs/test/create.test.ts @@ -0,0 +1,245 @@ +/* +DEVELOPMENT: + +pnpm exec jest --watch create.test.ts + + pnpm exec jest create.test.ts -b +*/ + +// application/typescript text +import { executeCode } from "@cocalc/backend/execute-code"; +import { createTestPools, deleteTestPools, init, describe } from "./util"; +import { + createFilesystem, + createBackup, + deleteFilesystem, + getPools, +} from "@cocalc/file-server/zfs"; +import { filesystemMountpoint } from "@cocalc/file-server/zfs/names"; +import { readFile, writeFile } from "fs/promises"; +import { join } from "path"; +import { exists } from "@cocalc/backend/misc/async-utils-node"; +import { uuid } from "@cocalc/util/misc"; +import { map as asyncMap } from "awaiting"; + +describe("creates project, clone project, delete projects", () => { + let x: any = null; + + beforeAll(async () => { + x = await createTestPools({ count: 1, size: "1G" }); + await init(); + }); + + afterAll(async () => { + if (x != null) { + await deleteTestPools(x); + } + }); + + it("verifies there is a pool", async () => { + const { stdout } = await executeCode({ + command: "zpool", + args: ["list", x.pools[0]], + }); + expect(stdout).toContain(x.pools[0]); + expect(Object.keys(await getPools()).length).toBe(1); + }); + + const project_id = "00000000-0000-0000-0000-000000000001"; + it("creates a project", async () => { + const project = await createFilesystem({ + project_id, + }); + expect(project.owner_id).toBe(project_id); + }); + + it("verify project is in output of zfs list", async () => { + const { stdout } = await executeCode({ + command: "zfs", + args: ["list", "-r", x.pools[0]], + }); + expect(stdout).toContain(project_id); + }); + + const FILE_CONTENT = "hello"; + const FILENAME = "cocalc.txt"; + it("write a file to the project", async () => { + const path = join( + filesystemMountpoint({ project_id, namespace: "default" }), + FILENAME, + ); + await writeFile(path, FILE_CONTENT); + }); + + const project_id2 = "00000000-0000-0000-0000-000000000002"; + it("clones our project to make a second project", async () => { + const project2 = await createFilesystem({ + project_id: project_id2, + clone: { project_id }, + }); + expect(project2.owner_id).toBe(project_id2); + }); + + it("verify clone is in output of zfs list", async () => { + const { stdout } = await executeCode({ + command: "zfs", + args: ["list", "-r", x.pools[0]], + }); + expect(stdout).toContain(project_id2); + }); + + it("read file from the clone", async () => { + const path = join( + filesystemMountpoint({ project_id: project_id2, namespace: "default" }), + FILENAME, + ); + const content = (await readFile(path)).toString(); + expect(content).toEqual(FILE_CONTENT); + }); + + let BUP_DIR; + it("make a backup of project, so can see that it gets deleted below", async () => { + const x = await createBackup({ project_id }); + BUP_DIR = x.BUP_DIR; + expect(await exists(BUP_DIR)).toBe(true); + }); + + it("attempt to delete first project and get error", async () => { + try { + await deleteFilesystem({ project_id }); + throw Error("must throw"); + } catch (err) { + expect(`${err}`).toContain("filesystem has dependent clones"); + } + }); + + it("delete second project, then first project, works", async () => { + await deleteFilesystem({ project_id: project_id2 }); + await deleteFilesystem({ project_id }); + const { stdout } = await executeCode({ + command: "zfs", + args: ["list", "-r", x.pools[0]], + }); + expect(stdout).not.toContain(project_id); + expect(stdout).not.toContain(project_id2); + }); + + it("verifies bup backup is also gone", async () => { + expect(await exists(BUP_DIR)).toBe(false); + }); +}); + +describe("create two projects with the same project_id at the same time, but in different namespaces", () => { + let x: any = null; + + beforeAll(async () => { + x = await createTestPools({ count: 2, size: "1G" }); + await init(); + }); + + afterAll(async () => { + if (x != null) { + await deleteTestPools(x); + } + }); + + it("there are TWO pools this time", async () => { + expect(Object.keys(await getPools()).length).toBe(2); + }); + + const project_id = "00000000-0000-0000-0000-000000000001"; + it("creates two projects", async () => { + const project = await createFilesystem({ + project_id, + namespace: "default", + }); + expect(project.owner_id).toBe(project_id); + + const project2 = await createFilesystem({ + project_id, + namespace: "test", + }); + expect(project2.owner_id).toBe(project_id); + // they are on different pools + expect(project.pool).not.toEqual(project2.pool); + }); + + it("two different entries in zfs list", async () => { + const { stdout: stdout0 } = await executeCode({ + command: "zfs", + args: ["list", "-r", x.pools[0]], + }); + expect(stdout0).toContain(project_id); + const { stdout: stdout1 } = await executeCode({ + command: "zfs", + args: ["list", "-r", x.pools[1]], + }); + expect(stdout1).toContain(project_id); + }); +}); + +describe("test the affinity property when creating projects", () => { + let x: any = null; + + beforeAll(async () => { + x = await createTestPools({ count: 2, size: "1G" }); + await init(); + }); + + afterAll(async () => { + if (x != null) { + await deleteTestPools(x); + } + }); + + const project_id = "00000000-0000-0000-0000-000000000001"; + const project_id2 = "00000000-0000-0000-0000-000000000002"; + const affinity = "math100"; + it("creates two projects with same afinity", async () => { + const project = await createFilesystem({ + project_id, + affinity, + }); + expect(project.owner_id).toBe(project_id); + + const project2 = await createFilesystem({ + project_id: project_id2, + affinity, + }); + expect(project2.owner_id).toBe(project_id2); + // they are on SAME pools, because of affinity + expect(project.pool).toEqual(project2.pool); + }); +}); + +describe("do a stress/race condition test creating a larger number of projects on a larger number of pools", () => { + let x: any = null; + + const count = 3; + const nprojects = 25; + + beforeAll(async () => { + x = await createTestPools({ count, size: "1G" }); + await init(); + }); + + afterAll(async () => { + if (x != null) { + await deleteTestPools(x); + } + }); + + it(`creates ${nprojects} projects in parallel on ${count} pools`, async () => { + const f = async (project_id) => { + await createFilesystem({ project_id }); + }; + const v: string[] = []; + for (let n = 0; n < nprojects; n++) { + v.push(uuid()); + } + // doing these in parallel and having it work is an important stress test, + // since we will get a bid speedup doing this in production, and there we + // will really need it. + await asyncMap(v, nprojects, f); + }); +}); diff --git a/src/packages/file-server/zfs/test/nfs.test.ts b/src/packages/file-server/zfs/test/nfs.test.ts new file mode 100644 index 0000000000..56729b2798 --- /dev/null +++ b/src/packages/file-server/zfs/test/nfs.test.ts @@ -0,0 +1,118 @@ +/* +DEVELOPMENT: + +pnpm exec jest --watch nfs.test.ts +*/ + +import { executeCode } from "@cocalc/backend/execute-code"; +import { + createTestPools, + deleteTestPools, + restartNfsServer, + init, + describe, +} from "./util"; +import { + createFilesystem, + createSnapshot, + get, + shareNFS, + unshareNFS, +} from "@cocalc/file-server/zfs"; +import { filesystemMountpoint } from "@cocalc/file-server/zfs/names"; +import { readFile, writeFile } from "fs/promises"; +import { join } from "path"; + +describe("create a project, put in a files, snapshot, another file, then share via NFS, mount and verify it works", () => { + let x: any = null; + const project_id = "00000000-0000-0000-0000-000000000001"; + let nsfMnt = ""; + + beforeAll(async () => { + x = await createTestPools({ count: 1, size: "1G" }); + nsfMnt = join(x.tempDir, project_id); + await init(); + }); + + afterAll(async () => { + if (x != null) { + await deleteTestPools(x); + } + }); + + const mnt = filesystemMountpoint({ project_id, namespace: "default" }); + const FILE_CONTENT = "hello"; + const FILENAME = "cocalc.txt"; + it("creates a project and write a file", async () => { + const project = await createFilesystem({ + project_id, + }); + expect(project.owner_id).toBe(project_id); + const path = join(mnt, FILENAME); + await writeFile(path, FILE_CONTENT); + }); + + let snapshot1, snapshot2; + const FILE_CONTENT2 = "hello2"; + const FILENAME2 = "cocalc2.txt"; + + it("create a snapshot and write another file, so there is a nontrivial snapshot to view through NFS", async () => { + snapshot1 = await createSnapshot({ project_id }); + expect(!!snapshot1).toBe(true); + const path = join(mnt, FILENAME2); + await writeFile(path, FILE_CONTENT2); + snapshot2 = await createSnapshot({ project_id, force: true }); + expect(snapshot2).not.toEqual(snapshot1); + }); + + let host = ""; + const client = "127.0.0.1"; + + const mount = async () => { + await executeCode({ + command: "sudo", + args: ["mkdir", "-p", nsfMnt], + }); + await executeCode({ + command: "sudo", + args: ["mount", host, nsfMnt], + }); + }; + + it("shares the project via NFS, and mounts it", async () => { + host = await shareNFS({ project_id, client }); + const project = get({ project_id }); + expect(project.nfs).toEqual([client]); + await mount(); + }); + + it("confirms our files and snapshots are there as expected", async () => { + const { stdout } = await executeCode({ + command: "sudo", + args: ["ls", nsfMnt], + }); + expect(stdout).toContain(FILENAME); + expect(stdout).toContain(FILENAME2); + expect((await readFile(join(nsfMnt, FILENAME))).toString()).toEqual( + FILE_CONTENT, + ); + expect((await readFile(join(nsfMnt, FILENAME2))).toString()).toEqual( + FILE_CONTENT2, + ); + }); + + it("stop NFS share and confirms it no longers works", async () => { + await executeCode({ + command: "sudo", + args: ["umount", nsfMnt], + }); + await restartNfsServer(); + await unshareNFS({ project_id, client }); + try { + await mount(); + throw Error("bug -- mount should fail"); + } catch (err) { + expect(`${err}`).toMatch(/not permitted|denied/); + } + }); +}); diff --git a/src/packages/file-server/zfs/test/pull.test.ts b/src/packages/file-server/zfs/test/pull.test.ts new file mode 100644 index 0000000000..a2689d7270 --- /dev/null +++ b/src/packages/file-server/zfs/test/pull.test.ts @@ -0,0 +1,315 @@ +/* +DEVELOPMENT: + +This tests pull replication by setting up two separate file-servers on disk locally +and doing pulls from one to the other over ssh. This involves password-less ssh +to root on localhost, and creating multiple pools, so use with caution and don't +expect this to work unless you really know what you're doing. +Also, these tests are going to take a while. + +Efficient powerful backup isn't trivial and is very valuable, so +its' worth the wait! + +pnpm exec jest --watch pull.test.ts +*/ + +import { join } from "path"; +import { createTestPools, deleteTestPools, init, describe } from "./util"; +import { + createFilesystem, + createSnapshot, + deleteSnapshot, + deleteFilesystem, + pull, + archiveFilesystem, + dearchiveFilesystem, +} from "@cocalc/file-server/zfs"; +import { context, setContext } from "@cocalc/file-server/zfs/config"; +import { filesystemMountpoint } from "@cocalc/file-server/zfs/names"; +import { readFile, writeFile } from "fs/promises"; +import { filesystemExists, get } from "@cocalc/file-server/zfs/db"; +import { SYNCED_FIELDS } from "../pull"; + +describe("create two separate file servers, then do pulls to sync one to the other under various conditions", () => { + let one: any = null, + two: any = null; + const prefix1 = context.PREFIX + ".1"; + const prefix2 = context.PREFIX + ".2"; + const remote = "root@localhost"; + + beforeAll(async () => { + one = await createTestPools({ count: 1, size: "1G", prefix: prefix1 }); + setContext({ prefix: prefix1 }); + await init(); + two = await createTestPools({ + count: 1, + size: "1G", + prefix: prefix2, + }); + setContext({ prefix: prefix2 }); + await init(); + }); + + afterAll(async () => { + await deleteTestPools(one); + await deleteTestPools(two); + }); + + it("creates a filesystem in pool one, writes a file and takes a snapshot", async () => { + setContext({ prefix: prefix1 }); + const fs = await createFilesystem({ + project_id: "00000000-0000-0000-0000-000000000001", + }); + await writeFile(join(filesystemMountpoint(fs), "a.txt"), "hello"); + await createSnapshot(fs); + expect(await filesystemExists(fs)).toEqual(true); + }); + + it("pulls filesystem one to filesystem two, and confirms the fs and file were indeed sync'd", async () => { + setContext({ prefix: prefix2 }); + expect( + await filesystemExists({ + project_id: "00000000-0000-0000-0000-000000000001", + }), + ).toEqual(false); + + // first dryRun + const { toUpdate, toDelete } = await pull({ + remote, + prefix: prefix1, + dryRun: true, + }); + expect(toDelete.length).toBe(0); + expect(toUpdate.length).toBe(1); + expect(toUpdate[0].remoteFs.owner_id).toEqual( + "00000000-0000-0000-0000-000000000001", + ); + expect(toUpdate[0].localFs).toBe(undefined); + + // now for real + const { toUpdate: toUpdate1, toDelete: toDelete1 } = await pull({ + remote, + prefix: prefix1, + }); + + expect(toDelete1).toEqual(toDelete); + expect(toUpdate1).toEqual(toUpdate); + const fs = { project_id: "00000000-0000-0000-0000-000000000001" }; + expect(await filesystemExists(fs)).toEqual(true); + expect( + (await readFile(join(filesystemMountpoint(fs), "a.txt"))).toString(), + ).toEqual("hello"); + + // nothing if we sync again: + const { toUpdate: toUpdate2, toDelete: toDelete2 } = await pull({ + remote, + prefix: prefix1, + }); + expect(toDelete2.length).toBe(0); + expect(toUpdate2.length).toBe(0); + }); + + it("creates another file in our filesystem, creates another snapshot, syncs again, and sees that the sync worked", async () => { + setContext({ prefix: prefix1 }); + const fs = { project_id: "00000000-0000-0000-0000-000000000001" }; + await writeFile(join(filesystemMountpoint(fs), "b.txt"), "cocalc"); + await createSnapshot({ ...fs, force: true }); + const { snapshots } = get(fs); + expect(snapshots.length).toBe(2); + + setContext({ prefix: prefix2 }); + await pull({ remote, prefix: prefix1 }); + + expect( + (await readFile(join(filesystemMountpoint(fs), "b.txt"))).toString(), + ).toEqual("cocalc"); + }); + + it("archives the project, does sync, and see the other one got archived", async () => { + const fs = { project_id: "00000000-0000-0000-0000-000000000001" }; + setContext({ prefix: prefix2 }); + const project2before = get(fs); + expect(project2before.archived).toBe(false); + + setContext({ prefix: prefix1 }); + await archiveFilesystem(fs); + const project1 = get(fs); + expect(project1.archived).toBe(true); + + setContext({ prefix: prefix2 }); + await pull({ remote, prefix: prefix1 }); + const project2 = get(fs); + expect(project2.archived).toBe(true); + expect(project1.last_edited).toEqual(project2.last_edited); + }); + + it("dearchives, does sync, then sees the other gets dearchived; this just tests that sync de-archives, but works even if there are no new snapshots", async () => { + const fs = { project_id: "00000000-0000-0000-0000-000000000001" }; + setContext({ prefix: prefix1 }); + await dearchiveFilesystem(fs); + const project1 = get(fs); + expect(project1.archived).toBe(false); + + setContext({ prefix: prefix2 }); + await pull({ remote, prefix: prefix1 }); + const project2 = get(fs); + expect(project2.archived).toBe(false); + }); + + it("archives project, does sync, de-archives project, adds another snapshot, then does sync, thus testing that sync both de-archives *and* pulls latest snapshot", async () => { + const fs = { project_id: "00000000-0000-0000-0000-000000000001" }; + setContext({ prefix: prefix1 }); + expect(get(fs).archived).toBe(false); + await archiveFilesystem(fs); + expect(get(fs).archived).toBe(true); + setContext({ prefix: prefix2 }); + await pull({ remote, prefix: prefix1 }); + expect(get(fs).archived).toBe(true); + + // now dearchive + setContext({ prefix: prefix1 }); + await dearchiveFilesystem(fs); + // write content + await writeFile(join(filesystemMountpoint(fs), "d.txt"), "hello"); + // snapshot + await createSnapshot({ ...fs, force: true }); + const project1 = get(fs); + + setContext({ prefix: prefix2 }); + await pull({ remote, prefix: prefix1 }); + const project2 = get(fs); + expect(project2.snapshots).toEqual(project1.snapshots); + expect(project2.archived).toBe(false); + }); + + it("deletes project, does sync, then sees the other does NOT gets deleted without passing the deleteFilesystemCutoff option, and also with deleteFilesystemCutoff an hour ago, but does get deleted with it now", async () => { + const fs = { project_id: "00000000-0000-0000-0000-000000000001" }; + setContext({ prefix: prefix1 }); + expect(await filesystemExists(fs)).toEqual(true); + await deleteFilesystem(fs); + expect(await filesystemExists(fs)).toEqual(false); + + setContext({ prefix: prefix2 }); + expect(await filesystemExists(fs)).toEqual(true); + await pull({ remote, prefix: prefix1 }); + expect(await filesystemExists(fs)).toEqual(true); + + await pull({ + remote, + prefix: prefix1, + deleteFilesystemCutoff: new Date(Date.now() - 1000 * 60 * 60), + }); + expect(await filesystemExists(fs)).toEqual(true); + + await pull({ + remote, + prefix: prefix1, + deleteFilesystemCutoff: new Date(), + }); + expect(await filesystemExists(fs)).toEqual(false); + }); + + const v = [ + { project_id: "00000000-0000-0000-0000-000000000001", affinity: "math" }, + { + account_id: "00000000-0000-0000-0000-000000000002", + name: "cocalc", + affinity: "math", + }, + { + group_id: "00000000-0000-0000-0000-000000000003", + namespace: "test", + name: "data", + affinity: "sage", + }, + ]; + it("creates 3 filesystems in 2 different namespaces, and confirms sync works", async () => { + setContext({ prefix: prefix1 }); + for (const fs of v) { + await createFilesystem(fs); + } + // write files to fs2 and fs3, so data will get sync'd too + await writeFile(join(filesystemMountpoint(v[1]), "a.txt"), "hello"); + await writeFile(join(filesystemMountpoint(v[2]), "b.txt"), "cocalc"); + // snapshot + await createSnapshot({ ...v[1], force: true }); + await createSnapshot({ ...v[2], force: true }); + const p = v.map((x) => get(x)); + + // do the sync + setContext({ prefix: prefix2 }); + await pull({ remote, prefix: prefix1 }); + + // verify that we have everything + for (const fs of v) { + expect(await filesystemExists(fs)).toEqual(true); + } + const p2 = v.map((x) => get(x)); + for (let i = 0; i < p.length; i++) { + // everything matches (even snapshots, since no trimming happened) + for (const field of SYNCED_FIELDS) { + expect({ i, field, value: p[i][field] }).toEqual({ + i, + field, + value: p2[i][field], + }); + } + } + }); + + it("edits some files on one of the above filesystems, snapshots, sync's, goes back and deletes a snapshot, edits more files, sync's, and notices that snapshots on sync target properly match snapshots on source.", async () => { + // edits some files on one of the above filesystems, snapshots: + setContext({ prefix: prefix1 }); + await writeFile(join(filesystemMountpoint(v[1]), "a2.txt"), "hello2"); + await createSnapshot({ ...v[1], force: true }); + + // sync's + setContext({ prefix: prefix2 }); + await pull({ remote, prefix: prefix1 }); + + // delete snapshot + setContext({ prefix: prefix1 }); + const fs1 = get(v[1]); + await deleteSnapshot({ ...v[1], snapshot: fs1.snapshots[0] }); + + // do more edits and make another snapshot + await writeFile(join(filesystemMountpoint(v[1]), "a3.txt"), "hello3"); + await createSnapshot({ ...v[1], force: true }); + const snapshots1 = get(v[1]).snapshots; + + // sync + setContext({ prefix: prefix2 }); + await pull({ remote, prefix: prefix1 }); + + // snapshots do NOT initially match, since we didn't enable snapshot deleting! + let snapshots2 = get(v[1]).snapshots; + expect(snapshots1).not.toEqual(snapshots2); + + await pull({ remote, prefix: prefix1, deleteSnapshots: true }); + // now snapshots should match exactly! + snapshots2 = get(v[1]).snapshots; + expect(snapshots1).toEqual(snapshots2); + }); + + it("test directly pulling one filesystem, rather than doing a full sync", async () => { + setContext({ prefix: prefix1 }); + await writeFile(join(filesystemMountpoint(v[1]), "a3.txt"), "hello2"); + await createSnapshot({ ...v[1], force: true }); + await writeFile(join(filesystemMountpoint(v[2]), "a4.txt"), "hello"); + await createSnapshot({ ...v[2], force: true }); + const p = v.map((x) => get(x)); + + setContext({ prefix: prefix2 }); + await pull({ remote, prefix: prefix1, filesystem: v[1] }); + const p2 = v.map((x) => get(x)); + + // now filesystem 1 should match, but not filesystem 2 + expect(p[1].snapshots).toEqual(p2[1].snapshots); + expect(p[2].snapshots).not.toEqual(p2[2].snapshots); + + // finally a full sync will get filesystem 2 + await pull({ remote, prefix: prefix1 }); + const p2b = v.map((x) => get(x)); + expect(p[2].snapshots).toEqual(p2b[2].snapshots); + }); +}); diff --git a/src/packages/file-server/zfs/test/util.ts b/src/packages/file-server/zfs/test/util.ts new file mode 100644 index 0000000000..36292bf0ce --- /dev/null +++ b/src/packages/file-server/zfs/test/util.ts @@ -0,0 +1,119 @@ +// application/typescript text +import { context, setContext } from "@cocalc/file-server/zfs/config"; +import { mkdtemp, rm } from "fs/promises"; +import { tmpdir } from "os"; +import { join } from "path"; +import { executeCode } from "@cocalc/backend/execute-code"; +import { initDataDir } from "@cocalc/file-server/zfs/util"; +import { resetDb } from "@cocalc/file-server/zfs/db"; +import { getPools } from "@cocalc/file-server/zfs/pools"; +import { execSync } from "child_process"; +import { map as asyncMap } from "awaiting"; + +// export "describe" from here that is a no-op if the zpool +// command is not available: +function isZpoolAvailable() { + try { + execSync("which zpool", { stdio: "ignore" }); + return true; + } catch { + return false; + } +} +const Describe = isZpoolAvailable() ? describe : describe.skip; +export { Describe as describe }; + +export async function init() { + if (!context.PREFIX.includes("test")) { + throw Error("context.PREFIX must contain 'test'"); + } + await initDataDir(); + resetDb(); +} + +export async function createTestPools({ + size = "10G", + count = 1, + prefix, +}: { + size?: string; + count?: number; + prefix?: string; +}): Promise<{ tempDir: string; pools: string[]; prefix?: string }> { + setContext({ prefix }); + if (!context.PREFIX.includes("test")) { + throw Error(`context.PREFIX=${context.PREFIX} must contain 'test'`); + } + // Create temp directory + const tempDir = await mkdtemp(join(tmpdir(), "test-")); + const pools: string[] = []; + // in case pools left from a failing test: + for (const pool of Object.keys(await getPools())) { + try { + await executeCode({ + command: "sudo", + args: ["zpool", "destroy", pool], + }); + } catch {} + } + for (let n = 0; n < count; n++) { + const image = join(tempDir, `${n}`, "0.img"); + await executeCode({ + command: "mkdir", + args: [join(tempDir, `${n}`)], + }); + await executeCode({ + command: "truncate", + args: ["-s", size, image], + }); + const pool = `${context.PREFIX}-${n}`; + pools.push(pool); + await executeCode({ + command: "sudo", + args: ["zpool", "create", pool, image], + }); + } + // ensure pool cache is cleared: + await getPools({ noCache: true }); + return { tempDir, pools, prefix }; +} + +// Even after setting sharefnfs=off, it can be a while (a minute?) until NFS +// fully frees up the share so we can destroy the pool. This makes it instant, +// which is very useful for unit testing. +export async function restartNfsServer() { + await executeCode({ + command: "sudo", + args: ["service", "nfs-kernel-server", "restart"], + }); +} + +export async function deleteTestPools(x?: { + tempDir: string; + pools: string[]; + prefix?: string; +}) { + if (!x) { + return; + } + const { tempDir, pools, prefix } = x; + setContext({ prefix }); + if (!context.PREFIX.includes("test")) { + throw Error("context.PREFIX must contain 'test'"); + } + + const f = async (pool) => { + try { + await executeCode({ + command: "sudo", + args: ["zpool", "destroy", pool], + }); + } catch (err) { +// if (!`$err}`.includes("no such pool")) { +// console.log(err); +// } + } + }; + await asyncMap(pools, pools.length, f); + await rm(tempDir, { recursive: true }); +} diff --git a/src/packages/file-server/zfs/types.ts b/src/packages/file-server/zfs/types.ts new file mode 100644 index 0000000000..d4c5e44526 --- /dev/null +++ b/src/packages/file-server/zfs/types.ts @@ -0,0 +1,195 @@ +import { context } from "./config"; +import { isValidUUID } from "@cocalc/util/misc"; + +export const OWNER_TYPES = ["account", "project", "group"] as const; + +export const OWNER_ID_FIELDS = OWNER_TYPES.map((x) => x + "_id"); + +export type OwnerType = (typeof OWNER_TYPES)[number]; + +export interface FilesystemPrimaryKey { + // The primary key (namespace, owner_type, owner_id, name): + + namespace: string; + // we support two types of filesystems: + // - 'project': owned by one project and can be used in various context associated with a single project; + // useful to all collaborators on a project. + // - 'account': owned by a user (a person) and be used in various ways on all projects they collaborator on. + // Other than the above distinction, the filesystems are treated identically by the server. + owner_type: OwnerType; + // owner_id is either a project_id or an account_id or an group_id. + owner_id: string; + // The name of the filesystem. + name: string; +} + +// This isn't exactly a FilesystemPrimaryKey, but it is convenient to +// work with and it uniquely *defines* one (or throws an error), after +// being fed through the primaryKey function below. +export interface PrimaryKey { + namespace?: string; + owner_type?: OwnerType; + owner_id?: string; + name?: string; + account_id?: string; + project_id?: string; + group_id?: string; +} + +const zfsSegmentRegex = /^[a-zA-Z0-9 _\-.:]+$/; + +export function primaryKey({ + namespace = context.namespace, + owner_type, + owner_id, + name, + account_id, + project_id, + group_id, +}: PrimaryKey): FilesystemPrimaryKey { + if ( + (account_id ? 1 : 0) + + (project_id ? 1 : 0) + + (group_id ? 1 : 0) + + (owner_type ? 1 : 0) != + 1 + ) { + throw Error( + `exactly one of account_id, project_id, group_id, or owner_type must be specified: ${JSON.stringify({ account_id, project_id, group_id, owner_type })}`, + ); + } + if ( + (account_id ? 1 : 0) + + (project_id ? 1 : 0) + + (group_id ? 1 : 0) + + (owner_id ? 1 : 0) != + 1 + ) { + throw Error( + `exactly one of account_id, project_id, group_id, or owner_type must be specified: ${JSON.stringify({ account_id, project_id, group_id, owner_id })}`, + ); + } + if (account_id) { + owner_type = "account"; + owner_id = account_id; + } else if (project_id) { + owner_type = "project"; + owner_id = project_id; + } else if (group_id) { + owner_type = "group"; + owner_id = group_id; + } + if (!owner_type || !OWNER_TYPES.includes(owner_type)) { + throw Error( + `unknown owner type '${owner_type}' -- must be one of ${JSON.stringify(OWNER_TYPES)}`, + ); + } + if (!name) { + if (owner_type == "project" && name == null) { + // the home directory of a project. + name = "home"; + } else { + throw Error("name must be nonempty"); + } + } + if (name.length >= 64) { + // this is only so mounting is reasonable on the filesystem... and could be lifted + throw Error("name must be at most 63 characters"); + } + if (!zfsSegmentRegex.test(name)) { + throw Error( + 'name must only contain alphanumeric characters, space, *, "-", "_", "." and ":"', + ); + } + + if (!isValidUUID(owner_id) || !owner_id) { + throw Error("owner_id must be a valid uuid"); + } + + return { namespace, owner_type, owner_id, name }; +} + +export interface Filesystem extends FilesystemPrimaryKey { + // Properties of the filesystem and its current state: + + // the pool is where the filesystem happened to get allocated. This can be influenced by affinity or usage. + pool: string; + // true if project is currently archived + archived: boolean; + // array of hosts (or range using CIDR notation) that we're + // granting NFS client access to. + nfs: string[]; + // list of snapshots as ISO timestamps from oldest to newest + snapshots: string[]; + // name of the most recent snapshot that was used for sending a stream + // (for incremental backups). This specified snapshot will never be + // deleted by the snapshot trimming process, until a newer send snapshot is made. + last_send_snapshot?: string; + // name of most recent bup backup + last_bup_backup?: string; + // Last_edited = last time this project was "edited" -- various + // operations cause this to get updated. + last_edited?: Date; + // optional arbitrary affinity string - we attempt if possible to put + // projects with the same affinity in the same pool, to improve chances of dedup. + affinity?: string; + // if this is set, then some sort of error that "should" never happen, + // has happened, and manual intervention is needed. + error?: string; + // when the last error actually happened + last_error?: Date; + + // Bytes used by the main project filesystem dataset, NOT counting snapshots (zfs "USED"). + // Obviously these used_by fields are NOT always up to date. They get updated on some + // standard operations, including making snapshots, so can be pretty useful for monitoring + // for issues, etc. + used_by_dataset?: number; + // Total amount of space used by snapshots (not the filesystem) + used_by_snapshots?: number; + + // Quota for dataset usage (in bytes), so used_by_dataset <= dataset_quota. This is the refquota property in ZFS. + quota?: number; +} + +// Used for set(...), main thing being each field can be FilesystemFieldFunction, +// which makes it very easy to *safely* mutate data (assuming only one process +// is using sqlite). +type FilesystemFieldFunction = (project: Filesystem) => any; +export interface SetFilesystem extends PrimaryKey { + pool?: string | FilesystemFieldFunction; + archived?: boolean | FilesystemFieldFunction; + nfs?: string[] | FilesystemFieldFunction; + snapshots?: string[] | FilesystemFieldFunction; + last_send_snapshot?: string | FilesystemFieldFunction; + last_bup_backup?: string | FilesystemFieldFunction; + last_edited?: Date | FilesystemFieldFunction; + affinity?: null | string | FilesystemFieldFunction; + error?: null | string | FilesystemFieldFunction; + last_error?: Date | FilesystemFieldFunction; + used_by_dataset?: null | number; + used_by_snapshots?: null | number; + quota?: null | number; +} + +// what is *actually* stored in sqlite +export interface RawFilesystem { + owner_type: OwnerType; + owner_id: string; + namespace: string; + pool: string; + // 0 or 1 + archived?: number; + // nfs and snasphots are v.join(',') + nfs?: string; + snapshots?: string; + last_send_snapshot?: string; + last_bup_backup?: string; + // new Date().ISOString() + last_edited?: string; + affinity?: string; + error?: string; + last_error?: string; + used_by_dataset?: number; + used_by_snapshots?: number; + quota?: number; +} diff --git a/src/packages/file-server/zfs/util.ts b/src/packages/file-server/zfs/util.ts new file mode 100644 index 0000000000..6f60893fd6 --- /dev/null +++ b/src/packages/file-server/zfs/util.ts @@ -0,0 +1,29 @@ +import { executeCode } from "@cocalc/backend/execute-code"; +import { context, DEFAULT_EXEC_TIMEOUT_MS } from "./config"; +import { fatalError } from "./db"; + +export async function exec(opts) { + try { + return await executeCode({ + ...opts, + timeout: DEFAULT_EXEC_TIMEOUT_MS / 1000, + }); + } catch (err) { + if (opts.what) { + fatalError({ + ...opts.what, + err, + desc: `${opts.desc ? opts.desc : ""} "${opts.command} ${opts.args?.join(" ") ?? ""}"`, + }); + } + throw err; + } +} + +export async function initDataDir() { + await executeCode({ command: "sudo", args: ["mkdir", "-p", context.DATA] }); + await executeCode({ + command: "sudo", + args: ["chmod", "a+rxw", context.DATA], + }); +} diff --git a/src/packages/frontend/account/account-page.tsx b/src/packages/frontend/account/account-page.tsx index 4a0427ee54..6d649d944c 100644 --- a/src/packages/frontend/account/account-page.tsx +++ b/src/packages/frontend/account/account-page.tsx @@ -15,7 +15,6 @@ and configuration. import { Flex, Menu, Space } from "antd"; import { useEffect } from "react"; import { useIntl } from "react-intl"; - import { SignOut } from "@cocalc/frontend/account/sign-out"; import { React, diff --git a/src/packages/frontend/account/table-error.tsx b/src/packages/frontend/account/table-error.tsx index f7dacbd491..9aa982e051 100644 --- a/src/packages/frontend/account/table-error.tsx +++ b/src/packages/frontend/account/table-error.tsx @@ -3,8 +3,8 @@ Show an error if something goes wrong trying to save the account settings table to the database. */ -import { Alert } from "antd"; -import { useTypedRedux } from "../app-framework"; +import ShowError from "@cocalc/frontend/components/error"; +import { redux, useTypedRedux } from "@cocalc/frontend/app-framework"; export default function AccountTableError() { const tableError = useTypedRedux("account", "tableError"); @@ -22,30 +22,29 @@ export default function AccountTableError() { } let description; - if (obj["name"] != null) { + if (obj?.["name"] != null) { // Issue trying to set the username. description = "Please try a different username. Names can be between 1 and 39 characters, contain upper and lower case letters, numbers, and dashes."; } else { - description = ( - <> - There was an error trying to save an account setting to the server. In - particular, the following change failed: - 
-          {JSON.stringify(obj, undefined, 2)}
-
- Try changing the relevant field below. - - ); + description = ` +There was an error trying to save an account setting to the server. In +particular, the following change failed: + +\`\`\`js +${JSON.stringify(obj, undefined, 2)} +\`\`\` +`; } return (
- + redux.getActions("account").setState({ tableError: undefined }) + } style={{ margin: "15px auto", maxWidth: "900px" }} - message={{error}} - description={description} - type="error" />
); diff --git a/src/packages/frontend/admin/json-editor.tsx b/src/packages/frontend/admin/json-editor.tsx index 5aa9760629..dd8a4084db 100644 --- a/src/packages/frontend/admin/json-editor.tsx +++ b/src/packages/frontend/admin/json-editor.tsx @@ -40,7 +40,7 @@ export const JsonEditor: React.FC = (props: Props) => { setError(""); if (save) onSave(oneLine); } catch (err) { - setError(err.message); + setError(`${err}`); } } @@ -48,7 +48,7 @@ export const JsonEditor: React.FC = (props: Props) => { try { setEditing(JSON.stringify(jsonic(editing), null, 2)); } catch (err) { - setError(err.message); + setError(`${err}`); } } diff --git a/src/packages/frontend/admin/site-settings/index.tsx b/src/packages/frontend/admin/site-settings/index.tsx index 4520698f33..babe4d562e 100644 --- a/src/packages/frontend/admin/site-settings/index.tsx +++ b/src/packages/frontend/admin/site-settings/index.tsx @@ -32,6 +32,7 @@ import { toCustomOpenAIModel, toOllamaModel, } from "@cocalc/util/db-schema/llm-utils"; +import ShowError from "@cocalc/frontend/components/error"; const { CheckableTag } = AntdTag; @@ -461,16 +462,11 @@ export default function SiteSettings({ close }) { }} > - {error && ( - setError("")} - style={{ margin: "30px auto", maxWidth: "800px" }} - /> - )} + diff --git a/src/packages/frontend/app-framework/Table.ts b/src/packages/frontend/app-framework/Table.ts index 306f17848a..caaeab2437 100644 --- a/src/packages/frontend/app-framework/Table.ts +++ b/src/packages/frontend/app-framework/Table.ts @@ -48,7 +48,7 @@ export abstract class Table { } this._table.on("error", (error) => { - console.warn(`Synctable error (table='${name}'): ${error}`); + console.warn(`Synctable error (table='${name}'):`, error); }); if (this._change != null) { diff --git a/src/packages/frontend/app-framework/index.ts b/src/packages/frontend/app-framework/index.ts index 579e65db73..fb1d04e29a 100644 --- a/src/packages/frontend/app-framework/index.ts +++ b/src/packages/frontend/app-framework/index.ts @@ -219,25 +219,36 @@ export class AppRedux extends AppReduxBase { // getEditorActions but for whatever editor -- this is mainly meant to be used // from the console when debugging, e.g., smc.redux.currentEditorActions() - public currentEditor(): { - actions: Actions | undefined; - store: Store | undefined; - } { + public currentEditor = (): { + project_id?: string; + path?: string; + account_id?: string; + actions?: Actions; + store?: Store; + } => { const project_id = this.getStore("page").get("active_top_tab"); + const current: { + project_id?: string; + path?: string; + account_id?: string; + actions?: Actions; + store?: Store; + } = { account_id: this.getStore("account")?.get("account_id") }; if (!is_valid_uuid_string(project_id)) { - return { actions: undefined, store: undefined }; + return current; } + current.project_id = project_id; const store = this.getProjectStore(project_id); const tab = store.get("active_project_tab"); if (!tab.startsWith("editor-")) { - return { actions: undefined, store: undefined }; + return current; } const path = tab.slice("editor-".length); - return { - actions: this.getEditorActions(project_id, path), - store: this.getEditorStore(project_id, path), - }; - } + current.path = path; + current.actions = this.getEditorActions(project_id, path); + current.store = this.getEditorStore(project_id, path); + return current; + }; } const computed = (rtype) => { diff --git a/src/packages/frontend/app/connection-indicator.tsx b/src/packages/frontend/app/connection-indicator.tsx index 35a1f4d1f0..5a8360d374 100644 --- a/src/packages/frontend/app/connection-indicator.tsx +++ b/src/packages/frontend/app/connection-indicator.tsx @@ -21,6 +21,7 @@ import { TOP_BAR_ELEMENT_CLASS, } from "./top-nav-consts"; import { blur_active_element } from "./util"; +import type { ConnectionStatus } from "./store"; interface Props { height: number; // px @@ -40,9 +41,12 @@ export const ConnectionIndicator: React.FC = React.memo( const { topPaddingIcons, sidePaddingIcons, fontSizeIcons } = pageStyle; const intl = useIntl(); - const connection_status = useTypedRedux("page", "connection_status"); + const hub_status = useTypedRedux("page", "connection_status"); const mesg_info = useTypedRedux("account", "mesg_info"); const actions = useActions("page"); + const nats = useTypedRedux("page", "nats"); + const nats_status = nats?.get("state") ?? "disconnected"; + const connection_status = worst(hub_status, nats_status); const connecting_style: CSS = { flex: "1", @@ -118,3 +122,15 @@ export const ConnectionIndicator: React.FC = React.memo( ); }, ); + +function worst(a?: ConnectionStatus, b?: ConnectionStatus): ConnectionStatus { + if (a == null || b == null) { + return "disconnected"; + } + for (const x of ["disconnected", "connecting", "connected"]) { + if (a == x || b == x) { + return x; + } + } + return a; +} diff --git a/src/packages/frontend/app/connection-info.tsx b/src/packages/frontend/app/connection-info.tsx index 46c0e0fa3d..9492b091db 100644 --- a/src/packages/frontend/app/connection-info.tsx +++ b/src/packages/frontend/app/connection-info.tsx @@ -25,6 +25,7 @@ export const ConnectionInfo: React.FC = React.memo(() => { const status = useTypedRedux("page", "connection_status"); const hub = useTypedRedux("account", "hub"); const page_actions = useActions("page"); + const nats = useTypedRedux("page", "nats"); function close() { page_actions.show_connection(false); @@ -69,12 +70,27 @@ export const ConnectionInfo: React.FC = React.memo(() => { ) : undefined} + + +

NATS.io client

+ + {nats != null && ( + + 
+                {JSON.stringify(nats.toJS(), undefined, 2)
+                  .replace(/{|}|,|\"/g, "")
+                  .trim()
+                  .replace("  data:", "data:")}
+
+ + )} +

@@ -91,7 +107,7 @@ export const ConnectionInfo: React.FC = React.memo(() => { -

{intl.formatMessage(labels.message_plural, { num: 10 })}

+

Hub {intl.formatMessage(labels.message_plural, { num: 10 })}

diff --git a/src/packages/frontend/app/monitor-pings.ts b/src/packages/frontend/app/monitor-pings.ts index 5147545102..6140dad46c 100644 --- a/src/packages/frontend/app/monitor-pings.ts +++ b/src/packages/frontend/app/monitor-pings.ts @@ -8,21 +8,8 @@ import { redux } from "../app-framework"; import { webapp_client } from "../webapp-client"; -import * as prom_client from "../prom-client"; export function init_ping(): void { - let prom_ping_time: any = undefined, - prom_ping_time_last: any = undefined; - if (prom_client.enabled) { - prom_ping_time = prom_client.new_histogram("ping_ms", "ping time", { - buckets: [50, 100, 150, 200, 300, 500, 1000, 2000, 5000], - }); - prom_ping_time_last = prom_client.new_gauge( - "ping_last_ms", - "last reported ping time" - ); - } - webapp_client.on("ping", (ping_time: number): void => { let ping_time_smooth = redux.getStore("page").get("avgping") ?? ping_time; @@ -34,10 +21,5 @@ export function init_ping(): void { ping_time_smooth = decay * ping_time_smooth + (1 - decay) * ping_time; } redux.getActions("page").set_ping(ping_time, Math.round(ping_time_smooth)); - - if (prom_client.enabled) { - prom_ping_time?.observe(ping_time); - prom_ping_time_last?.set(ping_time); - } }); } diff --git a/src/packages/frontend/app/store.ts b/src/packages/frontend/app/store.ts index 900d143544..3b3a36d661 100644 --- a/src/packages/frontend/app/store.ts +++ b/src/packages/frontend/app/store.ts @@ -52,6 +52,15 @@ export interface PageState { }; settingsModal?: string; + nats?: TypedMap<{ + state: ConnectionStatus; + data: { + inBytes?: number; + inMsgs?: number; + outBytes?: number; + outMsgs?: number; + }; + }>; } export class PageStore extends Store {} diff --git a/src/packages/frontend/billing/actions.ts b/src/packages/frontend/billing/actions.ts index dc8cd42b44..2751c309e8 100644 --- a/src/packages/frontend/billing/actions.ts +++ b/src/packages/frontend/billing/actions.ts @@ -3,6 +3,9 @@ * License: MS-RSL – see LICENSE.md for details */ +// COMPLETEY DEPRECATED -- DELETE THIS ? + + /* Billing actions. diff --git a/src/packages/frontend/chat/register.ts b/src/packages/frontend/chat/register.ts index 59cd8a12d8..7ff96d84ca 100644 --- a/src/packages/frontend/chat/register.ts +++ b/src/packages/frontend/chat/register.ts @@ -23,7 +23,7 @@ export function initChat(project_id: string, path: string): ChatActions { if (startswith(path_split(path).tail, ".")) { // Sidechat being opened -- ensure chat isn't marked as deleted: - redux.getProjectStore(project_id)?.get_listings()?.undelete(path); + redux.getProjectActions(project_id)?.setNotDeleted(path); } const syncdb = webapp_client.sync_client.sync_db({ diff --git a/src/packages/frontend/client/account.ts b/src/packages/frontend/client/account.ts index ae7af7dd84..5f8bc43b25 100644 --- a/src/packages/frontend/client/account.ts +++ b/src/packages/frontend/client/account.ts @@ -173,24 +173,6 @@ export class AccountClient { ); } - // legacy api: getting, setting, deleting, etc., the api key for this account - public async api_key( - action: "get" | "delete" | "regenerate", - password: string, - ): Promise { - if (this.client.account_id == null) { - throw Error("must be logged in"); - } - return ( - await this.call( - message.api_key({ - action, - password, - }), - ) - ).api_key; - } - // new interface: getting, setting, editing, deleting, etc., the api keys for a project public async api_keys(opts: { action: "get" | "delete" | "create" | "edit"; @@ -199,13 +181,6 @@ export class AccountClient { id?: number; expire?: Date; }): Promise { - if (this.client.account_id == null) { - throw Error("must be logged in"); - } - // because message always uses id, so we have to use something else! - const opts2: any = { ...opts }; - delete opts2.id; - opts2.key_id = opts.id; - return (await this.call(message.api_keys(opts2))).response; + return await this.client.nats_client.hub.system.manageApiKeys(opts); } } diff --git a/src/packages/frontend/client/admin.ts b/src/packages/frontend/client/admin.ts index 6deb16ab77..00a30d7315 100644 --- a/src/packages/frontend/client/admin.ts +++ b/src/packages/frontend/client/admin.ts @@ -4,19 +4,19 @@ */ import * as message from "@cocalc/util/message"; -import { AsyncCall } from "./client"; +import type { WebappClient } from "./client"; import api from "./api"; export class AdminClient { - private async_call: AsyncCall; + private client: WebappClient; - constructor(async_call: AsyncCall) { - this.async_call = async_call; + constructor(client: WebappClient) { + this.client = client; } public async admin_reset_password(email_address: string): Promise { return ( - await this.async_call({ + await this.client.async_call({ message: message.admin_reset_password({ email_address, }), @@ -36,12 +36,9 @@ export class AdminClient { } } - public async get_user_auth_token(account_id: string): Promise { - return ( - await this.async_call({ - message: message.user_auth({ account_id, password: "" }), - allow_post: false, - }) - ).auth_token; + public async get_user_auth_token(user_account_id: string): Promise { + return await this.client.nats_client.hub.system.generateUserAuthToken({ + user_account_id, + }); } } diff --git a/src/packages/frontend/client/client.ts b/src/packages/frontend/client/client.ts index 061e5e685b..d0ad03ceb8 100644 --- a/src/packages/frontend/client/client.ts +++ b/src/packages/frontend/client/client.ts @@ -8,7 +8,6 @@ import { delay } from "awaiting"; import { alert_message } from "../alerts"; import { StripeClient } from "./stripe"; import { ProjectCollaborators } from "./project-collaborators"; -import { SupportTickets } from "./support"; import { Messages } from "./messages"; import { QueryClient } from "./query"; import { TimeClient } from "./time"; @@ -22,13 +21,25 @@ import { SyncClient } from "@cocalc/sync/client/sync-client"; import { UsersClient } from "./users"; import { FileClient } from "./file"; import { TrackingClient } from "./tracking"; +import { NatsClient } from "@cocalc/frontend/nats/client"; import { HubClient } from "./hub"; import { IdleClient } from "./idle"; import { version } from "@cocalc/util/smc-version"; -import { start_metrics } from "../prom-client"; import { setup_global_cocalc } from "./console"; import { Query } from "@cocalc/sync/table"; import debug from "debug"; +import Cookies from "js-cookie"; +import { basePathCookieName } from "@cocalc/util/misc"; +import { ACCOUNT_ID_COOKIE_NAME } from "@cocalc/util/db-schema/accounts"; +import { appBasePath } from "@cocalc/frontend/customize/app-base-path"; +import type { NatsSyncTableFunction } from "@cocalc/nats/sync/synctable"; +import type { + CallNatsServiceFunction, + CreateNatsServiceFunction, +} from "@cocalc/nats/service"; +import type { NatsEnvFunction } from "@cocalc/nats/types"; +import { setNatsClient } from "@cocalc/nats/client"; +import { randomId } from "@cocalc/nats/names"; // This DEBUG variable comes from webpack: declare const DEBUG; @@ -42,14 +53,20 @@ const log = debug("cocalc"); // all the sync activity logging and everything that calls // client.dbg. +const ACCOUNT_ID_COOKIE = decodeURIComponent( + basePathCookieName({ + basePath: appBasePath, + name: ACCOUNT_ID_COOKIE_NAME, + }), +); + export type AsyncCall = (opts: object) => Promise; export interface WebappClient extends EventEmitter { account_id?: string; - + browser_id: string; stripe: StripeClient; project_collaborators: ProjectCollaborators; - support_tickets: SupportTickets; messages: Messages; query_client: QueryClient; time_client: TimeClient; @@ -63,6 +80,7 @@ export interface WebappClient extends EventEmitter { users_client: UsersClient; file_client: FileClient; tracking_client: TrackingClient; + nats_client: NatsClient; hub_client: HubClient; idle_client: IdleClient; client: Client; @@ -74,6 +92,11 @@ export interface WebappClient extends EventEmitter { get_username: Function; is_signed_in: () => boolean; synctable_project: Function; + synctable_nats: NatsSyncTableFunction; + callNatsService: CallNatsServiceFunction; + createNatsService: CreateNatsServiceFunction; + getNatsEnv: NatsEnvFunction; + pubsub_nats: Function; project_websocket: Function; prettier: Function; exec: Function; @@ -125,10 +148,10 @@ Connection events: */ class Client extends EventEmitter implements WebappClient { - account_id?: string; + account_id: string = Cookies.get(ACCOUNT_ID_COOKIE); + browser_id: string = randomId(); stripe: StripeClient; project_collaborators: ProjectCollaborators; - support_tickets: SupportTickets; messages: Messages; query_client: QueryClient; time_client: TimeClient; @@ -142,6 +165,7 @@ class Client extends EventEmitter implements WebappClient { users_client: UsersClient; file_client: FileClient; tracking_client: TrackingClient; + nats_client: NatsClient; hub_client: HubClient; idle_client: IdleClient; client: Client; @@ -154,6 +178,11 @@ class Client extends EventEmitter implements WebappClient { get_username: Function; is_signed_in: () => boolean; synctable_project: Function; + synctable_nats: NatsSyncTableFunction; + callNatsService: CallNatsServiceFunction; + createNatsService: CreateNatsServiceFunction; + getNatsEnv: NatsEnvFunction; + pubsub_nats: Function; project_websocket: Function; prettier: Function; exec: Function; @@ -185,7 +214,6 @@ class Client extends EventEmitter implements WebappClient { constructor() { super(); - if (DEBUG) { this.dbg = this.dbg.bind(this); } else { @@ -193,7 +221,6 @@ class Client extends EventEmitter implements WebappClient { return (..._) => {}; }; } - this.hub_client = bind_methods(new HubClient(this)); this.is_signed_in = this.hub_client.is_signed_in.bind(this.hub_client); this.is_connected = this.hub_client.is_connected.bind(this.hub_client); @@ -205,9 +232,6 @@ class Client extends EventEmitter implements WebappClient { this.project_collaborators = bind_methods( new ProjectCollaborators(this.async_call.bind(this)), ); - this.support_tickets = bind_methods( - new SupportTickets(this.async_call.bind(this)), - ); this.messages = new Messages(); this.query_client = bind_methods(new QueryClient(this)); this.time_client = bind_methods(new TimeClient(this)); @@ -218,19 +242,15 @@ class Client extends EventEmitter implements WebappClient { this.sync_string = this.sync_client.sync_string; this.sync_db = this.sync_client.sync_db; - this.admin_client = bind_methods( - new AdminClient(this.async_call.bind(this)), - ); + this.admin_client = bind_methods(new AdminClient(this)); this.openai_client = bind_methods(new LLMClient(this)); - //this.purchases_client = bind_methods(new PurchasesClient(this)); - this.purchases_client = bind_methods(new PurchasesClient()); + this.purchases_client = bind_methods(new PurchasesClient(this)); this.jupyter_client = bind_methods( new JupyterClient(this.async_call.bind(this)), ); - this.users_client = bind_methods( - new UsersClient(this.call.bind(this), this.async_call.bind(this)), - ); + this.users_client = bind_methods(new UsersClient(this)); this.tracking_client = bind_methods(new TrackingClient(this)); + this.nats_client = bind_methods(new NatsClient(this)); this.file_client = bind_methods(new FileClient(this.async_call.bind(this))); this.idle_client = bind_methods(new IdleClient(this)); @@ -241,7 +261,9 @@ class Client extends EventEmitter implements WebappClient { this.idle_reset = this.idle_client.idle_reset.bind(this.idle_client); this.exec = this.project_client.exec.bind(this.project_client); - this.touch_project = this.project_client.touch_project.bind(this.project_client); + this.touch_project = this.project_client.touch_project.bind( + this.project_client, + ); this.ipywidgetsGetBuffer = this.project_client.ipywidgetsGetBuffer.bind( this.project_client, ); @@ -252,6 +274,11 @@ class Client extends EventEmitter implements WebappClient { this.synctable_project = this.sync_client.synctable_project.bind( this.sync_client, ); + this.synctable_nats = this.nats_client.synctable; + this.pubsub_nats = this.nats_client.pubsub; + this.callNatsService = this.nats_client.callNatsService; + this.createNatsService = this.nats_client.createNatsService; + this.getNatsEnv = this.nats_client.getEnv; this.query = this.query_client.query.bind(this.query_client); this.async_query = this.query_client.query.bind(this.query_client); @@ -284,7 +311,6 @@ class Client extends EventEmitter implements WebappClient { this.time_client.ping(); // this will ping periodically }); - this.init_prom_client(); this.init_global_cocalc(); bind_methods(this); @@ -295,10 +321,6 @@ class Client extends EventEmitter implements WebappClient { setup_global_cocalc(this); } - private init_prom_client(): void { - this.on("start_metrics", start_metrics); - } - public dbg(f): Function { if (log.enabled) { return (...args) => log(new Date().toISOString(), f, ...args); @@ -340,6 +362,26 @@ class Client extends EventEmitter implements WebappClient { public set_deleted(): void { throw Error("not implemented for frontend"); } + + touchOpenFile = async ({ + project_id, + path, + setNotDeleted, + // id + }: { + project_id: string; + path: string; + id?: number; + // if file is deleted, this explicitly undeletes it. + setNotDeleted?: boolean; + }) => { + const x = await this.nats_client.openFiles(project_id); + if (setNotDeleted) { + x.setNotDeleted(path); + } + x.touch(path); + }; } export const webapp_client = new Client(); +setNatsClient(webapp_client); diff --git a/src/packages/frontend/client/console.ts b/src/packages/frontend/client/console.ts index 40b251445c..4e54923230 100644 --- a/src/packages/frontend/client/console.ts +++ b/src/packages/frontend/client/console.ts @@ -48,10 +48,9 @@ export function setup_global_cocalc(client): void { cocalc.misc = require("@cocalc/util/misc"); cocalc.immutable = require("immutable"); cocalc.done = cocalc.misc.done; - cocalc.sha1 = require("sha1"); - cocalc.prom_client = require("../prom-client"); cocalc.schema = require("@cocalc/util/schema"); cocalc.redux = redux; + cocalc.current = redux.currentEditor; cocalc.load_eruda = load_eruda; cocalc.compute = require("@cocalc/frontend/compute/api"); console.log( diff --git a/src/packages/frontend/client/file.ts b/src/packages/frontend/client/file.ts index 2acde86322..8000a8c999 100644 --- a/src/packages/frontend/client/file.ts +++ b/src/packages/frontend/client/file.ts @@ -18,7 +18,7 @@ export class FileClient { // Currently only used for testing and development in the console. public async syncdoc_history( string_id: string, - patches?: boolean + patches?: boolean, ): Promise { return ( await this.async_call({ @@ -33,15 +33,11 @@ export class FileClient { // Returns true if the given file in the given project is currently // marked as deleted. - public is_deleted(filename: string, project_id: string): boolean { + public is_deleted(path: string, project_id: string): boolean { return !!redux .getProjectStore(project_id) - ?.get_listings() - ?.isDeleted(filename); - } - - public undelete(filename: string, project_id: string): void { - redux.getProjectStore(project_id)?.get_listings()?.undelete(filename); + ?.get("recentlyDeletedPaths") + ?.get(path); } public set_deleted(_filename, _project_id): void { @@ -67,7 +63,7 @@ export class FileClient { } public async remove_blob_ttls( - uuids: string[] // list of sha1 hashes of blobs stored in the blobstore + uuids: string[], // list of sha1 hashes of blobs stored in the blobstore ) { if (uuids.length === 0) return; await this.async_call({ diff --git a/src/packages/frontend/client/hub.ts b/src/packages/frontend/client/hub.ts index c0487b2625..2771732a4b 100644 --- a/src/packages/frontend/client/hub.ts +++ b/src/packages/frontend/client/hub.ts @@ -113,7 +113,7 @@ export class HubClient { } public send(mesg: object): void { - //console.log("send at #{misc.mswalltime()}", mesg) + console.log("send to hub", mesg); const data = to_json_socket(mesg); this.mesg_data.sent_length += data.length; this.emit_mesg_data(); @@ -225,9 +225,6 @@ export class HubClient { } break; - case "start_metrics": - this.client.emit("start_metrics", mesg.interval_s); - break; } // the call f(null, mesg) below can mutate mesg (!), so we better save the id here. diff --git a/src/packages/frontend/client/llm.ts b/src/packages/frontend/client/llm.ts index b54f35b294..8ab9fbef4a 100644 --- a/src/packages/frontend/client/llm.ts +++ b/src/packages/frontend/client/llm.ts @@ -171,34 +171,31 @@ export class LLMClient { history = truncateHistory(history, maxTokens - n, model); } // console.log("chatgpt", { input, system, history, project_id, path }); - const mesg = message.chatgpt({ - text: input, + const options = { + input, system, project_id, path, history, model, tag: `app:${tag}`, - stream: chatStream != null, - }); + }; if (chatStream == null) { - return (await this.client.async_call({ message: mesg })).text; + // not streaming + return await this.client.nats_client.llm(options); } - chatStream.once("start", () => { + chatStream.once("start", async () => { // streaming version - this.client.call({ - message: mesg, - error_event: true, - cb: (err, resp) => { - if (err) { - chatStream.error(err); - } else { - chatStream.process(resp.text); - } - }, - }); + try { + await this.client.nats_client.llm({ + ...options, + stream: chatStream.process, + }); + } catch (err) { + chatStream.error(err); + } }); return "see stream for output"; @@ -343,14 +340,14 @@ class ChatStream extends EventEmitter { super(); } - process(text?: string) { + process = (text?: string) => { // emits undefined text when done (or err below) this.emit("token", text); - } + }; - error(err) { + error = (err) => { this.emit("error", err); - } + }; } export type { ChatStream }; diff --git a/src/packages/frontend/client/project.ts b/src/packages/frontend/client/project.ts index f0ad9f5021..05da8af05c 100644 --- a/src/packages/frontend/client/project.ts +++ b/src/packages/frontend/client/project.ts @@ -69,19 +69,35 @@ export class ProjectClient { return await this.client.async_call({ message }); } - public async write_text_file(opts: { + private natsApi = (project_id: string) => { + return this.client.nats_client.projectApi({ project_id }); + }; + + public async write_text_file({ + project_id, + path, + content, + }: { project_id: string; path: string; content: string; }): Promise { - return await this.call(message.write_text_file_to_project(opts)); + await this.natsApi(project_id).system.writeTextFileToProject({ + path, + content, + }); } - public async read_text_file(opts: { + public async read_text_file({ + project_id, + path, + }: { project_id: string; // string or array of strings path: string; // string or array of strings }): Promise { - return (await this.call(message.read_text_file_from_project(opts))).content; + return await this.natsApi(project_id).system.readTextFileFromProject({ + path, + }); } // Like "read_text_file" above, except the callback @@ -90,13 +106,21 @@ export class ProjectClient { public read_file(opts: { project_id: string; // string or array of strings path: string; // string or array of strings + compute_server_id?: number; }): string { const base_path = appBasePath; if (opts.path[0] === "/") { // absolute path to the root opts.path = HOME_ROOT + opts.path; // use root symlink, which is created by start_smc } - return encode_path(join(base_path, `${opts.project_id}/raw/${opts.path}`)); + let url = join( + base_path, + `${opts.project_id}/files/${encode_path(opts.path)}`, + ); + if (opts.compute_server_id) { + url += `?id=${opts.compute_server_id}`; + } + return url; } public async copy_path_between_projects(opts: { @@ -319,13 +343,6 @@ export class ProjectClient { return { files: listing }; } - public async public_get_text_file(opts: { - project_id: string; - path: string; - }): Promise { - return (await this.call(message.public_get_text_file(opts))).data; - } - public async find_directories(opts: { project_id: string; query?: string; // see the -iwholename option to the UNIX find command. @@ -457,7 +474,7 @@ export class ProjectClient { } this.touch_throttle[project_id] = Date.now(); try { - await this.call(message.touch_project({ project_id })); + await this.client.nats_client.hub.db.touch({ project_id }); } catch (err) { // silently ignore; this happens, e.g., if you touch too frequently, // and shouldn't be fatal and break other things. @@ -502,19 +519,17 @@ export class ProjectClient { description: string; image?: string; start?: boolean; - license?: string; // "license_id1,license_id2,..." -- if given, create project with these licenses applied - noPool?: boolean; // never use pool + // "license_id1,license_id2,..." -- if given, create project with these licenses applied + license?: string; + // never use pool + noPool?: boolean; }): Promise { - const { project_id } = await this.client.async_call({ - allow_post: false, // since gets called for anonymous and cookie not yet set. - message: message.create_project(opts), - }); - + const project_id = + await this.client.nats_client.hub.projects.createProject(opts); this.client.tracking_client.user_tracking("create_project", { project_id, title: opts.title, }); - return project_id; } @@ -613,20 +628,7 @@ export class ProjectClient { id?: number; expire?: Date; }): Promise { - if (this.client.account_id == null) { - throw Error("must be logged in"); - } - if (!is_valid_uuid_string(opts.project_id)) { - throw Error("project_id must be a valid uuid"); - } - if (opts.project_id == null && !opts.password) { - throw Error("must provide password for non-project api key"); - } - // because message always uses id, so we have to use something else! - const opts2: any = { ...opts }; - delete opts2.id; - opts2.key_id = opts.id; - return (await this.call(message.api_keys(opts2))).response; + return await this.client.nats_client.hub.system.manageApiKeys(opts); } computeServers = (project_id) => { diff --git a/src/packages/frontend/client/purchases.ts b/src/packages/frontend/client/purchases.ts index 7a74f9d1cc..12e0357f8b 100644 --- a/src/packages/frontend/client/purchases.ts +++ b/src/packages/frontend/client/purchases.ts @@ -14,12 +14,15 @@ import type { ProjectQuota } from "@cocalc/util/db-schema/purchase-quotas"; import * as purchasesApi from "@cocalc/frontend/purchases/api"; import type { Changes as EditLicenseChanges } from "@cocalc/util/purchases/cost-to-edit-license"; import { round2up } from "@cocalc/util/misc"; +import type { WebappClient } from "./client"; export class PurchasesClient { api: typeof purchasesApi; + client: WebappClient; - constructor() { + constructor(client: WebappClient) { this.api = purchasesApi; + this.client = client; } async getQuotas(): Promise<{ minBalance: number; @@ -29,7 +32,7 @@ export class PurchasesClient { } async getBalance(): Promise { - return await purchasesApi.getBalance(); + return await this.client.nats_client.hub.purchases.getBalance(); } async getSpendRate(): Promise { diff --git a/src/packages/frontend/client/query.ts b/src/packages/frontend/client/query.ts index 880477012a..e79748d9d4 100644 --- a/src/packages/frontend/client/query.ts +++ b/src/packages/frontend/client/query.ts @@ -3,90 +3,104 @@ * License: MS-RSL – see LICENSE.md for details */ -import * as message from "@cocalc/util/message"; import { is_array } from "@cocalc/util/misc"; import { validate_client_query } from "@cocalc/util/schema-validate"; import { CB } from "@cocalc/util/types/database"; +import { NatsChangefeed } from "@cocalc/sync/table/changefeed-nats"; +import { uuid } from "@cocalc/util/misc"; +import { client_db } from "@cocalc/util/schema"; declare const $: any; // jQuery export class QueryClient { private client: any; + private changefeeds: { [id: string]: NatsChangefeed } = {}; constructor(client: any) { this.client = client; } - private async call(message: object, timeout: number): Promise { - return await this.client.async_call({ - message, - timeout, - allow_post: false, // since that would happen via this.post_query - }); - } - // This works like a normal async function when // opts.cb is NOT specified. When opts.cb is specified, - // it works like a cb and returns nothing. For changefeeds + // it works like a cb and returns nothing. For changefeeds // you MUST specify opts.cb, but can always optionally do so. public async query(opts: { query: object; - changes?: boolean; options?: object[]; // if given must be an array of objects, e.g., [{limit:5}] - standby?: boolean; // if true and use HTTP post, then will use standby server (so must be read only) - timeout?: number; // default: 30 - no_post?: boolean; // DEPRECATED -- didn't turn out to be worth it. - ignore_response?: boolean; // if true, be slightly efficient by not waiting for any response or - // error (just assume it worked; don't care about response) - cb?: CB; // used for changefeed outputs if changes is true + changes?: boolean; + cb?: CB; // support old cb interface }): Promise { + // Deprecation warnings: + for (const field of ["standby", "timeout", "no_post", "ignore_response"]) { + if (opts[field] != null) { + console.trace(`WARNING: passing '${field}' to query is deprecated`); + } + } if (opts.options != null && !is_array(opts.options)) { // should never happen... throw Error("options must be an array"); } - if (opts.changes && opts.cb == null) { - throw Error("for changefeed, must specify opts.cb"); - } - - const err = validate_client_query(opts.query, this.client.account_id); - if (err) { - throw Error(err); - } - const mesg = message.query({ - query: opts.query, - options: opts.options, - changes: opts.changes, - multi_response: !!opts.changes, - }); - if (opts.timeout == null) { - opts.timeout = 30; - } - if (mesg.multi_response) { - if (opts.cb == null) { - throw Error("changefeed requires cb callback"); + if (opts.changes) { + const { cb } = opts; + if (cb == null) { + throw Error("for changefeed, must specify opts.cb"); + } + let changefeed; + try { + changefeed = new NatsChangefeed({ + client: this.client, + query: opts.query, // todo: regarding options + }); + // id for canceling this changefeed + const id = uuid(); + const rows = await changefeed.connect(); + const query = { [Object.keys(opts.query)[0]]: rows }; + this.changefeeds[id] = changefeed; + cb(undefined, { query, id }); + changefeed.on("update", (change) => { + cb(undefined, change); + }); + } catch (err) { + cb(`${err}`); + return; } - this.client.call({ - allow_post: false, - message: mesg, - error_event: true, - timeout: opts.timeout, - cb: opts.cb, - }); } else { - if (opts.cb != null) { - try { - const result = await this.call(mesg, opts.timeout); - opts.cb(undefined, result); - } catch (err) { - opts.cb(typeof err == "string" ? err : err.message ?? err); + try { + const err = validate_client_query(opts.query, this.client.account_id); + if (err) { + throw Error(err); + } + const query = await this.client.nats_client.hub.db.userQuery({ + query: opts.query, + options: opts.options, + }); + + if (query && !opts.options?.[0]?.["set"]) { // set thing isn't needed but doesn't hurt + // deal with timestamp versus Date and JSON using our schema. + for (const table in query) { + client_db.processDates({ table, rows: query[table] }); + } + } + + if (opts.cb == null) { + return { query }; + } else { + opts.cb(undefined, { query }); + } + } catch (err) { + if (opts.cb == null) { + throw err; + } else { + opts.cb(err); } - } else { - return await this.call(mesg, opts.timeout); } } } + // cancel a changefeed created above. This is ONLY used + // right now by the CRM code. public async cancel(id: string): Promise { - await this.call(message.query_cancel({ id }), 30); + this.changefeeds[id]?.close(); + delete this.changefeeds[id]; } } diff --git a/src/packages/frontend/client/stripe.ts b/src/packages/frontend/client/stripe.ts index 5f44c24500..6462047088 100644 --- a/src/packages/frontend/client/stripe.ts +++ b/src/packages/frontend/client/stripe.ts @@ -3,6 +3,8 @@ * License: MS-RSL – see LICENSE.md for details */ +// COMPLETEY DEPRECATED -- DELETE THIS ? + /* stripe payments api via backend hub */ diff --git a/src/packages/frontend/client/support.ts b/src/packages/frontend/client/support.ts deleted file mode 100644 index b085a84c03..0000000000 --- a/src/packages/frontend/client/support.ts +++ /dev/null @@ -1,33 +0,0 @@ -/* - * This file is part of CoCalc: Copyright © 2020 Sagemath, Inc. - * License: MS-RSL – see LICENSE.md for details - */ - -import { replace_all } from "@cocalc/util/misc"; -import * as message from "@cocalc/util/message"; -import { AsyncCall } from "./client"; - -export class SupportTickets { - private async_call: AsyncCall; - - constructor(async_call: AsyncCall) { - this.async_call = async_call; - } - - private async call(message: object): Promise { - return await this.async_call({ message, timeout: 30 }); - } - - public async create(opts): Promise { - if (opts.body != null) { - // Make it so the session is ignored in any URL appearing in the body. - // Obviously, this is not 100% bullet proof, but should help enormously. - opts.body = replace_all(opts.body, "?session=", "?session=#"); - } - return (await this.call(message.create_support_ticket(opts))).url; - } - - public async get(): Promise { - return (await this.call(message.get_support_tickets())).tickets; - } -} diff --git a/src/packages/frontend/client/time.ts b/src/packages/frontend/client/time.ts index 2b094c53af..76e2fb4060 100644 --- a/src/packages/frontend/client/time.ts +++ b/src/packages/frontend/client/time.ts @@ -4,20 +4,18 @@ */ import { delay } from "awaiting"; - import { get_local_storage, set_local_storage, } from "@cocalc/frontend/misc/local-storage"; -import * as message from "@cocalc/util/message"; export class TimeClient { private client: any; private ping_interval_ms: number = 30000; // interval in ms between pings - private last_ping: Date = new Date(0); - private last_pong?: { server: Date; local: Date }; + private last_ping: number = 0; + private last_pong?: { server: number; local: number }; private clock_skew_ms?: number; - private last_server_time?: Date; + private last_server_time?: number; private closed: boolean = false; constructor(client: any) { @@ -29,31 +27,26 @@ export class TimeClient { } // Ping server and also use the ping to determine clock skew. - public async ping(noLoop: boolean = false): Promise { - if (this.closed) return; - const start = (this.last_ping = new Date()); + ping = async (noLoop: boolean = false): Promise => { + if (this.closed) { + return; + } + const start = (this.last_ping = Date.now()); let pong; try { - pong = await this.client.async_call({ - allow_post: false, - message: message.ping(), - timeout: 10, // CRITICAL that this timeout be less than the @_ping_interval - }); + pong = await this.client.nats_client.hub.system.ping(); } catch (err) { if (!noLoop) { // try again **sooner** - setTimeout(this.ping.bind(this), this.ping_interval_ms / 2); + setTimeout(this.ping, this.ping_interval_ms / 2); } return; } - const now = new Date(); + const now = Date.now(); // Only record something if success, got a pong, and the round trip is short! // If user messes with their clock during a ping and we don't do this, then // bad things will happen. - if ( - pong?.event == "pong" && - now.valueOf() - this.last_ping.valueOf() <= 1000 * 15 - ) { + if (now - this.last_ping <= 1000 * 15) { if (pong.now == null) { console.warn("pong must have a now field"); } else { @@ -61,20 +54,20 @@ export class TimeClient { // See the function server_time below; subtract this.clock_skew_ms from local // time to get a better estimate for server time. this.clock_skew_ms = - this.last_ping.valueOf() + - (this.last_pong.local.valueOf() - this.last_ping.valueOf()) / 2 - - this.last_pong.server.valueOf(); + this.last_ping + + (this.last_pong.local - this.last_ping) / 2 - + this.last_pong.server; set_local_storage("clock_skew", `${this.clock_skew_ms}`); } } - this.emit_latency(now.valueOf() - start.valueOf()); + this.emit_latency(now - start); if (!noLoop) { // periodically ping the server, to ensure clocks stay in sync. - setTimeout(this.ping.bind(this), this.ping_interval_ms); + setTimeout(this.ping, this.ping_interval_ms); } - } + }; private emit_latency(latency: number) { if (!window.document.hasFocus()) { @@ -110,11 +103,11 @@ export class TimeClient { const last = this.last_server_time; if (last != null && last >= t) { // That's annoying -- time is not marching forward... let's fake it until it does. - t = new Date(last.valueOf() + 1); + t = last + 1; } if ( this.last_pong != null && - Date.now() - this.last_pong.local.valueOf() < 5 * this.ping_interval_ms + Date.now() - this.last_pong.local < 5 * this.ping_interval_ms ) { // We have synced the clock **recently successfully**, so // we now ensure the time is increasing. @@ -125,10 +118,10 @@ export class TimeClient { } else { delete this.last_server_time; } - return t; + return new Date(t); } - private unskewed_server_time(): Date { + private unskewed_server_time(): number { // Add clock_skew_ms to our local time to get a better estimate of the actual time on the server. // This can help compensate in case the user's clock is wildly wrong, e.g., by several minutes, // or even hours due to totally wrong time (e.g. ignoring time zone), which is relevant for @@ -141,9 +134,9 @@ export class TimeClient { } } if (this.clock_skew_ms != null) { - return new Date(Date.now() - this.clock_skew_ms); + return Date.now() - this.clock_skew_ms; } else { - return new Date(); + return Date.now(); } } @@ -152,7 +145,7 @@ export class TimeClient { timeout?: number; // any ping that takes this long in seconds is considered a fail delay_ms?: number; // wait this long between doing pings log?: Function; // if set, use this to log output - }) { + }={}) { if (opts.packets == null) opts.packets = 20; if (opts.timeout == null) opts.timeout = 5; if (opts.delay_ms == null) opts.delay_ms = 200; @@ -164,15 +157,12 @@ export class TimeClient { */ const ping_times: number[] = []; const do_ping: (i: number) => Promise = async (i) => { - const t = new Date(); + const t = Date.now(); const heading = `${i}/${opts.packets}: `; let bar, mesg, pong, ping_time; try { - pong = await this.client.async_call({ - message: message.ping(), - timeout: opts.timeout, - }); - ping_time = Date.now() - t.valueOf(); + pong = await this.client.nats_client.hub.system.ping(); + ping_time = Date.now() - t; bar = ""; for (let j = 0; j <= Math.floor(ping_time / 10); j++) { bar += "*"; diff --git a/src/packages/frontend/client/tracking.ts b/src/packages/frontend/client/tracking.ts index b3ce5bcda6..3a8dc2b13b 100644 --- a/src/packages/frontend/client/tracking.ts +++ b/src/packages/frontend/client/tracking.ts @@ -5,29 +5,29 @@ import { WebappClient } from "./client"; import * as message from "@cocalc/util/message"; +import { redux } from "@cocalc/frontend/app-framework"; export class TrackingClient { private client: WebappClient; private log_error_cache: { [error: string]: number } = {}; + private userTrackingEnabled?: string; constructor(client: WebappClient) { this.client = client; } - // Send metrics to the hub this client is connected to. - // There is no confirmation or response that this succeeded, - // which is fine, since dropping some metrics is fine. - public send_metrics(metrics: object): void { - this.client.hub_client.send(message.metrics({ metrics })); - } - - public async user_tracking(evt: string, value: object): Promise { - await this.client.async_call({ - message: message.user_tracking({ evt, value }), - }); - } + user_tracking = async (event: string, value: object): Promise => { + if (this.userTrackingEnabled == null) { + this.userTrackingEnabled = redux + .getStore("customize") + ?.get("user_tracking"); + } + if (this.userTrackingEnabled == "yes") { + await this.client.nats_client.hub.system.userTracking({ event, value }); + } + }; - public log_error(error: any): void { + log_error = (error: any): void => { if (typeof error != "string") { error = JSON.stringify(error); } @@ -39,9 +39,9 @@ export class TrackingClient { this.client.call({ message: message.log_client_error({ error }), }); - } + }; - public async webapp_error(opts: object): Promise { + webapp_error = async (opts: object): Promise => { await this.client.async_call({ message: message.webapp_error(opts) }); - } + }; } diff --git a/src/packages/frontend/client/users.ts b/src/packages/frontend/client/users.ts index 5d9bb964de..4598ef9e7e 100644 --- a/src/packages/frontend/client/users.ts +++ b/src/packages/frontend/client/users.ts @@ -3,48 +3,59 @@ * License: MS-RSL – see LICENSE.md for details */ -import { AsyncCall } from "./client"; import { User } from "../frame-editors/generic/client"; import { isChatBot, chatBotName } from "@cocalc/frontend/account/chatbot"; -import api from "./api"; import TTL from "@isaacs/ttlcache"; import { reuseInFlight } from "@cocalc/util/reuse-in-flight"; -import * as message from "@cocalc/util/message"; +import type { WebappClient } from "./client"; const nameCache = new TTL({ ttl: 60 * 1000 }); export class UsersClient { - private async_call: AsyncCall; + private client: WebappClient; - constructor(_call: Function, async_call: AsyncCall) { - this.async_call = async_call; + constructor(client) { + this.client = client; } + /* + There are two possible item types in the query list: email addresses + and strings that are not email addresses. An email query item will return + account id, first name, last name, and email address for the unique + account with that email address, if there is one. A string query item + will return account id, first name, and last name for all matching + accounts. + + We do not reveal email addresses of users queried by name to non admins. + + String query matches first and last names that start with the given string. + If a string query item consists of two strings separated by space, + the search will return accounts in which the first name begins with one + of the two strings and the last name begins with the other. + String and email queries may be mixed in the list for a single + user_search call. Searches are case-insensitive. + + Note: there is a hard limit of 50 returned items in the results, except for + admins that can search for more. + */ user_search = reuseInFlight( - async (opts: { + async ({ + query, + limit = 20, + admin, + only_email, + }: { query: string; limit?: number; - active?: string; // if given, would restrict to users active this recently admin?: boolean; // admins can do an admin version of the query, which also does substring searches on email address (not just name) only_email?: boolean; // search only via email address }): Promise => { - if (opts.limit == null) { - opts.limit = 20; - } - if (opts.active == null) { - opts.active = ""; - } - - const { results } = await this.async_call({ - message: message.user_search({ - query: opts.query, - limit: opts.limit, - admin: opts.admin, - active: opts.active, - only_email: opts.only_email, - }), + return await this.client.nats_client.hub.system.userSearch({ + query, + limit, + admin, + only_email, }); - return results; }, ); @@ -72,7 +83,11 @@ export class UsersClient { getNames = reuseInFlight(async (account_ids: string[]) => { const x: { [account_id: string]: - | { first_name: string; last_name: string } + | { + first_name: string; + last_name: string; + profile?: { color?: string; image?: string }; + } | undefined; } = {}; const v: string[] = []; @@ -84,7 +99,7 @@ export class UsersClient { } } if (v.length > 0) { - const { names } = await api("/accounts/get-names", { account_ids: v }); + const names = await this.client.nats_client.hub.system.getNames(v); for (const account_id of v) { // iterate over v to record accounts that don't exist too x[account_id] = names[account_id]; diff --git a/src/packages/frontend/codemirror/extensions/ai-formula.tsx b/src/packages/frontend/codemirror/extensions/ai-formula.tsx index b47c7ba391..45afbd14c3 100644 --- a/src/packages/frontend/codemirror/extensions/ai-formula.tsx +++ b/src/packages/frontend/codemirror/extensions/ai-formula.tsx @@ -242,7 +242,7 @@ function AiGenFormula({ mode, text = "", project_id, locale, cb }: Props) { setFullReply(""); } } catch (err) { - setError(err.message || err.toString()); + setError(`${err}`); } finally { setGenerating(false); } diff --git a/src/packages/frontend/components/api-keys.tsx b/src/packages/frontend/components/api-keys.tsx index 41870daa79..ab1d75de64 100644 --- a/src/packages/frontend/components/api-keys.tsx +++ b/src/packages/frontend/components/api-keys.tsx @@ -78,7 +78,7 @@ export default function ApiKeys({ manage, mode = "project" }: Props) { setError(null); } catch (err) { setLoading(false); - setError(err.message || "An error occurred"); + setError(`${err}`); } }; @@ -87,7 +87,7 @@ export default function ApiKeys({ manage, mode = "project" }: Props) { await manage({ action: "delete", id }); getAllApiKeys(); } catch (err) { - setError(err.message || "An error occurred"); + setError(`${err}`); } }; @@ -102,7 +102,7 @@ export default function ApiKeys({ manage, mode = "project" }: Props) { await manage({ action: "edit", id, name, expire }); getAllApiKeys(); } catch (err) { - setError(err.message || "An error occurred"); + setError(`${err}`); } }; @@ -117,6 +117,7 @@ export default function ApiKeys({ manage, mode = "project" }: Props) { getAllApiKeys(); Modal.success({ + width: 600, title: "New Secret API Key", content: ( <> @@ -137,7 +138,7 @@ export default function ApiKeys({ manage, mode = "project" }: Props) { }); setError(null); } catch (err) { - setError(err.message || "An error occurred"); + setError(`${err}`); } }; diff --git a/src/packages/frontend/compute/manager.ts b/src/packages/frontend/compute/manager.ts index 36ba37f46d..25cc99824a 100644 --- a/src/packages/frontend/compute/manager.ts +++ b/src/packages/frontend/compute/manager.ts @@ -6,159 +6,30 @@ are available and how they are used for a given project. When doing dev from the browser console, do: -cc.client.project_client.computeServers('...project_id...') + cc.client.project_client.computeServers(cc.current().project_id) */ -import { SYNCDB_PARAMS, decodeUUIDtoNum } from "@cocalc/util/compute/manager"; -import { webapp_client } from "@cocalc/frontend/webapp-client"; -import { redux } from "@cocalc/frontend/app-framework"; -import debug from "debug"; -import { once } from "@cocalc/util/async-utils"; -import { EventEmitter } from "events"; -import { excludeFromComputeServer } from "@cocalc/frontend/file-associations"; - -const log = debug("cocalc:frontend:compute:manager"); - -export class ComputeServersManager extends EventEmitter { - private sync_db; - private project_id; - - constructor(project_id: string) { - super(); - this.project_id = project_id; - this.sync_db = webapp_client.sync_db({ - project_id, - ...SYNCDB_PARAMS, - }); - this.sync_db.on("change", () => { - this.emit("change"); - }); - // It's reasonable to have many clients, e.g., one for each open file - this.setMaxListeners(100); - log("created", this.project_id); - } - - waitUntilReady = async () => { - const { sync_db } = this; - if (sync_db.get_state() == "init") { - // make sure project is running - redux.getActions("projects").start_project(this.project_id); - - // now wait for syncdb to be ready - await once(sync_db, "ready"); - } - if (sync_db.get_state() != "ready") { - throw Error("syncdb not ready"); - } - }; - - close = () => { - delete computeServerManagerCache[this.project_id]; - this.sync_db.close(); - }; - - // save the current state to the backend. This is critical to do, e.g., before - // opening a file and after calling connectComputeServerToPath, since otherwise - // the project doesn't even know that the file should open on the compute server - // until after it has opened it, which is disconcerting and not efficient (but - // does mostly work, though it is intentionally making things really hard on ourselves). - save = async () => { - await this.sync_db.save(); - }; - - getComputeServers = () => { - const servers = {}; - const cursors = this.sync_db.get_cursors({ excludeSelf: "never" }).toJS(); - for (const client_id in cursors) { - const server = cursors[client_id]; - servers[decodeUUIDtoNum(client_id)] = { - time: server.time, - ...server.locs[0], - }; - } - return servers; - }; - - // Call this if you want the compute server with given id to - // connect and handle being the server for the given path. - connectComputeServerToPath = ({ id, path }: { id: number; path: string }) => { - if (id == 0) { - this.disconnectComputeServer({ path }); - return; - } - assertSupportedPath(path); - this.sync_db.set({ id, path, open: true }); - this.sync_db.commit(); - }; - - // Call this if you want no compute servers to provide the backend server - // for given path. - disconnectComputeServer = ({ path }: { path: string }) => { - this.sync_db.delete({ path }); - this.sync_db.commit(); - }; - - // For interactive debugging -- display in the console how things are configured. - showStatus = () => { - console.log(JSON.stringify(this.sync_db.get().toJS(), undefined, 2)); - }; - - // Returns the explicitly set server id for the given - // path, if one is set. Otherwise, return undefined - // if nothing is explicitly set for this path. - getServerIdForPath = async (path: string): Promise => { - await this.waitUntilReady(); - const { sync_db } = this; - return sync_db.get_one({ path })?.get("id"); - }; - - // Get the server ids (as a map) for every file and every directory contained in path. - // NOTE/TODO: this just does a linear search through all paths with a server id; nothing clever. - getServerIdForSubtree = async ( - path: string, - ): Promise<{ [path: string]: number }> => { - const { sync_db } = this; - if (sync_db.get_state() == "init") { - await once(sync_db, "ready"); - } - if (sync_db.get_state() != "ready") { - throw Error("syncdb not ready"); - } - const x = sync_db.get(); - const v: { [path: string]: number } = {}; - if (x == null) { - return v; - } - const slash = path.endsWith("/") ? path : path + "/"; - for (const y of x) { - const p = y.get("path"); - if (p == path || p.startsWith(slash)) { - v[p] = y.get("id"); - } - } - return v; - }; -} - -function assertSupportedPath(path: string) { - if (excludeFromComputeServer(path)) { - throw Error( - `Opening '${path}' on a compute server is not yet supported -- copy it to the project and open it there instead`, - ); - } -} +import { + computeServerManager, + type ComputeServerManager, +} from "@cocalc/nats/compute/manager"; const computeServerManagerCache: { - [project_id: string]: ComputeServersManager; + [project_id: string]: ComputeServerManager; } = {}; -export const computeServers = (project_id: string) => { +// very simple cache with no ref counting or anything. +// close a manager only when closing the project. +export default function computeServers( + project_id: string, +): ComputeServerManager { if (computeServerManagerCache[project_id]) { return computeServerManagerCache[project_id]; } - const m = new ComputeServersManager(project_id); - computeServerManagerCache[project_id] = m; - return m; -}; - -export default computeServers; + const M = computeServerManager({ project_id }); + computeServerManagerCache[project_id] = M; + M.on("closed", () => { + delete computeServerManagerCache[project_id]; + }); + return M; +} diff --git a/src/packages/frontend/customize.tsx b/src/packages/frontend/customize.tsx index 63ea3c4eae..11a21e8e4d 100644 --- a/src/packages/frontend/customize.tsx +++ b/src/packages/frontend/customize.tsx @@ -181,6 +181,8 @@ export interface CustomizeState { insecure_test_mode?: boolean; i18n?: List; + + user_tracking?: string; } export class CustomizeStore extends Store { @@ -332,7 +334,7 @@ function process_customize(obj) { for (const k in site_settings_conf) { const v = site_settings_conf[k]; obj[k] = - obj[k] != null ? obj[k] : v.to_val?.(v.default, obj_orig) ?? v.default; + obj[k] != null ? obj[k] : (v.to_val?.(v.default, obj_orig) ?? v.default); } // the llm markup special case obj.llm_markup = obj_orig._llm_markup ?? 30; diff --git a/src/packages/frontend/file-upload.tsx b/src/packages/frontend/file-upload.tsx index e737631d4e..2ced3f697a 100644 --- a/src/packages/frontend/file-upload.tsx +++ b/src/packages/frontend/file-upload.tsx @@ -25,11 +25,12 @@ import { appBasePath } from "@cocalc/frontend/customize/app-base-path"; import { labels } from "@cocalc/frontend/i18n"; import { BASE_URL } from "@cocalc/frontend/misc"; import { MAX_BLOB_SIZE } from "@cocalc/util/db-schema/blobs"; -import { defaults, encode_path, is_array, merge } from "@cocalc/util/misc"; +import { defaults, is_array, merge } from "@cocalc/util/misc"; -// 3GB upload limit -- since that's the default filesystem quota -// and it should be plenty? -const MAX_FILE_SIZE_MB = 3000; +// very large upload limit -- should be plenty? +// there is no cost for ingress, and as cocalc is a data plaform +// people like to upload large data sets. +const MAX_FILE_SIZE_MB = 50 * 1000; const CHUNK_SIZE_MB = 8; @@ -61,14 +62,35 @@ given TIMEOUT_S. See also the discussion here: https://github.com/sagemathinc/cocalc-docker/issues/92 */ +// The corresponding server is in packages/hub/servers/app/upload.ts and significantly impacts +// our options! It uses formidable to capture each chunk and then rewrites it using NATS which +// reads the data and writes it to disk. const UPLOAD_OPTIONS = { maxFilesize: MAX_FILE_SIZE_MB, + // use chunking data for ALL files -- this is good because it makes our server code simpler. forceChunking: true, chunking: true, chunkSize: CHUNK_SIZE_MB * 1000 * 1000, - retryChunks: true, // might as well since it's a little more robust. - timeout: 1000 * TIMEOUT_S, // matches what cloudflare imposes on us; this + + // We do NOT support chunk retries, since our server doesn't. To support this, either our + // NATS protocol becomes much more complicated, or our server has to store at least one chunk + // in RAM before streaming it, which could potentially lead to a large amount of memory + // usage, especially with malicious users. If users really need a robust way to upload + // a *lot* of data, they should use rsync. + retryChunks: false, + + // matches what cloudflare imposes on us; this // is *per chunk*, so much larger uploads should still work. + // This is per chunk: + timeout: 1000 * TIMEOUT_S, + + // this is the default, but also I wrote the server (see packages/hub/servers/app/upload.ts) and + // it doesn't support parallel chunks, which would use a lot more RAM on the server. We might + // consider this later... + parallelChunkUploads: false, + + thumbnailWidth: 240, + thumbnailheight: 240, }; const DROPSTYLE = { @@ -77,6 +99,7 @@ const DROPSTYLE = { borderRadius: "5px", padding: 0, margin: "10px 0", + overflow: "auto", } as const; function Header({ close_preview }: { close_preview?: Function }) { @@ -91,7 +114,6 @@ function Header({ close_preview }: { close_preview?: Function }) { Drag and drop files from your computer {close_preview && (