Use case question: Sharing notes, ssh autostart, syncing, etc.

[mforbes]

I am trying to find a good solution for the following use case:

Use Case: A research group would like to share a set of "lab notebooks" - markdown files that we update and interact with heavily. The current limitation is that CoCalc does not provide a good way for users to synchronize or edit these files using their tools of choice.

Example: I want to use emacs running on my machine. I can do this using TRAMP, but this connects via SSH, and if the project is not running, then it will fail (after a long delay), requiring me to open a browser, trigger the project to launch, wait, then reconnect.

While this is not the end of the world, it is annoying enough that I often prefer to just edit a local file, promising to myself that I will update CoCalc later (but then inevitably forgetting). I want to make this as easy and transparent as possible (after an initial setup) so that there are no barriers. (It is hard enough to encourage people to keep rigorous lab notebooks even when there are no barriers!)

Any suggestions? I have a few ideas, but think they would require some modifications.

1. Provide a scriptable way of auto-starting a project when an SSH connection is attempted. auto start project on ssh connection attempt #4213. This might still incur an unacceptable delay when first trying to load/save the file.
2. Is there an inexpensive way to have an extremely lightweight CoCalc project always running? I.e. this project would only be used for editing markdown files, time travel, etc. but no heavy computation? It seems like a minimal configuration would cost $250.75/year which is a little steep for this use-case.
3. Provide some way of automatically syncing when the project starts up. This used to be possible with Dropbox, but they dropped support for the filesystem used by CoCalc. Is there another approach that currently works? (I suppose that git could be run when the project starts? But then people need to remember to commit their work. Perhaps something like git-annex would work. I tried to set this up in the past and, although I did not run into any real issues, it was not trivial.

Am I missing an obvious solution?

[williamstein]

Am I missing an obvious solution?

There is a simple solution, but it's probably not obvious, since it isn't documented yet. You can start the project using the API. Here's the endpoint for stopping a project:

https://doc.cocalc.com/api2/stop.html

There's also a start endpoint, but it's just not documented. https://github.com/sagemathinc/cocalc/blob/master/src/packages/next/pages/api/v2/projects/start.ts

Basically get an api key, then do this:

x='sk_xxxxx' # your API key

# project_id of the project you want to start
p='304506b4-262e-11ed-a279-2f22f36cbe91'

curl -sk \
  -u $x: \
  -d project_id=$p \
  https://cocalc.com/api/v2/projects/start

and your project will start. You can also do a post API call pretty easily from Python (using requests) or anywhere else. So in your script where you might use ssh or rsync or whatever, just put a single call to the api first.

You can actually do a frightening about on cocalc via the API, since all of cocalc is implemented using the api. However, the document lags.

E.g., this endpoint https://github.com/sagemathinc/cocalc/blob/master/src/packages/next/pages/api/v2/jupyter/execute.ts lets you execute a chunk of code in a Jupyter kernel in an anonymous sandbox, or in a project of your choice, and keeps another kernel around at the ready so the next execution is fast..

-- William

[mforbes]

When I run curl I get a request Enter host password for user (even with a project API KEY). This can be empty, but is there any way to suppress this?

(I don't recall this behaviour from before... did something change? Also... it can be really hard to stop a project sometimes: it keeps restarting. Is this known behavior? Maybe it is something with the browser? If I do not keep the project settings page open in Chrome, then I can stop without the project restarting.)

[mforbes]

Here is a little bash function that tries connecting with SSH, requesting to start the project if it is not running:

function sshcc () {
     ALIAS="$1"
     PID=$(lpass show --username ${ALIAS})
     #API_KEY=$(lpass show --password ${ALIAS})
     API_KEY=$(lpass show --notes "CoCalc API Key")
     USER=$(echo $PID | sed "s/-//g")
     local -r -i max_attempts=3
     local -i attempt_num=1
     local -i wait_time=5
     until ssh ${USER}@ssh.cocalc.com
     do
     if ((attempt_num==max_attempts))
        then
            echo "Could not connect after $attempt_num tries."
            return 1
        else
            echo "Attempt $attempt_num failed! Trying to start server..."
            expect << EOF
            spawn curl -sk -u $API_KEY -d project_id=$PID https://cocalc.com/api/v2/projects/start
            expect "Enter host password for user '$API_KEY':"
            send "\r"
            expect "{}"
EOF
            sleep $wait_time
            ((attempt_num++))
            
        fi
     done
  }

It requires the project information to be stored in LastPass (I am using the LastPass CLI):

echo "Username: <project id>" | lpass add <alias> --non-interactive
echo "<API KEY>" | lpass add --notes "CoCalc API Key" --non-interactive

Finally, I have something like the following configuration in my ~/.ssh/config as I have discussed elsewhere #370:

# ~/.ssh/config
Host ssh.cocalc.com
  ForwardAgent yes
  SendEnv LC_HG_USERNAME
  SendEnv LC_GIT_USERNAME
  SendEnv LC_GIT_USEREMAIL
  SetEnv LC_EDITOR=vi