Git Synch

[Parakoos]

I would love to be able to easily have my CircuitPython drive pull down the code from some git repo, either via the Web Workflow interface or automatically via the MCU's own internet connection.

I currently have my CIRCUITPY drive initialized as a git repo, with an external git directory, and a GitHub repo set up as a remote. So all I have to do to pull down code from GitHub is to connect to it via USB, open the CIRCUITPY drive in my terminal and run git pull. This works great, but I need that USB connection.

It would be great if I could do this via WIFI or BLE.

[dhalbert]

The git repo could live on your host computer drive, and be mirrored by some mechanism to CIRCUITPY. This is more important when CIRCUITPY is small, since a git repo is much larger than the snapshot of the latest commit.

mpremote, which currently works only on MicroPython (but could be made to work on CircuitPython easily) has a "mounting" mechanism which might work, but I haven't looked at in detail.

[tannewt]

The simplest way to do this may be to add zip support and then fetch the zip from the web periodically. set_next_code_file() can be used to switch to the new version.

[Parakoos]

The simplest way to do this may be to add zip support and then fetch the zip from the web periodically. set_next_code_file() can be used to switch to the new version.

That is an interesting option. Is there support for unzipping .zip files in circuitpython? I looked around the library bundle but didn't see anything.

Another option would be to write a function that:

1. Turn off auto-reload on file changes
2. Lists all the files on the MCU
3. For each (filename), make a request for https://raw.githubusercontent.com/(user)/(repo)/(branch)/(filename)
4. Download the file content and overwrite the file on the MCU.
5. supervisor.reload()

It won't pick up new files, but it would update all existing files. You could then have a trigger (maybe press a button 5 times) to initiate the download from github. (this all assumes you have a wifi board of course)

What do you think? Would this work? Does CP have access to overwrite its own files?

[Parakoos]

I put this idea to the test, and it seems to work!

BEWARE RUNNING THIS CODE! IT WILL OVERWRITE EVERY FILE ON THE MCU THAT IS ALSO ON THE GITHUB REPO BRANCH!

import storage
import ssl
import socketpool
import wifi
import time
import supervisor
from adafruit_requests import Session
from os import getenv

GITHUB_USER = getenv("GITHUB_USER")
GITHUB_REPO = getenv("GITHUB_REPO")
GITHUB_BRANCH = getenv("GITHUB_BRANCH")
GITHUB_TOKEN = getenv("GITHUB_TOKEN")

def run():
	try:
		storage.remount("/", readonly=False)
	except RuntimeError as e:
		print('Failed to remount the file-system to be writeable. Cannot pull data from GitHub without that.')
		raise e

	pool = socketpool.SocketPool(wifi.radio)
	ssl_context = ssl.create_default_context()
	session = Session(pool, ssl_context)
	header_auth={'Authorization':f"Bearer {GITHUB_TOKEN}"}

	with session.get(f'https://api.github.com/repos/{GITHUB_USER}/{GITHUB_REPO}/branches/{GITHUB_BRANCH}', headers=header_auth) as response:
		json = response.json()
		tree_url = json['commit']['commit']['tree']['url']

	with session.get(f"{tree_url}?recursive=1", headers=header_auth) as response:
		json = response.json()
		files = json['tree']

	for file in files:
		if file['type'] == 'blob':
			print(file['path'])
			with session.get(file['url'], headers=header_auth|{'accept': 'application/vnd.github.raw+json'}) as response:
				with open(file['path'], "w") as fp:
					fp.write(response.text)
	print('Pulled down the latest data from GitHub! Reloading in 1 second.')
	time.sleep(1)
	supervisor.reload()

[Parakoos]

Oh, and about the github token... You can make a read-only token only (content only) for the repo in question. That way the token isn't so sensitive to put in settings.toml since all it can do is read the repo.

https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens

[tannewt]

No zip file support yet. I'd be interested in seeing it added though.

We do have a zlib library that might be able to be used to decompress gzip and then you can write python code to unpack the tar. Looks like repos can be fetched as .tar.gz: https://github.com/adafruit/circuitpython/archive/refs/heads/main.tar.gz