Workstation

Introduction: A Repeatable Workstation Configuration

This is my workstation configuration.

By default, our workstations are a hodge-podge of tweaks that accumulate over the years. Quite often files etc are left over from old software we use. Worse, files will be left over from old versions of software that we still use. This makes it very confusing to tell what is essential and what is disjecta membra.

This is my attempt to fix all that for myself, and perhaps you, dear reader, may find it of use as well. If you have any questions, feel free to shoot me an email or open an issue (honestly, it would be somewhat nice to know that at least one other person has read this).

Rationale: The Problem

Have you ever had a tool or setup break and had a devil of a time trying to get it back to a working state? Over the years, I’ve set up many new tools to help me with my work. But, time passes, and things break. Sometimes I don’t even notice they’ve broken for a long time, so its even harder to fix. Something must have change must have changed: a dependency, perhaps? or filesystem contents? Or, even, perhaps I’ve forgotten how it is supposed to work.

This has been tricky to resolve. Especially if the tool is a hack I put together.

What is more, to reach higher levels of productivty, it is essential to research, develop, and use new tools for ourselves. Each of us has our own needs and constraints. However, without a firm foundation, these rube-goldberg development approaches are very fragile.

So, I have been trying to do just this: provide myself a repeatable, executable workstation setup which I can extend when needed and rely on if something unforseen happens (e.g. laptop breaks).

One final point: I’ve noticed over the years that there is a subtle pressure to not install new things because I know it will cause a problem for me in the future. But I think this is an anti-pattern, and needs to be fought against!

Design choices

Requirements

open source
hackable
where reasonable, use repeatable, automatable solutions

Invariants checking

It is necessary to check that the computer satisfies stated requirements. This might mean anything from “user can successfully use sudo” to “I can compile haskell source files”.

Basically, this can take shape as a script I run regularly, but more ideally it would (also) be something that is automatically run and have information reported to me.

Currently, the intention is that this will come in the form of running ws check, which is a part of the wshs sub-project in the wshs directory.

early problem reporting

Ideally, we’d like to know about problems asap. If a new update comes out, and our whole setup proces is flawed somehow with the new update, it is important to realize and address when I can.

Background and History

I have had a few different setups over the years. Accomplishing this is a lot harder than I expected.

Literate Programming

In my experience, these workstation projects are easy to put aside for a long time, and when you come back you can’t remember how things were built.

So, this project is mostly written with literate org mode. This gives me an easy way to document my thoughts as I work, and also explore using org-mode for this task. I did this same thing a long time ago with my old dotfiles setup, and I liked it, but everything else about it was a massive pain, so it was eventually abandoned.

If you want to see how literate programming works in org mode, view this raw file.

README

WARNING: this file is managed by tangling the file workstation.org. Do not edit directly!

# Instructions

1. download the bootstrap shell script:
`curl https://raw.githubusercontent.com/joelmccracken/workstation/master/bootstrap-workstation.sh > bootstrap-workstation.sh`
2. run `bash bootstrap-workstation.sh MACHINE_NAME BRANCH
3. Profit!
4. See workstation.org for manual setup documentation
# More Information

Much more information may be found in
<a href="workstation.org">workstation.org</a>.

Makefile

Used to tangle workstation.org. Tangling refers to the process of taking a literate program source and converting it to the target “source” file for execution.

Formerly had some other targets, but now they are OBE. It may make sense to delete this makefile if it becomes clearly unnecessary.

all: tangle

tangle:
	bash bin/tangle.sh

.PHONY: tangle

Which requires a shell script:

source ~/workstation/lib/shell/foundation.sh
$WORKSTATION_EMACS_CONFIG_DIR/bin/doomscript lib/emacs/tangle-file.el

And a little emacs lisp that goes with the tangle process:

;;; tangle-file.el --- description -*- lexical-binding: t; -*-

(setq safe-local-variable-values
      '((org-babel-noweb-wrap-start . "«")
        (org-babel-noweb-wrap-end . "»")))

(doom-require 'doom-start)

(defun do-tangle ()
  "Do the tangle"
  (find-file "workstation.org")
  (org-babel-tangle))

(do-tangle)

(provide 'tangle-file)

;;; tangle-file.el ends here

Bootstrap process

Bootstrapping is tricky. What do you actually start with? What can you assume? You want to keep the amount of manual steps which need to occur to a minimum. I start start with a shell script. This script could either do the entire setup process, or theoretically it could also prepare the way for another process.

For me, I currently basically have a single bash script. But what I want to do soon is change this so that its a bash script which invokes a haskell process asap. so the bash script would basically do the minimum amount required to set up everything for the haskell process. This is still a work in progress, and since this project is starting to stablize, I may abandon the haskell portion of the setup.

Bootstraping Script

This script is intended to be entrypoint to this project. It can be curled to a new machine and then run, and will set things up on that machine as necessary.

The steps to the setup are given more details in Bootstrap Script Execution Process.

set -xeuo pipefail

«bootstrap-steps»

Bootstrap Script Execution Process

This portion of the document explains the various execution steps of bootstrap.

Each of these steps are executed sequentially.

Argument Parsing

# Script should be passed a single argument, which is name of this workstation.

# When using script to set up a workstation, the "name" of the workstation should
# be provided as the first argument. This is used to pick which settings should be
# applied to this machine.
if [ -z "${1+x}" ]; then
    echo WORKSTATION_NAME must be provided as first argument
    exit 2
else
    export WORKSTATION_NAME="$1"
fi

# This argument generally should not be used by the user, but it is needed for
# the CI process.
# When the CI process starts, we start out with a check out of the code for this
# commit in a directory on the CI machine. However, this is not how workstation runs:
# - part of the job of workstation is getting its own code from the server
# - workstation expects the code to be in a specific directory, that is, ~/workstation
# Because of this (and possibly other reasons that escape me now), even though the
# source code of the current commit is checked out on the CI machine already,
# the CI process re-downloads the code (via this script). The specific SHA to get
# is passed via the argument below. However, if actually being used by a user,
# generally user will always want to use the most up to date content of the master
# branch, so this can be ignored.
# I think probably this sha should just be passed in as an environment variable
# instead of a CLI argument, as that seems a bit less confusing to me.
if [ -z "${2+x}" ]; then
    export WORKSTATION_BOOTSTRAP_COMMIT=origin/master
else
    export WORKSTATION_BOOTSTRAP_COMMIT="$2"
fi

Foundations

«workstation_foundation»

Set Version Settings

# These are the various versions of things that should be installed. Keeping them
# in one place like this make them easier to keep track of.
«workstation_setup_versions»

Helper and Component Functions

# hereafter, we use many helper functions. Here they are defined up front,
# as some of them are used throughout the other code.

«is_mac_function»

«is_linux_function»

«info_function»

«polite_git_checkout_function»

«mv_dated_backup_function»

«is_git_repo_cloned_at_function»

«clone_repo_and_checkout_at_function»

«xcode_setup_function»

«is_brew_installed_function»

«homebrew_setup_function»

«update_apt_install_git_function»

«is_git_repo_cloned_at_function»

«clone_repo_and_checkout_at_function»

Log that bootstrap is starting

info starting workstation bootstrap

macos only: ensure xcode, homebrew, and git are ready to go

is_mac && {
    info ensuring xcode is installed
    xcode_setup
    info finished ensuring xcode is installed

    info ensuring brew is installed
    if ! is_brew_installed; then
        homebrew_setup
    fi
    info finished ensuring brew is installed

    info installing git
    brew install git
    info finished installing git

}

(ubuntu) linux only: update apt and install git

is_linux && {
    info updating apt, installing git
    update_apt_install_git
    info finished updating apt, installing git
}

check out correct workstation commit

is_git_repo_cloned_at $WORKSTATION_DIR $WORKSTATION_GIT_ORIGIN || {
    clone_repo_and_checkout_at $WORKSTATION_DIR $WORKSTATION_GIT_ORIGIN_PUB \
        $WORKSTATION_BOOTSTRAP_COMMIT $WORKSTATION_GIT_ORIGIN
}

check out the dotfiles repository

# at this point, this is hardly necessary; however, the gitignore file is handy
# i may explore getting rid of this repo entirely and just having a fresh
# repo without any origin in ~
info ensuring dotfiles repo is checked out

DOTFILES_ORIGIN='git@github.com:joelmccracken/dotfiles.git'

is_git_repo_cloned_at ~ "$DOTFILES_ORIGIN" ||
    polite-git-checkout ~ 'https://github.com/joelmccracken/dotfiles.git' \
        "$DOTFILES_ORIGIN"

info finished ensuring dotfiles repo is checked out

set up host specific settings

# each workstaion host I use has different settings needs.
# For example, my remote cloud hosted server has a different setup than
# my mac laptop, which has a different set up from my work computer.
# the way I have these settings specified is by having a directory in my home
# directory which has all of the needed files I would need for such differences.
# there are different directories for each host I maintain, but on a given host,
# one of those directories are symlinked into 'current' host, which other things
# can then refer to

export WORKSTATION_HOST_SETTINGS_SRC_DIR=$WORKSTATION_DIR/hosts/$WORKSTATION_NAME

info setting current host settings directory...
info workstation host settings directory: $WORKSTATION_HOST_SETTINGS_SRC_DIR

if [ -d $WORKSTATION_HOST_SETTINGS_SRC_DIR ]; then
    info setting current host directory to $WORKSTATION_HOST_SETTINGS_SRC_DIR;
    ln -s $WORKSTATION_HOST_SETTINGS_SRC_DIR $WORKSTATION_HOST_CURRENT_SETTINGS_DIR;
else
    echo ERROR $WORKSTATION_HOST_SETTINGS_SRC_DIR does not exist, must exit
    exit 5
fi

set up nix

info ensuring nix is installed
~/workstation/lib/shell/setup/ensure_nix_installed.sh

info finished ensuring nix is installed

info setting up nix.conf
~/workstation/lib/shell/setup/install_system_nix_conf.sh

info restarting nix daemon
~/workstation/lib/shell/setup/restart_nix_daemon.sh
info nix daemon restarted

NIX_DAEMON_PATH='/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
set +u
source "$NIX_DAEMON_PATH";
set -u

install nix darwin

is_mac && {
    info installing darwin-nix
    ~/workstation/lib/shell/setup/install_nix_darwin.sh
    info finished installing darwin-nix
}

install home manager

~/workstation/lib/shell/setup/install_home_manager.sh

~/workstation/lib/shell/setup/home-manager-flake-switch.sh

set +u
# evaluating this with set -u will cause an unbound variable error
source $HOME/.nix-profile/etc/profile.d/hm-session-vars.sh
set -u

install doom emacs

Unfortunately, because of some problems that are highly detailed (which are too much to get into here), the project nix-doom-emacs doesn’t work for my purposes, and so while emacs itself is installed via nix, doom does its own package management.

~/workstation/lib/shell/setup/install_doom_emacs_no_nix.sh

link dotfiles into place

Note: this must be before wshs is run otherwise when ws runs brew bundle, it will fail.

info linking dotfiles that should be symlinked
bash ~/workstation/lib/shell/setup/link-dotfiles.sh -f -c
info finished linking dotfiles

set up ‘ws’ workstation tool

info "building the 'ws' script"
~/workstation/lib/shell/setup/build_ws_tool.sh

info "running the 'ws install' process"
~/workstation/lib/shell/setup/ws_install.sh
info "'ws install' process completed"

link dotfiles into place

info linking dotfiles that should be symlinked
bash ~/workstation/lib/shell/setup/link-dotfiles.sh -f -c
info finished linking dotfiles

set up workstation secrets

bash ~/workstation/lib/shell/setup/initial_bitwarden_sync.sh

output final manual setup notes

cat <<-EOF
Success! However, there are some remaining manual set up steps required.
«manual-setup-instructions»
EOF

Bootstrapping Components

There are many components Many of these snippets are also provided as separate scripts in the workstation repository. It is handy to have these quickly available if I am debugging a problem; the alternative, frequently, is to reconstruct these things ad-hoc.

For that matter, thinking about extracting things from the giant install file into pieces is helpful.

Helper Functions

is_mac

Detects if is running on a mac.

function is_mac() {
    [[ "$(uname)" == 'Darwin' ]]
}

is_linux

Detects if running on linux

function is_linux() {
    [[ "$(uname)" == 'Linux' ]]
}

info

A simple function for logging.

function info() {
    echo "INFO ========= $(date) $@"
}

mv_dated_backup

function mv_dated_backup() {
    local THEDIR="$1"
    if test -e "$THEDIR"; then
        mv "$THEDIR" "${THEDIR}-$(date +"%s")"
    fi
}

is_git_repo_cloned_at

Utility function to see if a git repo is checked out Use the origin url as an approximate way to check if its checked out

function is_git_repo_cloned_at(){
    cd $1 && [[ "$(git remote get-url origin)" == "$2" ]]
}

clone_repo_and_checkout_at

function clone_repo_and_checkout_at() {
    mv_dated_backup $1
    info cloning repo into $1
    git clone $2 $1
    cd $1
    info checking out commit $3
    git checkout $3
    info setting origin
    git remote set-url origin $4
}

Versions

The versions of various things that are installed as part of the bootstrapping process. Sometimes I need to update these, having them contained in one spot is helpful.

export WORKSTATION_NIX_PM_VERSION=nix-2.11.1
export WORKSTATION_NIX_DARWIN_VERSION=f6648ca0698d1611d7eadfa72b122252b833f86c
export WORKSTATION_HOME_MANAGER_VERSION=0f4e5b4999fd6a42ece5da8a3a2439a50e48e486

setting up xcode (macos)

function xcode_setup() {
    # this will accept the license that xcode requires from the command line
    # and also install xcode if required.
    sudo bash -c '(xcodebuild -license accept; xcode-select --install) || exit 0'
}

External Script:

«xcode_setup_function»
xcode_setup

check if brew is installed (macos)

function is_brew_installed() {
    which brew > /dev/null
}

External Script:

«is_brew_installed_function»
is_brew_installed

set up homebrew (macos)

function homebrew_setup() {
    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
}

External Script:

«homebrew_setup_function»
homebrew_setup

update apt and install git

function update_apt_install_git() {
    sudo bash -c 'apt-get update && apt-get install git'
}

External Script:

«update_apt_install_git_function»
update_apt_install_git

install nix configuration file

I wish I could do this with a nix-like thing, but sadly, there are several complications.

for MacOS, this is nix-darwin.
for Ubuntu, there is nothing that can do it.
There is a way to do something similar with home manager, but it sets the user nix settings, not the system settings. This is not overly surprising, but it does mean that it can’t be the sole solution for setting configurations, if you need to set up caches/substituters. At the very least, I would need some other way besides home manager to sepecify that my user is a trusted user. But, then, there becomes a question of bootstrapping (nix settings needed before home manager ever runs), so I think its overall easier to just hack a thing with bash.

function emit_nix_conf_content () {
    cat - <<-EOF
# Generated at $(date)
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ=
substituters = https://cache.nixos.org https://cache.iog.io
experimental-features = nix-command flakes
trusted-users = root $(whoami) runner
build-users-group = nixbld
# END OF /etc/nix/nix.conf
EOF
}

emit_nix_conf_content | \
    sudo bash -c 'mkdir -p /etc/nix; cat > /etc/nix/nix.conf'

restart nix daemons

Sometimes we need to restart the nix daemons, e.g. after editing the nix config file.

source ~/workstation/lib/shell/funcs.sh
function restart_nix_daemon_linux() {
    sudo systemctl restart nix-daemon.service;
}

function restart_nix_daemon_mac() {
    set +e
    sudo launchctl unload /Library/LaunchDaemons/org.nixos.nix-daemon.plist
    sudo launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist
    set -e
}

if is_mac; then  restart_nix_daemon_mac; fi
if is_linux; then restart_nix_daemon_linux; fi

Install Nix

source ~/workstation/lib/shell/setup/workstation_setup_versions.sh
source ~/workstation/lib/shell/funcs.sh

if which nix > /dev/null; then
    info "nix exists in path, not installing"
else
    info "nix not in path, installing"
    sh <(curl -L https://releases.nixos.org/nix/$WORKSTATION_NIX_PM_VERSION/install) --daemon;
fi

Install doom emacs without nix

source ~/workstation/lib/shell/foundation.sh

{
    cd $WORKSTATION_EMACS_CONFIG_DIR
    [[ "$(git remote get-url origin)" == 'https://github.com/hlissner/doom-emacs' ]]
} || {
    mv_dated_backup $WORKSTATION_EMACS_CONFIG_DIR
    time git clone --depth 1 https://github.com/doomemacs/doomemacs $WORKSTATION_EMACS_CONFIG_DIR/
    # alternative: use this if encounter problems
    # ~/.emacs.d/bin/doom -y install;
    # time timeout 45m bash -c 'yes | ~/.emacs.d/bin/doom install' || exit 0
    # time bash -c 'yes | ~/.emacs.d/bin/doom install' || exit 0
    time timeout 60m bash -c "yes | $WORKSTATION_EMACS_CONFIG_DIR/bin/doom install" || exit 0
    $WORKSTATION_EMACS_CONFIG_DIR/bin/doom sync
    echo FINISHED INSTALLING DOOM;
}

Build WS tool

cd  ~/workstation/wshs
nix build --no-link -L .#"wshs:exe:bww" .#"wshs:exe:ws"

Run WS install

cd  ~/workstation/wshs
$(nix path-info .#"wshs:exe:ws")/bin/ws install -m "$WORKSTATION_NAME";

Install nix-darwin

source ~/workstation/lib/shell/foundation.sh
source ~/workstation/lib/shell/setup/workstation_setup_versions.sh

cd $WORKSTATION_DIR
nix-build https://github.com/LnL7/nix-darwin/archive/${WORKSTATION_NIX_DARWIN_VERSION}.tar.gz -A installer
./result/bin/darwin-installer

source ~/workstation/lib/shell/setup/nix-darwin-rebuild-flake.sh

Install home manager

export HOME_MANAGER_BACKUP_EXT=old

nix run home-manager/$WORKSTATION_HOME_MANAGER_VERSION -- init ~/workstation

Initial Bitwarden Sync

# The initial BitWarden Sync process. Requires wshs/bww executable to
# be built and available. This could all be more robust
# extracting it is theoretically useful as it provides a mechanism for
# resetting the secrets.
# Likely this should be broken down into separate functions that can be reused.
function initial_bitwarden_sync() {
    # why is bash so cryptic
    if [ ! -z "${BW_CLIENTID+x}" ] && \
       [ ! -z "${BW_CLIENTSECRET+x}" ] && \
       [ ! -z "${WS_BW_MASTER_PASS+x}" ]; then
        info variables requried to run bww force-sync are set, running
        if [ ! -d ~/secrets ]; then
            mkdir ~/secrets;
        fi
        cd  ~/workstation/wshs
        # overwriting anything that was previously in the file
        echo "${WS_BW_MASTER_PASS}" > ~/secrets/bw_pass
        bw login --apikey
        bw_unlock
        bw sync
        $(nix path-info .#"wshs:exe:bww")/bin/bww force-sync
    else
        info variables required to run bww force sync are MISSING, skipping
    fi
}

source ~/workstation/lib/shell/funcs.sh

«initial_bitwarden_sync_function»
initial_bitwarden_sync

Linking dotfiles

source ~/workstation/lib/shell/funcs.sh

export FORCE=false;
export VERBOSE=false;
export CHECK=false;

function error() {
    printf "$@" >&2
    exit 1
}

function handle_force() {
    if [ "$FORCE" = "true" ]; then
        mv_dated_backup "$1"
    fi
}

function verbose() {
    if [ "$VERBOSE" = "true" ]; then
        echo "$@"
    fi
}

function check () {
    if [ "$CHECK" = "true" ] || [ "$VERBOSE" = "true" ]; then
        echo "$@"
    fi
}


function ln_helper() {
    dest=~/$2$1
    src=~/workstation/dotfiles/$1
    curr=$(readlink -f "$dest")

    if [ -L "$dest" ] && [ "$curr" = "$src" ]; then
        check "OK: $dest already points to $src"
    else
        check "NOT OK: $dest does not point to $src"
        if [ "$CHECK" = "true" ] && ! [ "$FORCE" = "true" ]; then
            exit 11
        fi

        handle_force $dest
        ln -s "$src" "$dest"
    fi
}

function ln_dotfile() {
    ln_helper $1 "."
}

function ln_norm() {
    ln_helper $1 ""
}

function ln_dotfile_n() {
    src=~/workstation/dotfiles/$1
    dest=~/.$1
    destdir=$(dirname $dest)

    if [ ! -d $destdir ]; then
        mkdir -p $destdir
    fi

    ln_helper $1 "."
}

while (( $# > 0 )); do
    opt="$1"
    shift

    case $opt in
        -f)
            FORCE=true
            ;;
        -v)
            VERBOSE=true
            ;;
        -c)
            CHECK=true
            ;;
        *)
            error "%s: error, unknown option '%s'" "$0" "$opt"
            exit 1
            ;;
    esac
done

ln_dotfile bashrc
ln_dotfile ghci
ln_dotfile gitconfig
ln_dotfile hammerspoon
ln_dotfile nix-channels
ln_dotfile npmrc
ln_dotfile reddup.yml
ln_dotfile zshrc

ln_norm Brewfile
ln_norm Brewfile.lock.json
ln_norm bitbar

ln_dotfile_n config/git
ln_dotfile_n config/doom

Nix components

Home Manager

I use home manager as the primary method for installing and configuring software

flake-world equivalent to ‘home-manager switch’

The pre-flake way of using home manager had a home-manager switch command which would build and then activate the next home manager generation. This is the flake “equivalent”. Having it as a shell command makes it easier to run.

Also, this script obviously requires the WORKSTATION_NAME environment variable to be set, which provides the ‘identity’ of the current machine – not all machines have the same home manager configurations.

set -u # error in case WORKSTATION_NAME is not set

nix build --no-link ~/workstation/#homeConfigurations.${WORKSTATION_NAME}.$(whoami).activationPackage --show-trace

"$(nix path-info ~/workstation/#homeConfigurations.${WORKSTATION_NAME}.$(whoami).activationPackage)"/activate --show-trace

Nix darwin

I actually don’t use this for much of anything now, but I do know that since home manager can’t manage daemons on macos, I want to keep nix darwin around so that I can use it for that. I had used this for setting up nix.conf, but I decided to just unify how it was done since I have to do it another way anyway for non-darwin machines.

Also, this script obviously requires the WORKSTATION_NAME environment variable to be set, which provides the ‘identity’ of the current machine – not all machines have the same home manager configurations. This environment variable is set by other mechanisms withing the workstation system.

command to have darwin build and switch to next generation

set -u # error in case WORKSTATION_NAME is not set

nix build --extra-experimental-features "nix-command flakes" \
    ~/workstation\#darwinConfigurations.${WORKSTATION_NAME}.system
./result/sw/bin/darwin-rebuild switch --flake ~/workstation#${WORKSTATION_NAME}

rm -rf ./result

Post-bootstrap manual installation and setup

There are unfortunately a number of things need to install and set up
manually:
- lastpass firefox extension
- vimium-ff etension
- dropbox
- icloud
- slack
- spotify
- install haskell language server in ~/bin (or somwewhere else?) for hls

These are the settings I use for slack:
- accessibility then at bottom changbe up arrow to move focus to last message
- advanced
  - when in markdown block backticks, enter should do a newline
  - format messages with markup

mac settings
- enable screen sharing, _not_ remote management
- enable remote login
- configure hammerspoon
  - open it
  - enable accessability settings
  - launch at login

System update process

this is still incomplete, but some things I think

fetch ~/worksation and ~, if can clealy rebase, do so
run any other kind of “sync”
on macos, run darwin-rebuild
run home-manager switch
run bww sync

System “check”

I need to have a process to check that system is OK.

Utilities

Passwordless sudo

Occasionally, sudo is extremely annoying. Having to type “sudo” in the middle of a nix-darwin rebuild really interrupts the flow. So here are a couple of scripts to toggle passwordless sudo.

set -eo pipefail

if [[ -z "$SUDO_USER" ]]; then
    echo ERROR: run as sudo
    exit 1
fi

TEMPFILE=$(mktemp)

cat > $TEMPFILE <<EOF
$SUDO_USER  ALL=(ALL) NOPASSWD: ALL
EOF

visudo -c $TEMPFILE

mv $TEMPFILE /etc/sudoers.d/me-passwordless-sudo

set -euo pipefail

rm /etc/sudoers.d/me-passwordless-sudo

Foundation settings

This is the kind of thing that sets up the “foundation” for everything else.

export WORKSTATION_DIR="$HOME/workstation"
export WORKSTATION_EMACS_CONFIG_DIR=~/.config/emacs
export WORKSTATION_GIT_ORIGIN='git@github.com:joelmccracken/workstation.git'
export WORKSTATION_GIT_ORIGIN_PUB='https://github.com/joelmccracken/workstation.git'
export WORKSTATION_HOST_CURRENT_SETTINGS_DIR=$WORKSTATION_DIR/hosts/current

sourceIfExists () {
    if [ -f "$1" ]; then
        source "$1"
    fi
}

if [ -z "${WORKSTATION_NAME+x}" ] ; then
    sourceIfExists "$WORKSTATION_HOST_CURRENT_SETTINGS_DIR/settings.sh"
fi


if [ -z "${WORKSTATION_NAME+x}" ] ; then
    echo WARNING: no environment variable WORKSTATION_NAME provided.
    echo This variable should be exported by a script at:
    echo $WORKSTATION_DIR/hosts/current/settings.sh
    echo see workstation.org for more information
else
    export WORKSTATION_HOST_SETTINGS_SRC_DIR=$WORKSTATION_DIR/hosts/$WORKSTATION_NAME
fi

library of shell functions

This single file contains many of the general-purpose functions that I use in numerous scripts etc.

«is_mac_function»

«is_linux_function»

«info_function»

«bw_unlock_function»

«polite_git_checkout_function»

«mv_dated_backup_function»

«is_git_repo_cloned_at_function»

«clone_repo_and_checkout_at_function»

Polite git checkout

This script provides a way to check out a repository in a directory without clobbering the existing contents of the directory. This is useful in case the directory might have contents that you wish to save, and you think it might be handy to be able to i.e. git diff the contents against what git knows about in the repository, once all of the trivial differences have been resolved (i.e. files missing are put into place).

I used to use this for setting up dotfiles, however, I’ve changed the approach, but I still think this script is handy and want to hang on to it.

«polite_git_checkout_function»
polite-git-checkout $1 $2

function polite-git-checkout () {
    DIR=$1
    REPO=$2
    ORIGIN=$3

    cd $DIR
    git init
    git remote add origin $REPO
    git fetch

    # wont work (it will have already been deleted from the index)
    git reset --mixed origin/master
    # This formulation of the checkout command seems to work most reliably
    git status -s | grep -E '^ D' | sed -E 's/^ D //' | xargs -n 1 -- git checkout
    # fixing; used public to start, but want to be able to push
    git remote set-url origin $ORIGIN
}

Bitwarden and personal secrets

I have a script to set up and download various “private” information. for various reasons I’ve decided to try bitwarden for this, but out of the box bitwarden doesn’t really do what I need it to.

This restores SSH keys to my local computer. These can’t be in git, and really they are essential for any meaningfully complete workstation setup.

bw_unlock bash function

The bw_unlock function sets the BW_SESSION environment variable in the current shell process, which is required in order to query the bitwarden password database.

# unlocks bitwarden, so that the `bw` program can access the bitwarden database.
bw_unlock () {
    # authtenticates bitwarden for this shell session only
    export BW_SESSION=`bw unlock --passwordfile ~/secrets/bw_pass --raw`;
}

Hosts: profiles for individual machines

One of the issues that is inherent in workstation configuration is variation between individual workstations. I have different configuation needs on machines for work use and for personal use.

There are many possible ways to handle this, but the one I use here is to have the configurations unique to each machine in specific subdirectories (i.e. workstation/hosts/glamdring, workstation/hosts/anduril, etc), and then have the current configuration specified by a symlink from /workstation/hosts/current to one of the other directories. I can then put whatever is convenient in those directories (shell scripts, emacs lisp), and have other systems read from there.

Kinda ugly, but it works.

glamdring/

# About
glamdring: My primary computer/workstation

(after! org
  (setq org-directory "~/EF/")
  (setq org-roam-directory "~/EF/")
  (setq org-roam-db-location "~/EF/org-roam.glamdring.db")

  ;; for now wont be able to org-mobile-push from glamdring
  (setq org-mobile-directory "~/Dropbox/Apps/MobileOrg")
  (setq org-directory "~/EF")
  (setq org-id-locations-file "~/EF/.orgids.el")
  (setq org-agenda-files '("~/EF/actions.org" "~/EF/projects.org"))
  (setq +org-capture-notes-file "inbox.org")
  (setq org-mobile-files (org-agenda-files))
  (setq org-mobile-inbox-for-pull "~/EF/inbox-mobile.org"))

source ~/workstation/hosts/glamdring/settings.sh

export WORKSTATION_NAME=glamdring

anduril/

aeglos/

(after! org
  (setq org-directory "~/Dropbox/EF/")
  (setq org-roam-directory "~/Dropbox/EF/")
  (setq org-roam-db-location "~/Dropbox/EF/org-roam.aeglos.db")

  (setq org-mobile-directory "~/Dropbox/Apps/MobileOrg")
  (setq org-directory "~/Dropbox/EF")
  (setq org-id-locations-file "~/Dropbox/EF/.orgids.el")
  (setq org-agenda-files '("~/EF/actions.org" "~/EF/projects.org"))
  (setq +org-capture-notes-file "inbox.org")
  (setq org-mobile-files (org-agenda-files))
  (setq org-mobile-inbox-for-pull "~/Dropbox/EF/inbox-mobile.org"))

source ~/workstation/hosts/aeglos/settings.sh

export WORKSTATION_NAME=aeglos

ci/

belthronding

# About
belthronding: my cloud ubuntu machine on DO

(after! org
  (setq org-directory "~/EF/")
  (setq org-roam-directory "~/EF/")
  (setq org-roam-db-location "~/EF/org-roam.belthronding.db")

  (setq org-mobile-directory "~/Dropbox/Apps/MobileOrg")
  (setq org-directory "~/EF")
  (setq org-id-locations-file "~/EF/.orgids.el")
  (setq org-agenda-files '("~/EF/actions.org" "~/EF/projects.org"))
  (setq +org-capture-notes-file "inbox.org")
  (setq org-mobile-files (org-agenda-files))
  (setq org-mobile-inbox-for-pull "~/EF/inbox-mobile.org"))

source ~/workstation/hosts/belthronding/settings.sh

export WORKSTATION_NAME=belthronding

Cron file

I have set up a crontab process for automatic synchronization of my personal notes.

# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command
*/5 * * * * /home/joel/workstation/bin/cron-5.sh

#* * * * * /home/joel/workstation/bin/cron-5.sh
# uncomment ^^ for use during development

# run to install:
#   $ crontab ~/workstation/lib/misc/crontab

Testing

test.sh

At this point in time, this test actually checks very little, but what it DOES check is things that indicate that everything went right. Specifically, checking the doom version means emacs, doom, and the whole doom setup process worked out.

I plan to move this to a Haskell project at some point, probably do it with hspec instead. Or maybe that bats testing library. We’ll see.

set -euox pipefail

set +u
# evaluating this with set -u will cause an unbound variable error
source $HOME/.nix-profile/etc/profile.d/hm-session-vars.sh
set -u

source ~/workstation/lib/shell/foundation.sh

function find_emacs_init() {
  init_file="";
  for x in "$WORKSTATION_EMACS_CONFIG_DIR/early-init.el" "$WORKSTATION_EMACS_CONFIG_DIR/init.el"; do
    if [[ -f "$x" ]]; then
      init_file="$x"
      break;
    fi;
  done;
  if [[ "$init_file" = "" ]]; then
    echo "Error: Could not find emacs init file" 1>&2
    exit 43
  else
    echo "$init_file"
  fi
}

emacs_init="$(find_emacs_init)"


function assert_input() {
  local label=$1
  local expected=$2
  local actual
  read actual

  if [[ "$expected" == "$actual" ]]; then
    echo "$label is correct"
  else
    echo "$label is not correct, found '$actual', expected '$expected'"
    exit 1
  fi
}

echo "RUNNING TESTS"

EMACS_PATH=~/.nix-profile/bin/emacs
# emacs
if [ -x "$EMACS_PATH" ]; then
    echo found emacs
else
    echo EMACS NOT FOUND
    exit 1
fi

$EMACS_PATH -Q --batch --eval '(progn (princ emacs-version) (terpri))' | {
  read actual
  if [[ "$actual" == "27.1" || "$actual" == "27.2" || "$actual" == "28.1" || "$actual" == "28.2" ]]; then
    echo "emacs version is correct"
  else
    echo "emacs version is not correct, found '$actual', expected 27.1, 27.2, 28.1, or 28.2"
    exit 1
  fi
}

$EMACS_PATH -l "$emacs_init" --batch --eval '(progn (princ doom-version) (terpri))' | {
  read actual;
  if [[ "$actual" == "21.12.0-alpha" || "$actual" == "3.0.0-dev" || "$actual" == "3.0.0-pre" ]]; then
    echo "doom version is correct"
  else
    echo "doom version is not correct, found '$actual', expected 21.12.0-alpha, 3.0.0-dev, or 3.0.0-pre"
    exit 1
  fi
}

if $EMACS_PATH -l "$emacs_init" --batch --eval "(progn (require 'vterm-module nil t))"; then
  echo "emacs is able to load vterm-module, so vterm-module is compiled and ready to go";
else
  echo "error: emacs was not able to load vterm-module";
  exit 1
fi

if [ -f ~/secrets/test_secret ]; then
    echo "test secret file sucessfully synced"
    cat ~/secrets/test_secret
else
    echo "error: test secret file was missing"
fi

echo "TESTS COMPLETE"

Github Actions CI

Importantly, github CI support macos environments.

Daily build to ensure that potential problems get caught (NB: I have had issues where a working setup no longer worked due to bit rot, which would have been caught with a regular build like this).

I am running up close to maximum execution time, so very likely I will need to refactor/come up with some other way to do this.

docs on different available runners can be found here
- https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories

name: CI

on:
  push:
  schedule:
  - cron: '0 0 * * *'  # every day at midnight

jobs:
  build:
    strategy:
      matrix:
        os:
        - macos-13 # x86
        - macos-latest # aarch
        - ubuntu-latest
    runs-on: ${{ matrix.os }}
    steps:
    - uses: actions/checkout@v3

    - name: Run a one-line script
      env:
        BW_CLIENTID: ${{ secrets.BW_CLIENTID }}
        BW_CLIENTSECRET: ${{ secrets.BW_CLIENTSECRET }}
        WS_BW_MASTER_PASS: ${{ secrets.WS_BW_MASTER_PASS }}
      run: ./test/ci.sh

The environment setup script

To run CI, we have a script which, thankfully, basically mirrors the install instructions.

Importantly, this does a LOT of things, such as install nix, home-manager, etc, and eventually runs the test script.

set -xeuo pipefail

# env # are there environment variables where I can get the commit sha?

cd ~

if [ "$GITHUB_SHA" == "" ]; then
    WORKSTATION_BOOTSTRAP_COMMIT=master
else
    WORKSTATION_BOOTSTRAP_COMMIT="$GITHUB_SHA"
fi

curl https://raw.githubusercontent.com/joelmccracken/workstation/$WORKSTATION_BOOTSTRAP_COMMIT/bootstrap-workstation.sh > bootstrap-workstation.sh

echo BEGINNING INITIAL INSTALL

# disable native compilation, too slow for CI
export DOOM_DISABLE_NATIVE_COMPILE=true

if [ "$RUNNER_OS" == "macOS" ]; then
    bash bootstrap-workstation.sh ci-macos $WORKSTATION_BOOTSTRAP_COMMIT
else
    bash bootstrap-workstation.sh ci-ubuntu $WORKSTATION_BOOTSTRAP_COMMIT
fi

echo INSTALL PROCESS COMPLETE, TESTING

bash ~/workstation/test/test.sh

tasks

cron thing
- document how to work with it
- and script cronfile installation
check if current username is different from expected username
use flake.nix to generate the different username/pw settings;
- or… generate the targets of flake.nix from expected combos?
[ ] there are lots of weird little things that have accumulated in bootstrap-workstation.sh; try to clean some of them up
- many things in bootstrap-workstation.sh should also become helper scripts in bin
  - mv_dated_backup
  - install git (mac and linux)? homebrew? nix?
  - restart nix daemon, linux/macos
  - build_wshs
  - ws (an executable to run the currently built wshs), bww
- for various installations, document the interesting parts of each and have subsections of the workstation config
[ ] change setting of WORKSTATION_BOOTSTRAP_COMMIT to use env var its awkward having it be a cli arg, I feel the need to explain when we need to use it
[ ] better document all of the workstation names that are available
[ ] create mechanism to run for updating workstations
- download updates to master branch of dotfiles and workstation
- run nix stuffs when appropriate
- maybe do bww sync?
[ ] move various code not in workstation.org into this file
- machine settings for each machine in workstation/hosts
- bww
- the ws code
- various code in lib
[ ] finish filling in the numerous incomplete sections of prose in this document
[ ] devise method to prevent committing manually-edited target files
- git pre-commit-hook?
- github action CI that runs tangle, checks for differences
[ ] rebuild my personal laptop once all of this is stable
[ ] port test/test.sh into wshs/haskell
[ ] laptop-state-checking script/features
- ensure secrets/bw_pass exists
- ensure other secrets are there
  - check that no new secrets need to be synced
    - if any secrets have changed, list the changes
- check if can access/ssh/etc into some other machines
- check for updates on workstation origin
- checks for nix (nix store verify --all and nix-doctor)
- check for brew updates/state and presence of brew executable
  - brew doctor?
- check reddup state
- check for various execuables I care about
  - (e.g. each thing specifially installed)
  - haskell language server versions
- check for any changes/differences in ~/
[ ] setup hammerspoon, and especially spacehammer
[ ] rebuild belthronding/my cloud machine
- (has had lots of manual hacking)
[ ] build nixos sever on gandi
[ ] document various components of bww sync
- how it works
- how to use it
- document/alert if going to replace a file with server version
  - display diff of files
[ ] get rid of rming results when darwin-rebuild script finishes (use path technique from home manager script)
[ ] move host settings into this file

quick planning for server config needs

(in rough order need to complete, to get to server-config phase of project) goal is to get ready to provision/set up nixos cloud machines

[ ] use bww to sync/restore secrets on workstations
[ ] create a new host for nixos server
[ ] create script to sync updates from changes to workstation and dotfiles
- does something like https://nixos.wiki/wiki/NixOps help?
[ ] figure out way to run update script on hosts that need it.
[ ] experiement with https://docs.hercules-ci.com/arion/ for running nextcloud (most urgent cloud service I want to use)
[ ] set up “intelligent” s3 bucket for

Files

workstation.org

Latest commit

History

workstation.org

File metadata and controls

Workstation

Introduction: A Repeatable Workstation Configuration

Rationale: The Problem

Design choices

Requirements

Invariants checking

early problem reporting

Background and History

Literate Programming

README

Makefile

Bootstrap process

Bootstraping Script

Bootstrap Script Execution Process

Argument Parsing

Foundations

Set Version Settings

Helper and Component Functions

Log that bootstrap is starting

macos only: ensure xcode, homebrew, and git are ready to go

(ubuntu) linux only: update apt and install git

check out correct workstation commit

check out the dotfiles repository

set up host specific settings

set up nix

install nix darwin

install home manager

install doom emacs

link dotfiles into place

set up ‘ws’ workstation tool

link dotfiles into place

set up workstation secrets

output final manual setup notes

Bootstrapping Components

Helper Functions

is_mac

is_linux

info

mv_dated_backup

is_git_repo_cloned_at

clone_repo_and_checkout_at

Versions

setting up xcode (macos)

check if brew is installed (macos)

set up homebrew (macos)

update apt and install git

install nix configuration file

restart nix daemons

Install Nix

Install doom emacs without nix

Build WS tool

Run WS install

Install nix-darwin

Install home manager

Initial Bitwarden Sync

Linking dotfiles

Nix components

Home Manager

flake-world equivalent to ‘home-manager switch’

Nix darwin

command to have darwin build and switch to next generation

Post-bootstrap manual installation and setup

System update process

System “check”

Utilities

Passwordless sudo

Foundation settings

library of shell functions

Polite git checkout

Bitwarden and personal secrets

bw_unlock bash function

Hosts: profiles for individual machines

glamdring/

anduril/