Today I Learned
Tags: shell bash zsh
When you run commands in a pipeline, $? only tells you the exit status of the last command in the pipeline.
If you need the exit code of a command earlier in a pipeline, you can use the $PIPESTATUS array:
#!/bin/bash
grep some.machine.example.com /etc/hosts 2>&1 | tee /tmp/hosts-results.txt
if [ "${PIPESTATUS[0]}" -ne "0" ]; then
# The grep command failed to find "some.machine.example.com" in /etc/hosts file
echo "I don't know the IP address of some.machine.example.com"
exit 2
fi
https://www.shellscript.sh/examples/pipestatus/
Git: If a subrepo breaks due to rebasing in the upstream subrepo, force re-cloning it is the simplest resolution
Tags: git subrepo
With git subrepo, you can force reclone a subrepo with -f, if you hit issues with rebases breaking subrepo's tracking of parentage.
Tags: shell bash zsh
Backtick command substitution can be abused in order to include comments in multi-line commands, where the continuation slash needing to be the last character, and lines needing to be adjacent, would otherwise preclude adding comments:
echo CommandName InputFiles `#1st comment` \
--option1 arg1 `#2nd comment` \
--option2 arg2 `#3rd comment`https://stackoverflow.com/a/12797512/1108828
Tags: osx clipboard
It can be useful to copy HTML to the clipboard with a HTML data type, so that applications that accept HTML clipboard content accept it correctly (rather than as the HTML as text).
It's possible to add a HTML string to the clipboard as a HTML copy type by using a hexdumped version with AppleScript:
hex=`echo -n "your html code here" | hexdump -ve '1/1 "%.2x"'`
osascript -e "set the clipboard to «data HTML${hex}»"https://stackoverflow.com/a/11089226/1108828
Tags: shell bash
until can be used to run commands until a given command no longer fails.
until docker-compose exec app bash -c 'cat /app/webpack-stats/webpack-stats.json.invalid' | grep '\"status\":\"done\"' ; do echo "*** No webpack-stats.json found, waiting..." ; sleep 10 ; donehttps://stackoverflow.com/a/35412943/1108828
Tags: jira
Jira panels are really nice for organising issue descriptions.
In older versions of Jira, these panels could be custom styled with Wiki markup.
In the newer versions of Jira rich text editors, the panel styles are more limited.
The follow Wiki markup panels map to the new-style panels.
{panel:bgColor=#ffebe6}
This error panel
{panel}
{panel:bgColor=#deebff}
This info panel
{panel}
{panel:bgColor=#e3fcef}
This success panel
{panel}
{panel:bgColor=#fefae6}
This warning panel
{panel}
{panel:bgColor=#eae6ff}
This note panel
{panel}
Tags: vscode
From the ... context menu in the extensions pane, there's an option to "Start Extension Bisect".
VSCode will run a git bisect-like halving of enabled extensions (by disabling half), and you to perform any actions to identify undesired behaviour, then report whether the set of enabled extensions is good or bad. This is repeated until the offending extension(s) are identified.
Handy!
Tags: terraform
Via https://alexwlchan.net/2019/01/debugging-a-stuck-terraform-plan/
Stop endless attempted connections to AWS APIs that are exhibiting issues (permissions or otherwise):
provider "aws" {
# other settings
max_retries = 1
}The behaviour of retrying with max_retries can cause very long hangs where API(s) are not accessible:
The delay between the subsequent API calls increases exponentially. If omitted, the default value is 25.
Tags: terraform
The following snippet removes all items from Terraform state, and does so efficiently even if there is remote state locking (i.e. it doesn't acquire the lock for removing each individual state item.
terraform state list | cut -f 1 -d '[' | xargs -L 0 terraform state rmTags: python
https://docs.python.org/3/library/zipapp.html
Tags: javascript i18n
JS has a built-in object for formatting numbers with internationalisation:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat
Tags: rdkit docker wasm javascript
The minimal RDKit JS build (emscripten compiled to WASM) allows a lot of cheminformatics work (including 2D depictions, fingerprint generation and similarity searching) to be done in the browser.
Using Docker it's easy to build the library using a single Dockerfile.
If the Dockerfile is placed on it's own in a folder, build with:
docker build -t rdkitjs . The build will take some time... ☕️
The built JS and WASM file can then be copied out of a running instance of the container:
CONTAINER_ID=$(docker run -d rdkitjs sleep 20)
docker cp $CONTAINER_ID:/src/rdkit/build/Code/MinimalLib/RDKit_minimal.js .
docker cp $CONTAINER_ID:/src/rdkit/build/Code/MinimalLib/RDKit_minimal.wasm .
docker stop $CONTAINER_ID
docker rm $CONTAINER_IDUse in your web app with
<script src="<path/to>/RDKit_minimal.js"></script>It may be a good idea to add cache-buster to the filename, so that the large-ish (~4.3 MB without gzip) RDKit WASM can be cached with a long lifetime.
If you do, you'll need to rename both the JS and WASM file, and in the JS file, find:
wasmBinaryFile="RDKit_minimal.wasm"and replace the filename with your altered name.
When loaded, an initRDKitModule function is added to window. window.initRDKitModule returns a Promise, which resolves to a handler object for RDKit functionality.
Example Typescript wrapper around initRDKitModule:
const setupRDKit = async () => {
let rdkit
try {
rdkit = await (window as any).initRDKitModule()
if (process.env.NODE_ENV !== 'production') {
console.log('RDKit initialized')
}
(window as any).rdkit = rdkit
return rdkit
} catch (error) {
console.error('Error initializing RDKit', error)
}
}
export default setupRDKitUsing the above, check out window.rdkit in browser development tools to see the functionality available. window.rdkit.get_mol allows you to get a molecule from e.g. SMILES; that Mol object provides further methods such as get_svg.
Tags: caddy dns cloudflare pihole
Hosting sites on a local network with SSL (using the Cloudflare DNS validation module for ACME / Let'sEncrypt validation) on a Pi-hole DHCP / DNS managed network with Caddy: can be tricky as making a local DNS record can throw Caddy's DNS validation off.
In this very specific case, rather than giving a limited config:
tls {env.ACME_EMAIL} {
dns cloudflare {env.CLOUDFLARE_API_TOKEN}
}
A full config can be provided giving resolvers to use for DNS validation:
example.com {
root * /var/caddy/example.com
file_server
tls {
issuer acme {
dir https://acme-staging-v02.api.letsencrypt.org/directory
dns cloudflare <api_token>
resolvers 1.1.1.1 1.0.0.1
}
}
}
caddy-dns/cloudflare#13 (comment)
Tags: python administration
You can customise a Python install as a system administrator by creating a sitecustomize module in the install's site-packages directory, and including code to be applied on every Python run there.
https://docs.python.org/3/library/site.html
You can use this to do things like globally opt out of SSL verification (not that you should in the vast majority of cases!)
https://www.python.org/dev/peps/pep-0476/#opting-out
Tags: ssl letsencrypt certbot docker
If you need a certificate quickly (and free) and you're not able to set up automated Let'sEncrypt renewal, you can grab a three month valid certificate by running Certbot manually in Docker, e.g.
docker run -it --rm --name certbot \
-v "$(pwd)/etc:/etc/letsencrypt" \
-v "$(pwd)/lib:/var/lib/letsencrypt" \
certbot/certbot -d '*.domain.com' --manual --preferred-challenges dns certonly
Certbot will interactively take you through validating via setting a DNS TXT record manually. The certificates live in etc/live/domain.com on your host system (from the pwd bind mount), will last for three months. fullchain.pem is the full certificate chain used for web servers, privkey.pem is the private key.
Tags: postgres database
You can update the default template database, template1, by running commands against it (e.g. CREATE EXTENSION). Subsequent created databases will be based off of the updated template1, which is handy for setting defaults for all creating databases to inherit.
If you mess up template1 somehow, it can be restored to it's original setup from the uneditable template0 database.
https://www.postgresql.org/docs/13/manage-ag-templatedbs.html
Tags: javascript typescript webpack frontend
If you have a folder of files you want to import at build-time, Webpack can traverse the folder structure for you and import them. For example, to import every Vue file (assuming you're using a Vue Webpack loader):
const cache = {}
function importAll (r) {
r.keys().forEach(key => { cache[key] = r(key) })
}
importAll(require.context('./plugins', true, /\.vue$/))cache has key:value pairs of filename: Module.
In Vue for example, this can be used to import a large set of components without importing them all individually or maintaining a manifest:
const plugins = Object.fromEntries(
Object.entries(cache).map(([filename, esModule]) => [
esModule.default.name, esModule.default
])
)
export default Vue.extend({
name: 'Plugins',
components: {
...plugins
}
})More info at https://webpack.js.org/guides/dependency-management/#context-module-api
Tags: git
Avoid a lot of messy merge commits by rebasing on pull by default:
git config --global pull.rebase trueTags: git
While cherry-pick is great for copying over specific commits, if you want to grab a whole file onto your current branch from another, it's a simple as a checkout:
git checkout my-awesome-source-branch the/path/to/yourfile.radhttps://github.com/thoughtbot/til/blob/master/git/grab-a-file-from-another-branch.md
Tags: aws cloud bash shell *nix
Log userdata scripts for cloud instances in detail, to allow easier debugging of any issues when the script is run when the instance is initialised.
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-log-user-data/
Tags: networking bash shell *nix
https://netbeez.net/blog/telnet-to-test-connectivity-to-tcp/
Tags: nginx docker bash shell *nix
If you want to bake an nginx config into a Docker image, and/or dynamically set configuration in a volume mounted config, you can have the COMMAND for the container use envsubst to substitute in environment variable values in the config before the nginx process is started, for example in a docker-compose.yml:
web:
image: nginx
volumes:
- ./mysite.template:/etc/nginx/conf.d/mysite.template
ports:
- "8080:80"
environment:
- NGINX_HOST=foobar.com
- NGINX_PORT=80
command: /bin/bash -c "envsubst < /etc/nginx/conf.d/mysite.template > /etc/nginx/conf.d/default.conf && exec nginx -g 'daemon off;'"From https://hub.docker.com/_/nginx
Tags: python django graphql graphene graphene-django
When using Graphene-Django to use GraphQL with Django, GraphQL swallows your error messages and tracebacks, making it more difficult to debug exceptions in resolvers.
There is a GitHub issue discussing the problem, a number of solutions are suggested including using middleware, however middleware has the caveat that it has to be applied individually on each request.
A generally applicable solution is subclassing GraphQLView and adding back the traceback there (adapted from Martin Samami on Medium adding Sentry logging):
import traceback
from pprint import pprint
from graphene_django.views import GraphQLView
class ErrorLoggingGraphQLView(GraphQLView):
"""
GraphQL view that logs errors better for easier debugging.
https://github.com/graphql-python/graphene-django/issues/124#issuecomment-368283927
https://medium.com/@martin.samami/make-graphene-django-and-sentry-exceptions-work-together-f796be60a901
"""
def execute_graphql_request(self, *args, **kwargs):
"""Extract any exceptions and log them better than the default"""
result = super().execute_graphql_request(*args, **kwargs)
if result.errors:
for error in result.errors:
try:
raise error.original_error
except Exception as original_error:
pprint(original_error)
traceback.print_exc()
return resultThen in your urls.py:
from django.urls import include, path
from django.views.decorators.csrf import csrf_exempt
from your_schema_location import schema
from your_module import ErrorLoggingGraphQLView
urlpatterns = [
path(
"graphql/", csrf_exempt(ErrorLoggingGraphQLView.as_view(schema=schema, graphiql=True))
),
]Tags: bash zsh
When running a long terminal command, it would be nice to be able to do something else while it runs, but be notified when it's finished.
The following bash/zsh function rings the terminal bell when the command passed to it either finishes or fails:
function ring () {
($@ && tput bel) || tput bel
}ring could also be adapted to send a notification with a custom message, using e.g. Applescript osascript.
Alternatively, there are richer solutions, such as NotiFyre.
Tags: python python3.8
F-strings outperform every other method. Interestingly, concatenation outperforms .format.
https://twitter.com/raymondh/status/1205969258800275456
Tags: docker docker-compose
docker-compose.override.yml is read by docker-compose by default, laid on top of docker-compose.yml.
Using docker-compose.override.yml allows for use cases where, for example, an image is built locally for development, but pulled in production: the build (placed in docker-compose.override.yml) can be avoided entirely with docker-compose -f docker-compose.yml -f docker-compose.production.yml to exclude the override entirely.
More in the compose documentation.
Tags: vim neovim
Use g; and g, to quickly jump backwards and forwards through the history of cursor positions of changes.
See :h g;
Tags: html
HTML 5.2 includes <summary>, <details>, <dialog>, <meter>, and <progress> tags, providing native alternatives to many components usually supplied by libraries or frameworks.
Via AppsFlyer
Tags: javascript ES2019
ES2019 adds Array.prototype.flat, which you can use to flatten arrays to a supplied depth.
The proposal is Stage 4 along with flatMap.
Tags: python python3.5 python3.6 python3.7 python3.8 typing
New accepted or experimental typing PEPs are backported to older versions of Python that already included typing in the standard library on a provisional basis, via the typing_extensions PyPI package.
pip install typing-extensions
#!/usr/bin/python3.6
from typing_extensions import Literaltyping_extensions allows you to use e.g. types added in Python 3.8 in Python 3.7 or 3.6, for example typing.Literal.
For versions older than 3.5 (before typing was added to the standard library), there is also the typing module on PyPI to backport typing to those versions.
Tags: python python3.8 typing
PEP-589 adds support for typing dictionaries requiring a fixed set of string keys with specific value types.
Example from the Python docs:
class Point2D(TypedDict):
x: int
y: int
label: str
a: Point2D = {'x': 1, 'y': 2, 'label': 'good'} # OK
b: Point2D = {'z': 3, 'label': 'bad'} # Fails type checkTags: python python3.8 typing
PEP-586 adds support for type annotations where literal values are expected.
Example from the PEP:
from typing import Literal
def accepts_only_four(x: Literal[4]) -> None:
pass
accepts_only_four(4) # OK
accepts_only_four(19) # RejectedSee also the Python docs.
Tags: python
ast.literal_eval can be used to safely parse strings as a limited set of Python literals.
Tags: python
Yappi can profile Python across multiple threads:
https://github.com/sumerc/yappi
Tags: docker raspberrypi
Feels a little obvious: Docker images (for example on DockerHub) that were built for one processing architecture may throw errors on another one; for example on Raspberry Pis using Arm processors for x86 built images.
The solution is to build the image locally on the Pi, wherein the build will be specific to the correct architecture.
Tags: python python3.8 async
asyncio.run, a much nicer interface to managing an async event loop, is stable in Python 3.8.
async def main():
await asyncio.sleep(1)
print('hello')
asyncio.run(main())https://docs.python.org/3/whatsnew/3.8.html#asyncio
https://docs.python.org/3/library/asyncio-task.html#asyncio.run
Via @redraw
Tags: javascript frontend
You can turn a Blob or similar into a URL for download with createObjectURL.
https://developer.mozilla.org/en-US/docs/Web/API/URL/createObjectURL
https://gist.github.com/danallison/3ec9d5314788b337b682
Tags: python performance
time.perf_counter is an equivalent to time.time that has the highest possible resolution for performance measuring purposes.
Tags: vim remote
vim scp://server/file
https://medium.com/usevim/vim-101-editing-remote-files-a6d2f9c8d9fb
Tags: python
Remove cache files for all sub-directories in the current directory:
pyclean .
http://manpages.ubuntu.com/manpages/trusty/man1/py3clean.1.html
Tags: python statistics
Python has a built-in statistics module containing basic statistical functions: