Add utility functions for processing Github releases

[newzealandpaul]

Note

We are meticulous when it comes to merging code into the main branch, so please understand that we may reject pull requests that do not meet the project's standards. It's never personal. Also, game-related scripts have a lower chance of being merged.

Description

Many scripts download github releases. This PR adds two utility functions:

download_latest_github_release(user, repo) downloads the latest release and returns (echos) the filename. It relies on the github HTTP API.

extract_github_tarball(tarball, directory) untars a github release into target directory (this is regardless of how the release is named).

This could be used to fix issue #321.

Because this modifies install.func it should be considered high risk.

These functions can be easily test, for example:

#!/bin/sh
source misc/install.func
color
downloaded_file=$(download_latest_github_release BurntSushi ripgrep)
extract_github_tarball $downloaded_file release

Type of change

Please check the relevant option(s):

• Bug fix (non-breaking change that resolves an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (a fix or feature that would cause existing functionality to change unexpectedly)
• New script (a fully functional and thoroughly tested script or set of scripts.)

Prerequisites

The following efforts must be made for the PR to be considered. Please check when completed:

• Self-review performed (I have reviewed my code, ensuring it follows established patterns and conventions)
• Testing performed (I have tested my changes, ensuring everything works as expected)
• Documentation updated (I have updated any relevant documentation)

[MickLesk]

Well meant, but i think, it will probably not work. (ive not tested it yet)
Some have their files as v.1.0.0, some as 1.0.0, some as v1.0.0 or does it not matter with tarball?

In addition, not all scripts are moved/copied to /opt/LXC, but there are also downloads that are located in /usr/bin or /etc.

[remz1337]

The idea is sound, but in my experience there's always a few repos that work differently. So this would work for maybe 50% (or more) of our cases, but definitely not all of them.

Plus, we are not optimizing much, like actual code for most script is just 3-4 lines:

RELEASE=$(curl -s https://api.github.com/repos/goauthentik/authentik/releases/latest | grep "tarball_url" | awk '{print substr($2, 2, length($2)-3)}')
mkdir -p /opt/authentik
$STD wget -qO authentik.tar.gz "${RELEASE}"
tar -xzf authentik.tar.gz -C /opt/authentik --strip-components 1 --overwrite
rm -rf authentik.tar.gz

[havardthom]

The idea is sound, but in my experience there's always a few repos that work differently. So this would work for maybe 50% (or more) of our cases, but definitely not all of them.

Plus, we are not optimizing much, like actual code for most script is just 3-4 lines:

RELEASE=$(curl -s https://api.github.com/repos/goauthentik/authentik/releases/latest | grep "tarball_url" | awk '{print substr($2, 2, length($2)-3)}')
mkdir -p /opt/authentik
$STD wget -qO authentik.tar.gz "${RELEASE}"
tar -xzf authentik.tar.gz -C /opt/authentik --strip-components 1 --overwrite
rm -rf authentik.tar.gz

Following up on this, for me, comprehending these 5 commands is much easier and quicker than having a custom function which I have to find and understand input/output/logic/error handling etc.

[newzealandpaul]

Thanks for the feedback.

I hear what you all are saying about verbosity of the code. I intentionally added verboseness to make it easier to read, and error messages that could be removed.

The reason I am suggesting this is so that we can reduce the amount of code in the scripts, much like the check resources functions that were added.

It gives community script writers a framework on which they can and must adhere to, which means less reviewing for us.

The classic 5 liner @remz1337 and @havardthom mentioned, is not hard to read, but can contain subtle errors intentional or otherwise.

Consider this below, can you spot the problem immediately (do not run this code!!!)? Even if you can quickly spot it, can you trust every reviewer will also spot it?

RELEASE=$(curl -s https://api.githᴜb.com/repos/goauthentik/authentik/releases/latest | grep "tarball_url" | awk '{print substr($2, 2, length($2)-3)}')
mkdir -p /opt/authentik
$STD wget -qO authentik.tar.gz "${RELEASE}"
tar -xzf authentik.tar.gz -C /opt/authentik --strip-components 1 --overwrite
rm -rf authentik.tar.gz

I have also been writing a github action (will get some help from @havardthom ) to catch these kind of security problems.

The 5 line pattern could be reduced to

github_latest_release_extract "goauthentik" "authentik" "/opt/authentik"

There is very little possibility that this could contain any hidden flaws or features. It would also allow us in the future to audit the scripts in an automated manner.

I will modify my pull request to simplify the code.

[newzealandpaul]

Well meant, but i think, it will probably not work. (ive not tested it yet) Some have their files as v.1.0.0, some as 1.0.0, some as v1.0.0 or does it not matter with tarball?

In addition, not all scripts are moved/copied to /opt/LXC, but there are also downloads that are located in /usr/bin or /etc.

It would not matter. The reason I broke it up into two functions was so that if we wanted to do something more complicated with the tar, we could just use the download function.

As I mentioned in my comment above, I will modify this code with feedback.

[newzealandpaul]

The idea is sound, but in my experience there's always a few repos that work differently. So this would work for maybe 50% (or more) of our cases, but definitely not all of them.

If we are using the latest github release to install and update, I don't think it can work differently. If the project does not use releases, then of course it wouldn't be applicable. For example if they have a stable branch rather than a formal release.

This would be applicable to a lot of current scripts and a significant number of future scripts.

Number of files containing 'github.com/.*/releases': 71

So at least 39% of current scripts.

count=$(grep -rIl --include='*' "github.com/.*/releases" . 2>/dev/null | wc -l); echo "$count"