Module-ify and add some type hinting; move to async sqlite3 lib (#572)

* relocate all files into an actual overarching python module
this facilitates packaging and relative imports
also fixes some await issues and adds some type hints

* rename main and add some hints/format strings

* linting and hinting

* aaaaa

* lint

* docs: reformat readme and update a few things

* add script for running the bot and install uvloop in actual main function

* fstrings

* typing/linting

* type hinting work

* linting / code style / small fixes to strings

* hinting

* add aiosqlite and mypy typings for (optional) typechecking

* relocate config & remove configurable schema location

* oop

* add new base cog which factors out some common functionality
use it in basic cogs that dont need additional changes

* relocate schema

* add package manifest

* docs: update info on config

* rewrite bot.py with async database + some code style changes

* rewrite main.py with aiosqlite and code style changes

* rewrite banner cog with aiosqlite and code style changes

* rewrite currency cog with aiosqlite and code style changes

* rewrite customreactions cog with aiosqlite and refactoring

* rewrite games cog with new base cog

* rewrite helpers cog and images cog with new base cog

* rewrite mod cog with aiosqlite and new base cog

* change music cog to use new base cog

* change quote cog to use aiosqlite and new base cog

* change reminder cog to use new base cog

* make add_member_if_needed util async

* change score cog to new base cog + aiosqlite

* comment

* move Reminder.check_reminders to aiosqlite

* convert mod cog to use aiosqlite

* convert reminder cog to use aiosqlite

* type hints for roles cog

* remove unused import in score cog

* cleanup in role check utils

* add settings key del function in base cog

* convert games cog to aiosqlite

* convert helpers cog to aiosqlite

* convert role restoration to aiosqlite

* fix fetch_saved_roles not being async

* respond with error msg if keydates encounters an http error

* add error message if tex rendering fails

* linting and autoformatting fixes

* type fixes

* lint

* fix some typing complaints

* fix another typing complaint

* add bot health check function which checks if guild ID is set right

* switch subscribers cog to new base cog
fix guild errors
switch to async fetch for metro status

* change: switched to using lxml parser instead of python html.parser when invoking using bs4. (#573)

* fix STM subscriber (?) by disabling content type checking in custom requests

* add hashbang to hangman

* mild refactor of customreactions cog

* feat: higher level abstraction for fetching lists

* lint

* score code style

* score reformatting

* fix some funky strings

* add a fetch_one higher level fn to base cog
do a bit of reminder refactoring

* clarifying parens

* more reminders refactoring + lint

* add (de)serialization to settings key methods; some related refactoring

* f string

* chore: more typing

* chore: update dev dependencies & lockfile

* lint

* lint&hint

* banner cog cleanup

* more banner cleanup

* currency cleanup

* a bit of spacing

* lint

* update bot.py year

* redo config class

* type hint context in main.load fn

* update dependencies, require poetry 1.4+

* update python & actions ver

* new config integration work

* single year range var for course url templates

* new config: impl bet roll

* ignore poetry toml

* add example env file

* fix supported python ver

* ci: remove old ubuntus from matrix

* chore [noci]: update copyright

* chore: update more deps

* chore: update pillow

* lint

* lint

* fix: defaults for currency and music nested settings

* fix: setting log levels with new config

* fix: developer/moderator role checks

* chore: port image config to new config object

* fix: missing await for db fetchone

* chore: port games config to new config model

* fix: muted role checks

* fix: music config model default start vol value

* chore: port legacy assignable roles config to new config model

* ci: tweak formatting check

* fix: music config access

* chore: update discordpy to 1.7.3

* fix: type hints causing bad conversions in currency cog

* fix: type hints causing bad conversions in memes and roles cogs

* chore: add log webhook env vars to example env

* chore: remove old config class

* fix: listing all roles not working

* example env wording

* [fix]: fixed field name in example.env

* update course year range to 24-25

* steal #519

* update dependencies

* dockerfile work

* some paginator type hinting

* fix add_quote full name

* fix some member converters in currency/quotes

* custom reactions elifs

* lint: run black

* remove old config.ini

* docs: update config / deployment details

* ci: update actions

* chore: clear apt lists + combine apt RUN statements

* chore(deps): update uvloop

* ci: build an edge (master-branch) image too

---------

Co-authored-by: namemcguffin <[email protected]>
Co-authored-by: Mathieu B <[email protected]>

.dockerignore

@@ -1,4 +1,6 @@
-config/config.ini
+canary/config/config.ini
+data/runtime/*.db
+data/runtime/*.obj
 env/
 tests/
 venv/

.github/workflows/formatting_check.yml

@@ -3,15 +3,14 @@ name: formatting check
 on:
   push:
     paths:
-      "**.py"
+      - "**.py"
 
 jobs:
   lint:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
-      - uses: actions/setup-python@v2
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
       - uses: psf/black@stable
         with:
-          options: "--diff --check -t py310 --fast"
-
+          options: "--diff --check -t py311 canary"

.github/workflows/poetry-dev.yml

@@ -24,18 +24,16 @@ jobs:
     strategy:
       matrix:
         os:
-          - 'ubuntu-20.04'
-          - 'ubuntu-18.04'
-          # - 'macos-10.15'
-          - 'macos-11.0'
-          - 'windows-2019'
+          - 'ubuntu-24.04'
+          - 'macos-14.0'
+          - 'windows-2022'
         python-version:
-          - '3.10'
+          - '3.11'
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
 
       - name: Set up Python
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
 
@@ -50,4 +48,3 @@ jobs:
 
       - name: install canary development dependencies
         run: poetry install --no-interaction
-

.github/workflows/poetry-prod.yml

@@ -17,15 +17,14 @@ jobs:
     strategy:
       matrix:
         os:
-          - 'ubuntu-20.04'
-          - 'ubuntu-18.04'
+          - 'ubuntu-24.04'
         python-version:
-          - '3.10'
+          - '3.11'
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
 
       - name: Set up Python
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
 
@@ -40,4 +39,3 @@ jobs:
 
       - name: install canary production dependencies
         run: poetry install --no-dev --no-interaction
-

.github/workflows/publish-docker.yml

@@ -4,34 +4,64 @@ on:
   release:
     types: [published]
 
+  push:
+    branches:
+      - master
+
 jobs:
   publish:
     runs-on: ubuntu-latest
     permissions:
       packages: write
       contents: read
     steps:
-      - uses: actions/checkout@v2
-      - uses: actions/setup-python@v2
+      - uses: actions/checkout@v4
+
+      - name: Set up master (edge) image metadata
+        id: meta-master
+        uses: docker/metadata-action@v5
+        if: ${{ github.event_name != 'release' }}
         with:
-          python-version: '3.10'
-      - run: pip install poetry && poetry update && poetry export -f requirements.txt > requirements.txt
-      - uses: docker/metadata-action@v3
-        id: meta
+          images: |
+            ${{ inputs.image-name }}
+          flavor: |
+            latest=false
+          tags: |
+            type=raw,value=master,enable={{is_default_branch}}
+            type=sha,prefix=sha-
+
+      - name: Set up release image metadata
+        id: meta-release
+        uses: docker/metadata-action@v5
+        if: ${{ github.event_name == 'release' }}
         with:
           images: ghcr.io/idoneam/Canary
+          flavor: |
+            latest=true
           tags: |
             type=semver,pattern={{major}}
             type=semver,pattern={{major}}.{{minor}}
             type=semver,pattern={{version}}
-      - uses: docker/setup-buildx-action@v1
-      - uses: docker/login-action@v1
+
+      - uses: docker/setup-buildx-action@v3
+
+      - uses: docker/login-action@v3
         with:
           registry: ghcr.io
-          username: ${{ github.repository_owner }}
-          password: ${{ github.token }}
-      - uses: docker/build-push-action@v2
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build and push master (edge) image
+        uses: docker/build-push-action@v6
+        if: ${{ github.event_name != 'release' }}
+        with:
+          context: .
+          push: true
+          tags: ${{ steps.meta-master.outputs.tags }}
+
+      - uses: docker/build-push-action@v6
+        if: ${{ github.event_name == 'release' }}
         with:
           context: .
           push: true
-          tags: ${{ steps.meta.outputs.tags }}
+          tags: ${{ steps.meta-release.outputs.tags }}

.gitignore

@@ -103,6 +103,7 @@ ENV/
 
 # configuration files
 **/*.ini
+poetry.toml
 
 # other
 *~

Dockerfile

@@ -1,28 +1,38 @@
-FROM python:3.10-slim-bullseye
+FROM python:3.11-slim-bookworm
 
-# Install base apt dependencies
-RUN apt-get update && apt-get install -y git sqlite3
+# Install base apt dependencies + auxiliary dependencies (for GL, Tex, etc.)
+RUN apt-get update && \
+    apt-get install -y  \
+      git \
+      sqlite3 \
+      libgl1-mesa-glx \
+      texlive-latex-extra \
+      texlive-fonts-extra \
+      texlive-lang-greek \
+      dvipng \
+      ffmpeg \
+      gcc && \
+    rm -rf /var/lib/apt/lists/*
 
-# Install auxiliary dependencies (for GL, Tex, etc.)
-RUN apt-get install -y \
-  libgl1-mesa-glx \
-  texlive-latex-extra \
-  texlive-fonts-extra \
-  texlive-lang-greek \
-  dvipng \
-  ffmpeg \
-  gcc
+# Update pip, install poetry
+RUN pip install --no-cache-dir -U pip; \
+    pip install --no-cache-dir poetry==1.8.3
 
-# Install requirements with pip to use Docker cache independent of project metadata
-COPY requirements.txt /
-RUN pip install -r /requirements.txt
+COPY pyproject.toml .
+COPY poetry.lock .
+RUN poetry config virtualenvs.create false && \
+    poetry --no-cache install --no-root --without dev
 
-# Copy code to the `canary` directory in the image and run the bot from there
-COPY . /canary
-WORKDIR /canary
+# Copy code and pre-made data to the /app directory, where the bot will be run from
+WORKDIR /app
+COPY canary canary
+COPY data data
+
+RUN pip install -e .
 
 # Notes:
-#   Users will have to mount their config.ini in by hand
-#   Users should mount a read/writable volume for /canary/data/runtime
+#   Users will have to configure their instance using environment variables, described by the Pydantic settings object
+#     in /app/canary/config/config.py
+#   Users should mount a read/writable volume for /app/data/runtime
 
-CMD ["python3.10", "Main.py"]
+CMD ["canary"]

MANIFEST.in

@@ -0,0 +1 @@
+include canary/Martlet.schema