Merge pull request #35 from ItsDrike/add-database

Add sqlalchemy

.gitignore

@@ -1,3 +1,6 @@
+# SQLite database
+database.db
+
 # Files generated by the interpreter
 __pycache__/
 *.py[cod]

README.md

@@ -60,13 +60,19 @@ The bot is configured using environment variables. You can either create a `.env
 there, or you can set / export them manually. Using the `.env` file is generally a better idea and will likely be more
 convenient.
 
-| Variable name        | Type   | Description                                                                                         |
-| -------------------- | ------ | --------------------------------------------------------------------------------------------------- |
-| `BOT_TOKEN`          | string | Bot token of the discord application (see: [this guide][bot-token-guide] if you don't have one yet) |
-| `TVDB_API_KEY`       | string | API key for TVDB (see [this page][tvdb-api-page] if you don't have one yet)                         |
-| `DEBUG`              | bool   | If `1`, debug logs will be enabled, if `0` only info logs and above will be shown                   |
-| `LOG_FILE`           | path   | If set, also write the logs into given file, otherwise, only print them                             |
-| `TRACE_LEVEL_FILTER` | custom | Configuration for trace level logging, see: [trace logs config section](#trace-logs-config)         |
+<!--
+TODO: Separate these to variables necessary to run the bot, and those only relevant during development.
+-->
+
+| Variable name          | Type   | Default       | Description                                                                                                        |
+| ---------------------- | ------ | ------------- | ------------------------------------------------------------------------------------------------------------------ |
+| `BOT_TOKEN`            | string | N/A           | Bot token of the discord application (see: [this guide][bot-token-guide] if you don't have one yet)                |
+| `TVDB_API_KEY`         | string | N/A           | API key for TVDB (see [this page][tvdb-api-page] if you don't have one yet)                                        |
+| `SQLITE_DATABASE_FILE` | path   | ./database.db | Path to sqlite database file, can be relative to project root (if the file doesn't yet exists, it will be created) |
+| `ECHO_SQL`             | bool   | 0             | If `1`, print out every SQL command that SQLAlchemy library runs internally (can be useful when debugging)         |
+| `DEBUG`                | bool   | 0             | If `1`, debug logs will be enabled, if `0` only info logs and above will be shown                                  |
+| `LOG_FILE`             | path   | N/A           | If set, also write the logs into given file, otherwise, only print them                                            |
+| `TRACE_LEVEL_FILTER`   | custom | N/A           | Configuration for trace level logging, see: [trace logs config section](#trace-logs-config)                        |
 
 [bot-token-guide]: https://guide.pycord.dev/getting-started/creating-your-first-bot#creating-the-bot-application
 [tvdb-api-page]: https://www.thetvdb.com/api-information

alembic-migrations/README.md

@@ -0,0 +1,93 @@
+<!-- vi: tw=119
+-->
+
+# Welcome to `alembic-migrations` directory
+
+This directory contains all of our database migrations.
+
+## What are database migrations?
+
+In case you aren't familiar, a database migration is essentially just a set of SQL instructions that should be
+performed, to get your database into the state that we expect it to be in.
+
+The thing is, as software changes, the requirements for the database structure change alongside with it and that means
+that anyone who would like to update this application to a newer version will also need to find a way to get their
+database up to date with any changes that were made.
+
+If people had to do this manually, it would mean going through diffs and checking what exactly changed in the relevant
+files, then using some tool where they can run SQL commands, figuring out what commands to even run to properly get
+everything up to date without losing any existing information and finally actually running them.
+
+Clearly, this isn't ideal, especially for people who just want to use this bot as it's being updated and don't know
+much about SQL and databases. For that reason, we decided to instead keep the instructions for these migrations in
+individual version files, which people can simply execute to get their database up to date (or even to downgrade it,
+in case someone needs to test out an older version of the bot).
+
+## How to use these migrations?
+
+We're using [`alembic`](https://alembic.sqlalchemy.org/en/latest/index.html), which is a tool that makes generating and
+applying migrations very easy. Additionally, we have some custom logic in place, to make sure that all migrations that
+weren't yet applied will automatically be ran once the application is started, so you don't actually need to do
+anything to apply them.
+
+That said, if you would want to apply the migrations manually, without having to start the bot first, you can do so
+with the command below:
+
+```bash
+alembic upgrade head
+```
+
+This will run all of the migrations one by one, from the last previously executed migrations (if you haven't run the
+command before, it will simply run each one). Alembic will then keep track of the revision id (basically the specific
+migration file) that was applied and store that id into your database (Alembic will create it's own database table for
+this). That way, alembic will always know what version is your current database at.
+
+> [!TIP]
+> You can also run `alembic check`, to see if there are any pending migrations that you haven't yet applied.
+
+## How to create migrations? (for developers)
+
+If you're adding a new database table, deleting it, or just changing it somehow, you will want to create a new
+migration file for it. Thankfully, alembic makes this very easy. All you need to do is run:
+
+```bash
+alembic revision --autogenerate -m "Some message (e.g.: Added users table)"
+```
+
+Alembic will actually load the python classes that represent all of the tables and compare that with what you currently
+have in the database, automatically generating all of the instructions that need to be ran in a new migration script.
+This script will be stored in `alembic-migrations/versions/` directory.
+
+Note that after you did this, you will want to apply the migrations. You can do that by simply running the bot for a
+while, to let the custom logic we have in place run alembic migrations for you, or you can run them manually with
+`alembic upgrade head`.
+
+### Manual migrations
+
+In most cases, running the command to auto-generate the migration will be all that you need to do.
+
+That said, alembic has it's limitations and in some cases, the automatic generation doesn't work, or doesn't do what
+we'd like it to do. For example, if you rename a table, alembic can't understand that this was a rename, rather than a
+deletion of one table and a creation of another. This is a problem, because instead of simply renaming while keeping
+the old existing data, alembic will generate instructions that would lead to losing those old data.
+
+In these cases, you will need to do some extra work and edit the migration files yourself. In case auto-generation
+fails completely, you can run the same command without that `--autogenerate` flag, which will generate an empty
+migration file, that you'll need to fill out.
+
+That said, in vast majority of cases, you will not need to write your migrations manually. For more info on when you
+might need to, check [the documentation][alembic-autogeneration-autodetection].
+
+[alembic-autogeneration-autodetection]: https://guide.pycord.dev/getting-started/creating-your-first-bot#creating-the-bot-application
+
+### Stamping
+
+In case you've made modifications to your database already (perhaps by manually running some SQL commands to test out a
+manually written migration), you might want to skip applying a migration and instead just tell Alembic that the
+database is already up to date with the latest revision.
+
+Thankfully, alembic makes this really simple, all you need to do is run:
+
+```bash
+alembic stamp head
+```

alembic-migrations/env.py

@@ -0,0 +1,83 @@
+import asyncio
+
+from alembic import context
+from sqlalchemy import MetaData, engine_from_config, pool
+from sqlalchemy.engine import Connection
+from sqlalchemy.ext.asyncio import AsyncEngine
+
+from src.utils.database import Base, SQLALCHEMY_URL, load_db_models
+from src.utils.log import get_logger
+
+# Obtain a custom logger instance
+# This will also set up logging with our custom configuration
+log = get_logger(__name__)
+
+# This is the Alembic Config object, which provides access to the values within the .ini file in use.
+config = context.config
+
+
+def run_migrations_offline(target_metadata: MetaData) -> None:
+    """Run migrations in 'offline' mode.
+
+    This configures the context with just a URL and not an Engine, though an Engine is acceptable
+    here as well. By skipping the Engine creation we don't even need a DBAPI to be available.
+
+    Calls to context.execute() here emit the given string to the script output.
+    """
+    url = config.get_main_option("sqlalchemy.url")
+    context.configure(
+        url=url,
+        target_metadata=target_metadata,
+        literal_binds=True,
+        dialect_opts={"paramstyle": "named"},
+    )
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+async def run_migrations_online(target_metadata: MetaData) -> None:
+    """Run migrations in 'online' mode.
+
+    In this scenario we need to create an Engine and associate a connection with the context.
+    """
+
+    def do_run_migrations(connection: Connection) -> None:
+        context.configure(connection=connection, target_metadata=target_metadata)
+
+        with context.begin_transaction():
+            context.run_migrations()
+
+    configuration = config.get_section(config.config_ini_section)
+    if configuration is None:
+        raise RuntimeError("Config ini section doesn't exists (should never happen)")
+
+    connectable = AsyncEngine(
+        engine_from_config(
+            configuration,
+            prefix="sqlalchemy.",
+            poolclass=pool.NullPool,
+            future=True,
+        )
+    )
+
+    async with connectable.connect() as connection:
+        await connection.run_sync(do_run_migrations)
+
+    await connectable.dispose()
+
+
+def main() -> None:
+    """Main entry point function."""
+    config.set_main_option("sqlalchemy.url", SQLALCHEMY_URL)
+    load_db_models()
+
+    target_metadata = Base.metadata
+
+    if context.is_offline_mode():
+        run_migrations_offline(target_metadata)
+    else:
+        asyncio.run(run_migrations_online(target_metadata))
+
+
+main()

alembic-migrations/script.py.mako

@@ -0,0 +1,25 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+${imports if imports else ""}
+# revision identifiers, used by Alembic.
+revision: str = ${repr(up_revision)}
+down_revision: str | None = ${repr(down_revision)}
+branch_labels: str | Sequence[str] | None = ${repr(branch_labels)}
+depends_on: str | Sequence[str] | None = ${repr(depends_on)}
+
+
+def upgrade() -> None:
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade() -> None:
+    ${downgrades if downgrades else "pass"}

alembic-migrations/versions/2024_07_21_1854-c55da3c62644_initial_migration.py

@@ -0,0 +1,26 @@
+"""Initial migration
+
+Revision ID: c55da3c62644
+Revises:
+Create Date: 2024-07-21 18:54:26.159716
+"""
+
+from collections.abc import Sequence
+
+# revision identifiers, used by Alembic.
+revision: str = "c55da3c62644"
+down_revision: str | None = None
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    pass
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    pass
+    # ### end Alembic commands ###

alembic.ini

@@ -0,0 +1,118 @@
+# A generic, single database configuration.
+
+[alembic]
+# path to migration scripts
+# Use forward slashes (/) also on windows to provide an os agnostic path
+script_location = alembic-migrations
+
+# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
+# Uncomment the line below if you want the files to be prepended with date and time
+# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
+# for all available tokens
+file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
+
+# sys.path path, will be prepended to sys.path if present.
+# defaults to the current working directory.
+prepend_sys_path = .
+
+# timezone to use when rendering the date within the migration file
+# as well as the filename.
+# If specified, requires the python>=3.9 or backports.zoneinfo library.
+# Any required deps can installed by adding `alembic[tz]` to the pip requirements
+# string value is passed to ZoneInfo()
+# leave blank for localtime
+# timezone =
+
+# max length of characters to apply to the "slug" field
+# truncate_slug_length = 40
+
+# set to 'true' to run the environment during
+# the 'revision' command, regardless of autogenerate
+# revision_environment = false
+
+# set to 'true' to allow .pyc and .pyo files without
+# a source .py file to be detected as revisions in the
+# versions/ directory
+# sourceless = false
+
+# version location specification; This defaults
+# to alembic-migrations/versions.  When using multiple version
+# directories, initial revisions must be specified with --version-path.
+# The path separator used here should be the separator specified by "version_path_separator" below.
+# version_locations = %(here)s/bar:%(here)s/bat:alembic-migrations/versions
+
+# version path separator; As mentioned above, this is the character used to split
+# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
+# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
+# Valid values for version_path_separator are:
+#
+# version_path_separator = :
+# version_path_separator = ;
+# version_path_separator = space
+version_path_separator = os  # Use os.pathsep. Default configuration used for new projects.
+
+# set to 'true' to search source files recursively
+# in each "version_locations" directory
+# new in Alembic version 1.10
+# recursive_version_locations = false
+
+# the output encoding used when revision files
+# are written from script.py.mako
+# output_encoding = utf-8
+
+# Connection string to the database to perform migrations on
+# **This is a placeholder, we obtain the real value dynamically from the environment**
+sqlalchemy.url = driver://user:pass@localhost/dbname
+
+
+[post_write_hooks]
+# post_write_hooks defines scripts or Python functions that are run
+# on newly generated revision scripts.  See the documentation for further
+# detail and examples
+# https://alembic.sqlalchemy.org/en/latest/autogenerate.html#basic-post-processor-configuration
+
+hooks = ruff-autofix, ruff-format
+
+ruff-autofix.type = exec
+ruff-autofix.executable = %(here)s/.venv/bin/ruff
+ruff-autofix.options = check --fix REVISION_SCRIPT_FILENAME
+
+ruff-format.type = exec
+ruff-format.executable = %(here)s/.venv/bin/ruff
+ruff-format.options = format REVISION_SCRIPT_FILENAME
+
+
+# Logging configuration
+[loggers]
+keys = root,sqlalchemy,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARN
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARN
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = INFO
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S