-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* mysql WIP * mysql WIP * WIP tokenize fully tested except for unicode * WIP tokenize fully tested * token strip * mysql prefixed literals * Added Skip() * mysql WIP * first pass of mysql tests * mysql: check test * mysql skip helpers * remove remaining idempotent reference * code review typos * add some comments in reaction to Aaron's comments * add some comments in reaction to Aaron's comments * mysql: add missing file: skip.go with utility functions * mysql: add github actions CI for mysql (#4) * mysql: add github action * mysql: github action: fix * mysql: github action: fix2 * mysql: github action: fix3 * sqltoken is now its own repo * update deps * update test name
- Loading branch information
Showing
14 changed files
with
981 additions
and
27 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| name: Run Go tests | ||
| name: Go tests | ||
| on: [ push ] | ||
| jobs: | ||
| Build-and-test: | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| name: MySQL tests | ||
| on: [ push ] | ||
|
|
||
| jobs: | ||
| Test-mysql-integration: | ||
| runs-on: ubuntu-latest | ||
|
|
||
| services: | ||
| mysql: | ||
| image: mysql:8 | ||
| env: | ||
| MYSQL_DATABASE: libschematest | ||
| MYSQL_ROOT_PASSWORD: mysql | ||
| options: >- | ||
| --health-cmd "mysqladmin ping" | ||
| --health-interval 10s | ||
| --health-timeout 5s | ||
| --health-retries 5 | ||
| ports: | ||
| - 3306:3306 | ||
|
|
||
| steps: | ||
| - name: Check out repository code | ||
| uses: actions/checkout@v2 | ||
|
|
||
| - name: Set up Go | ||
| uses: actions/setup-go@v2 | ||
| with: | ||
| go-version: 1.16 | ||
|
|
||
| - name: Build | ||
| run: go build -v ./... | ||
|
|
||
| - name: Test | ||
| env: | ||
| LIBSCHEMA_MYSQL_TEST_DSN: "root:mysql@tcp(127.0.0.1:3306)/libschematest?tls=false" | ||
| run: go test ./lsmysql/... -v | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
|
|
||
| # libschema/lsmysql - mysql support for libschema | ||
|
|
||
| [](https://pkg.go.dev/github.com/muir/libschema/lsmysql) | ||
|
|
||
| Install: | ||
|
|
||
| go get github.com/muir/libschema | ||
|
|
||
| --- | ||
|
|
||
| ## DDL Transactions | ||
|
|
||
| MySQL and MariaDB do not support DDL (Data Definition Language) transactions like | ||
| `CREATE TABLE`. Such commands cause the current transaction to switch to `autocommit` | ||
| mode. | ||
|
|
||
| The consequence of this is that it is not possible for a schema migration tool, | ||
| like libschema, to track if a migration has been applied or not by tracking the status | ||
| of a transaction. | ||
|
|
||
| When working with MySQL and MariaDB, schema-changing migrations should be done | ||
| separately from data-changing migrations. Schema-changing transactions that are | ||
| idempotent are safe and require no special handling. | ||
|
|
||
| Schema-changing transactions that are not idempotent need to be guarded with conditionals | ||
| so that they're skipped if they've already been applied. | ||
|
|
||
| Fortunately, `IF EXISTS` and `IF NOT EXISTS` clauses can be most of the DDL statements. | ||
|
|
||
| ### Conditionals | ||
|
|
||
| The DDL statements missing `IF EXISTS` and `IF NOT EXISTS` include: | ||
|
|
||
| ```sql | ||
| ALTER TABLE ... | ||
| ADD CONSTRAINT | ||
| ALTER COLUMN SET SET DEFAULT | ||
| ALTER COLUMN SET DROP DEFAULT | ||
| ADD FULLTEXT | ||
| ADD SPATIAL | ||
| ADD PERIOD FOR SYSTEM TIME | ||
| ADD {INDEX|KEY} index_name [NOT] INVISIBLE | ||
| DROP PRIMARY KEY | ||
| RENAME COLUMN | ||
| RENAME INDEX | ||
| RENAME KEY | ||
| DISCARD TABLESPACE | ||
| IMPORT TABLESPACE | ||
| COALESCE PARTITION | ||
| REORGANIZE PARTITION | ||
| EXCHANGE PARTITION | ||
| REMOVE PARTITIONING | ||
| DISABLE KEYS | ||
| ENABLE KEYS | ||
| ``` | ||
|
|
||
| To help make these conditional, the lsmysql provides some helper functions to easily | ||
| check the current database state. | ||
|
|
||
| For example: | ||
|
|
||
| ```go | ||
| schema := libschema.NewSchema(ctx, libschema.Options{}) | ||
|
|
||
| sqlDB, err := sql.Open("mysql", "....") | ||
|
|
||
| database, mysql, err := lsmysql.New(logger, "main-db", schema, sqlDB) | ||
|
|
||
| database.Migrations("MyLibrary", | ||
| lsmysql.Script("createUserTable", ` | ||
| CREATE TABLE users ( | ||
| name text, | ||
| id bigint, | ||
| PRIMARY KEY (id) | ||
| ) ENGINE=InnoDB` | ||
| }), | ||
| lsmysql.Script("dropUserPK", ` | ||
| ALTER TABLE users | ||
| DROP PRIMARY KEY`, | ||
| libschema.SkipIf(func() (bool, error) { | ||
| hasPK, err := mysql.HasPrimaryKey("users") | ||
| return !hasPK, err | ||
| })), | ||
| ) | ||
| ``` | ||
|
|
||
| ### Some notes on MySQL | ||
|
|
||
| While most identifiers (table names, etc) can be `"`quoted`"`, you cannot use quotes around | ||
| a schema (database really) name with `CREATE SCHEMA`. | ||
|
|
||
| MySQL does not support schemas. A schema is just a synonym for `DATABASE` in the MySQL world. | ||
| This means that it is easier to put migrations tracking table in the same schema (database) as | ||
| the rest of the tables. It also means that to run migration unit tests, the DSN for testing | ||
| has to give access to a user that can create and drop databases. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| package lsmysql | ||
|
|
||
| import ( | ||
| "regexp" | ||
| "strings" | ||
|
|
||
| "github.com/muir/sqltoken" | ||
| ) | ||
|
|
||
| type CheckResult string | ||
|
|
||
| const ( | ||
| Safe CheckResult = "safe" | ||
| DataAndDDL = "dataAndDDL" | ||
| NonIdempotentDDL = "nonIdempotentDDL" | ||
| ) | ||
|
|
||
| var ifExistsRE = regexp.MustCompile(`(?i)\bIF (?:NOT )?EXISTS\b`) | ||
|
|
||
| // CheckScript attempts to validate that an SQL command does not do | ||
| // both schema changes (DDL) and data changes. | ||
| func CheckScript(s string) CheckResult { | ||
| var seenDDL int | ||
| var seenData int | ||
| var idempotent int | ||
| ts := sqltoken.TokenizeMySQL(s) | ||
| for _, cmd := range ts.Strip().CmdSplit() { | ||
| word := strings.ToLower(cmd[0].Text) | ||
| switch word { | ||
| case "alter", "rename", "create", "drop", "comment": | ||
| seenDDL++ | ||
| if ifExistsRE.MatchString(cmd.String()) { | ||
| idempotent++ | ||
| } | ||
| case "truncate": | ||
| seenDDL++ | ||
| idempotent++ | ||
| case "use", "set": | ||
| // neither | ||
| case "values", "table", "select": | ||
| // doesn't modify anything | ||
| case "call", "delete", "do", "handler", "import", "insert", "load", "replace", "update", "with": | ||
| seenData++ | ||
| } | ||
| } | ||
| if seenDDL > 0 && seenData > 0 { | ||
| return DataAndDDL | ||
| } | ||
| if seenDDL > idempotent { | ||
| return NonIdempotentDDL | ||
| } | ||
| return Safe | ||
| } |
Oops, something went wrong.