Skip to content

Commit

Permalink
Drop legacy schema, and update to 2.1.66 backend (ankidroid#14171)
Browse files Browse the repository at this point in the history
* Update target API to 23

* Update to 2.1.66, and switch to new schema by default

+ Temporarily disable unused resource warnings

* Drop legacy syncing code

* Drop legacy import/export code

* Drop legacy tag handling

* Drop legacy config handling

* Drop legacy database check

* Stub out unused v1 scheduler code

Can't be fully removed, as we still need to be able to open a v1 collection
so the user can upgrade.

moveVersions test has been removed, as backend code is used for moving
outside of unit tests.

* Drop legacy deck handling

* Drop legacy search code

* Drop legacy notetype code

This breaks the card template editor, so I've temporarily disabled it
in the GUI. Getting it working again will require switching to the new
template rendering code in the new backend.

Also breaks an "empty cards" action in CardContentProvider,
as I was not sure what it was trying to accomplish.

* Drop legacy media code

This removes oakkitten's custom media checking code, as keeping it
would require keeping a bunch of the duplicated Kotlin logic. His comment
about cancellation is incorrect: the call can be cancelled with backend.
setWantsAbort() on a background thread. And since the app enforces filenames
are normalized when they're added anyway, I don't think avoiding automatic
normalization is worth the duplicated code.

* Drop legacy Collection/DB code

* Drop most references to defaultLegacySchema

* Remove legacy schema tests from CI

CI won't work yet, because backend version needs updating

* Fix import CSV call

* Remove some unused symbols from anki module

* Drop majority of old stats code

* Remove unused symbols from libanki

* Remove some broken Android tests, and fix one

checkIfStudyOptionsIsDisplayedOnTablet() is also consistently failing,
but was doing so before I started on these changes.

* Move config methods into col.config

* Simplify config API

- Use kotlinx.serialization so that arbitrary typed objects can be
(de)serialized
- There is a single generic get(), that always returns an optional
value, which will be null if the key is missing, is null, or is not the
correct type. There will always be collections that have invalid data,
so the calling code always needs to be prepared to substitute a reasonable
default in such cases.
- Expose the typed getBool() method from the backend.

* Remove unused legacy importer

* Drop jackson

* Models -> Notetypes

* Remove some more unused symbols

* Remove generic type aliases

* Rename Model -> Notetype

I had to leave ModelTest.kt's filename alone, as changing it to Notetype.kt
reorders the unit tests, and causes about 4 tests to start flaking.

* Get card template editor preview working again

* Enable undo on edits; fix test hang

* Use backend for bulk suspend/undo

By default, undoableOp() notifies all screens listening for opExecuted,
so the refreshing happens automatically, and the manual refresh code
at the end of the old routine is no longer required. It is possible to
bypass this when you want to manually update the UI, but this is probably
not worth attempting until the card browser is either switched over to
a recycler view, or reimplemented as a web component.

The LongArrays have been switched to simple lists, as Google's protobuf
code expects Iterable<Long>

* Use backend for marking/undo, and change deck/undo

* Migrate reviewer actions/undo to backend

* Migrate remainder of undo code to backend, apart from v2 undo

Couple of notes:

- The legacy v2 undo no longer returns the undone card, and instead
rebuilds the queue. In some circumstances (eg a learning card has become
due), this can result in a different card being shown after undoing.
This is not ideal, but v2 does not have long to live at this point.
- The reset/reposition tests were reusing the old card cache values,
which have to be reloaded after an operation.
- A test in AbstractSched had to be moved to ReviewerTest so it can have
access to an activity.
- UndoTest has been removed, as it mainly focused on removed code.

* GetCard() -> getNextCardAndRedraw()

* render_output -> renderOutput()

* Drop AsyncTask

Probably the hardest part of AnkiDroid's code base to follow; very
glad to see the end of it.

Closes ankidroid#7108

* Make note addition undoable

* Make bulk tag update undoable

* Remove remaining explicit transaction handling

The backend automatically wraps backend actions in a transaction,
and removing the explicit calls will allow us to drop the redundant
mutex that rsdroid acquires.

* Remove remaining unused classes/methods/properties

* Remove FunctionalInterfaces

* Switch to backend answer comparison

* Remove unused locking code + legacy storage code

* Switch ContentProviderTest to new schema, and fix bug

* Remove legacy Deck Options settings

* fix: remove minSdkVersion < 23 code

* Use short time in snackbars

People complained about snackbars interrupting their reviews before

* fix: don't duplicate undo label

It were showing a message like `Undo Undo Add Note`

* Remove "Advanced statistics"

* Remove legacy_schema local property

* Leave zip validation up to the backend

The backend takes care of validating the zip file and ensuring files aren't
written outside of the media folder.

I've expanded the mime match to include the zip mime, on the assumption
that's what the mime type is being changed to on some devices. If that
proves to be insufficient, a much simpler approach would be to look
for 50 4B 03 04 at the start of the file to determine if it's a zip file
or not.

This leaves only Mani's add-on code using commons-compression, so you
could potentially switch to a simpler tgz-specific library in the future
if that is easier.

* Disable two flaky tests

They started to flaky reliably after the legacy deck options test
was removed.

* Improve logging of HTTP requests

* Remove ModelBrowser

Replaced by ManageNotetypes

* Remove legacy CardInfo

replaced by com.ichi2.anki.pages.CardInfo

* Change comments at AdvancedSettingsFragment

* test: check if prefs analytics list don't have extra elements

* ContextCompat.getColor -> getColor

that compat method is for implementations in API < 23

* Remove more unused files

* refactor: move SECONDS_PER_DAY to a separate util file

in order to be able to remove Stats.kt later

* refactor: remove Stats and OverviewStatsBuilder

* Access sched.card directly in tests

The helper was adding a 500ms sleep on every fetch, which slowed the
scheduler tests down considerably.

* Make v3 scheduler the default; drop support for v2

The v3 scheduler was originally released in 2021, and we've been waiting
for a stable AnkiDroid release to support it before we could switch users
over to it. Now that 2.16 is out, we can finally push v3 out across the
ecosystem.

While we could theoretically make v3 the default without removing v2,
here are the reasons why I think we're better off switching to v3 only:

- AnkiWeb's review interface will likely switch over to v3-only in the
coming weeks, and AnkiMobile will likely drop v2 around the same time
2.17 comes out.
- v2 and v3 differ in a few ways that makes maintaining the two separate
paths more complicated: things like the different undo paths, counts not
including the current card, and the way the v3 scheduler supports custom
scheduler js. 2.17 is a chance to clean up a lot of old cruft in the
code base, and the old scheduling code is part of that.
- v2 and v3 are compatible with each other, and don't require a full sync
to change, so it doesn't break syncing with old clients (though depending
on settings, due counts may differ, which needs explanation)

I had to disable a couple of tests for this that we'll probably want
to restore in some form:

- corruptVersion16CollectionShowsDatabaseError() needs an update, and
it may be time to rip out the old CollectionHelper colIsOpen(), getColSafe()
and so on, migrating code that uses it over to withCol() instead. The
'collection inaccessible' dialog also needs a rethink - perhaps the various
exceptions could be handled in launchCatchingTask instead.
- testUndoResetsCardCountsToCorrectValue() is failing because initLayout()
creates a gesture listener, which fails with an error about the looper
not being initialized. I am not sure what's going on there - one option
would be to move the test to androidTest.

* Drop v2 sched file; rename files

* Drop AbstractSched and legacy undo code/queue code

* Migrate BaseSched into Scheduler

* More unused symbol removal

* Remove DeckTreeNode

* Remove unused processChildren()

* Remove shouldDisplayCounts(), as counts always available

* Remove manual hashCode()/compareTo() implementations

* Drop AbstractDeckTreeNode

* Turn DeckNode into a wrapper for DeckTreeNode

* Drop TreeNode and simplify filtering

* Remove `New card position` global preference

Overridden by per-deck configuration

Closes ankidroid#12319

* fix: remove chess.css from card_template_html

* Store current queue state in reviewer

Prerequisite for solving ankidroid#12620

Also dropped answerButtons(), as it's always 4

* Use local HTTP server for serving flashcard content

Prerequisite for solving ankidroid#12620

* Implement support for custom JS scheduling

Enables FSRS and closes ankidroid#12620

* Remove more unused code

* Remove separate Deck(Config)V16 objects and more unused code

* Move some NotetypeJson methods into its file

* Finish tidying Decks.kt

* Drop redundant col property on Collection

* Remove some usages of CollectionGetter

* Rename col->getColUnsafe to encourage migration

* Remove Reviewer.sched

* Handle deck tree not initialized in empty collection

* refactor: move syncStatus() to Collection

to avoid direct calls to the backend

* fix: do not show error when updating the menu if there is no internet

Reproduction steps:
1. have the collection synced
2. disable the device internet connection
3. Restart the app

* Remove unused resources and re-enable lint

* Restore sync cancel strings

* Defer loading webpage until server has initialized

* Ignore translations without a base entry

ankidroid#14171 (comment)

* Remove the SortOrder deprecation

* Remove the apparently-unused Backup.kt

* Remove last remaining use of Collection's context field

* Stop passing Context to backend

The library loading is now AnkiDroid's responsibility, so the backend
does not require access to an Android context.

Also remove the unused context field from Collection.

* Use backend for rendering next time labels

* Use backend for rendering finished message

The custom study & unbury descriptions are slightly different, but
should suffice until AnkiDroid can start using the Svelte finished
screen.

* Run SchedulerTest without Robolectric

Halves the run time. To do this, I implemented a new JvmTest class
and copied some of the helpers in RobolectricTest into it.

* Remove unused Kotlin implementation of template parsing

* Move Util code dealing with resources out of libanki

* Migrate remaining libanki tests to raw JVM

+ Remove some commented-out tests

Total savings are a drop from about 3m40s to 2m40s, with the scheduler
test class dropping its runtime by about half.

* Lazy-initialize col in tests

* Restore 'experimental' string

* Add comments to update-localizations.py

* Restore AcraAnalyticsInteraction.kt

---------

Co-authored-by: Brayan Oliveira <[email protected]>
  • Loading branch information
dae and BrayanDSO authored Aug 30, 2023
1 parent 4d39b5b commit 7a65160
Show file tree
Hide file tree
Showing 476 changed files with 4,822 additions and 50,359 deletions.
4 changes: 0 additions & 4 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,3 @@ updates:
- dependency-name: org.apache.commons:commons-compress
versions:
- ">= 1.12, < 1.23"
# We cannot update this one until minimum API24. Ignore range should slide with known versions so we stay informed.
- dependency-name: com.fasterxml.jackson.core:jackson-databind
versions:
- ">= 2.13.5, < 2.16"
9 changes: 1 addition & 8 deletions .github/workflows/tests_unit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,16 +39,12 @@ concurrency:

jobs:
unit:
name: JUnit Tests (${{ matrix.os}}, legacy_schema = ${{ matrix.legacy_schema }})
name: JUnit Tests (${{ matrix.os}}
timeout-minutes: 40
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
legacy_schema: [true, false]
exclude:
- os: macos-latest
legacy_schema: false
runs-on: ${{ matrix.os }}
#env:
# CODACY_TOKEN: ${{ secrets.CODACY_TOKEN }}
Expand Down Expand Up @@ -103,9 +99,6 @@ jobs:
max_attempts: 3
command: ./gradlew robolectricSdkDownload --daemon

- name: Set legacy schema property
run: echo "legacy_schema = ${{ matrix.legacy_schema }}" >> local.properties

- name: Run Unit Tests
uses: gradle/gradle-build-action@v2
with:
Expand Down
22 changes: 6 additions & 16 deletions AnkiDroid/build.gradle
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
plugins {
// Gradle plugin portal
id 'com.github.triplet.play' version '3.8.4'
id 'org.jetbrains.kotlin.plugin.serialization' version '1.9.0'
}

apply plugin: 'com.android.application'
Expand Down Expand Up @@ -46,7 +47,6 @@ android {
defaultConfig {
applicationId "com.ichi2.anki"
buildConfigField "Boolean", "CI", (System.getenv("CI") == "true").toString()
buildConfigField "Boolean", "LEGACY_SCHEMA", "true"
buildConfigField "String", "ACRA_URL", '"https://ankidroid.org/acra/report"'
buildConfigField "String", "BACKEND_VERSION", "\"$ankidroid_backend_version\""
buildConfigField "Boolean", "ENABLE_LEAK_CANARY", "false"
Expand Down Expand Up @@ -77,7 +77,7 @@ android {
// needed for upgrades to be offered correctly.
versionCode=21700101
versionName="2.17alpha1"
minSdkVersion 21
minSdkVersion 23
// change api/build.gradle
// change robolectricDownloader.gradle
// After #13695: change .tests_emulator.yml
Expand Down Expand Up @@ -113,10 +113,6 @@ android {
if (localProperties['enable_languages'] == "false") {
android.defaultConfig.resConfigs "en"
}
// allow overriding default schema version
if (localProperties["legacy_schema"] != null) {
buildConfigField "Boolean", "LEGACY_SCHEMA", localProperties["legacy_schema"]
}
// allows the scoped storage migration when the user is not logged in
if (localProperties["allow_unsafe_migration"] != null) {
buildConfigField "Boolean", "ALLOW_UNSAFE_MIGRATION", localProperties["allow_unsafe_migration"]
Expand Down Expand Up @@ -237,7 +233,6 @@ android {
kotlinOptions {
jvmTarget = JavaVersion.VERSION_11
}
ndkVersion "22.0.7026061"
}

play {
Expand Down Expand Up @@ -317,24 +312,21 @@ dependencies {
// Note: the design support library can be quite buggy, so test everything thoroughly before updating it
// Changing the version from 1.8.0 to 1.7.0 because the item in navigation drawer is getting bold unnecessarily
implementation 'com.google.android.material:material:1.7.0'
// noinspection GradleDependency jackson-databind 2.13 requires SDK 24+: https://github.com/FasterXML/jackson-databind#compatibility
implementation 'com.fasterxml.jackson.core:jackson-databind:2.13.5'
implementation 'com.vanniktech:android-image-cropper:4.5.0'
implementation 'org.nanohttpd:nanohttpd:2.3.1'
implementation "org.jetbrains.kotlinx:kotlinx-serialization-json:1.5.1"

// Backend libraries

implementation 'com.google.protobuf:protobuf-kotlin:3.24.1' // This is required when loading from a file
implementation 'com.google.protobuf:protobuf-kotlin-lite:3.24.1' // This is required when loading from a file

Properties localProperties = new Properties()
if (project.rootProject.file('local.properties').exists()) {
localProperties.load(project.rootProject.file('local.properties').newDataInputStream())
}
if (localProperties['local_backend'] == "true") {
implementation files("../../Anki-Android-Backend/rsdroid/build/outputs/aar/rsdroid-release.aar")
testImplementation files("../../Anki-Android-Backend/rsdroid-testing/build/libs/rsdroid-testing-${ankidroid_backend_version}.jar")
// On Windows, you can use something like
// implementation files("C:\\GitHub\\Rust-Test\\rsdroid\\build\\outputs\\aar\\rsdroid-release.aar")
testImplementation files("../../Anki-Android-Backend/rsdroid-testing/build/libs/rsdroid-testing.jar")
} else {
implementation "io.github.david-allison-1:anki-android-backend:$ankidroid_backend_version"
testImplementation "io.github.david-allison-1:anki-android-backend-testing:$ankidroid_backend_version"
Expand All @@ -352,10 +344,8 @@ dependencies {

implementation 'com.afollestad.material-dialogs:core:3.3.0'
implementation 'com.afollestad.material-dialogs:input:3.3.0'
// io.github.java-diff-utils:java-diff-utils is the natural successor here, but requires API24, #7091
implementation 'org.bitbucket.cowwoc:diff-match-patch:1.2'
// noinspection GradleDependency - commons-compress 1.12 - later versions use `File.toPath`; API26 can remove?
implementation 'org.apache.commons:commons-compress:1.12' // #6419 - handle >2GB apkg files
implementation 'org.apache.commons:commons-compress:1.12'
implementation 'org.apache.commons:commons-collections4:4.4' // SetUniqueList
implementation 'commons-io:commons-io:2.13.0' // FileUtils.contentEquals
implementation 'net.mikehardy:google-analytics-java7:2.0.13'
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,6 @@ class NoteEditorTabOrderTest : NoteEditorTest() {
}

private fun ensureCollectionLoaded() {
CollectionHelper.instance.getCol(targetContext)
CollectionHelper.instance.getColUnsafe(targetContext)
}
}
Loading

0 comments on commit 7a65160

Please sign in to comment.