Consider adding "Progress sync" kosync / KoReader Sync support

[clach04]

When viewing a book (not the book/library browser) under the tools/settings menu here is a "Progress sync" sub menu.
Config defaults to "Binary. Only identical files will be kept in sync." (versus" Filename. Files with matching names will be kept in...")

• Progress Sync is not enabled by default, needs a login (or login creation).
• Then either manually sync (push/pull) or auto sync (which requires setting options for when)
• One setup, default is to sync using partial md5 checksum of file contents, can change this from default of "binary" (partial) to "filename"

Resources and notes

• https://github.com/koreader/koreader/wiki/Progress-sync - User docs
• https://github.com/koreader/koreader-sync-server - reference server implementation
  • https://github.com/koreader/koreader-sync-server/blob/master/spec/controllers/1/syncs_controller_spec.lua - tests?
• Plugin for Progress Sync, kosync - shipped/included with KoReader
  • https://github.com/koreader/koreader/tree/master/plugins/kosync.koplugin Progess Sync plugin
  • https://github.com/koreader/koreader/blob/master/plugins/kosync.koplugin/api.json API
• https://github.com/harmtemolder/koreader-calibre-plugin - A calibre plugin to synchronize metadata from KOReader to calibre .
• Implementations
  • https://github.com/topics/koreader-sync
  • https://github.com/clach04?submit=Search&q=koreader+sync&tab=stars&type=&sort=&direction=&submit=Search

API

API - Hashing

password is ran run plain md5sum.

Parts of ebook file content are ran through plain md5sum (default, "Binary" option) OR filename ran through md5 (non-default, "Filename" Progress Sync option) is used as document key:

Binary option - partial file contents, https://github.com/koreader/koreader/blob/master/frontend/apps/reader/readerui.lua look for util.partialMD5() which is then stored in meta data for document/book.file partial_md5_checksum in KoReader.

https://github.com/koreader/koreader/blob/master/frontend/util.lua has code implementation for partialMD5(), code unchanged for a while, snapshot from 2024-08-03:

    --- Calculate partial digest of an open file. To the calculating mechanism itself,
    -- since only PDF documents could be modified by KOReader by appending data
    -- at the end of the files when highlighting, we use a non-even sampling
    -- algorithm which samples with larger weight at file head and much smaller
    -- weight at file tail, thus reduces the probability that appended data may change
    -- the digest value.
    -- Note that if PDF file size is around 1024, 4096, 16384, 65536, 262144
    -- 1048576, 4194304, 16777216, 67108864, 268435456 or 1073741824, appending data
    -- by highlighting in KOReader may change the digest value.
    function util.partialMD5(filepath)
        if not filepath then return end
        local file = io.open(filepath, "rb")
        if not file then return end
        local step, size = 1024, 1024
        local update = md5()
        for i = -1, 10 do
            file:seek("set", lshift(step, 2*i))
            local sample = file:read(size)
            if sample then
                update(sample)
            else
                break
            end
        end
        file:close()
        return update()
    end

API - details

All requests to API endpoints require the following headers:

"x-auth-user": "REPLACE_WITH_USERNAME"
"x-auth-key": "REPLACE_WITH_MD5_HASH_OF_PASSWORD"

API - doc refs

Worth a refresher read https://blog.stoplight.io/api-keys-best-practices-to-authenticate-apis

• https://github.com/koreader/koreader/blob/master/plugins/kosync.koplugin/api.json - client side
  • https://github.com/koreader/koreader/blob/master/spec/unit/kosync_spec.lua#L213 - client side
• server side
  • reference lua server
    • https://github.com/koreader/koreader-sync-server/blob/master/spec/controllers/1/syncs_controller_spec.lua
    • https://github.com/koreader/koreader-sync-server/blob/master/app/controllers/1/syncs_controller.lua
  • https://github.com/relkochta/koreader-sync/blob/main/kosync.py

API - examples

API - get progress

curl -X GET \
    -H "Accept: application/vnd.koreader.v1+json" \
    -H 'X-AUTH-USER: REPLACE_WITH_USERNAME' \
    -H "X-AUTH-KEY: REPLACE_WITH_MD5_HASH_OF_PASSWORD" \
     https://sync.koreader.rocks:443/syncs/progress/REPLACE_WITH_MD5_HASH_OF_FILE

Storage Model

Consider initial implementation using https://github.com/imbolc/sqlite_dbm for storage.

Implementations

• https://github.com/relkochta/koreader-sync/blob/main/backend/sqlite.py

  CREATE TABLE IF NOT EXISTS users (
      username text, 
      userkey text
  )
  
  CREATE TABLE IF NOT EXISTS documents (
          username text, 
          document text, 
          progress text,
          percentage float, 
          device text, 
          device_id text,
          timestamp int
  )
  

• https://github.com/yeeac/kosyncsrv/blob/main/handledb.go

  CREATE TABLE IF NOT EXISTS "user" (
      "username"  TEXT(255),
      "password"  TEXT(255)
  );
  
  CREATE TABLE IF NOT EXISTS "document" (
      "username"  TEXT(255),
      "documentid"  TEXT(255),
      "percentage"  REAL(64,4),
      "progress"  TEXT(255),
      "device"  TEXT(255),
      "device_id"  TEXT(255),
      "timestamp"  INTEGER
  

• https://github.com/aremmiw/ksync/blob/main/db.c

  CREATE TABLE IF NOT EXISTS users(
          username TEXT COLLATE NOCASE, 
          password TEXT, 
          UNIQUE(username)
  ); 
  
  CREATE TABLE IF NOT EXISTS progress(
          username TEXT COLLATE NOCASE, 
          document TEXT, 
          progress TEXT,
          percentage REAL, 
          device TEXT, 
          device_id TEXT, 
          timestamp DATETIME, 
          UNIQUE(username, document)
  );
  

• https://github.com/pborzenkov/koreader-syncd/blob/master/migrations/01_init.sql

CREATE TABLE users (
    username TEXT PRIMARY KEY NOT NULL,
    password TEXT NOT NULL
);
 
CREATE TABLE progress (
    document TEXT NOT NULL,
    username TEXT NOT NULL,
    device TEXT NOT NULL,
    device_id TEXT NOT NULL,
    progress TEXT NOT NULL,
    percentage REAL NOT NULL,
    timestamp INTEGER NOT NULL,
    PRIMARY KEY (document, username)
);

Other progress information

KoReader also stores in local files (see rough notes below, SDR).

From comment janeczku/calibre-web#2122 (comment) SomeOtherDev wrote:

Funnily enough, I wanted to implement something related to this to. I use a Kobo Libra H2O, and I wanted to write a daemon that syncs reading progress between KOReader and Calibre. Like people have said before, there's not really a standard way to do this.

Calibre seems to use epubcfi, location, references etc. KOReader stores this information roughly in a statistics.sqlite3 database (pages read, total pages), and as a percentage read/xpointer in metadata.epub.lua (the latter being in the folder of a given epub. Not sure about other document types).

Converting between the two doesn't seem easy. You'd probably have to reverse engineer the xpointer piece so you can translate it back and fourth between epubcfi.

Edit: To add to this, I've just realised Cool Reader 3 is used with this project, and it seems the function CR3View::goToXPointer in that project might hold the answer.

---

Rough unsorted notes

Koreader sync progress whisper sync

fastdigest appears to be partial md5sum and used as document key. partial_md5_checksum
Koreader can (md5) hash filename instead. NOTE filename (not directory) and includes file extension.

The md5 hash is calculated only the first time book is opened, and then it's stored in bookname.sdr/metadata.lua. As long as the md5 hash in that file doesn't change, the progress will sync, even if the file's hash changes.

SiDe caR files.
sdr files have partial md5sum an page progress
https://git.sr.ht/~harmtemolder/koreader-to-markdown

slpp.py load lua dict into python

Sync servers

• https://github.com/myelsukov/koreader-sync python dict
• https://github.com/relkochta/koreader-sync - An alternative reading progress sync server for KoReader (Python sqlite3 storage)
• https://github.com/b1n4ryj4n/koreader-sync
• https://github.com/yeeac/kosyncsrv - a tiny koreader sync server written by golang (sqlite3 storage)
• https://github.com/aremmiw/ksync - KOReader sync server implementation in C with kcgi (sqlite3 storage)
• https://github.com/search?q=koreader+sync
• https://github.com/koreader/koreader-sync-server
• https://github.com/koreader/koreader/search?q=fastDigest&type=
• https://github.com/koreader/koreader/tree/master/plugins/kosync.koplugin
• https://github.com/koreader/koreader/blob/fadee1f5dc5ffb6cd386a2fdadea4290e174069d/frontend/document/document.lua
• https://github.com/koreader/koreader/blob/8403154d4d36f4279828e0af90667a92685a4ebe/frontend/document/document.lua#L125
• Ability to export Reading Statistics data koreader/koreader#6454
• https://github.com/noembryo/KoHighlights
• https://github.com/noembryo/slppu

https://github.com/koreader/koreader/blob/master/plugins/statistics.koplugin/main.lua