-
Notifications
You must be signed in to change notification settings - Fork 25
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #22 from pryv/release/1.9.2
Initating release 1.9.2
- Loading branch information
Showing
70 changed files
with
2,952 additions
and
2,708 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 1.9.1-open | ||
| 1.9.2-open |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| name: Open Pryv.io CI | ||
|
|
||
| on: | ||
| pull_request: | ||
| branches: | ||
| - '*' | ||
| push: | ||
| branches: | ||
| - '*' | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-22.04 | ||
|
|
||
| strategy: | ||
| matrix: | ||
| node-version: [18.16.0] | ||
|
|
||
| steps: | ||
| - name: Install `just` | ||
| uses: extractions/setup-just@v2 | ||
|
|
||
| - name: Checkout repository with submodules | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Install Node.js with version ${{ matrix.node-version }} | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: ${{ matrix.node-version }} | ||
|
|
||
|
|
||
| - name: Setup Open Pryv.io | ||
| run: | | ||
| npm run setup-dev-env | ||
| npm install | ||
| sudo apt-get install graphicsmagick | ||
| - name: Run tests on Open Pryv.io with coverage | ||
| run: | | ||
| IS_CI=true just test-cover-lcov | ||
| - name: Upload coverage to Codecov | ||
| uses: codecov/[email protected] | ||
| with: | ||
| token: ${{ secrets.CODECOV_TOKEN }} | ||
| slug: pryv/open-pryv.io |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,182 @@ | ||
|
|
||
| ## 1.9 | ||
|
|
||
| ### 1.9.2 | ||
| - Refactored Attachments (Event Files) Logic to be modular for future cloud storage of files such as S3. | ||
|
|
||
| ### 1.9.1 | ||
| - Implemented ferretDB compatibility allowing full-open source modules | ||
| - Replaced rec.la by backloop.dev | ||
|
|
||
| ### 1.9.0 | ||
|
|
||
| - Remove FlowType and convert (best-effort) typing information into JSDoc comments | ||
| - Update to MongoDB v6 | ||
| - Update to node v18 | ||
| - Stream deletion eventIds when deleting streams to avoid timeout | ||
| - Introduce platform DB for future cross-cores usage | ||
| - Unify SQLite usage across audit and storage | ||
| - Move attachments to per-user directories | ||
| - Finalize data-store API for first public release | ||
| - Many linting fixes | ||
| - Support for multiple CAA (certificate autorities issuer) | ||
| - Bug fixes: | ||
| - Non-reusable deleted streamIds when following auth process #484 | ||
| - SQLITE_BUSY error thrown in multi-core #487 | ||
|
|
||
| ## 1.8 | ||
|
|
||
| ### 1.8.1 | ||
|
|
||
| - Fix migration 1.6.x to 1.8.0 bug | ||
|
|
||
| ### 1.8.0 | ||
|
|
||
| - Add support for password rules: complexity, age, reuse; see API server's `auth.password*` settings | ||
| - Affected methods are: create user (`POST /users`), change password (`{user endpoint}/account/change-password`), reset password (`{user endpoint}/account/reset-password`) and login (`{user endpoint}/auth/login`) | ||
| - Add undocumented support for external stores (a.k.a. "data mapping" feature); see component `pryv-datastore` (will be published separately when appropriate) | ||
|
|
||
| ## 1.7 | ||
|
|
||
| ### 1.7.14 | ||
| - Fix crash caused by permissions selfRevoke used in combinaison with BACKWARD_COMPATIBILITY_SYSTEM_STREAMS_PREFIX set to true. | ||
| - Fix issue with `accesses.create` theand selfRevoke permissions that was only possible with a personalToken. | ||
|
|
||
| ### 1.7.13 | ||
|
|
||
| - Fix another issue when BACKWARD_COMPATIBILITY_SYSTEM_STREAMS_PREFIX is set to "true" - children streams' ids were not following the correct format | ||
| - Fix a performance issue when querying events by type | ||
| - Fix an issue which caused the service not to restart properly in some situations | ||
|
|
||
| ### 1.7.12 | ||
|
|
||
| - Fix issue when BACKWARD_COMPATIBILITY_SYSTEM_STREAMS_PREFIX is set to "true" - "account" streamId was handled as ".account" | ||
|
|
||
| ### 1.7.10 | ||
|
|
||
| - API change: Don't coerce event content and simplify known type validation process in api-server | ||
| - serviceInfo:eventTypes URL now supports `file://` protocol allowing it to load definition from file system | ||
|
|
||
| ### 1.7.9 | ||
|
|
||
| - Fix issue with events.getAttachment making core crash if filename contained fancy characters by putting it in the 'Content-disposition' header | ||
| - Security fix: make password reset token single-use | ||
| - Security fix: hide "newPassword" in logs when an error occurs in account.resetPassword | ||
|
|
||
| ### 1.7.7 | ||
|
|
||
| - Fix issue where a deleted user was kept in the cache, thus rendering the reuse of username possible, but failing all subsequent calls as the password and tokens were not returned (since the wrong userId was returned by the cache) | ||
| - Fix issue where attempting to create streams with id 'size' would return an error | ||
| - Fix socket.io CORS issue | ||
|
|
||
| ### 1.7.6 | ||
|
|
||
| - Fix access-info permissions | ||
|
|
||
| ### 1.7.5 | ||
|
|
||
| - add missing system stream permissions accesses | ||
| - change __unique properties cleanup, just match them by key suffix, not from current serializer unique props. Avoids migration error if uniqueness has been modified. | ||
|
|
||
|
|
||
| ### 1.7.1 | ||
|
|
||
| - migrate tags into streams | ||
|
|
||
| ### 1.7.0 | ||
|
|
||
| - introduce mall abstraction | ||
| - add integrity | ||
| - refactor access permissions logic | ||
|
|
||
| ## 1.6 | ||
|
|
||
| ### 1.6.21 | ||
|
|
||
| Fixes: | ||
|
|
||
| - fix boolean/bool event type that was not allowed | ||
| - fix HF null values for optional values that was not fully working | ||
|
|
||
| Changes: | ||
|
|
||
| - increase username characters limit to 60 | ||
|
|
||
| ### 1.6.20 | ||
|
|
||
| - Implement system route to deactivate MFA | ||
|
|
||
| ### 1.6.18 | ||
|
|
||
| - Fix welcome email: don't wait for welcome email sending before replying to client. | ||
|
|
||
| ### 1.6.16 | ||
|
|
||
| - Fix versioning: update unique system events bug | ||
|
|
||
| ### 1.6.15 | ||
|
|
||
| - Fix user deletion | ||
|
|
||
| ### 1.6.14 | ||
|
|
||
| - personal token can delete an account | ||
| - add external licenser: pryv/app-node-licenser | ||
| - fix security issue with users registration conflicts leaking random email addresses | ||
|
|
||
| ### 1.6.13 | ||
|
|
||
| - Unify configuration into boiler | ||
| - Fixes for Open Pryv.io | ||
|
|
||
| ### 1.6.12 | ||
|
|
||
| Fixes: | ||
|
|
||
| - versioning now works when trashing event | ||
|
|
||
| ### 1.6.7 | ||
|
|
||
| New Features: | ||
|
|
||
| - Stream queries for events.get | ||
|
|
||
| Fixes: | ||
|
|
||
| - usernames starting with "system" are available | ||
| - personal token expiration now fixed | ||
| - Users create call on core username error message now specifies that letters must be lowercase | ||
|
|
||
| Changes: | ||
|
|
||
| - In configuration, rename "singleNode" to "dnsLess", keeping retro-compatibility for "singleNode" with warning message | ||
|
|
||
| Removals: | ||
|
|
||
| - Deprecated "GET /who-am-i" API method removed | ||
| - Remove pryvuser-cli code (the image was not built since July) | ||
|
|
||
| ### 1.6.3 | ||
|
|
||
| Custom Auth function now has access to all headers. | ||
|
|
||
| ### 1.6.2 | ||
|
|
||
| - Fix migration that was skipping passwordHash leading to users not being able to login | ||
| - add errors if this cases arises | ||
|
|
||
| ### 1.6.1 | ||
|
|
||
| Fixes for dnsLess/openSource: | ||
|
|
||
| - /reg/service/info | ||
| - dependencies | ||
| - boost POST payload to 10MB for HF server | ||
|
|
||
| ### 1.6.0 | ||
|
|
||
| system streams: | ||
|
|
||
| - customizable (& extendable) unique and indexed account properties | ||
| - access to account properties through the events API with its access management | ||
| - user account deletion through administration API |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| # Pryv.io Databases | ||
|
|
||
| Initially Pryv.io was built on top of MongoDB with separated collection per user. This initial design allowed to isolate peruser data on the file system. | ||
|
|
||
| This design has a drawback as MongoDB was consuming a fixed amount of RAM per collection and the with growing sets of users (over 40'000 per node) 16Gb was needed. In v1.6.0 an option to merge the Mongo's collection was added, resulting in an average RAM requirement of 4Gb for 100'000 users. | ||
|
|
||
| From v1.7.0 Sqlite has been investigated in order to provide back the ability to isolate peruser data on the file system. The motivation is to provide full control over the user's data in order to facilitate and prove the "right to be forgotten." | ||
|
|
||
| From v1.8.0 a Sqlite version for Event has been provided on top of the [datastore](https://github.com/pryv/pryv-datastore) abstraction. | ||
|
|
||
| From v1.9.x [FerretDB](https://www.ferretdb.com) has been implemented as on optional replacement of MongoDB. | ||
|
|
||
| Since v1.9.2 Pryv.io can be deployed in "full-cloud" setup without relying on the file system. This can be done by configuring all storage modules to use MongoDB. For the attachments and S3 implementation is in development. | ||
|
|
||
| For future v1.9.3 Pryv.io will be also capable in being "full local" with only SQLite databases. | ||
|
|
||
| ## List of storage used in Pryv.io | ||
|
|
||
| #### User local directory | ||
|
|
||
| base code: [components/storage/src/userLocalDirectory.js](components/storage/src/userLocalDirectory.js) | ||
|
|
||
| Localization of user data on the host file system, usually in `var-pryv/users` then a directory path is constructed using the 3 last characters of the userId and the userId. | ||
|
|
||
| Exemple with userId `c123456789abc`: `var-pryv/users/c/b/a/c123456789abc/` | ||
|
|
||
| In this directory, the attachments and any user attributed data and sqlite db should be stored. | ||
|
|
||
| #### User local index | ||
|
|
||
| base code: [components/storage/src/userLocalIndex.js](components/storage/src/userLocalIndex.js) | ||
|
|
||
| This database is a per-server index to map userId and userName. In the future it could be extended to allow user aliases. | ||
|
|
||
| - With SQLite (default) the db file can be usually found at `var-pryv/user-index.db` | ||
| - With MongoDB the collection is `id4name` and stored in the main host database `pryv-node` | ||
|
|
||
| Settings to activate MongoDB/ferretDB instead of SQLite: `storageUserIndex:engine = 'mongodb'` | ||
|
|
||
| Script to migrate userIndex from SQLite to MongoDB: [read first](#sql2mongo) | ||
| `LOGS=info node components/storage/src/migrations/switchSqliteMongo/usersIndex.js --config configs/api.yml` | ||
|
|
||
| #### User account storage | ||
|
|
||
| base code: [components/storage/src/userAccountStorage*.js](components/storage/src/) *: Mongo or Sqlite | ||
|
|
||
| This database contains the password and passwords history of the user. | ||
|
|
||
| - With SQLite (default) it can be found in the "User local directory" named as `account-1.0.0.sqlite` . | ||
| - With MongoDB the collection is `passwords` and stored in the main host database `pryv-node` | ||
|
|
||
| Settings to activate MongoDB/ferretDB instead of SQLite: `storageUserAccount:engine = 'mongodb'` | ||
|
|
||
| Script to migrate from SQLite to MongoDB: [read first](#sql2mongo) | ||
| `LOGS=info node components/storage/src/migrations/switchSqliteMongo/userAccountStorage.js --config configs/api.yml` | ||
|
|
||
| #### Platform Wide Shared Storage | ||
|
|
||
| base code: [components/platform](components/platform) | ||
|
|
||
| This database contains all indexed and unique fields for users such as emails and custom systems streams data. | ||
|
|
||
| In the Enterprise version of Pryv, it acts as a local cache and report to `service-register` being the main index. For Open-Pryv.io platformDB should evolve in a shared database between running service-core. | ||
|
|
||
| - With SQLite (default) the db file can be usually found at `var-pryv/platform-wide.db` | ||
| - With MongoDB | ||
|
|
||
| Settings to activate MongoDB/ferretDB instead of SQLite:`storagePlatform:engine = 'mongodb'` | ||
|
|
||
| Script to migrate from SQLite to MongoDB: [read first](#sql2mongo) | ||
|
|
||
| `LOGS=info node components/storage/src/migrations/switchSqliteMongo/platformDB.js --config configs/api.yml` | ||
|
|
||
| #### Events, Streams & Attachments Storage | ||
|
|
||
| base code: [components/storage/src/localDataStore](components/storage/src/localDataStore) and [localDataStoreSQLite](components/storage/src/localDataStoreSqlite) | ||
|
|
||
| Main storage for `events` , `streams` & `attachments` this implementation follows the modular API of [datastore](https://github.com/pryv/pryv-datastore) abstraction. | ||
|
|
||
| - Fully implemented with MongoDB/FerretDB | ||
| - Only events are implemented with SQLite - Expecting full SQLite implementation in v1.9.3 | ||
|
|
||
| #### Profile, Accesses, FollowedSlices & Webhooks Storage | ||
|
|
||
| base code: [components/storage/src/user](components/storage/src/user) | ||
|
|
||
| Only implemented for MongoDB/FerretDB - Expecting full SQLite implementation in v1.9.3 | ||
|
|
||
| ### Notes | ||
|
|
||
| #### Known issues | ||
|
|
||
| - [ ] test B2I7 is failing when testing `storage` with `full-mongo` as indexes for password is not yet created. Run `just test-full-mongo storage` to reproduce | ||
|
|
||
| #### <a name="sql2mongo"/>Using SQlite to MongoDB migration scripts | ||
|
|
||
| 1. Make sure that all Pryv.io components are stopped but `MongoDB` | ||
| 2. Do not set the `storage*:engine` setting to `mongodb` yet ! | ||
| 3. Run the scripts | ||
| 4. Change appropriate setting to `storage*:engine = 'mongodb'` | ||
| 5. Start all services and check | ||
| 6. If all is fine, related SQLite DB should be deleted manually |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.