Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🧧 Recommendations #325

Open
wants to merge 50 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 48 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
6748e4d
Initian setup
WRadoslaw Jan 8, 2024
c36b5b1
Override schema
WRadoslaw Jan 8, 2024
e1a8dc4
Add resolver to update video view
WRadoslaw Jan 8, 2024
d302d46
Switch account id for user id
WRadoslaw Jan 9, 2024
723466a
Ineractions service: initial setup
WRadoslaw Jan 12, 2024
e099099
Interactions service: handlers
WRadoslaw Jan 12, 2024
3bb5a1e
Rename methods
WRadoslaw Jan 15, 2024
eb52536
Add user upsert to auth api
WRadoslaw Jan 16, 2024
7bd0359
Add video upsert to the mappings and processor
WRadoslaw Jan 16, 2024
91384ef
Add resolvers for videos scenarios
WRadoslaw Jan 16, 2024
1626888
Small fixes to resolver
WRadoslaw Jan 16, 2024
cf7495f
Handle next recommendation error
WRadoslaw Jan 17, 2024
59706d7
DEV: homepageVideos debugging
WRadoslaw Jan 18, 2024
5a7263d
Small logic improvements
WRadoslaw Jan 23, 2024
340b314
Initial docs work
WRadoslaw Jan 23, 2024
2fabee9
New env vars
WRadoslaw Jan 24, 2024
32dddd8
Prettier
WRadoslaw Jan 24, 2024
bff051e
Interactions server docs
WRadoslaw Jan 24, 2024
36c72c0
Decrease queue size after reaching export block
WRadoslaw Jan 24, 2024
54540ea
Add update loop and change queue based batch logic
WRadoslaw Jan 24, 2024
85d25c3
Switch to single queue and make scheduling sync
WRadoslaw Jan 25, 2024
1950d19
Remove awaits for sync methods
WRadoslaw Jan 25, 2024
17c6d6e
Remove not existing method call
WRadoslaw Jan 25, 2024
39ca0a1
Always init batch loop
WRadoslaw Jan 25, 2024
e0afbf7
Remove initially enabled manager on dev
WRadoslaw Jan 25, 2024
41c622b
Parse IDs
WRadoslaw Jan 26, 2024
579662d
Handle custom item id
WRadoslaw Jan 26, 2024
45721a8
Fix response error if where is empty
WRadoslaw Jan 26, 2024
6e10adf
Init schedule loop on auth and interactions servers
WRadoslaw Jan 26, 2024
73bf1a7
Add missing user id parsing in methods
WRadoslaw Jan 26, 2024
ed225c7
Change timestamp format to avoid error
WRadoslaw Jan 26, 2024
1420552
Decrease ip limiting on interactions server
WRadoslaw Jan 26, 2024
9304bcf
Adjust provider logic and fix similiar items query
WRadoslaw Jan 27, 2024
85bb1d5
Watch next video resolver
WRadoslaw Jan 30, 2024
48d6090
Add remove video event to the recombee
WRadoslaw Jan 30, 2024
bb5330d
Script to quickly setup recombee database
WRadoslaw Jan 30, 2024
cfc6291
Add channel oriented schedules to the processor
WRadoslaw Jan 30, 2024
6455b1e
Add logic for channel clicked handler
WRadoslaw Jan 30, 2024
77144ce
Add logic for channel follow
WRadoslaw Jan 30, 2024
58f64fe
Add n-trigram language detection for recommendation system
WRadoslaw Jan 31, 2024
a44c381
Remove channel from sync
WRadoslaw Feb 7, 2024
aaba1f4
Setup mongodb for interactions
WRadoslaw Mar 25, 2024
567c202
Save interactions to mongodb
WRadoslaw Mar 26, 2024
ee8755d
Avoid running interactions-api container if env vars are not present
WRadoslaw Mar 26, 2024
acd5c21
Merge branch 'master' into feat/recommendations
WRadoslaw Apr 8, 2024
c99c599
Post merge fixes and self review fixes
WRadoslaw Apr 8, 2024
c1ec1d5
CI fixes
WRadoslaw Apr 8, 2024
e285b34
Fix checks
WRadoslaw Apr 8, 2024
4a666fc
Initial PR fixes
WRadoslaw Apr 17, 2024
edba6ec
Persisted queue idea
WRadoslaw Apr 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .env
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ DB_PASS=squid
DB_ADMIN_USER=admin
DB_ADMIN_PASS=admin
DB_PORT=23798
INTERACTIONS_DB_PORT=23799

# Processor service host
PROCESSOR_HOST=localhost
# Processor service prometheus port
Expand All @@ -17,6 +19,16 @@ PROCESSOR_PROMETHEUS_PORT=3337
GQL_PORT=4350
# Auth api port
AUTH_API_PORT=4074
# Recommendation service config
INTERACTIONS_API_PORT=4075
# Set private key and database name for recommendation service
# If you set only private key, Orion will support only local interactions service (for now at least)
RECOMMENDATION_SERVICE_PRIVATE_KEY=save-interactions-locally
# jsgenesis-dev
RECOMMENDATION_SERVICE_DATABASE=
# this will disable export block check for recommendation system - sync all data from the beginning
FORCE_RECOMMENDATION_DATA_SYNC=false


# Archive gateway url
ARCHIVE_GATEWAY_URL=${CUSTOM_ARCHIVE_GATEWAY_URL:-http://localhost:8888/graphql}
Expand Down
27 changes: 20 additions & 7 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,16 @@ serve:
serve-auth-api:
@npm run auth-server-start

serve-interactions-api:
@npm run interactions-server-start

migrate:
@npx squid-typeorm-migration apply

dbgen:
@npx squid-typeorm-migration generate

generate-migrations:
generate-migrations:
@docker run -d --name temp_migrations_db \
-e POSTGRES_DB=squid \
-e POSTGRES_HOST_AUTH_METHOD=trust \
Expand All @@ -49,27 +52,37 @@ codegen:
@npm run generate:schema || true
@npx squid-typeorm-codegen

setup-recommendations:
@npm run recommendations:setup

network-tests:
@npm run generate:network-tests || true

typegen:
@npx squid-substrate-typegen typegen.json

prepare: install typegen codegen build
prepare: install typegen codegen build setup-recommendations

up-squid:
@docker network create joystream_default || true
@docker-compose up -d
@set -a; \
source .env; \
set +a; \
docker network create joystream_default || true; \
if [ -z "$$RECOMMENDATION_SERVICE_PRIVATE_KEY" ] && [ -z "$$RECOMMENDATION_SERVICE_DATABASE"]; then \
docker-compose up -d; \
else \
docker-compose --profile interactions up -d; \
fi \
Comment on lines +67 to +75
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change will start the recommendation service containers even if only one of the env var is present.

Expected behaviour should be to start with --profile interactions when both of these are present


up-archive:
@docker network create joystream_default || true
@docker network create joystream_default || truee
@docker-compose -f archive/docker-compose.yml up -d

up: up-archive up-squid

down-squid:
@docker-compose down -v
@docker-compose down -v --remove-orphans

down-archive:
@docker-compose -f archive/docker-compose.yml down -v

Expand Down
39 changes: 39 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,22 @@ services:
- orion_db_data:/var/lib/postgresql/data
- ./db/postgres.conf:/etc/postgresql/postgresql.conf

interactions_db:
image: mongo:latest
WRadoslaw marked this conversation as resolved.
Show resolved Hide resolved
container_name: interactions_db
hostname: interactions_db
profiles: ['interactions']
restart: unless-stopped
shm_size: 1g
command: ['mongod', '--port', '${INTERACTIONS_DB_PORT}']
ports:
- '${INTERACTIONS_DB_PORT}:${INTERACTIONS_DB_PORT}'
volumes:
- interactions_db_data:/data/db
environment:
MONGO_INITDB_ROOT_USERNAME: squid
MONGO_INITDB_ROOT_PASSWORD: squid

orion_processor:
container_name: orion_processor
hostname: orion_processor
Expand Down Expand Up @@ -82,8 +98,31 @@ services:
ports:
- '${AUTH_API_PORT}:${AUTH_API_PORT}'

orion_interactions-api:
container_name: orion_interactions-api
hostname: orion_interactions-api
profiles: ['interactions']
image: node:18
restart: unless-stopped
env_file:
- .env
- docker.env
environment:
- SQD_TRACE=interactions
depends_on:
- orion_db
volumes:
- type: bind
source: .
target: /orion
working_dir: /orion
command: ['make', 'serve-interactions-api']
ports:
- '${INTERACTIONS_API_PORT}:${INTERACTIONS_API_PORT}'

volumes:
orion_db_data:
interactions_db_data:

networks:
default:
Expand Down
1 change: 1 addition & 0 deletions docs/developer-guide/architecture-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ The services can be divided into 2 main categories:
- [Squid processor](https://docs.subsquid.io/basics/squid-processor/): It's responsible for processing Joystream blockchain events and creating, updating or deleting the database entities (such as `Channel` or `Video`) accordingly, so that they remain in-sync with the current Joystream chain state.
- [GraphQL server](https://docs.subsquid.io/graphql-api/overview/): Orion's main GraphQL server which handles all the queries and mutations.
- [Auth server](../../src/auth-server/docs/README.md): Orion's authentication server (REST API) responsible for handling user authentication.
- [Interactions server](../../src/interactions-server/docs/README.md): Orion's interactions proxy server (REST API) responsible for limiting the amount of interactions sent to the recommendations' system.


## Directory structure and important files
Expand Down
3 changes: 2 additions & 1 deletion docs/operator-guide/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ To learn how to run your own instance of Atlas, please refer to the [Atlas opera
- Production deployment:
- [Deploying Orion to production](./tutorials/deploying-orion.md)
- [Upgrading Orion](./tutorials/upgrading-orion.md)
- [Recommendations in Orion](./tutorials/recommendations.md)


## Orion architecture
Expand All @@ -46,4 +47,4 @@ To learn how to deploy Orion to production, please read the _[Deployment](./tuto

### Upgrading a production instance

The process of upgrading a production instance of Orion in the event of a new release is described in the _[Upgrading](./tutorials/upgrading-orion.md)_ tutorial.
The process of upgrading a production instance of Orion in the event of a new release is described in the _[Upgrading](./tutorials/upgrading-orion.md)_ tutorial.
21 changes: 21 additions & 0 deletions docs/operator-guide/tutorials/recommendations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# User recommendations in Orion
User recommendations are outsourced to the external provider.
Initial implementation was done with the Recombee API, and it is out of the box solution offered.

### Abstraction
All the logic for the provider was abstracted and ecapsulated in [`RecommendationServiceManager`](../../../src/utils/RecommendationServiceManager.ts) this means, that if you want to change
the provider you just have to adjust the logic inside it and Orion will use the service to send data and get recommendations.

### Syncing data
- Video data gets upserted only through processor
- User data get upserted through `auth-server`
- Interactions data is proxied thorugh `interactions-server` where the request are limited both by IP and interactions.
Only iteraction that is not coming thorugh the server is rating interaction (like, dislike), which will be synced through processor on MemberRemark event.

### Env variables
Recombee client is constructed with 2 variables `RECOMMENDATION_SERVICE_PRIVATE_KEY` and `RECOMMENDATION_SERVICE_DATABASE` both self-explanatory.
`FORCE_RECOMMENDATION_DATA_SYNC` variable is a boolean that indicate whether processor should consider export block height in terms of syncing videos and ratings with recommendations service.
If set to true - it will sync all the data for the first block even if last exported block is greater.

If those two values are provided, when you use `make prepare` command, it will set up item properties and segments supported by Orion.
If you want to run set up script directly use `make setup-recommendations`.
Loading
Loading