Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite Grades v2 data synchronizer in async Rust #797

Closed

Conversation

junlarsen
Copy link
Member

@junlarsen junlarsen commented Feb 9, 2024

Still not too sure how good of an idea introducing Rust to the monorepo is but here goes

Ad-hoc Benchmarks

  • Performs on average 3 times faster than Node version (35 minutes for 132k subjects, vs 9 minutes for 137k)
  • Performed processing of 295k grades in 20 minutes (vs expected 3.5 hours)

A lot of this speed is probably not rust, but...

  • The old implementation ran inside a Next.js app with the dev compiler. Not ideal
  • Was ran with 56 pg pool pool connections vs the default 10 of pg.Pool
  • It looks cool

Copy link

linear bot commented Feb 9, 2024

@junlarsen junlarsen changed the title Bootstrap rust application Rewrite Grades v2 data synchronizer in async Rust Feb 9, 2024
@vercel vercel bot temporarily deployed to Preview – rif-dev February 9, 2024 21:19 Inactive
Copy link

vercel bot commented Feb 9, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
rif-prd ✅ Ready (Inspect) Visit Preview 💬 Add feedback Feb 11, 2024 4:40pm

@junlarsen
Copy link
Member Author

Also quite a bit of spaghetti here but it's good enough for an MVP imo

apps/grades-sync/src/main.rs Outdated Show resolved Hide resolved
apps/grades-sync/src/grade_repository.rs Outdated Show resolved Hide resolved
apps/grades-sync/src/grade_repository.rs Outdated Show resolved Hide resolved
apps/grades-sync/src/subject_repository.rs Outdated Show resolved Hide resolved
apps/grades-sync/src/job.rs Outdated Show resolved Hide resolved
apps/grades-sync/src/job.rs Outdated Show resolved Hide resolved
apps/grades-sync/src/job.rs Outdated Show resolved Hide resolved
@joleeee
Copy link
Contributor

joleeee commented Feb 10, 2024

So the point of this is to update the grades.no backend db?

@junlarsen
Copy link
Member Author

So the point of this is to update the grades.no backend db?

We are replacing the entire site

@junlarsen junlarsen requested a review from joleeee February 11, 2024 16:35
Copy link

Quality Gate Passed Quality Gate passed

Issues
0 New issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

let database_url =
std::env::var("DATABASE_URL").expect("missing DATABASE_URL environment variable");
let max_connections = std::env::var("MAX_CONNECTIONS")
.unwrap_or_else(|_| "10".to_string())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this could be a constant variable instead of a magic number?

@jotjern
Copy link
Member

jotjern commented Feb 18, 2024

Wild pull request, love it

@junlarsen junlarsen closed this Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants