Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce database queries on dashboards #3561

Merged
merged 8 commits into from
Feb 13, 2025
Merged

Conversation

eemeli
Copy link
Member

@eemeli eemeli commented Feb 11, 2025

Fixes #3545

This should speed things up a bit. Two related changes are included to resolve the identified inefficiencies, and a couple of the Locale methods are dropped/simplified to match current usage. The ProjectLocale.get_latest_activity() function is split into two separate implementations on Locale and Project, as they are now rather different, and the get_object_or_none() utility is dropped as obsolete.

@eemeli eemeli requested a review from mathjazz February 11, 2025 12:35
@mathjazz
Copy link
Collaborator

I'm getting this error on the locale pages:

[server] [INFO:django.server] 2025-02-11 19:45:53,493 "GET /de/ HTTP/1.1" 200 3956
[server] [ERROR:django.request] 2025-02-11 19:45:54,292 Internal Server Error: /de/ajax/
[server] Traceback (most recent call last):
[server]   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
[server]     response = get_response(request)
[server]                ^^^^^^^^^^^^^^^^^^^^^
[server]   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
[server]     response = wrapped_callback(request, *callback_args, **callback_kwargs)
[server]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server]   File "/app/pontoon/base/utils.py", line 84, in wrap
[server]     return f(request, *args, **kwargs)
[server]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server]   File "/app/pontoon/teams/views.py", line 135, in ajax_projects
[server]     return render(
[server]            ^^^^^^^
[server]   File "/usr/local/lib/python3.11/site-packages/django/shortcuts.py", line 24, in render
[server]     content = loader.render_to_string(template_name, context, request, using=using)
[server]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server]   File "/usr/local/lib/python3.11/site-packages/django/template/loader.py", line 62, in render_to_string
[server]     return template.render(context, request)
[server]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server]   File "/usr/local/lib/python3.11/site-packages/django_jinja/backend.py", line 59, in render
[server]     return mark_safe(self._process_template(self.template.render, context, request))
[server]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server]   File "/usr/local/lib/python3.11/site-packages/django_jinja/backend.py", line 105, in _process_template
[server]     return handler(context)
[server]            ^^^^^^^^^^^^^^^^
[server]   File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1304, in render
[server]     self.environment.handle_exception()
[server]   File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 939, in handle_exception
[server]     raise rewrite_traceback_stack(source=source)
[server]   File "/app/pontoon/teams/templates/teams/includes/projects.html", line 28, in top-level template code
[server]     {% set latest_activity = project.get_latest_activity(locale) %}
[server]     ^^^^^^^^^^^^^^^^^^^^^^^^^
[server]   File "/app/pontoon/base/models/project.py", line 293, in get_latest_activity
[server]     if project_locale and project_locale.latest_translation
[server]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server] AttributeError: 'list' object has no attribute 'latest_translation'

@eemeli
Copy link
Member Author

eemeli commented Feb 12, 2025

I'm getting this error on the locale pages:

Should be fixed now, was a silly mistake.

@codecov-commenter
Copy link

codecov-commenter commented Feb 12, 2025

Codecov Report

Attention: Patch coverage is 53.84615% with 18 lines in your changes missing coverage. Please review.

Project coverage is 78.94%. Comparing base (48ed7e0) to head (decac2f).
Report is 7 commits behind head on main.

Additional details and impacted files

Copy link
Collaborator

@mathjazz mathjazz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Tested locally and on stage and performance seems better. I also didn't spot any issues with the stats and latest activity data.

Left some comments inline.

pontoon/base/models/project.py Outdated Show resolved Hide resolved
cast(
BaseManager[Project], Project.objects.visible_for(request.user).available()
),
slug=slug,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really necessary?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cast() is a no-op at runtime, but with it the type of project is correct within this function.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I appreciate the intent behind improving type safety, our Python codebase does not use static typing consistently. Introducing it in just this one place creates an inconsistency that might confuse other developers who are not expecting type hints.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the other hand, type hints were how I was able to catch the bug fixed in #3562. Without them, working with Pontoon code is much more difficult and more error-prone, in particular for developers not previously familiar with it.

We should be working towards more type hinting being available in Pontoon.

pontoon/projects/views.py Show resolved Hide resolved
@eemeli
Copy link
Member Author

eemeli commented Feb 12, 2025

I got ajax_projects() down to 4 queries for anonymous users, and up to 7 if you're logged in with admin rights. That's probably the best that can be done here easily.

@eemeli eemeli requested a review from mathjazz February 12, 2025 18:00
Copy link
Collaborator

@mathjazz mathjazz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deployed latest version to stage.

cast(
BaseManager[Project], Project.objects.visible_for(request.user).available()
),
slug=slug,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I appreciate the intent behind improving type safety, our Python codebase does not use static typing consistently. Introducing it in just this one place creates an inconsistency that might confuse other developers who are not expecting type hints.

Co-authored-by: Matjaž Horvat <[email protected]>
@eemeli eemeli requested a review from mathjazz February 13, 2025 12:44
@mathjazz
Copy link
Collaborator

@flodolo Could you take this for a spin on stage?

@flodolo
Copy link
Collaborator

flodolo commented Feb 13, 2025

Seems to work fine for me, couldn't find inconsistencies.

@mathjazz mathjazz merged commit b0526d3 into mozilla:main Feb 13, 2025
2 checks passed
@eemeli eemeli deleted the reduce-queries branch February 13, 2025 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

N+1 query problem on dashboards
4 participants