-
-
Notifications
You must be signed in to change notification settings - Fork 57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix snuba admin's query tracing to connect to right storage and query nodes #6268
Conversation
❌ 1 Tests Failed:
View the top 1 failed tests by shortest run time
To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and actually this does not seem like a good idea at all. what is the problem you're solving? having everyone edit /etc/hosts
on their mac is not a good solution and is going to cause endless pain for developers
@asottile-sentry thanks for the review. Let's take step back to the larger problem. I will edit the description if this explanation makes sense. At Sentry, snuba developers use a tool called snuba admin. It has a query tracing section that allows snuba developers to trace how a query is executed in clickhouse. We wanted to enhance that tool to also show clickhouse's profile events. Without this enhancement, snuba developers have to do these manual steps to get profile events:
You can see that these manual steps are error-prone and there is scope to automate them. The query trace output has a filed called
The To be clear, not every developer needs this entry in |
wouldn't it be simpler, and a better experience, to write a tool which does all of the manual translation automatically (including replacing with the appropriate localhost / loopback addresses? -- we already utilize |
I resolved the issues based on the feedback in our call. No network settings need to be changed to make this work now. |
@@ -85,6 +92,9 @@ services: | |||
depends_on: | |||
- zookeeper | |||
image: "${CLICKHOUSE_IMAGE:-ghcr.io/getsentry/image-mirror-altinity-clickhouse-server:23.3.19.33.altinitystable}" | |||
hostname: clickhouse-query.local | |||
extra_hosts: | |||
- "clickhouse-query.local:127.0.0.1" # Add entry to /etc/hosts file |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suggest rewording the comment to explain why this is desirable, rather than restating what the code does.
Snuba admin's tracing tool shows query trace output. In addition to that, it will now show the profile events data. This PR parses the trace output and builds a
dict
of node name to query id. Then system query is executed on those nodes to get the profile events for the corresponding query id.Before connecting to the nodes, a socket connection is attempted to see if the hostname resolves. If it does not, then the node is assumed to be
127.0.0.1
. This is required because while running locally, snuba admin process will connect to clickhouse container. But the container's hostname does not resolve, defaulting to127.0.0.1
as the hostname.To make sure hostnames in the CI jobs resolve, the CI docker-compose file is updated.