Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds profiler for knn query #2450

Open
wants to merge 4 commits into
base: 2.x
Choose a base branch
from
Open

Conversation

shatejas
Copy link
Collaborator

@shatejas shatejas commented Jan 27, 2025

Description

Adds exact and ann timings and counts with knn query profile API

You can find sample response here:
concurrent_nofilter.json
concurrent.json
profile_filter.json
faiss_profile.json

CIs will fail since it require a change in Opensearch-core

Pending:

Related Issues

Resolves #2286

Partial response

"profile": {
		"shards": [
			{
				"id": "[6lfotEMuRUqgq0kM6BfLSQ][target_index_faiss][0]",
				"inbound_network_time_in_millis": 0,
				"outbound_network_time_in_millis": 0,
				"searches": [
					{
						"query": [
							{
								"type": "NativeEngineKnnVectorQuery",
								"description": "NativeEngineKnnVectorQuery[]...KNNQuery[]",
								"time_in_nanos": 27173585,
								"breakdown": {
									"set_min_competitive_score_count": 0,
									"exact_knn_search": 0,
									"match_count": 0,
									"shallow_advance_count": 0,
									"next_doc": 3542,
									"score_count": 3,
									"compute_max_score_count": 0,
									"advance": 0,
									"advance_count": 0,
									"score": 2459,
									"shallow_advance": 0,
									"create_weight_count": 1,
									"build_scorer": 614792,
									"ann_search": 0,
									"set_min_competitive_score": 0,
									"match": 0,
									"next_doc_count": 4,
									"compute_max_score": 0,
									"build_scorer_count": 2,
									"create_weight": 26552792,
									"ann_search_count": 0,
									"exact_knn_search_count": 0
								},
								"children": [
									{
										"type": "KNNQuery",
										"description": "",
										"time_in_nanos": 2821209,
										"breakdown": {
											"set_min_competitive_score_count": 0,
											"exact_knn_search": 0,
											"match_count": 0,
											"shallow_advance_count": 0,
											"next_doc": 0,
											"score_count": 0,
											"compute_max_score_count": 0,
											"advance": 0,
											"advance_count": 0,
											"score": 0,
											"shallow_advance": 0,
											"create_weight_count": 0,
											"build_scorer": 0,
											"ann_search": 2821209,
											"set_min_competitive_score": 0,
											"match": 0,
											"next_doc_count": 0,
											"compute_max_score": 0,
											"build_scorer_count": 0,
											"create_weight": 0,
											"ann_search_count": 1,
											"exact_knn_search_count": 0
										}
									}
								]
							}
						],
						"rewrite_time": 29292,
						"collector": [
							{
								"name": "SimpleTopScoreDocCollector",
								"reason": "search_top_hits",
								"time_in_nanos": 381916
							}
						]
					}
				],
				"aggregations": []
			}

Check List

  • New functionality includes testing.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Tejas Shah <[email protected]>
@VijayanB
Copy link
Member

@shatejas Can you post the sample response as text in comment if it is small so that we don't have to download?

@shatejas
Copy link
Collaborator Author

shatejas commented Jan 28, 2025

@VijayanB there are multiple of them and aren't small so uploaded them. I can add partial if it helps

@shatejas shatejas removed the v2.19.0 label Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants