Telemetry: At Mentions

[RXminuS]

Before the cody.at-mention event fired as soon as the chat view was rendered leading to an overcount of DAUs. Duplicate mention events were also fired for other mention providers such as when typing a file name where each character would fire an event.

Now events only fire when actually triggered. We also clean up and add additional public metadata that can help evaluate the actual effectiveness of different mention providers.

Fixes CODY-3405

Test plan

• Added new Telemetry testing framework. Run cd vscode & pnpm run test:e2e2 --grep "telemetry.
• See the snapshots to view the included data

Changelog

[RXminuS]

• Migrate logout test #5617
• Telemetry: At Mentions #5558 👈
• E2Ev2 #5278 : 1 other dependent PR (#5026 )
• Status Bar: Observable Migration #5757
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @RXminuS and the rest of your teammates on Graphite

[github-actions]

‼️ Hey @sourcegraph/cody-security, please review this PR carefully as it introduces the usage of an unsafe_ function or abuses PromptString.

[akalia25]

Left a few clarifying comments :)

[RXminuS]

@akalia25 what do you want to do with those failing telemetry events in the other tests?

Can we somehow just remove the event from the general checks as we can do more detailed testing now? Or do I have to go in and update them all?

[akalia25]

Left a few comments regarding the collection of transcripts, and changes introduced to that. After those, I can approve :)

[akalia25]

For the collection of transcript data, we'd need to do the same isDotCom() check that was happening before, is this happening somewhere else in the code now?

The reason for this is two-fold, first prevents the local PG instance from storing this data and increasing db load and second for additional security around this sensitive data.

Suggested change

	promptText: inputText,
	promptText:
	isDotCom(authStatus) &&
	(await truncatePromptString(inputText, CHAT_INPUT_TOKEN_BUDGET)),
	},

[akalia25]

@akalia25 what do you want to do with those failing telemetry events in the other tests?

Can we somehow just remove the event from the general checks as we can do more detailed testing now? Or do I have to go in and update them all?

hmm I'd love to have those updated 😅 , those have been critical in us spot-checking events. Happy to assist and update them as well!

[abeatrix]

manually tested

Asked @jan Hartman before and he said the prompt text wasn't used. For now we're prioritizing making sure DAU count is correct. We can easily add back prompt text back later.