[ENG-2342] ai_first_segment_delay metric added

[pwilczynskiclearcode]

Adding ai_first_segment_delay prometheus metric.
It measures the time between AI Gateway receiving a GenLiveVideoToVideo request and successfully receiving the first processed segment from O.

[codecov]

Codecov Report

Attention: Patch coverage is 36.84211% with 24 lines in your changes missing coverage. Please review.

Project coverage is 33.72276%. Comparing base (17832e7) to head (ef06672).
Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
monitor/census.go	53.84615%	11 Missing and 1 partial ⚠️
server/ai_live_video.go	0.00000%	7 Missing ⚠️
server/ai_process.go	0.00000%	5 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                 @@
##              master       #3341         +/-   ##
===================================================
- Coverage   33.72469%   33.72276%   -0.00193%     
===================================================
  Files            141         141                 
  Lines          37391       37405         +14     
===================================================
+ Hits           12610       12614          +4     
- Misses         24061       24070          +9     
- Partials         720         721          +1     

Files with missing lines	Coverage Δ
server/ai_process.go	0.59880% <0.00000%> (-0.00206%)	⬇️
server/ai_live_video.go	0.00000% <0.00000%> (ø)
monitor/census.go	61.98675% <53.84615%> (+0.16736%)	⬆️

... and 1 file with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17832e7...ef06672. Read the comment docs.

Files with missing lines	Coverage Δ
server/ai_process.go	0.59880% <0.00000%> (-0.00206%)	⬇️
server/ai_live_video.go	0.00000% <0.00000%> (ø)
monitor/census.go	61.98675% <53.84615%> (+0.16736%)	⬆️

... and 1 file with indirect coverage changes

[leszko]

I wonder if we may want to tag it with Orchestrator's address?

I can imagine that for the given Gateway, only some Os may be slow.

[pwilczynskiclearcode]

Yes. That would be interesting information.

The question is how to get Orchestrator info in startTrickleSubscribe() function.
Where of if does it live on the Gateway code? We'd need OrchestratorInfo struct, I think.

[leszko]

You can pass host to startTrickleSubscribe().

This host is good as a tag. It's the IP/DNS of the Orchestrator.

[pwilczynskiclearcode]

OK. I took entire orchInfo and unified it a bit on the census-side

[leszko]

Added one more comment on how to tag with host