rls: Fix flaky test Test/ControlChannelConnectivityStateMonitoring

[eshitachandwani]

fixes #5468
The test flakes because

1. • Test creates a ClientConn with service config setting the LB policy to "rls"
   • RLS LB policy initializes the control channel to the RLS server. As part of this, it spawns a goroutine to monitor the control channel connectivity state changes.
   • But by the time the first RPC is successfully made, the above goroutine has not gotten a chance to run yet.
   • And at this time, the test stops the RLS server. This moves the control channel to IDLE and it is only now that the monitoring goroutine gets to run, and it has already missed the first transition to READY.

FIX : Use channel to make sure the go routine starts

2. Our current state change API is lossy because state changes can be lost between the former returning and the caller invoking GetState
   FIX : The fix is to use grpcsync.pubsub to subscribe to the state changes so that we do not loose state changes.

RELEASE NOTES: N/A

[codecov]

Codecov Report

Attention: Patch coverage is 92.85714% with 2 lines in your changes missing coverage. Please review.

Project coverage is 82.29%. Comparing base (e0d191d) to head (990c949).

Files with missing lines	Patch %	Lines
balancer/rls/control_channel.go	92.85%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8055      +/-   ##
==========================================
- Coverage   82.29%   82.29%   -0.01%     
==========================================
  Files         387      387              
  Lines       39065    39081      +16     
==========================================
+ Hits        32150    32163      +13     
+ Misses       5584     5581       -3     
- Partials     1331     1337       +6     

Files with missing lines	Coverage Δ
balancer/rls/control_channel.go	92.51% <92.85%> (-0.62%)	⬇️

... and 20 files with indirect coverage changes

[easwars]

Can you try 10K or 1M runs in forge before and after the fix to ensure that flakes are eliminated by the fix?

[easwars]

This is an error we don't expect to happen in practice and if it does, it indicates a severe programming error. I would be OK to add a panic here that includes the type being received.

[easwars]

When a new subscriber is added to the pubsub, it receives the most recent message posted on the pubsub. This means that if there were N messages posted on the pubsub when a new subscriber is added, the subscriber only receives the most recently posted message. This might not be good enough for our purposes here. So, I suggest making the following changes.

• Get rid of the ccStateSubscriber. Instead store the buffer.Unbounded as a field of controlChannel.
• Initialize the unbounded buffer when the control channel is created in newControlChannel.
• Change grpc.Dial to grpc.NewClient in newControlChannel.
• Register the subscriber right after creating the ClientConn to the RLS server, but before calling Connect on it. This will ensure that the subscriber will receive every single state change on the ClientConn.
  • Implement the OnMessage method on the controlChannel type and pass it to the call to internal.SubscribeToConnectivityStateChanges

[easwars]

This would move to controlChannel.close.

[easwars]

Can we define this variable to be of the concrete type connectivity.State instead?

[easwars]

Do we need a for loop here? We know for a fact that we are in READY. So, the first time we actually read anything out of the unbounded buffer, we can be sure that we have moved out of READY. Am I missing something?