[Licensing plugin] Reduce startup time by applying a more aggresive retry strategy

[afharo]

Our current licensing plugin polls the licensing information periodically (default: every 30s).

Relevant pieces of code:

• Trigger definition:

  kibana/x-pack/plugins/licensing/server/plugin.ts

  Lines 116 to 122 in f3f53e0

  	// Trigger whenever the timer ticks or ES becomes available
  	const intervalRefresh$ = this.isElasticsearchAvailable$.pipe(
  	distinctUntilChanged(),
  	filter((isElasticsearchAvailable) => isElasticsearchAvailable),
  	switchMap(() => timer(0, pollingFrequency).pipe(takeUntil(isElasticsearchNotAvailable$))),
  	throttleTime(pollingFrequency) // avoid triggering too often
  	);

• Actual fetching:

  kibana/x-pack/plugins/licensing/common/license_update.ts

  Lines 32 to 36 in 47a497c

  	const fetched$ = merge(triggerRefresh$, manuallyRefresh$).pipe(
  	takeUntil(stop$),
  	exhaustMap(fetcher),
  	share()
  	);

When the license fails to be fetched, Kibana waits for the next polling trigger to kick in, which leads to Kibana taking 30s for the next retry to occur. The main issue here is that network issues or ES starting up might lead to Kibana taking extra 30s to be ready to address authenticated HTTP traffic after startup.

We could improve this situation by applying a retry with back-off strategy that can retry the license fetch on failure, after a progressively increasing interval until we reach the actual polling interval (1s, 2s, 4s, 8s, 16s, 30s, 30s, ...).

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)