v1: add support for multiple endpoints

[rboyer]

If you run a replicated mesos-master setup (say N=3) then the current state of how you inform the HTTP API client of the masters is:

1. pick one, and use httpcli.Endpoint to save it
2. hope it never goes down
3. let the 307 redirection code ask any master for the leading master and update your favorite

This works ok enough until the selection from (1) is not responsive. Then you have to cobble together some sort of much higher level round-robin over a list of mesos-master addresses.

It would be cool if the httpcli.Client could take one of the following:

// NextEndpointFunc returns a new endpoint to try.  
type NextEndpointFunc func() string

// example of NextEndpointFunc
func simpleRoundRobin(urls []string) httpcli.NextEndpointFunc {
       var pos int
       return func() string {
               if pos >= len(urls) {
                       pos = 0
               }
               s := urls[pos]
               pos++
               return s
       }
}

// EndpointProviderFunc could hook into a service-discovery option or flat config file
type EndpointProviderFunc func() []string

// example of EndpointProviderFunc from a flat file:
func staticHosts(urls []string) httpcli.EndpointProviderFunc {
    return urls
}

And then have it rotate through the options down wherever c.url is referenced when there's an error indicative of a dead or bad master.

[rboyer]

This would relate to #338

[jdef]

I like the proposed interface. Which specific failures would trigger calls to such an interface? "no such host"? "connection timeout"? something else?

[mlowicki]

#352 is marked as merged. Does it mean that this ticket has been solved?

[jdef]

Yes I believe this is resolved. Closing out. Please re-open if you feel that something here has been left unaddressed.