Containers spawned through docker service doesn't get DNS names through Contiv IPAM driver

[rkharya]

Description

Containers spawned using Contiv network and Contiv IPAM driver fails to get DNS names and host names. Container host name takes container id instead of container names as should be provided by Contiv IPAM driver as part of service discovery feature.

Expected Behavior

Contiv IPAM driver should generated the DNS names for containers while giving IP address to them and use those name as host names for the containers through docker service create work-flow.

Observed Behavior

Containers are getting names as randomly generated containers IDs

Steps to Reproduce (for bugs)

1. Create Contiv network with nw-tag
2. Create Docker network using contiv-tag & contiv network driver and IPAM driver
3. Spawn containers using docker service create using contiv network
4. Once service gets deployed log-in to the containers and check the hostnames

Your Environment

• netctl version - 1.1.1
• Orchestrator version (e.g. kubernetes, mesos, swarm): UCP/Swarm - 2.2.2
• Operating System and version - RHEL 7.3 Bare Metal cluster nodes

Initial triage -

1).
[root@DEE-Ctrl-1 ~]# netctl network ls -a
Tenant Network Nw Type Encap type Packet tag Subnet Gateway IPv6Subnet IPv6Gateway Cfgd Tag

---

tenant-1 contiv-test-1 data vlan 1001 100.100.100.0/24 100.100.100.254 tenant-1-test-1
tenant-2 contiv-test-2 data vlan 1002 200.200.200.0/24 200.200.200.254 tenant-2-test-2

2).
[root@DEE-Ctrl-1 ~]# docker network ls
NETWORK ID NAME DRIVER SCOPE
701a8b77777f bridge bridge local
0xebqmj20v6s contiv-test-1 contiv/v2plugin:1.1.1 swarm
gegwyxkagj7u contiv-test-2 contiv/v2plugin:1.1.1 swarm
799fe436bc46 docker_gwbridge bridge local
o6a7ob7q8aqa dtr-ol overlay swarm
61735572149c host host local
swcu9ofu7amv ingress overlay swarm
gzs3dqjm2eln my-overlay overlay swarm
a2295b13a327 none null local

3).
RKHARYA-M-H07C:ucp-17.06-B rkharya$ docker service create --name apline-tenant-1 --network contiv-test-1 alpine sleep 100000
xdz75i68l9p4dxv7zvwrm01yp

4).
RKHARYA-M-H07C:ucp-17.06-B rkharya$ docker service scale apline-tenant-1=8
apline-tenant-1 scaled to 8

[root@DEE-Ctrl-1 ~]# docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
8wqtn92lavk9 ucp-agent-s390x global 0/0 docker/ucp-agent-s390x:2.2.2
i3ttsfu22lcv ucp-agent global 10/10 docker/ucp-agent:2.2.2
vvqbvxt8zdas busybox-tenant-2 replicated 8/8 busybox:latest
xaapf3q2n34x ucp-agent-win global 0/0 docker/ucp-agent-win:2.2.2
xdz75i68l9p4 apline-tenant-1 replicated 8/8 alpine:latest

5).
RKHARYA-M-H07C:ucp-17.06-B rkharya$ docker service ps apline-tenant-1
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
8msxuxk1wlf0 apline-tenant-1.1 alpine:latest DEE-Ctrl-1.cisco.com Running Running about a minute ago
e5n36ut19kcu apline-tenant-1.2 alpine:latest DEE-Wrk-1.cisco.com Running Running 36 seconds ago
9qqln1emoz28 apline-tenant-1.3 alpine:latest DEE-Wrk-2.cisco.com Running Running 36 seconds ago
gc97stbogy20 apline-tenant-1.4 alpine:latest DEE-DTR-1.cisco.com Running Running 36 seconds ago
2aop3g8x2349 apline-tenant-1.5 alpine:latest DEE-DTR-2.cisco.com Running Running 36 seconds ago
khsby8641gvw apline-tenant-1.6 alpine:latest DEE-Ctrl-2.cisco.com Running Running 36 seconds ago
u1qbd6gzj9qn apline-tenant-1.7 alpine:latest DEE-DTR-3.cisco.com Running Running 36 seconds ago
hzjgcik16cvf apline-tenant-1.8 alpine:latest DEE-Wrk-4.cisco.com Running Running 36 seconds ago

6).
RKHARYA-M-H07C:ucp-17.06-B rkharya$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
df679daf7461 alpine:latest "sleep 100000" 10 seconds ago Up 7 seconds DEE-DTR-3.cisco.com/apline-tenant-1.7.u1qbd6gzj9qn2evgm7vukfz16
deb250d36ca1 alpine:latest "sleep 100000" 10 seconds ago Up 6 seconds DEE-Ctrl-2.cisco.com/apline-tenant-1.6.khsby8641gvwcjwzvep8blxf7
e99534004598 alpine:latest "sleep 100000" 10 seconds ago Up 7 seconds DEE-Wrk-1.cisco.com/apline-tenant-1.2.e5n36ut19kcu3ywj0yevyaopf
f663d5dd6291 alpine:latest "sleep 100000" 10 seconds ago Up 6 seconds DEE-Wrk-4.cisco.com/apline-tenant-1.8.hzjgcik16cvf15rfrgmq9lm3n
9ced4eec08d1 alpine:latest "sleep 100000" 10 seconds ago Up 7 seconds DEE-Wrk-2.cisco.com/apline-tenant-1.3.9qqln1emoz28un9tunx8alzlx
647fad882b94 alpine:latest "sleep 100000" 10 seconds ago Up 7 seconds DEE-DTR-1.cisco.com/apline-tenant-1.4.gc97stbogy20vmubfe8fddvf9
350756ffb379 alpine:latest "sleep 100000" 10 seconds ago Up 7 seconds DEE-DTR-2.cisco.com/apline-tenant-1.5.2aop3g8x23

7).
RKHARYA-M-H07C:ucp-17.06-B rkharya$ docker exec -it df679daf7461 /bin/sh
/ # cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
100.100.100.7 df679daf7461

8).
[root@DEE-Ctrl-1 ~]# netctl network inspect contiv-test-1 -t tenant-1
{
"Config": {
"key": "tenant-1:contiv-test-1",
"cfgdTag": "tenant-1-test-1",
"encap": "vlan",
"gateway": "100.100.100.254",
"networkName": "contiv-test-1",
"nwType": "data",
"pktTag": 1001,
"subnet": "100.100.100.0/24",
"tenantName": "tenant-1",
"link-sets": {},
"links": {
"Tenant": {
"type": "tenant",
"key": "tenant-1"
}
}
},
"Oper": {
"allocatedAddressesCount": 9,
"allocatedIPAddresses": "100.100.100.1-100.100.100.9, 100.100.100.254",
"availableIPAddresses": "100.100.100.10-100.100.100.253",
"endpoints": [
{


{
"endpointID": "22f00a7fd5444afaffbc3650d25452db0a0c5b5359b5b307cd043de6c8a5bbeb",
"homingHost": "DEE-DTR-3.cisco.com",
"ipAddress": [
"100.100.100.7",
""
],

[rchirakk]

docker service create has
--hostname option if that is what you are looking to configure.

[rkharya]

we are not looking for docker provided --hostname option. That option gives the container hostname to the first container only. Rest of the containers in the service get the same name as first container.

This issue is to report Contiv IPAM driver's service discovery feature not working correctly with docker service model.

If we use docker run, legacy way of deploying container, container name specified to docker run command also gets attached to /etc/hosts file inside container correctly by contiv IPAM driver. so that containers are knows in the network with their names. With docker service its broken.

[rhim]

@rkharya Looking at https://docs.docker.com/engine/reference/commandline/service_create/, I don't see any requirement around container naming. Can you please point us to the requirement you mention above (i.e., the network driver must name containers based of some scheme?)

My quick experiment locally w/o Contiv shows the same behavior as what you see with Contiv above:

➜  ~ docker service create --name redis --replicas=5 redis:3.0.6
0t9yw26srx2tps3xktj6fvuod
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.
➜  ~ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
0t9yw26srx2t        redis               replicated          0/5                 redis:3.0.6

➜  ~ docker service ps redis
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
0a7w5j1f8h5o        redis.1             redis:3.0.6         moby                Running             Running 11 seconds ago
9jn8v3vvj1tj        redis.2             redis:3.0.6         moby                Running             Running 11 seconds ago
u6hahgnepxkm        redis.3             redis:3.0.6         moby                Running             Running 11 seconds ago
f9mythqsdbpf        redis.4             redis:3.0.6         moby                Running             Running 11 seconds ago
6ig0j6zyrwkj        redis.5             redis:3.0.6         moby                Running             Running 11 seconds ago
➜  ~ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
ee2fb01bd10f        redis:3.0.6         "/entrypoint.sh re..."   27 seconds ago      Up 25 seconds       6379/tcp            redis.5.6ig0j6zyrwkjce7thho3zimi9
fb35f8eb5103        redis:3.0.6         "/entrypoint.sh re..."   27 seconds ago      Up 25 seconds       6379/tcp            redis.1.0a7w5j1f8h5obc8qiaccon8we
7969f8895a38        redis:3.0.6         "/entrypoint.sh re..."   27 seconds ago      Up 25 seconds       6379/tcp            redis.3.u6hahgnepxkmo7non1n90ackf
59a01c9aa89d        redis:3.0.6         "/entrypoint.sh re..."   27 seconds ago      Up 25 seconds       6379/tcp            redis.2.9jn8v3vvj1tjgub9iryzijmoo
3d217641c73e        redis:3.0.6         "/entrypoint.sh re..."   27 seconds ago      Up 25 seconds       6379/tcp            redis.4.f9mythqsdbpfb8imilrrlzwlf

➜  ~ docker exec -it redis.5.6ig0j6zyrwkjce7thho3zimi9 /bin/sh
# more /etc/hosts
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.17.0.6	ee2fb01bd10f
#

[rkharya]

When we don't use docker service and use docker run and deploy container with Contiv IPAM driver, we do get /etc/hosts appended with container name. And we can reach out to deployed container on the network using that name.

Same is not true with docker service, which you also saw. So the ask is - as part of Contiv's service discovery feature, will this not be possible to have containers attached to a docker service known on the network through their host names? Essentially, /etc/hosts should get appended with the container names rather than container IDs. Is that falls under the scope of Contiv service discovery feature or its a docker service limitation.

Hope i am able to explain what is intended cause of the issue.

[gkvijay]

@rkharya, network plugin does not update the /etc/hosts inside the container. This is done by docker in CNM. For Contiv to know the containers attached to the service, Docker should expose the service APIs. Without that the plugins will not know about the service VIP and backend IP mapping. Docker also services the DNS requests and contiv doesn't see it. We need to discuss with docker folks to see how your use case can be addressed

[uday-shetty]

@rkharya moby/libnetwork#1855