Skip to content
This repository has been archived by the owner on Aug 28, 2023. It is now read-only.

Can't see container CPU/Memory usage. #3

Open
shabbirkagalwala opened this issue Jun 30, 2017 · 8 comments
Open

Can't see container CPU/Memory usage. #3

shabbirkagalwala opened this issue Jun 30, 2017 · 8 comments
Assignees
Labels

Comments

@shabbirkagalwala
Copy link

I started the service for swi-ui and containers for the agent using docker run on each node of the swarm. I was also able to add the add nodes using the web-ui but now for some reason, the container stats i.e CPU and Memory just show 0.00% and null values. (see screenshot)

My service create command:
docker service create \ --name swi-ui \ --replicas 1 \ --network webhelpdesk --network proxy \ --label com.df.notify=true --label com.df.distribute=true \ --label com.df.servicePath=/ --label com.df.port=80 \ solarwinds/container-ui

Docker run command:
docker run -d --name swi-agent --privileged --net=host --restart always -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro solarwinds/container-agent

capture1

Port 9090 is open and I can see ./agent using this port via the command netstat -tuplen.

Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 12060078 15412/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 17088 2406/master
tcp6 0 0 :::9090 :::* LISTEN 0 237464630 6037/./agent
tcp6 0 0 :::9000 :::* LISTEN 0 219767543 21355/dockerd
tcp6 0 0 :::4040 :::* LISTEN 0 29844626 17522/scope-app
tcp6 0 0 :::7946 :::* LISTEN 0 29240890 21355/dockerd
tcp6 0 0 :::80 :::* LISTEN 0 114601075 21355/dockerd
tcp6 0 0 :::8081 :::* LISTEN 0 196936481 11774/docker-proxy
tcp6 0 0 :::8082 :::* LISTEN 0 79188916 28217/docker-proxy
tcp6 0 0 :::22 :::* LISTEN 0 12060080 15412/sshd
tcp6 0 0 :::8089 :::* LISTEN 0 103626226 2435/docker-proxy
tcp6 0 0 ::1:25 :::* LISTEN 0 17089 2406/master
tcp6 0 0 :::443 :::* LISTEN 0 114602087 21355/dockerd
tcp6 0 0 :::8443 :::* LISTEN 0 79188905 28205/docker-proxy
udp 0 0 0.0.0.0:4789 0.0.0.0:* 0 29240926 -
udp6 0 0 :::7946 :::* 0 29240891 21355/dockerd

Are there any additional ports that are being used to display the container stats? If not, what may be causing the issue here?

@derhally
Copy link
Contributor

What version of docker are you running?

Can you take a post logs from the agent and the ui containers. to get the logs you can run

docker logs swi-ui

docker logs swi-agent

@shabbirkagalwala
Copy link
Author

This is the log for the agent:

[root@ip-192-168-4-4 centos]# docker logs f32
starting agent
network mode[host] container: f32ddf3afb81292fbdaf0a6f5491c93692c158dd675870521cebee361cfcc34d pid: 14537
inventory - NetworkMode is host skipping container f32ddf3afb81292fbdaf0a6f5491c93692c158dd675870521cebee361cfcc34d
network mode[default] container: d7be3eca828ec06cfb3f065e197ef7097b886f93289f026235b0b10b1d439648 pid: 3739
network mode[default] container: bdc64876969c184c3c8bffa7f9073522665685c76ebdff468a7c64b17faf4d07 pid: 2380
network mode[default] container: f0a44f06dcdd5a69c4a232193954a21d82caba418c18d1617e59e3c0aebac065 pid: 2408
2017/07/01 02:37:21 serving on port 9090
2017/07/01 02:37:34 Client connecting from '192.168.0.108:56082'
message received [{"type":"subscription","modules":["api.Pcap", "api.Iperf", "api.Inventory", "api.StatsSummary", "api.Event", "api.PingResult"]}]

2017/07/01 02:19:57 Sending request to http://192.168.4.4:9090/ping
2017/07/01 02:19:57 Sending request to http://192.168.3.78:9090/ping
2017/07/01 02:19:57 Sending request to http://192.168.3.131:9090/ping
2017/07/01 02:19:57 192.168.3.78 ping 1ms
2017/07/01 02:19:57 192.168.4.4 ping 1ms
2017/07/01 02:19:57 192.168.3.131 ping 1ms
2017/07/01 02:20:27 Running ping against 3 hosts
2017/07/01 02:20:27 Sending request to http://192.168.4.4:9090/ping
2017/07/01 02:20:27 Sending request to http://192.168.3.78:9090/ping
2017/07/01 02:20:27 Sending request to http://192.168.3.131:9090/ping
2017/07/01 02:20:27 192.168.3.78 ping 0ms
2017/07/01 02:20:27 192.168.3.131 ping 1ms
2017/07/01 02:20:27 192.168.4.4 ping 2ms
2017/07/01 02:20:57 Running ping against 3 hosts
2017/07/01 02:20:57 Sending request to http://192.168.4.4:9090/ping
2017/07/01 02:20:57 Sending request to http://192.168.3.78:9090/ping
2017/07/01 02:20:57 Sending request to http://192.168.3.131:9090/ping
2017/07/01 02:20:57 192.168.3.131 ping 1ms
2017/07/01 02:20:57 192.168.4.4 ping 1ms
2017/07/01 02:20:57 192.168.3.78 ping 4ms
2017/07/01 02:21:27 Running ping against 3 hosts
2017/07/01 02:21:27 Sending request to http://192.168.4.4:9090/ping
2017/07/01 02:21:27 Sending request to http://192.168.3.78:9090/ping
2017/07/01 02:21:27 Sending request to http://192.168.3.131:9090/ping
2017/07/01 02:21:27 192.168.4.4 ping 2ms
2017/07/01 02:21:27 192.168.3.131 ping 2ms
2017/07/01 02:21:27 192.168.3.78 ping 3ms
2017/07/01 02:21:57 Running ping against 3 hosts
2017/07/01 02:21:57 Sending request to http://192.168.4.4:9090/ping
2017/07/01 02:21:57 Sending request to http://192.168.3.78:9090/ping
2017/07/01 02:21:57 Sending request to http://192.168.3.131:9090/ping
2017/07/01 02:21:57 192.168.3.78 ping 0ms
2017/07/01 02:21:57 192.168.4.4 ping 1ms
2017/07/01 02:21:57 192.168.3.131 ping 1ms
2017/07/01 02:22:27 Running ping against 3 hosts
2017/07/01 02:22:27 Sending request to http://192.168.4.4:9090/ping
2017/07/01 02:22:27 Sending request to http://192.168.3.78:9090/ping
2017/07/01 02:22:27 Sending request to http://192.168.3.131:9090/ping
2017/07/01 02:22:27 192.168.3.131 ping 1ms
2017/07/01 02:22:27 192.168.3.78 ping 1ms
2017/07/01 02:22:27 192.168.4.4 ping 1ms
2017/07/01 02:22:57 Running ping against 3 hosts
2017/07/01 02:22:57 Sending request to http://192.168.4.4:9090/ping
2017/07/01 02:22:57 Sending request to http://192.168.3.78:9090/ping
2017/07/01 02:22:57 Sending request to http://192.168.3.131:9090/ping
2017/07/01 02:22:57 192.168.3.131 ping 1ms
2017/07/01 02:22:57 192.168.4.4 ping 1ms
2017/07/01 02:22:57 192.168.3.78 ping 4ms
2017/07/01 02:23:27 Running ping against 3 hosts
2017/07/01 02:23:27 Sending request to http://192.168.4.4:9090/ping
2017/07/01 02:23:27 Sending request to http://192.168.3.78:9090/ping
2017/07/01 02:23:27 Sending request to http://192.168.3.131:9090/ping
2017/07/01 02:23:27 192.168.4.4 ping 3ms
2017/07/01 02:23:27 192.168.3.131 ping 4ms
2017/07/01 02:23:27 192.168.3.78 ping 4ms
2017/07/01 02:23:57 Running ping against 3 hosts
2017/07/01 02:23:57 Sending request to http://192.168.4.4:9090/ping
2017/07/01 02:23:57 Sending request to http://192.168.3.78:9090/ping
2017/07/01 02:23:57 Sending request to http://192.168.3.131:9090/ping
2017/07/01 02:23:57 192.168.3.131 ping 0ms
2017/07/01 02:23:57 192.168.3.78 ping 1ms
2017/07/01 02:23:57 192.168.4.4 ping 1ms
2017/07/01 02:24:27 Running ping against 3 hosts
2017/07/01 02:24:27 Sending request to http://192.168.4.4:9090/ping
2017/07/01 02:24:27 Sending request to http://192.168.3.78:9090/ping
2017/07/01 02:24:27 Sending request to http://192.168.3.131:9090/ping
2017/07/01 02:24:27 192.168.3.78 ping 1ms
2017/07/01 02:24:27 192.168.3.131 ping 1ms
2017/07/01 02:24:27 192.168.4.4 ping 2ms
2017/07/01 02:24:57 Running ping against 3 hosts
2017/07/01 02:24:57 Sending request to http://192.168.4.4:9090/ping
2017/07/01 02:24:57 Sending request to http://192.168.3.78:9090/ping
2017/07/01 02:24:57 Sending request to http://192.168.3.131:9090/ping
2017/07/01 02:24:57 192.168.3.131 ping 1ms
2017/07/01 02:24:57 192.168.3.78 ping 1ms
2017/07/01 02:24:57 192.168.4.4 ping 1ms
2017/07/01 02:25:27 Running ping against 3 hosts
2017/07/01 02:25:27 Sending request to http://192.168.4.4:9090/ping
2017/07/01 02:25:27 Sending request to http://192.168.3.78:9090/ping
2017/07/01 02:25:27 Sending request to http://192.168.3.131:9090/ping
2017/07/01 02:25:27 192.168.4.4 ping 3ms
2017/07/01 02:25:27 192.168.3.131 ping 4ms
2017/07/01 02:25:27 192.168.3.78 ping 5ms
2017/07/01 02:25:57 Running ping against 3 hosts
2017/07/01 02:25:57 Sending request to http://192.168.4.4:9090/ping
2017/07/01 02:25:57 Sending request to http://192.168.3.78:9090/ping
2017/07/01 02:25:57 Sending request to http://192.168.3.131:9090/ping
2017/07/01 02:25:57 192.168.3.131 ping 0ms
2017/07/01 02:25:57 192.168.3.78 ping 1ms
2017/07/01 02:25:57 192.168.4.4 ping 1ms

For the web ui:

[root@ip-192-168-0-108 centos]# docker logs e6d
Hosting environment: Production
Content root path: /app
Now listening on: http://+:80
Application started. Press Ctrl+C to shut down.
info: CMonService.Services.AgentConnectionManager[0]
Added listener for host 'ip-192-168-0-108.us-west-2.compute.internal'
info: CMonService.Services.AgentConnectionManager[0]
Added listener for host 'ip-192-168-3-131.us-west-2.compute.internal'
info: CMonService.Services.AgentConnectionManager[0]
Added listener for host 'ip-192-168-3-78.us-west-2.compute.internal'
info: CMonService.Services.AgentConnectionManager[0]
Added listener for host 'ip-192-168-4-4.us-west-2.compute.internal'
fail: CMonService.Agent.AgentEventListener[0]
Error reading from socket for 'ws://192.168.4.4:9090/ws/stream'.
System.Net.WebSockets.WebSocketException: The 'System.Net.WebSockets.InternalClientWebSocket' instance cannot be used for communication because it has been transitioned into the 'Aborted' state. ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'ClientWebSocket'.
at System.Net.WebSockets.ManagedWebSocket.d__69.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.WebSockets.ManagedWebSocket.d__60.MoveNext()
--- End of inner exception stack trace ---
at System.Net.WebSockets.ManagedWebSocket.d__60.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CMonService.Agent.AgentEventListener.d__12.MoveNext() in D:\source\cmontestui\src\CMonService\Agent\AgentEventListener.cs:line 100
fail: CMonService.Services.AgentRegistry[0]
Error adding agent on host '192.168.4.4:9090'
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.HttpClient.d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CMonService.Agent.AgentClient.d__3.MoveNext() in D:\source\cmontestui\src\CMonService\Agent\AgentClient.cs:line 32
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CMonService.Services.AgentRegistry.d__5.MoveNext() in D:\source\cmontestui\src\CMonService\Services\AgentRegistry.cs:line 46
info: CMonService.Services.AgentConnectionManager[0]
Added listener for host 'ip-192-168-4-4.us-west-2.compute.internal'
fail: CMonService.Agent.AgentEventListener[0]
Error reading from socket for 'ws://192.168.4.4:9090/ws/stream'.
System.Net.WebSockets.WebSocketException: The 'System.Net.WebSockets.InternalClientWebSocket' instance cannot be used for communication because it has been transitioned into the 'Aborted' state. ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'AsyncEventArgsNetworkStream'.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.WebSockets.ManagedWebSocket.d__69.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.WebSockets.ManagedWebSocket.d__60.MoveNext()
--- End of inner exception stack trace ---
at System.Net.WebSockets.ManagedWebSocket.d__60.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CMonService.Agent.AgentEventListener.d__12.MoveNext() in D:\source\cmontestui\src\CMonService\Agent\AgentEventListener.cs:line 100
info: CMonService.Services.AgentConnectionManager[0]
Added listener for host 'ip-192-168-4-4.us-west-2.compute.internal'

Docker Version

Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:05:44 2017
OS/Arch: linux/amd64

Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:05:44 2017
OS/Arch: linux/amd64
Experimental: false

OS Version

CentOS Linux release 7.3.1611 (Core)

@derhally
Copy link
Contributor

derhally commented Jul 3, 2017 via email

@shabbirkagalwala
Copy link
Author

I am using a service for the web ui, i did try removing the service, adding it again, adding the nodes/hosts again but still couldn't see the CPU and memory usage. It is strange as I am able to add nodes/hosts, see the containers running on the nodes, can even see the topology (without any connections) but can't see the CPU and memory usage.

Are there any additional ports that the web ui uses to communicate with the agent containers except for 9090?

@derhally
Copy link
Contributor

derhally commented Jul 5, 2017

Port 9090 is the only port that needs to be open between the UI service and the agents. The log shows that the websocket is being closed right after it is established. The UI tries to maintain a persistent websocket connection with the agent in order to receive the stats.

@shabbirkagalwala
Copy link
Author

shabbirkagalwala commented Jul 5, 2017 via email

@leecalcote
Copy link
Contributor

@shabbirkagalwala you bet. Here's another troubleshooting tactic - you can test for connectivity as well as hunt for the statistics you're missing for via the Agent's REST API by pointing your browser at any of these endpoints:

  • http://a-host-running-the-agent:9090/environment
  • http://a-host-running-the-agent:9090/networks
  • http://a-host-running-the-agent:9090/containers
  • http://a-host-running-the-agent:9090/containers/container-id

@derhally can point to which of these should have the CPU & memory statistics.

@shabbirkagalwala
Copy link
Author

shabbirkagalwala commented Jul 6, 2017

@leecalcote Thank you for the response.

I tried all the links as you mentioned and I can see the output. For example:

{
"hostname": "ip-192-168-3-131.us-west-2.compute.internal",
"architecture": "x86_64",
"totalmemory": 7933190144,
"numberofcpu": 2,
"ostype": "linux",
"osversion": "CentOS Linux 7 (Core)",
"containerplatform": "Docker",
"containerplatformversion": "17.03.1-ce",
"containercount": 5,
"containerspaused": 0,
"containersrunning": 5,
"containersstopped": 0,
"numberofimages": 71,
"typesofnetworks": [
"bridge",
"host",
"macvlan",
"null",
"overlay"
]

I guess as mentioned by @derhally the agents not being able to communicate with the web-ui might be the issue. I am going to try tweaking my security groups and Network ACL's to see if anything helps and will get back here to report on what i find.

Thank you for all the help so far, truly appreciate it!

EDIT

Looking at the :9090/containers/container-id, this is what i found, it says 404 page not found at the end, could that be an issue?

{
"id": "b752b3b32d2849608e31d0a1413425782c19cd957846ad04998d736a1ec6141f",
"Names": [
"/swi-agent"
],
"Image": "solarwinds/container-agent",
"ImageID": "sha256:fb5347c04f225d0bae6ef2a42f5ba79f9c08c66886e537cd661586c34d25b864",
"Created": 1498850039,
"Ports": [],
"Labels": {},
"State": "running",
"Status": "Up 5 days",
"NetworkSettings": {
"Networks": {
"host": {
"Links": null,
"Aliases": null,
"NetworkID": "33811c2c6a416be08e65dfd67f3a16ee336833ff441a792002a0e4c4f4075f76",
"EndpointID": "962f69808fa9122e46e11fdc958102a01a2edd8eba86c7f01187298b82dae104",
"IPAddress": "",
"IPPrefixLen": 0,
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": ""
}
}
}
}404 page not found

EDIT 2

I opened up all ports and allowed traffic still cant see the stats. I think the ports aren't the issue now, let me know if I need to do anything else to troubleshoot.

@leecalcote leecalcote added question and removed bug labels Jul 17, 2017
@leecalcote leecalcote self-assigned this Jul 17, 2017
leecalcote pushed a commit that referenced this issue Jan 25, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Development

No branches or pull requests

3 participants