100% when running the docker image


#1

Hi,

When starting the docker image and not using the api yet i already see 100% cpu usage. Is this normal?


#2

Hello, can you share your usage logs?


#3

*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh…
*** Running /etc/my_init.d/10_syslog-ng.init…
Mar 22 20:03:03 a322b4034202 syslog-ng[10]: syslog-ng starting up; version=‘3.5.6’
Mar 22 20:03:04 a322b4034202 syslog-ng[10]: EOF on control channel, closing connection;
*** Running /etc/rc.local…
*** Booting runit daemon…
*** Runit started as PID 16
Mar 22 20:03:04 a322b4034202 cron[23]: (CRON) INFO (pidfile fd = 3)
Mar 22 20:03:04 a322b4034202 cron[23]: (CRON) INFO (Skipping @reboot jobs – not system startup)

DeepStack is Active

Active APIs:

/v1/vision/detection

v1/vision/addmodel

v1/vision/listmodels

v1/vision/deletemodel

v1/vision/setadminkey

v1/vision/setapikey

v1/vision/backup

v1/vision/restore
[GIN] 2019/03/22 - 20:03:13 | 200 | 7.108284541s | 172.17.0.1 | POST /v1/vision/detection
[GIN] 2019/03/22 - 20:03:16 | 200 | 2.373163841s | 172.17.0.1 | POST /v1/vision/detection
[GIN] 2019/03/22 - 20:03:20 | 200 | 2.365908038s | 172.17.0.1 | POST /v1/vision/detection
[GIN] 2019/03/22 - 20:03:23 | 200 | 2.189618926s | 172.17.0.1 | POST /v1/vision/detection
Process Process-1:
Traceback (most recent call last):
File “/usr/lib/python3.5/multiprocessing/process.py”, line 249, in _bootstrap
self.run()
File “/usr/lib/python3.5/multiprocessing/process.py”, line 93, in run
self._target(*self._args, **self._kwargs)
File “/app/intelligence.py”, line 643, in detection
for i in range(len(labels)):
TypeError: object of type ‘NoneType’ has no len()
*** Shutting down runit daemon (PID 16)…
*** Running /etc/my_init.post_shutdown.d/10_syslog-ng.shutdown…
Mar 22 20:04:01 a322b4034202 syslog-ng[10]: syslog-ng shutting down; version=‘3.5.6’
Mar 22 20:04:01 a322b4034202 syslog-ng[10]: EOF on control channel, closing connection;
*** Killing all processes…


#4

But I see the 100% cpu when i’m not polling at all as well.


#5

Hello, did you obtain your cpu usage through docker stats?

Also, there is an error in your log, the error has been fixed in the latest image


#6

Yes, this error was from the previous version. Issues is the same with current version. 100% cpu is seen via top. nothing but normal api calls in the log. when not doing any api calls cpu is steady at 100%.

this is the process:

root 21683 21680 96 10:50 ? 04:07:22 python3 /app/intelligence.py


#7

Thanks for the clarification. We have investigated this and found the cause of the problem.

In about a week, we shall release a new update that will fix this and also increase the speed of most of the APIs by about 2X.


#8

awesome, can’t wait! :stuck_out_tongue:


#9

Thanks for your patience. This has been fixed in the latest release.


#10

@john, I guess that version with the fix was not released yet, right?
was it fixed for GPU (AVX) version as well?

Can I raise the log level to see also info messages?
the reason I’m asking is that at some point after 5-10 minutes of sending requests every 2 seconds to detect, the intelligence.py process gets to 100% CPU

Tried it in both GPU 3.4 AVX and non-AVX versions


#11

Hello @elad.bar, occasionally, cpu utilization might go up to 100%, however, it will drop down depending on the number of requests you are sending. The previous version was always at 100% all the time. However, that has been fixed. Does this 100% utilization persist permanently, or just an occasional burst of usage?


#12

Stays until I restart the container, I will upload later graphs of yesterday’s statistics (CPU / MEM / requests)


#13

hi @john, image below represent yesterday’s statsiticsm
as you can see, when the CPU goes up, the memory get reduced, number of messages gets to 0, at ~15:00 I moved back to the implementation that calls just the recognize endpoint and problem solved, but less efficiant.

Since I’m using the GPU version,I assume that when the CPU goes up it can indicate that something is getting stuck


#14

Hi, thanks for sharing the graph. We are working on releasing a minor update in the next couple of days. We have observed some performance issues on some GPUs and would fix them in the next update.


#15

Hi, I saw that CPU version was updated while GPU not yet, when do you think it will be updated

Thanks


#16

Hello, Happy easter holidays!. The cpu update fixed a bug, the gpu update would be released soon. Thanks for your patience.