100% when running the docker image


When starting the docker image and not using the api yet i already see 100% cpu usage. Is this normal?

Hello, can you share your usage logs?

*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh…
*** Running /etc/my_init.d/10_syslog-ng.init…
Mar 22 20:03:03 a322b4034202 syslog-ng[10]: syslog-ng starting up; version=‘3.5.6’
Mar 22 20:03:04 a322b4034202 syslog-ng[10]: EOF on control channel, closing connection;
*** Running /etc/rc.local…
*** Booting runit daemon…
*** Runit started as PID 16
Mar 22 20:03:04 a322b4034202 cron[23]: (CRON) INFO (pidfile fd = 3)
Mar 22 20:03:04 a322b4034202 cron[23]: (CRON) INFO (Skipping @reboot jobs – not system startup)

DeepStack is Active

Active APIs:








[GIN] 2019/03/22 - 20:03:13 | 200 | 7.108284541s | | POST /v1/vision/detection
[GIN] 2019/03/22 - 20:03:16 | 200 | 2.373163841s | | POST /v1/vision/detection
[GIN] 2019/03/22 - 20:03:20 | 200 | 2.365908038s | | POST /v1/vision/detection
[GIN] 2019/03/22 - 20:03:23 | 200 | 2.189618926s | | POST /v1/vision/detection
Process Process-1:
Traceback (most recent call last):
File “/usr/lib/python3.5/multiprocessing/process.py”, line 249, in _bootstrap
File “/usr/lib/python3.5/multiprocessing/process.py”, line 93, in run
self._target(*self._args, **self._kwargs)
File “/app/intelligence.py”, line 643, in detection
for i in range(len(labels)):
TypeError: object of type ‘NoneType’ has no len()
*** Shutting down runit daemon (PID 16)…
*** Running /etc/my_init.post_shutdown.d/10_syslog-ng.shutdown…
Mar 22 20:04:01 a322b4034202 syslog-ng[10]: syslog-ng shutting down; version=‘3.5.6’
Mar 22 20:04:01 a322b4034202 syslog-ng[10]: EOF on control channel, closing connection;
*** Killing all processes…

But I see the 100% cpu when i’m not polling at all as well.

Hello, did you obtain your cpu usage through docker stats?

Also, there is an error in your log, the error has been fixed in the latest image

Yes, this error was from the previous version. Issues is the same with current version. 100% cpu is seen via top. nothing but normal api calls in the log. when not doing any api calls cpu is steady at 100%.

this is the process:

root 21683 21680 96 10:50 ? 04:07:22 python3 /app/intelligence.py

Thanks for the clarification. We have investigated this and found the cause of the problem.

In about a week, we shall release a new update that will fix this and also increase the speed of most of the APIs by about 2X.

awesome, can’t wait! :stuck_out_tongue:

1 Like

Thanks for your patience. This has been fixed in the latest release.

@john, I guess that version with the fix was not released yet, right?
was it fixed for GPU (AVX) version as well?

Can I raise the log level to see also info messages?
the reason I’m asking is that at some point after 5-10 minutes of sending requests every 2 seconds to detect, the intelligence.py process gets to 100% CPU

Tried it in both GPU 3.4 AVX and non-AVX versions

Hello @elad.bar, occasionally, cpu utilization might go up to 100%, however, it will drop down depending on the number of requests you are sending. The previous version was always at 100% all the time. However, that has been fixed. Does this 100% utilization persist permanently, or just an occasional burst of usage?

Stays until I restart the container, I will upload later graphs of yesterday’s statistics (CPU / MEM / requests)

hi @john, image below represent yesterday’s statsiticsm
as you can see, when the CPU goes up, the memory get reduced, number of messages gets to 0, at ~15:00 I moved back to the implementation that calls just the recognize endpoint and problem solved, but less efficiant.

Since I’m using the GPU version,I assume that when the CPU goes up it can indicate that something is getting stuck

Hi, thanks for sharing the graph. We are working on releasing a minor update in the next couple of days. We have observed some performance issues on some GPUs and would fix them in the next update.

1 Like

Hi, I saw that CPU version was updated while GPU not yet, when do you think it will be updated


1 Like

Hello, Happy easter holidays!. The cpu update fixed a bug, the gpu update would be released soon. Thanks for your patience.

I am having a similar issue when using the GPU image.

Deepstack will sit there quite happily for hours detecting objects, then as soon as night time falls and my cameras switch to IR, after 1 or 2 scans the CPU will sit at 100% and the API becomes unresponsive until I restart the container.

Happy to provide additional logs / debugging info because currently I have had to add a healthcheck the stack in order to kill the container off when this happens, which is throughout the night… but this is less than ideal as the healthcheck itself is consuming CPU and GPU cycles.

The way I have achieved the healthcheck is by adding ‘curl’ to the image, bind mounting a directory with a sample image inside and then adding this to my docker-compose.yaml:

  test: ["CMD-SHELL", "curl -F 'image=@/images/person.jpg' http://localhost:5000/v1/vision/detection || exit 1"]
  interval: 60s
  timeout: 1s
  retries: 1

Any thoughts?