GPU docker stops working after image post with no objects

Running Debian Buster with GPU docker version c7a476ed07b5 (created 2 weeks ago) with GTX 1650 4GB RAM
I’ve spent days trying to narrow this issue down, and it comes down to if I post an image to the docker that has no objects in it can find, the docker stops processing anything and the request just hangs. I can monitor the GPU using nvtop, which shows GPU usage and GPU mem usage. GPU memory usage stays at 25% and GPU usage hovers around 4% without submitting an image, but only jumps up to about 10-15% when sending an image. The image I’m sending are from a security camera, 640x480. I know the size isn’t the problem because I can send 4k+ images of other things (with objects in them) and it works fine. I even stressed tested it by sending about 5 parallel requests every second for over an hour (it was an image that actually worked). I finally made the connection of what was wrong when I was pulling an image from my camera every second and it worked for a while, the suddenly stopped. It was detecting a truck, and I think when the light changed, there were no objects to detect.

Now how I tested this was when running the docker container, I left off the “–gpus all” option so that it was only using the CPU. Sure enough, the same images I had been sending to deepstack that were causing it to stop responding, now worked and I verified through nvtop and looking at the CPU usage that deepstack was not using the GPU. So, there is something about using the GPU that causes the container to stop processing requests when there are no objects found. Also, when this happens, I see the GPU memory usage drop to 0 after this happens. To get it to work again, I have to restart the container. The response I get when I disable the GPU on the container and submit the image is {"success":true,"predictions":[]} I should be getting the same response when using the GPU, but instead, it just hangs.

You can recreate this error by posting a simple, single color image (for example, a 640x480 image that is nothing but the color orange, white, etc…) to the docker and it will hang.

I’m getting the same issue as well. Have to restart the container to get it working again.

1 Like

Yep, same issue here, HomeAssistant is forward security camera images but after a while Deepstack (GPU version) stops working and will only work after reboot.

Breaking the use case a bit :stuck_out_tongue:

Same here. This was driving me crazy. Couldn’t understand where the error is coming from

I am having a similar issue when using the GPU image.

Deepstack will sit there quite happily for hours detecting objects, then as soon as night time falls and my cameras switch to IR, after 1 or 2 scans the CPU will sit at 100% and the API becomes unresponsive until I restart the container.

Happy to provide additional logs / debugging info because currently I have had to add a healthcheck the stack in order to kill the container off when this happens, which is throughout the night… but this is less than ideal as the healthcheck itself is consuming CPU and GPU cycles.

The way I have achieved the healthcheck is by adding ‘curl’ to the image, bind mounting a directory with a sample image inside and then adding this to my docker-compose.yaml:

healthcheck:
  test: ["CMD-SHELL", "curl -F 'image=@/images/person.jpg' http://localhost:5000/v1/vision/detection || exit 1"]
  interval: 60s
  timeout: 1s
  retries: 1

DdI you guys install NVIDIA Container Toolkit?

Yes, it doesn’t run without out it installed (I tried). As stated, it works with GPU, but dies once there are NO objects in the picture)

Using the gpunightly issue seems to have been resolved.

I guess the same happens with the CPU version. Every once in a while the container will just hang. Using the latest container. Can this be fixed? I am using this program quite a bit!

I think that must be a different problem. I use the CPU version as a secondary check for person detection. I use Shinobi + YOLO, but sometimes YOLO has some false positives. So, for now, when Shinobi thinks there is a person, the image snapshot is sent to the CPU version of deepstack to check if there is an actual person. I’ve been running like this for many weeks and the CPU version hasn’t had any issues.

Where do you get the nightly version? I may just wait until the official docker release though.

Docker GPUNIGHTLY gpu has a lot more false positive than cpu especially in images that has lot of shadow. CPU seems to be best.

Hello @DevanNull , @ne0ark, @doodieb15, @zeliant, @Flight777, @jonwilliams84,

Sorry about this issues, we are working on a new version of DeepStack that will be free and opensource.
In the meantime, see below for the latest cpu and gpu images with fixes for the above issues, including a much more accurate object detection model.

Thanks for your patience

The same problem persists in this beta version where you can no longer process data through the server on that particular IP address I found using another IP address from different server still processes images whilst the first IP stops , and then eventually the second IP address no longer processes data, it seems after a few processes it’s blocking the IP addresses.

Hello @racingpassion, thanks for reporting this. Would investigate and try to reproduce this issue.

1 Like