GPU 2022.01.1 - no objects detected

can you share more details about the GPU specifications on your machine?

Sorry I’m not sure what more detail you are after - it’s a GTX 1660 Ti 6GB, Nvidia Linux driver 495.44 and CUDA 11.5. Previous container version (2021.09.1) works with the same configuration. All I changed was the container image tag in the compose file and redeployed it.

Can you debug DeepStack has detailed in the documentation linked below and post any errors found?

Both stdout.txt and stderr.txt are empty - this was after posting 4 images to the detection endpoint.

DeepStack: Version 2022.01.01
/v1/vision/detection
---------------------------------------
---------------------------------------
v1/backup
---------------------------------------
v1/restore
[GIN] 2022/01/21 - 12:25:15 | 200 |  897.495596ms |      172.17.0.1 | POST     "/v1/vision/detection"
[GIN] 2022/01/21 - 12:25:35 | 200 |  367.345615ms |      172.17.0.1 | POST     "/v1/vision/detection"
[GIN] 2022/01/21 - 12:25:54 | 200 |  371.430695ms |      172.17.0.1 | POST     "/v1/vision/detection"
[GIN] 2022/01/21 - 12:26:04 | 200 |  392.772015ms |      172.17.0.1 | POST     "/v1/vision/detection"

@OlafenwaMoses

cd /app/logs && cat stderr.txt
Traceback (most recent call last):
  File "/app/intelligencelayer/shared/detection.py", line 68, in <module>
    detector = YOLODetector(model_path, reso, cuda=CUDA_MODE)
  File "/app/intelligencelayer/shared/./process.py", line 31, in __init__
    self.model = attempt_load(model_path, map_location=self.device)
  File "/app/intelligencelayer/shared/./models/experimental.py", line 158, in attempt_load
    torch.load(w, map_location=map_location)["model"].float().fuse().eval()
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 607, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 882, in _load
    result = unpickler.load()
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 857, in persistent_load
    load_tensor(data_type, size, key, _maybe_decode_ascii(location))
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 846, in load_tensor
    loaded_storages[key] = restore_location(storage, location)
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 827, in restore_location
    return default_restore_location(storage, str(map_location))
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 175, in default_restore_location
    result = fn(storage, location)
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 157, in _cuda_deserialize
    return obj.cuda(device)
  File "/usr/local/lib/python3.7/dist-packages/torch/_utils.py", line 79, in _cuda
    return new_type(self.size()).copy_(self, non_blocking)
  File "/usr/local/lib/python3.7/dist-packages/torch/cuda/__init__.py", line 606, in _lazy_new
    return super(_CudaBase, cls).__new__(cls, *args, **kwargs)
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Traceback (most recent call last):
  File "/app/intelligencelayer/shared/scene.py", line 64, in <module>
    SharedOptions.CUDA_MODE,
  File "/app/intelligencelayer/shared/scene.py", line 40, in __init__
    self.model = self.model.cuda()
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 680, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 570, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 593, in _apply
    param_applied = fn(param)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 680, in <lambda>
    return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Traceback (most recent call last):
  File "/app/intelligencelayer/shared/face.py", line 45, in <module>
    cuda=SharedOptions.CUDA_MODE,
  File "/app/intelligencelayer/shared/./recognition/process.py", line 31, in __init__
    self.model = self.model.cuda()
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 680, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 570, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 570, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 593, in _apply
    param_applied = fn(param)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 680, in <lambda>
    return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

I posted this issue to IPCT the day the new version was released.

I’m running a GTX 1650 Super, and a P106-90, all are failing.

@ne0ark Thanks for sharing the error logs. As you can see, there is a RuntimeError: CUDA error: out of memory error from DeepStack, which means the compute required to process the request exceeded your available GPU memory. I will suggest you do the following

  • If you only need the detection endpoint but other endpoints are enabled, stop and start DeepStack again with only DETECTION=True enabled
  • If you have other programs using your GPU memory but you don’t need at the time of using DeepStack, stop the other programs
  • If you didn’t specify a MODE while running DeepStack, it will run in Medium mode. I will advice specify -e MODE=Low as instructed in this documentation , to reduce GPU memory usage

@speedst3r Interesting. Is it possible to share the images you are sending to DeepStack with via (probably via inbox) to test the latest GPU version on our side here?

It works just okay for with all our test.

@speedst3r, @ne0ark , @kidrcth also can you try this solution mentioned by @elad.bar for the Nvidia Drivers?

I think @ne0ark 's issue is different and shouldn’t be conflated with mine or @kidrcth . We are not getting 500 errors but 200 responses with an empty set. I am getting no errors in the stderr.log

I’m using the Ubuntu packaged software, the latest being 495 and CUDA 11.5. I’d rather not start using unpackaged software as it then has to be managed manually.

@OlafenwaMoses I removed all the others except detection and I am no longer getting any memory errors but it’s not detecting anything like @speedst3r.

[GIN] 2022/01/23 - 02:22:25 | 200 |  163.276691ms |      172.17.0.1 | POST     "/v1/vision/detection"
[GIN] 2022/01/23 - 02:22:43 | 200 |  148.442871ms |      172.17.0.1 | POST     "/v1/vision/detection"

2022-01-22T21:23:30-05:00 [Trigger Back Side] /MotionEye/BackSide/images/BackSide-2022-01-22-21-23-27.jpg: No objects detected. (164 ms)

I setup Deepstack on another PC (ubuntu 21.10, Docker, manual install of nvidia 510 driver) and it works.

Can you confirm 495 drivers are not supported with 2022.01.1? This would impact anyone using Deepstack on a Ubuntu LTS release as the 510 drivers are not packaged yet.

You can download 510 drivers for Ubuntu 20.04 from the following repo:

http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64

I am still unable to get it to work. Maybe something else is new in 21.10?

I am getting no objects detected for Windows CPU and GPU. I get a 200/OK returned - just nothing detected. Everything identical/same picture with the previous version and objects are detected.

Sorry, I posted this on a couple of other topics with no response. Thought I’d mention that here too.

1 Like

510.54 driver and CUDA 11.6 are available through the graphics-drivers PPA so I updated and… no change, still getting an empty set returned.

@OlafenwaMoses I’m just using one of the stock images from @robmarkcole 's deepstack-ui container: https://github.com/robmarkcole/deepstack-ui/raw/master/app/street.jpg

1 Like

I am also having a similar issue with objects not being detected with the new GPU version. I created a separate thread with all the information I was able to gather - Unable to get GPU version working on docker

any updates here? I just pulled the gpu docker image, and it returns 200, but no objects detected.

I’m running the CPU version side-by-side, and that works just fine with the same source.

For me only this version works with the GPU - gpu-2021.09.1

Hi folks, we have performed a number of testing on the January 2022 release of DeepStack for Windows (CPU & GPU) and Docker (CPU & GPU) and we have not being able to replicate this bug.

We recognize a number of users are experiencing this and we will continue to work on replicating and fixing the issue.

Temporary suggestion: Have you tried testing it on a different machine of on a virtual machine perhaps? Does the problem persists?

2 Likes

My instance is running on a virtual machine. The issue is present even after a fresh reinstall on a new VM.

1 Like

This only appears to be an issue with newer cards. The Jan release works fine with a Quadro k1200/4gb (BlueIris/W10 system) but when swapped for a T600/4gb (Turing based card similar to a GTX1650), it stops detecting objects. Everything is the same (driver, cuda version 11.5 and cudnn 8.3.3). In that case, uninstalling DS and going back to the Sept 2021 release fixes the issue.

4 Likes

I can report something similar as vulture1. Using linux OS with a P400 and it works just fine. When I switch the card to a T600, it stops working. Nothing changes in between.

2 Likes

I have the same problem. I changed my old GTX 650 which didn’t work with Deepstask-GPU to Nvidia T600, it works, but doesn’t return any objects to “Agent DVR”.