it seems sending multiple (5) requests at once seems to cause the GPU docker to stop responding. It stays running with no errors in the logs, but wont accept any new requests.
To resolve this I need to reboot the docker. Is this a known issue, or can this be addressed in the next build?
Hello @jaburges1 , this issue is caused by deepstack running out of GPU memory. I assume you are running in high mode. We shall provide better error reports in the next releases. The best way to avoid this is to use a GPU with more memory or use the standard or low modes in DeepStack
actually its in Low mode - but i’ll check the docker limits
its a i7 + 1080Ti + 32Gb RAM host.
there are no docker limits on that container.
CPU is at 5% idle
Memory is 1.8Gb idle
CPU spikes to 90+% then sits at 45% when it fails.
There is definitely something not right with the docker (both API through HA and curl no longer works)
Thanks for the extra details. We shall investigate this further and provide an update soon.
i’ve returned to the non GPU docker and this works fine.
idle at around 10% CPU, busy around 12%
Responding fine. Issue seems to be with the GPU docker.
What are the requirements on the host?
The picture lacks of part of data. Please use this picture to test your AI GPU mode.
The docker will be stuck and stop response to external request.
Nice to know the cpu version is working fine.
The only requirement for the GPU version is Nvidia Docker 2. Cuda and Cuddn ships with DeepStack. If you don’t have cuda installed on the host, it doesn’t affect deepstack as it runs with its own cuda environment. However, when cuda is installed on the host, the host version should not be less than cuda version 9.1 which is contained in deepstack.
Generally, it is preferrable not to have cuda on the host due to this incompatibility. By normal container principles, this should not arise, the issue is with nvidia docker.
Hello @JDU This is an interesting case. Does this happen on the cpu version?
We didn’t test it on CPU mode. Maybe you can test it.
Did you test the picture?