|
I haven't been back to the machine yet, but the system info screen shows GPU usage=0.
Looked at the performance monitor just now, that shows occasional 3d activity, but nothing else. 1% memory constant. GPU0
Maybe it's not using the GPU.
I again clicked enable GPU, log says GPU enabled when the module restarts.
Maybe uninstall reinstall the module? I've uninstalled and reinstalled ver. 2.6.2 a couple of times already trying to figure out the Coral failed inferences. I just don't know.
|
|
|
|
|
In my case, GPU is not used.
CPU (DirectML) is utilised despite selecting Enable GPU
|
|
|
|
|
This is a naming issue. Here's the comment for the EnableGPU property:
public bool? EnableGPU { get; set; } = true;
cheers
Chris Maunder
|
|
|
|
|
How can we force the use of GPU?
|
|
|
|
|
You can't. That module uses DirectML which means it will use the GPU if it can, meaning if the correct drivers are installed, and if the GPU is supported by DirectML. With DirectML, the use of the internal GPU is assumed, so if it's not actually using the GPU it's not something (AFAIK) we can really influence.
The only thing I would say is to be careful when trying to determine if the GPU is being used or not. A quote from an issue regarding DirectML using Tensorflow:
Quote: One thing to keep in mind is that if you're using Task Manager to monitor GPU usage, it can sometimes be misleading because the default Task Manager GPU usage graph looks for 3D workloads which are different from compute workloads like tensorflow-directml
cheers
Chris Maunder
|
|
|
|
|
Interesting. I do recall that prior to upgrading to CPAI ver2.6.2 (using 2.5.6) the YoloV5.NET model displayed that it was utilising the GPU, however, it seems to utilise CPU with this upgrade. I haven’t rolled back to CPAI ver2.5.6 to confirm my recollection though.
|
|
|
|
|
Yes.
I'm going to try rolling back to 2.5.4.
|
|
|
|
|
Me too.
Rolled back to CPAI ver. 2.5.6. YOLOv5 .NET connected to GPU on startup, and shows it is using GPU (Intel 630) without any user intervention.
Server version: 2.5.6
System: Windows
Operating System: Windows (Microsoft Windows 10.0.19045)
CPUs: Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz (Intel)
1 CPU x 4 cores. 4 logical processors (x64)
GPU (Primary): Intel(R) HD Graphics 630 (1,024 MiB) (Intel Corporation)
Driver: 31.0.101.2111
System RAM: 16 GiB
Platform: Windows
BuildConfig: Release
Execution Env: Native
Runtime Env: Production
Runtimes installed:
.NET runtime: 7.0.5
.NET SDK: Not found
Default Python: Not found
Go: Not found
NodeJS: Not found
Video adapter info:
Intel(R) HD Graphics 630:
Driver Version 31.0.101.2111
Video Processor Intel(R) HD Graphics Family
System GPU info:
GPU 3D Usage 10%
GPU RAM Usage 0
Global Environment variables:
CPAI_APPROOTPATH = <root>
CPAI_PORT = 32168
Module 'Object Detection (YOLOv5 .NET)' 1.9.3 (ID: ObjectDetectionYOLOv5Net)
Valid: True
Module Path: <root>\modules\ObjectDetectionYOLOv5Net
AutoStart: True
Queue: objectdetection_queue
Runtime: dotnet
Runtime Loc: Shared
FilePath: bin\ObjectDetectionYOLOv5Net.exe
Pre installed: False
Start pause: 1 sec
Parallelism: 0
LogVerbosity:
Platforms: all
GPU Libraries: installed if available
GPU Enabled: enabled
Accelerator:
Half Precis.: enable
Environment Variables
CUSTOM_MODELS_DIR = <root>\modules\ObjectDetectionYOLOv5Net\custom-models
MODELS_DIR = <root>\modules\ObjectDetectionYOLOv5Net\assets
MODEL_SIZE = MEDIUM
Status Data: {
"inferenceDevice": "GPU",
"inferenceLibrary": "DirectML",
"canUseGPU": true,
"successfulInferences": 179,
"failedInferences": 0,
"numInferences": 179,
"averageInferenceMs": 245,
"histogram": {
"person": 63,
"car": 262,
"bus": 19,
"truck": 2
},
"numItemsFound": 346
}
Started: 16 May 2024 7:52:55 AM Central Standard Time
LastSeen: 16 May 2024 8:07:00 AM Central Standard Time
Status: Started
Requests: 179 (includes status calls)
|
|
|
|
|
So it looks like its a CPAI ver2.6.2 issue then.
YoloV5 .NET GPU works successfully on CPAI ver2.5.6 but doesn't in CPAI ver2.6.2
|
|
|
|
|
This is what I was referring to.
https://www.codeproject.com/Messages/6002999/Re-stupid-me-Windows-11-upgrade
Thanks very much for the report. There is a bug with 2.6.2 where the dashboard displays CPU, but it is actually using GPU (which the DirectML indicates). So you should actually be using GPU. Unless you have some monitoring tools that indicate that's not happening, you might actually be OK here.
Thanks,
Sean Ewington
CodeProject
|
|
|
|
|
Just installed CPAI ver 2.6.5 this morning and can confirm that YOLOv5.NET is utilising and appears with GPU on the dashboard!
Thank you for your help with fixing this issue team!
|
|
|
|
|
i have a fresh install of code project 2.6.4 on ubuntu server 24.04, object detection installed fine on on the install but trying to install LPR and Training and i get a 404 error for both of them
Server version: 2.6.4
System: Linux
Operating System: Linux (Ubuntu 24.04)
CPUs: 12th Gen Intel(R) Core(TM) i7-12700K (Intel)
1 CPU x 12 cores. 20 logical processors (x64)
GPU (Primary): NVIDIA GeForce RTX 2060 (6 GiB) (NVIDIA)
Driver: 535.161.08, CUDA: 12.2 (up to: 12.2), Compute: 7.5, cuDNN:
System RAM: 31 GiB
Platform: Linux
BuildConfig: Release
Execution Env: Native
Runtime Env: Production
Runtimes installed:
.NET runtime: 7.0.18
.NET SDK: 7.0.118
Default Python: 3.12.3
Go: Not found
NodeJS: Not found
Rust: Not found
Video adapter info:
TU104 [GeForce RTX 2060] (rev a1):
Driver Version
Video Processor
System GPU info:
GPU 3D Usage 10%
GPU RAM Usage 1.9 GiB
Global Environment variables:
CPAI_APPROOTPATH = <root>
CPAI_PORT = 32168
|
|
|
|
|
Are you still getting 404s?
cheers
Chris Maunder
|
|
|
|
|
well ive gotten past the 404 errors, Now im getting a error with fiftyone
15:42:51:Training for YoloV5 6.2: [FiftyOneConfigError] : Traceback (most recent call last):
File "/usr/bin/codeproject.ai-server-2.6.5/modules/TrainingObjectDetectionYOLOv5/bin/linux/python38/venv/lib/python3.8/site-packages/fiftyone/core/odm/database.py", line 195, in establish_db_conn
_db_service = fos.DatabaseService()
File "/usr/bin/codeproject.ai-server-2.6.5/modules/TrainingObjectDetectionYOLOv5/bin/linux/python38/venv/lib/python3.8/site-packages/fiftyone/core/service.py", line 80, in __init__
self.start()
File "/usr/bin/codeproject.ai-server-2.6.5/modules/TrainingObjectDetectionYOLOv5/bin/linux/python38/venv/lib/python3.8/site-packages/fiftyone/core/service.py", line 209, in start
super().start()
File "/usr/bin/codeproject.ai-server-2.6.5/modules/TrainingObjectDetectionYOLOv5/bin/linux/python38/venv/lib/python3.8/site-packages/fiftyone/core/service.py", line 118, in start
+ self.command,
File "/usr/bin/codeproject.ai-server-2.6.5/modules/TrainingObjectDetectionYOLOv5/bin/linux/python38/venv/lib/python3.8/site-packages/fiftyone/core/service.py", line 238, in command
DatabaseService.find_mongod(),
File "/usr/bin/codeproject.ai-server-2.6.5/modules/TrainingObjectDetectionYOLOv5/bin/linux/python38/venv/lib/python3.8/site-packages/fiftyone/core/service.py", line 286, in find_mongod
raise ServiceExecutableNotFound("Could not find `mongod`")
fiftyone.core.service.ServiceExecutableNotFound: Could not find `mongod`
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/codeproject.ai-server-2.6.5/modules/TrainingObjectDetectionYOLOv5/training_objectdetection_YOLOv5_adapter.py", line 177, in long_process
result = self.method_to_execute(data)
File "/usr/bin/codeproject.ai-server-2.6.5/modules/TrainingObjectDetectionYOLOv5/training_objectdetection_YOLOv5_adapter.py", line 223, in create_dataset
result = self.dataset_creator.create_dataset(dataset_name, classes, num_images)
File "/usr/bin/codeproject.ai-server-2.6.5/modules/TrainingObjectDetectionYOLOv5/fiftyone_dataset_creator.py", line 118, in create_dataset
if fo.dataset_exists(dataset_name):
File "/usr/bin/codeproject.ai-server-2.6.5/modules/TrainingObjectDetectionYOLOv5/bin/linux/python38/venv/lib/python3.8/site-packages/fiftyone/core/dataset.py", line 85, in dataset_exists
conn = foo.get_db_conn()
File "/usr/bin/codeproject.ai-server-2.6.5/modules/TrainingObjectDetectionYOLOv5/bin/linux/python38/venv/lib/python3.8/site-packages/fiftyone/core/odm/database.py", line 395, in get_db_conn
_connect()
File "/usr/bin/codeproject.ai-server-2.6.5/modules/TrainingObjectDetectionYOLOv5/bin/linux/python38/venv/lib/python3.8/site-packages/fiftyone/core/odm/database.py", line 234, in _connect
establish_db_conn(fo.config)
File "/usr/bin/codeproject.ai-server-2.6.5/modules/TrainingObjectDetectionYOLOv5/bin/linux/python38/venv/lib/python3.8/site-packages/fiftyone/core/odm/database.py", line 201, in establish_db_conn
raise FiftyOneConfigError(
fiftyone.core.config.FiftyOneConfigError: MongoDB could not be installed on your system. Please define a `database_uri` in your `fiftyone.core.config.FiftyOneConfig` to connect to yourown MongoDB instance or cluster
just trying to figure out the database part and how to get fiftyone to connect to it, i was able to install Mongod no problem
|
|
|
|
|
I've seen that before and frankly I'm tempted to simply disable the training module on Ubuntu. If you find your way through the FiftyOne docs and come up with a solution please let us know.
cheers
Chris Maunder
|
|
|
|
|
Upgraded to 2.6.2, Blue Iris intermittent AI Error 500 are back.
Can anyone explain the actual root cause of these errors? I'm getting a bit tired of the game between BI and CodeProject where updating one or the other has a 75% chance of re-introducing these errors.
Thanks
|
|
|
|
|
I'm sure you've reported stuff before but it's extremely difficult for us to remember everyone's details on everything.
Can you please let us know
- what version of the server you're using
- on which module you're seeing the error
- The actual error (screen shot would be good)
- Have you tested the image that's throwing the 500 using the CodeProject.AI Explorer. It uses the same API as Blue Iris
cheers
Chris Maunder
|
|
|
|
|
For sure. For reference I was on 2.5.4 w/ Blue Iris 5.8.8.12 and had no AI errors.
Updated to CodeProject 2.6.2. Intermittent errors started every 15-30min on random cameras.
Decided to upgrade BI to latest 5.9.0.5. Still exist.
I'm using Coral TPU, tried various models and sizes. Multi-TPU disabled.
I don't know what images it's failing on unfortunately; BI doesn't give me the optics into that.
09-05-24-1817 hosted at ImgBB — ImgBB[^]
|
|
|
|
|
Does it work better if multi-TPU is enabled?
|
|
|
|
|
It does not. I've now shifted to 2.6.4 on Ubuntu at the recommendation that it's stable, looked good for a day or so but 500 errors are back sporadically.
|
|
|
|
|
Hm. I wonder if it’s affect d by something like the ‘parallelism’ parameter? I think right now it’s set to 16 for Coral.
|
|
|
|
|
I don't know if it's relevant, but if I sit and watch the web interface I can often see the Coral module on the status tab switching between CPU/TPU and the latency will increase for the CPU processed images then it flips back to TPU.
I see nothing in the logs about why it is flapping between them.
|
|
|
|
|
Huh. Weird. Does anything work better or worse as it flips between them? What is the timing like, does it spend a lot of time in one or the other? Does it work any better if you added a time.sleep(1) before the allocator to give the driver a second to catch up?
|
|
|
|
|
Coral on Windows is not very reliable I'm afraid. It works, but intermittent errors are not uncommon. I use the .NET Object Detection module on Windows since it uses DirectML and so will make the best use of available hardware.
cheers
Chris Maunder
|
|
|
|
|
What do you recommend for Coral? Docker? Linux? I can pivot if it's known to be more reliable.
|
|
|
|