|
No plans at the moment, sorry.
cheers
Chris Maunder
|
|
|
|
|
Error message:
14:15:31:Started Object Detection (YOLOv5 6.2) module
14:15:31:detect_adapter.py: Traceback (most recent call last):
14:15:31:detect_adapter.py: File "C:\Program Files\CodeProject\AI\modules\ObjectDetectionYOLOv5-6.2\detect_adapter.py", line 13, in
14:15:31:detect_adapter.py: from module_runner import ModuleRunner
14:15:31:detect_adapter.py: File "../../SDK/Python\module_runner.py", line 30, in
14:15:31:detect_adapter.py: import aiohttp
14:15:31:detect_adapter.py: ModuleNotFoundError: No module named 'aiohttp'
modified 16-Apr-24 12:05pm.
|
|
|
|
|
It fails if you choose to only install Yolo 6.2, if you set it to install all the default options of Yolo 5 and face, it works.
|
|
|
|
|
|
Is there a way to prevent the TPU from automatically shutting down after 120 seconds of inactivity? It takes quite a while for an image to be recognized afterwards.
12:41:19:System: Docker (fa0761f44809)
12:41:19:Operating System: Linux (Ubuntu 22.04)
12:41:19:CPUs: 13th Gen Intel(R) Core(TM) i5-1340P (Intel)
12:41:19: 1 CPU x 12 cores. 16 logical processors (x64)
12:41:19:System RAM: 31 GiB
12:41:19:Platform: Linux
12:41:19:BuildConfig: Release
12:41:19:Execution Env: Docker
12:41:19:Runtime Env: Production
12:41:19:Runtimes installed:
12:41:19: .NET runtime: 7.0.17
12:41:19: .NET SDK: Not found
12:41:19: Default Python: 3.10.12
12:41:19: Go: Not found
12:41:19: NodeJS: Not found
12:41:19: Rust: Not found
12:41:19:App DataDir: /etc/codeproject/ai
12:41:19:Video adapter info:
12:41:19:STARTING CODEPROJECT.AI SERVER
12:41:19:RUNTIMES_PATH = /app/runtimes
12:41:19:PREINSTALLED_MODULES_PATH = /app/preinstalled-modules
12:41:19:DEMO_MODULES_PATH = /app/demos/modules
12:41:19:MODULES_PATH = /app/modules
12:41:19:PYTHON_PATH = /bin/linux/%PYTHON_NAME%/venv/bin/python3
12:41:19:Data Dir = /etc/codeproject/ai
12:41:19:Server version: 2.6.2
12:45:04:Started Object Detection (Coral) module
12:45:08:objectdetection_coral_adapter.py: TPU detected
12:45:08:objectdetection_coral_adapter.py: Attempting multi-TPU initialisation
12:45:08:objectdetection_coral_adapter.py: Supporting multiple Edge TPUs
12:45:48:Response rec'd from Object Detection (Coral) command 'detect' (...8c51a5) ['Found car, car, car...'] took 245ms
12:45:56:Response rec'd from Object Detection (Coral) command 'detect' (...3a38f2) ['Found car, car, car...'] took 75ms
12:45:57:Response rec'd from Object Detection (Coral) command 'detect' (...1c094c) ['Found car, car, car...'] took 80ms
12:45:57:Response rec'd from Object Detection (Coral) command 'detect' (...cb4a86) ['Found car, car, car...'] took 74ms
12:45:58:Response rec'd from Object Detection (Coral) command 'detect' (...08b14a) ['Found car, car, car...'] took 75ms
12:45:58:Response rec'd from Object Detection (Coral) command 'detect' (...ec2923) ['Found car, car, car...'] took 78ms
12:46:45:Response rec'd from Object Detection (Coral) command 'custom' (...229416) ['Found car, car, car...'] took 80ms
12:46:46:Response rec'd from Object Detection (Coral) command 'detect' (...d7ac0a) ['Found car, car, car...'] took 74ms
12:46:47:Response rec'd from Object Detection (Coral) command 'custom' (...883f24) ['Found car, car, car...'] took 74ms
12:48:50:objectdetection_coral_adapter.py: WARNING:root:No work in 120.0 seconds, watchdog shutting down TPUs.
12:56:18:Response rec'd from Object Detection (Coral) command 'detect' (...0b333f) ['Found car, person, car...'] took 3430ms
13:04:01:Response rec'd from Object Detection (Coral) command 'custom' (...14da1e) ['Found car, car, car...'] took 54ms
modified 5-Apr-24 17:46pm.
|
|
|
|
|
Thanks for pointing this out. I'm not sure why it'd take so long to start up. (For example, it looks like your first iteration took 245ms.) The intent is if something is stalled for whatever reason in the processing pipeline, the watchdog will reset things in the background so CPAI doesn't sit stalled for hours.
I've added some code that should make the recycle process faster and I've increased the watchdog timeout so it takes longer to trigger. Likely in the next CPAI release. Hopefully that'll fix the problem for you.
|
|
|
|
|
I also experience this, with the first detection after the TPU has shutdown often taking > 2 seconds.. which seems to have the knock on effect of making a TPU equipped host report a very high detection time to mesh hosts, and thus never receiving any mesh offload work unless I run a load of detections through manually to get the time down
|
|
|
|
|
With object detection yolov5.net 1.10.1 i just update and i cant get it to use the gpu ( i have a P1000) i select it in the menu it restarts and go back to CPU (DirectML) is this a bug ?
What I have tried:
ReInstall object detection yolov5.net 1.10.1 , pc restart , reinstalling CPAI
modified 5-Apr-24 17:46pm.
|
|
|
|
|
Thanks very much for your report. Could you please share your System Info tab from your CodeProject.AI Server dashboard?
Thanks,
Sean Ewington
CodeProject
|
|
|
|
|
|
Sean
It looks like a code typo mine is showing CPU, I tested it and it is definitely using the GPU.
|
|
|
|
|
Mike,
Could you please send my your System Info tab contents.
Thanks,
"Mistakes are prevented by Experience. Experience is gained by making mistakes."
|
|
|
|
|
Both of my systems are showing CPU
Server version: 2.6.2
System: Windows
Operating System: Windows (Microsoft Windows 10.0.19045)
CPUs: AMD Ryzen 9 5950X 16-Core Processor (AMD)
1 CPU x 16 cores. 32 logical processors (x64)
GPU (Primary): NVIDIA GeForce RTX 4090 (24 GiB) (NVIDIA)
Driver: 551.86, CUDA: 12.4 (up to: 12.4), Compute: 8.9, cuDNN: 8.9
System RAM: 128 GiB
Platform: Windows
BuildConfig: Release
Execution Env: Native
Runtime Env: Production
Runtimes installed:
.NET runtime: 7.0.17
.NET SDK: 7.0.203
Default Python: 3.11.0
Go: Not found
NodeJS: 20.8.1
Rust: Not found
Video adapter info:
NVIDIA GeForce RTX 4090:
Driver Version 31.0.15.5186
Video Processor NVIDIA GeForce RTX 4090
System GPU info:
GPU 3D Usage 1%
GPU RAM Usage 6 GiB
Global Environment variables:
CPAI_APPROOTPATH = <root>
CPAI_PORT = 32168
Server version: 2.6.2
System: Windows
Operating System: Windows (Microsoft Windows 10.0.19045)
CPUs: AMD Ryzen 9 3950X 16-Core Processor (AMD)
1 CPU x 16 cores. 32 logical processors (x64)
GPU (Primary): NVIDIA GeForce RTX 3060 (12 GiB) (NVIDIA)
Driver: 551.86, CUDA: 12.4 (up to: 12.4), Compute: 8.6, cuDNN: 8.9
System RAM: 64 GiB
Platform: Windows
BuildConfig: Release
Execution Env: Native
Runtime Env: Production
Runtimes installed:
.NET runtime: 7.0.5
.NET SDK: Not found
Default Python: 3.10.1
Go: Not found
NodeJS: Not found
Rust: Not found
Video adapter info:
NVIDIA GeForce RTX 3060:
Driver Version 31.0.15.5186
Video Processor NVIDIA GeForce RTX 3060
System GPU info:
GPU 3D Usage 3%
GPU RAM Usage 6.8 GiB
Global Environment variables:
CPAI_APPROOTPATH = <root>
CPAI_PORT = 32168
|
|
|
|
|
Mike, you are correct, it must be a type. DirectML will not be displayed without whatever graphics driver installed on the Windows system being used.
"Mistakes are prevented by Experience. Experience is gained by making mistakes."
|
|
|
|
|
Does object detection yolov5.net is it normal to take 20% of a i7 6700 cpu even when it set to use GPU
|
|
|
|
|
I have confirmed it is a typo in the code. Not the one you would expect, but it still causes the error.
I have fixed the code and it will be in the next release.
"Mistakes are prevented by Experience. Experience is gained by making mistakes."
|
|
|
|
|
|
What would this be?
08:59:27:
08:59:27:Started Object Detection (YOLOv8) module
09:00:19:Object Detection (YOLOv8): [AttributeError] : Traceback (most recent call last):
File "/app/modules/ObjectDetectionYOLOv8/detect.py", line 147, in do_detection
results = detector.predict(img, imgsz=resolution, half=use_half_precision,
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/model.py", line 266, in predict
self.predictor.setup_model(model=self.model, verbose=is_cli)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 341, in setup_model
self.model = AutoBackend(
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/autobackend.py", line 141, in __init__
model = model.fuse(verbose=verbose) if fuse else model
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/tasks.py", line 176, in fuse
delattr(m, "bn") # remove batchnorm
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1266, in __delattr__
object.__delattr__(self, name)
AttributeError: bn
09:00:19:Response rec'd from Object Detection (YOLOv8) command 'detect' (...8d4e22)
Everything installed successfully without error, but the YOLOv8 didn't automattically start, I had to start it.
Here is my SystemInfo:
Server version: 2.6.2
System: Docker (Linux 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022)
Operating System: Linux (Ubuntu 22.04)
CPUs: AMD Ryzen Threadripper 3960X 24-Core Processor (AMD)
1 CPU x 24 cores. 48 logical processors (x64)
GPU (Primary): NVIDIA GeForce RTX 3070 (8 GiB) (NVIDIA)
Driver: 551.23, CUDA: 12.4 (up to: 12.4), Compute: 8.6, cuDNN: 8.9.6
System RAM: 63 GiB
Platform: Linux
BuildConfig: Release
Execution Env: Docker
Runtime Env: Production
Runtimes installed:
.NET runtime: 7.0.17
.NET SDK: Not found
Default Python: 3.10.12
Go: Not found
NodeJS: Not found
Rust: Not found
Video adapter info:
System GPU info:
GPU 3D Usage 25%
GPU RAM Usage 7.5 GiB
Global Environment variables:
CPAI_APPROOTPATH = <root>
CPAI_PORT = 32168
modified 5-Apr-24 17:46pm.
|
|
|
|
|
Thanks very much for your message. The YOLOv8 module isn't set to start automatically. What should happen is, once you start the module once, it should auto-start next time.
Thanks,
Sean Ewington
CodeProject
|
|
|
|
|
OK, I get that it will restart on it's own.
However, what about these errors? This is a new install, uninstalled YOLOv8 and reinstalled it.
When it starts, I get the following (maybe I shouldn't be concerned about it...but maybe others are seeing it as well...or maybe you just need to know it)
Here is what I see - HINT: LOOK AT 9:45:19 timestamp :
09:44:20:Server: This is the latest version
09:44:21:
09:44:21:Module 'Object Detection (YOLOv8)' 1.4.2 (ID: ObjectDetectionYOLOv8)
09:44:21:Valid: True
09:44:21:Module Path: <root>/modules/ObjectDetectionYOLOv8
09:44:21:AutoStart: True
09:44:21:Queue: objectdetection_queue
09:44:21:Runtime: python3.8
09:44:21:Runtime Loc: Shared
09:44:21:FilePath: detect_adapter.py
09:44:21:Start pause: 1 sec
09:44:21:Parallelism: 0
09:44:21:LogVerbosity:
09:44:21:Platforms: all
09:44:21:GPU Libraries: installed if available
09:44:21:GPU Enabled: enabled
09:44:21:Accelerator:
09:44:21:Half Precis.: enable
09:44:21:Environment Variables
09:44:21:APPDIR = <root>/modules/ObjectDetectionYOLOv8
09:44:21:CUSTOM_MODELS_DIR = <root>/modules/ObjectDetectionYOLOv8/custom-models
09:44:21:MODELS_DIR = <root>/modules/ObjectDetectionYOLOv8/assets
09:44:21:MODEL_SIZE = Medium
09:44:21:USE_CUDA = True
09:44:21:YOLO_VERBOSE = false
09:44:21:
09:44:21:Started Object Detection (YOLOv8) module
09:44:23:face.py: GPU in use: NVIDIA GeForce RTX 3070
09:45:19:Object Detection (YOLOv8): [AttributeError] : Traceback (most recent call last):
File "/app/modules/ObjectDetectionYOLOv8/detect.py", line 147, in do_detection
results = detector.predict(img, imgsz=resolution, half=use_half_precision,
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/model.py", line 266, in predict
self.predictor.setup_model(model=self.model, verbose=is_cli)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 341, in setup_model
self.model = AutoBackend(
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/autobackend.py", line 141, in __init__
model = model.fuse(verbose=verbose) if fuse else model
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/tasks.py", line 176, in fuse
delattr(m, "bn") # remove batchnorm
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1266, in __delattr__
object.__delattr__(self, name)
AttributeError: bn
09:45:20:Object Detection (YOLOv8): [AttributeError] : Traceback (most recent call last):
File "/app/modules/ObjectDetectionYOLOv8/detect.py", line 147, in do_detection
results = detector.predict(img, imgsz=resolution, half=use_half_precision,
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/model.py", line 266, in predict
self.predictor.setup_model(model=self.model, verbose=is_cli)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 341, in setup_model
self.model = AutoBackend(
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/autobackend.py", line 141, in __init__
model = model.fuse(verbose=verbose) if fuse else model
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/tasks.py", line 176, in fuse
delattr(m, "bn") # remove batchnorm
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1266, in __delattr__
object.__delattr__(self, name)
AttributeError: bn
09:45:20:Object Detection (YOLOv8): [AttributeError] : Traceback (most recent call last):
File "/app/modules/ObjectDetectionYOLOv8/detect.py", line 147, in do_detection
results = detector.predict(img, imgsz=resolution, half=use_half_precision,
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/model.py", line 266, in predict
self.predictor.setup_model(model=self.model, verbose=is_cli)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 341, in setup_model
self.model = AutoBackend(
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/autobackend.py", line 141, in __init__
model = model.fuse(verbose=verbose) if fuse else model
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/tasks.py", line 176, in fuse
delattr(m, "bn") # remove batchnorm
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1266, in __delattr__
object.__delattr__(self, name)
AttributeError: bn
09:45:20:Object Detection (YOLOv8): [AttributeError] : Traceback (most recent call last):
File "/app/modules/ObjectDetectionYOLOv8/detect.py", line 147, in do_detection
results = detector.predict(img, imgsz=resolution, half=use_half_precision,
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/model.py", line 266, in predict
self.predictor.setup_model(model=self.model, verbose=is_cli)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 341, in setup_model
self.model = AutoBackend(
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/autobackend.py", line 141, in __init__
model = model.fuse(verbose=verbose) if fuse else model
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/tasks.py", line 176, in fuse
delattr(m, "bn") # remove batchnorm
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1266, in __delattr__
object.__delattr__(self, name)
AttributeError: bn
09:45:20:Response rec'd from Object Detection (YOLOv8) command 'detect' (...7832cb)
09:45:20:Response rec'd from Object Detection (YOLOv8) command 'detect' (...c4899c)
09:45:20:Response rec'd from Object Detection (YOLOv8) command 'detect' (...345625)
09:45:20:Object Detection (YOLOv8): [AttributeError] : Traceback (most recent call last):
File "/app/modules/ObjectDetectionYOLOv8/detect.py", line 147, in do_detection
results = detector.predict(img, imgsz=resolution, half=use_half_precision,
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/model.py", line 266, in predict
self.predictor.setup_model(model=self.model, verbose=is_cli)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 341, in setup_model
self.model = AutoBackend(
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/autobackend.py", line 141, in __init__
model = model.fuse(verbose=verbose) if fuse else model
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/tasks.py", line 176, in fuse
delattr(m, "bn") # remove batchnorm
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1266, in __delattr__
object.__delattr__(self, name)
AttributeError: bn
09:45:20:Object Detection (YOLOv8): [AttributeError] : Traceback (most recent call last):
File "/app/modules/ObjectDetectionYOLOv8/detect.py", line 147, in do_detection
results = detector.predict(img, imgsz=resolution, half=use_half_precision,
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/model.py", line 266, in predict
self.predictor.setup_model(model=self.model, verbose=is_cli)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 341, in setup_model
self.model = AutoBackend(
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/autobackend.py", line 141, in __init__
model = model.fuse(verbose=verbose) if fuse else model
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/tasks.py", line 176, in fuse
delattr(m, "bn") # remove batchnorm
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1266, in __delattr__
object.__delattr__(self, name)
AttributeError: bn
09:45:20:Object Detection (YOLOv8): [AttributeError] : Traceback (most recent call last):
File "/app/modules/ObjectDetectionYOLOv8/detect.py", line 147, in do_detection
results = detector.predict(img, imgsz=resolution, half=use_half_precision,
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/model.py", line 266, in predict
self.predictor.setup_model(model=self.model, verbose=is_cli)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 341, in setup_model
self.model = AutoBackend(
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/autobackend.py", line 141, in __init__
model = model.fuse(verbose=verbose) if fuse else model
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/tasks.py", line 176, in fuse
delattr(m, "bn") # remove batchnorm
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1266, in __delattr__
object.__delattr__(self, name)
AttributeError: bn
09:45:20:Object Detection (YOLOv8): [AttributeError] : Traceback (most recent call last):
File "/app/modules/ObjectDetectionYOLOv8/detect.py", line 147, in do_detection
results = detector.predict(img, imgsz=resolution, half=use_half_precision,
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/model.py", line 266, in predict
self.predictor.setup_model(model=self.model, verbose=is_cli)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 341, in setup_model
self.model = AutoBackend(
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/autobackend.py", line 141, in __init__
model = model.fuse(verbose=verbose) if fuse else model
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/tasks.py", line 176, in fuse
delattr(m, "bn") # remove batchnorm
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1266, in __delattr__
object.__delattr__(self, name)
AttributeError: bn
09:45:20:Object Detection (YOLOv8): [AttributeError] : Traceback (most recent call last):
File "/app/modules/ObjectDetectionYOLOv8/detect.py", line 147, in do_detection
results = detector.predict(img, imgsz=resolution, half=use_half_precision,
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/model.py", line 266, in predict
self.predictor.setup_model(model=self.model, verbose=is_cli)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 341, in setup_model
self.model = AutoBackend(
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/autobackend.py", line 141, in __init__
model = model.fuse(verbose=verbose) if fuse else model
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/tasks.py", line 176, in fuse
delattr(m, "bn") # remove batchnorm
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1266, in __delattr__
object.__delattr__(self, name)
AttributeError: bn
09:45:20:Object Detection (YOLOv8): [AttributeError] : Traceback (most recent call last):
File "/app/modules/ObjectDetectionYOLOv8/detect.py", line 147, in do_detection
results = detector.predict(img, imgsz=resolution, half=use_half_precision,
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/model.py", line 266, in predict
self.predictor.setup_model(model=self.model, verbose=is_cli)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 341, in setup_model
self.model = AutoBackend(
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/autobackend.py", line 141, in __init__
model = model.fuse(verbose=verbose) if fuse else model
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/ultralytics/nn/tasks.py", line 176, in fuse
delattr(m, "bn") # remove batchnorm
File "/app/modules/ObjectDetectionYOLOv8/bin/linux/python38/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1266, in __delattr__
object.__delattr__(self, name)
AttributeError: bn
09:45:36:Response rec'd from Object Detection (YOLOv8) command 'detect' (...226f6d) ['Found chair, chair, tv'] took 13376ms
|
|
|
|
|
Thank you kindly. We patched this issue once before, it looks like we have to patch it again.
Thanks,
Sean Ewington
CodeProject
|
|
|
|
|
This is a threading issue with PyTorch. I've added a lock around the interpreter (really only needs to be around the fuse code, so there's scope to optimise this) which should help.
Out of interest, how many requests per second are you sending to CodeProject.AI?
cheers
Chris Maunder
|
|
|
|
|
How do I get this information?
|
|
|
|
|
I experience the same issue, YOLOv8 + ALPR on 2.6.2, when request count becomes fairly high. I have attached a log from Blue Iris showing requests and errors, as well as Event Viewer output.
Event Viewer:
Faulting application name: python.exe, version: 3.9.6150.1013, time stamp: 0x60d9eb23
Faulting module name: nvcuda64.dll, version: 31.0.15.5212, time stamp: 0x660c7129
Exception code: 0xc0000005
Fault offset: 0x00000000000bcdc0
Faulting process id: 0x0x6FE4
Faulting application start time: 0x0x1DA8A57F069CC87
Faulting application path: C:\Program Files\CodeProject\AI\runtimes\bin\windows\python39\python.exe
Faulting module path: C:\Windows\system32\DriverStore\FileRepository\nv_dispi.inf_amd64_268e85175aa9e991\nvcuda64.dll
Report Id: 38480e22-c755-469d-a9d9-0229903fe94c
Faulting package full name:
Faulting package-relative application ID:
The real problem is... when it starts to behave like this (after "remove batchnorm" error), Cp.AI hardly recovers, there are crashes between 1-20 minutes, and in many cases YOLOv8 module stays down. I basically have to babysit CP.AI continuously in those cases.
Below, an excerpt from log with requests and timestamps (actually these were handled fine by CP.AI). All requests within a span of 3 seconds.
For what it's worth, the GPU is an RTX 3090.
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'detect' (...18f7a0) ['Found potted plant, potted plant, potted plant...'] took 28ms
11:39:15:Object Detection (YOLOv8): Detecting using ipcam-general
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'detect' (...da8d5a) ['Found person, boat, potted plant...'] took 53ms
11:39:15:Object Detection (YOLOv8): Detecting using ipcam-general
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...b4e42b) ['No objects found'] took 31ms
11:39:15:Object Detection (YOLOv8): Detecting using license-plate
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'detect' (...967f3a) ['Found potted plant, potted plant, potted plant...'] took 86ms
11:39:15:Object Detection (YOLOv8): Detecting using ipcam-general
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...3543c2) ['Found person'] took 38ms
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...e079b9) ['No objects found'] took 35ms
11:39:15:Response rec'd from License Plate Reader command 'alpr' (...751f74) ['No plates found'] took 45ms
11:39:15:Object Detection (YOLOv8): Detecting using license-plate
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'detect' (...f10bb7) ['Found boat, potted plant, potted plant...'] took 121ms
11:39:15:Object Detection (YOLOv8): Detecting using ipcam-general
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...cb0ff0) ['No objects found'] took 39ms
11:39:15:Object Detection (YOLOv8): Detecting using license-plate
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...9010b6) ['No objects found'] took 45ms
11:39:15:Response rec'd from License Plate Reader command 'alpr' (...7f1e6d) ['No plates found'] took 57ms
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'detect' (...59eeec) ['Found potted plant, potted plant, boat...'] took 187ms
11:39:15:Object Detection (YOLOv8): Detecting using ipcam-general
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...31d4ae) ['Found person'] took 74ms
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...a78bee) ['No objects found'] took 76ms
11:39:15:Response rec'd from License Plate Reader command 'alpr' (...a8a7d3) ['No plates found'] took 85ms
11:39:15:Object Detection (YOLOv8): Detecting using license-plate
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'detect' (...c0a06b) ['Found potted plant, potted plant, potted plant...'] took 244ms
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...f17d45) ['No objects found'] took 51ms
11:39:15:Object Detection (YOLOv8): Detecting using ipcam-general
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...56efa0) ['No objects found'] took 39ms
11:39:15:Response rec'd from License Plate Reader command 'alpr' (...3c1aaf) ['No plates found'] took 51ms
11:39:15:Object Detection (YOLOv8): Detecting using license-plate
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...85eb35) ['No objects found'] took 33ms
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...34894d) ['No objects found'] took 28ms
11:39:15:Response rec'd from License Plate Reader command 'alpr' (...8c583c) ['No plates found'] took 41ms
11:39:15:Object Detection (YOLOv8): Detecting using license-plate
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...25192a) ['No objects found'] took 16ms
11:39:15:Response rec'd from License Plate Reader command 'alpr' (...c6ea2f) ['No plates found'] took 27ms
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'detect' (...d12720) ['Found boat, potted plant, potted plant...'] took 11ms
11:39:15:Object Detection (YOLOv8): Detecting using ipcam-general
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...24a0d6) ['Found person'] took 10ms
11:39:15:Object Detection (YOLOv8): Detecting using license-plate
11:39:15:Response rec'd from Object Detection (YOLOv8) command 'custom' (...a7ac68) ['No objects found'] took 10ms
11:39:15:Response rec'd from License Plate Reader command 'alpr' (...1d93f7) ['No plates found'] took 18ms
11:39:16:Response rec'd from Object Detection (YOLOv8) command 'detect' (...f738a4) ['Found person, potted plant, potted plant...'] took 17ms
11:39:16:Object Detection (YOLOv8): Detecting using ipcam-general
11:39:16:Response rec'd from Object Detection (YOLOv8) command 'custom' (...b850fe) ['Found person'] took 12ms
11:39:16:Object Detection (YOLOv8): Detecting using license-plate
11:39:16:Response rec'd from Object Detection (YOLOv8) command 'custom' (...0bc07e) ['No objects found'] took 9ms
11:39:16:Response rec'd from License Plate Reader command 'alpr' (...4b2cb2) ['No plates found'] took 17ms
11:39:17:Response rec'd from Object Detection (YOLOv8) command 'detect' (...05f9ec) ['Found potted plant, boat, person...'] took 18ms
11:39:17:Object Detection (YOLOv8): Detecting using ipcam-general
11:39:17:Response rec'd from Object Detection (YOLOv8) command 'custom' (...20adb8) ['Found person'] took 12ms
11:39:17:Object Detection (YOLOv8): Detecting using license-plate
11:39:17:Response rec'd from Object Detection (YOLOv8) command 'custom' (...5c0901) ['No objects found'] took 11ms
11:39:17:Response rec'd from License Plate Reader command 'alpr' (...023b91) ['No plates found'] took 19ms
11:39:18:Response rec'd from Object Detection (YOLOv8) command 'detect' (...f213b0) ['Found person, potted plant, potted plant...'] took 18ms
11:39:18:Object Detection (YOLOv8): Detecting using ipcam-general
11:39:18:Response rec'd from Object Detection (YOLOv8) command 'custom' (...c02e67) ['Found person'] took 15ms
11:39:18:Object Detection (YOLOv8): Detecting using license-plate
11:39:18:Response rec'd from Object Detection (YOLOv8) command 'custom' (...3e3aaf) ['No objects found'] took 13ms
11:39:18:Response rec'd from License Plate Reader command 'alpr' (...8d3f62) ['No plates found'] took 21ms
If you need more details or any information, please let me know.
|
|
|
|
|
Running 2.1.6 version of the Coral object detection which works fine using tiny or small object size, however when switching to medium or large the TPU is no longer found and it rolls back to CPU instead. Enclosed is the error output from the console. I have fully uninstalled / reinstalled both CP and the module with the same result.
08:38:21:Module 'Object Detection (Coral)' 2.1.6 (ID: ObjectDetectionCoral)
08:38:21:Valid: True
08:38:21:Module Path: <root>\modules\ObjectDetectionCoral
08:38:21:AutoStart: True
08:38:21:Queue: objectdetection_queue
08:38:21:Runtime: python3.9
08:38:21:Runtime Loc: Local
08:38:21:FilePath: objectdetection_coral_adapter.py
08:38:21:Pre installed: False
08:38:21:Start pause: 1 sec
08:38:21:Parallelism: 1
08:38:21:LogVerbosity:
08:38:21:Platforms: all
08:38:21:GPU Libraries: installed if available
08:38:21:GPU Enabled: enabled
08:38:21:Accelerator:
08:38:21:Half Precis.: enable
08:38:21:Environment Variables
08:38:21:CPAI_CORAL_MODEL_NAME = YOLOv8
08:38:21:CPAI_CORAL_MULTI_TPU = True
08:38:21:MODELS_DIR = <root>\modules\ObjectDetectionCoral\assets
08:38:21:MODEL_SIZE = medium
08:38:21:
08:38:21:Started Object Detection (Coral) module
08:38:24:objectdetection_coral_adapter.py: ERROR:root:TFLite file C:\Program Files\CodeProject\AI\modules\ObjectDetectionCoral\assets\yolov5m-int8_edgetpu.tflite doesn't exist
08:38:24:objectdetection_coral_adapter.py: WARNING:root:Exception creating interpreter: [Errno 2] No such file or directory: 'C:\\Program Files\\CodeProject\\AI\\modules\\ObjectDetectionCoral\\assets\\yolov5m-int8_edgetpu.tflite'
08:38:24:objectdetection_coral_adapter.py: Exception ignored in:
08:38:24:objectdetection_coral_adapter.py: Traceback (most recent call last):
08:38:24:objectdetection_coral_adapter.py: File "C:\Program Files\CodeProject\AI\modules\ObjectDetectionCoral\tpu_runner.py", line 352, in __del__
08:38:24:objectdetection_coral_adapter.py: self.delete()
08:38:24:objectdetection_coral_adapter.py: File "C:\Program Files\CodeProject\AI\modules\ObjectDetectionCoral\tpu_runner.py", line 375, in delete
08:38:24:objectdetection_coral_adapter.py: self._halt_interpreters(q_idx)
08:38:24:objectdetection_coral_adapter.py: File "C:\Program Files\CodeProject\AI\modules\ObjectDetectionCoral\tpu_runner.py", line 357, in _halt_interpreters
08:38:24:objectdetection_coral_adapter.py: for i in self.interpreters[seg_idx]:
08:38:24:objectdetection_coral_adapter.py: IndexError: list index out of range
08:38:24:objectdetection_coral_adapter.py: WARNING:root:No Coral TPUs found or able to be initialized. Using CPU.
08:38:24:objectdetection_coral_adapter.py: ERROR:root:TFLite file C:\Program Files\CodeProject\AI\modules\ObjectDetectionCoral\assets\yolov5m-int8.tflite doesn't exist
08:38:24:objectdetection_coral_adapter.py: WARNING:root:Unable to create interpreter for CPU using edgeTPU library: [Errno 2] No such file or directory: 'C:\\Program Files\\CodeProject\\AI\\modules\\ObjectDetectionCoral\\assets\\yolov5m-int8.tflite'
08:38:24:objectdetection_coral_adapter.py: Exception ignored in:
08:38:24:objectdetection_coral_adapter.py: Traceback (most recent call last):
08:38:24:objectdetection_coral_adapter.py: File "C:\Program Files\CodeProject\AI\modules\ObjectDetectionCoral\tpu_runner.py", line 352, in __del__
08:38:24:objectdetection_coral_adapter.py: self.delete()
08:38:24:objectdetection_coral_adapter.py: File "C:\Program Files\CodeProject\AI\modules\ObjectDetectionCoral\tpu_runner.py", line 375, in delete
08:38:24:objectdetection_coral_adapter.py: self._halt_interpreters(q_idx)
08:38:24:objectdetection_coral_adapter.py: File "C:\Program Files\CodeProject\AI\modules\ObjectDetectionCoral\tpu_runner.py", line 357, in _halt_interpreters
08:38:24:objectdetection_coral_adapter.py: for i in self.interpreters[seg_idx]:
08:38:24:objectdetection_coral_adapter.py: IndexError: list index out of range
08:38:29:objectdetection_coral_adapter.py: TPU detected
08:38:29:objectdetection_coral_adapter.py: Attempting multi-TPU initialisation
08:38:29:objectdetection_coral_adapter.py: Failed to init multi-TPU. Falling back to single TPU.
08:38:29:objectdetection_coral_adapter.py: Unable to find or initialise the Coral TPU. Falling back to CPU-only.
08:38:29:objectdetection_coral_adapter.py: Error creating interpreter: Could not open 'C:\Program Files\CodeProject\AI\modules\ObjectDetectionCoral\assets\yolov5m-int8.tflite'.
08:38:29:objectdetection_coral_adapter.py: Using CPU
modified 5-Apr-24 17:46pm.
|
|
|
|
|