Resource Optimization
I profiled argus, my live object-detection pipeline, with Nsight Systems and took the same 16-stream test from 27 seconds to 7 on one laptop GPU. Most of the fixes came from looking at gaps in the timeline and figuring out what each thread was waiting for.
How argus works
argus takes video streams in and puts them back out with boxes drawn on every object. Each frame goes through six stages: FFmpeg decodes it, preprocess letterboxes it to 640×640, YOLO11s finds objects on the GPU with TensorRT, postprocess turns the model's output into boxes, draw paints them into the frame, and NVENC, the video encoder built into the GPU, compresses the result. Every stage has its own pool of threads, and the stages pass frames to each other through small blocking queues.
That design makes the question "which part is slow?" hard to answer by reading code. A stage that looks slow might just be waiting for its input queue, or for space in its output queue. So I marked every unit of work with NVTX, a small library that labels time ranges for Nsight Systems, under one rule: a range wraps only the work, never a queue push or pop. So on the timeline, a bar means the thread is working and a gap means it's waiting.
The test was the same every time: 16 copies of a 300-frame, 768×576 clip, so 4,800 frames, with all 16 outputs encoded to MP4. I timed every change without the profiler attached, two to four runs each, and used the Nsight recordings only to find out why.
I kept a single report file and overwrote it every time, so the original recordings are gone. For the timelines in this post I added temporary switches that turn each change back off, recorded every state again with the same test, then removed the switches. The re-recordings land within a few percent of the original numbers. Each timeline below is a 250 ms slice of steady state, with tabs for before and after that step.
Wall time for the 16-stream test, step by step
Unprofiled, average of 2 to 4 runs per step. Hover a bar for the individual runs.
Slow startup: a CUDA context per encoder
The first recording showed the whole pipeline almost stalled for its first few seconds. Between 2 and 6 seconds into the run, argus encoded between 1 and 6 frames per half second. Then it caught up with a burst and settled at about 200 frames per second.
The encode threads were busy the whole time, inside a range I had labelled open output. argus opens each stream's encoder lazily, on its first frame, so 16 encoders opened at once on 3 encode threads. While those threads were stuck opening, the encode queue filled, and that back-pressure stalled every stage before it. The opens overlapped, but each one finished about 0.25 s after the one before.
Nsight also showed threads I didn't create: 16 pairs of CUDA driver threads, showing up one after another. The driver starts a pair of those for every new CUDA context. FFmpeg's NVENC encoder, when nobody gives it a GPU device, creates its own context with cuCtxCreate. So 16 encoders meant 16 contexts, plus TensorRT's.
A context is to the GPU roughly what a process is to the CPU: its own GPU memory address space, its own loaded kernels, its own queues of work. Creating one is slow (the driver sets all of that up), each one costs about 100 MiB of GPU memory here, and the driver creates them one at a time. Work from different contexts also can't run on the GPU at the same moment; the GPU switches between them in time slices. This comes up again later.
The first 7 seconds of the run
Yellow is encoder setup (open output), the strip on top is frames encoded per 100 ms. Before: the encoders open from 0.6 s to 4.2 s and almost nothing comes out until they're done. After: setup is over at 1.1 s.
How long each encoder took to open
The 16 open output ranges in the order they started, profiled run, before the fix. They overlap; the driver serializes the context creation inside them.
The fix was to create one CUDA device with av_hwdevice_ctx_create before any encoder opens, and hand a reference to it to every NVENC encoder through hw_device_ctx. Each stream still gets its own encoder session; they just share one context. It's about 20 lines.
I first tried to share TensorRT's own context instead of making a new one. FFmpeg refused: Primary context already active with incompatible flags. So I went with one extra context shared by all the encoders. I came back to this at the end.
One thing this didn't change: 4 of the 16 encoders still failed with incompatible client key and fell back to libx264 on the CPU. GeForce drivers cap how many NVENC sessions can run at once. On this machine the cap is 12.
The inference thread never stopped
With startup fixed I moved to the steady state. One row on the timeline never had a gap: the single inference thread. Each batch of 16 frames took about 80 ms there, in three parts: gather (copy 16 preprocessed frames into one batch buffer), copy in (upload the batch to the GPU), and enqueue (run the model).
Nsight's CUDA summary said 64% of the thread's time was inside cudaMemcpyAsync, so I assumed it was a memory problem. Turns out most of that 64% was one 4-byte copy that had to wait for the whole model to finish.
TensorRT's enqueueV3 is meant to queue the model's work on the GPU and return immediately. My model ends in ONNX NonMaxSuppression, which removes duplicate boxes, and its output size depends on the data: how many boxes survive. TensorRT can't size the layers after it without knowing that number. So it queues everything up to NMS, copies the 4-byte count back to the CPU, waits for it, then queues the rest. That wait sits behind the upload and every layer before NMS, and the profiler files all of it under one memcpy call.
The profiler tells you which call the time went to, not why. The memcpy was 64% of the thread, but actual data movement was about 44%. The rest was waiting for the GPU to compute.
250 ms of the baseline
One inference thread. Every batch it gathers and uploads (orange), then waits (grey) while the GPU computes. The copy and the compute never overlap, and the GPU sits idle while the thread gathers the next batch.
Pinned memory
The GPU copies data with DMA: its copy engine reads host RAM directly by physical address, without the CPU. Ordinary memory from new or malloc is pageable: the OS can move or swap its pages at any time, so the GPU can't safely read it. When you copy from pageable memory, the driver first copies your data on the CPU into a small hidden pinned staging buffer, chunk by chunk, and DMAs from there. Pinned memory (cudaHostAlloc) is locked in place, so the GPU reads it directly and the CPU does nothing.
argus was moving each frame three times on the CPU: preprocess wrote it to a buffer, gather copied it into the batch buffer, and the driver copied the batch again into its staging buffer. I gave the engine a pool of pinned frame buffers. Preprocess now writes each frame straight into one, and the upload is one cudaMemcpyAsync per frame into its slot of the GPU batch.
Before and after pinned memory
Gather and copy in shrink to slivers on the inference thread, which now only waits. On the GPU the upload still comes first and the compute after it, one at a time.
The upload itself didn't get any faster though. Pinned frames crossed at 5.55 GB/s; pageable had done about 6. The link was already the limit: this laptop's GPU sits on PCIe Gen3 with 8 lanes, and the driver's staging kept up with it. The speedup came from the CPU not copying 78 MB twice per batch anymore.
If the NMS readback blocks the thread, why not remove it? I tried two ways. A raw model with NMS on the CPU moved 2.8 MB of output per frame instead of 7 KB, and was about 30% slower. A fixed-shape NMS built from ordinary ops (top-K candidates, a pairwise IoU matrix, four rounds of suppression) gave the same 6,098 boxes on my test frames and had no readback at all. It wasn't faster either: the wait just moved to the final output copy. With one stream the thread has to wait for the GPU somewhere. What actually helped was giving the GPU something else to do in the meantime, which is the next step.
Two CUDA streams
After step 1 the GPU still did one thing at a time: about 15 ms of upload with the compute units idle, then about 44 ms of compute with the copy engine idle.
A CUDA stream is an in-order queue of GPU work. Work in one stream runs one after another; work in different streams may overlap. The GPU has separate hardware for each kind of work: SMs (the processors that run kernels) and copy engines (my GPU has two, one per direction). So if batch A computes in stream 1 while batch B uploads in stream 2, the copy engine and the SMs both stay busy.
I split the CUDA backend into two slots. Each slot has its own TensorRT execution context, its own GPU buffers and its own stream, and argus runs one inference thread per slot, both pulling frames from the same queue.
Less than I expected. The profile showed why: batches shrank to 9.7 frames on average, many of them only 1 to 5. Two inference threads now drained the queue faster than the single preprocess thread could fill it. It was 96% busy; the 4 decode threads were 4 to 5% busy.
Before and after two streams
Two slots now, and an upload can run under the other slot's compute. But look at preprocess 0: one thread that never rests, feeding batches of about 5 frames in this recording.
Rebalancing the threads
argus sizes its thread pools from the core count. Decode had 40% of the threads, but with this input decode was cheap and preprocess (scale, letterbox, convert to floats) was the real CPU work. I changed the split to preprocess 40%, encode 30%, decode 20%, and 10% each for postprocess and draw, after reserving one thread per inference slot. On 16 cores that's 4 preprocess threads instead of 1, and 2 decode threads instead of 4.
How busy each stage's threads are
Share of time inside NVTX work ranges, averaged over the stage's threads. Inference is "busy" while it waits for the GPU inside infer, so it reads near 100% whenever the GPU is the limit.
Before and after the thread rebalance
Four preprocess threads keep up, the batches are full again, and the two GPU compute rows are close to solid. The GPU is the bottleneck now.
With two copy engines, why not upload with both? Because each engine handles one direction, host to GPU or GPU to host, and the API doesn't let you choose. I checked the hardware anyway: uploading 2 × 79 MB of pinned memory took 23.5 ms (6.7 GB/s) from one stream and 23.5 ms from two streams at once. The PCIe link is the limit, so the only way to make uploads cheaper is to send fewer bytes.
Sending uint8 instead of floats
Preprocess turned every pixel into a 4-byte float between 0 and 1, in planar layout, so each frame was 4.9 MB. The model can do that conversion itself. tools/add_nms.py --uint8-input edits the ONNX file: the input becomes uint8 [batch, 640, 640, 3], interleaved RGB exactly as FFmpeg's scaler writes it, followed by three new nodes: Cast to float, divide by 255, transpose to planar. Because it's in the model file, it works the same in TensorRT, OpenVINO and MIGraphX.
And the whole test? In alternating runs, 17.0 s with float input and 16.4 s with uint8, about 3%. By now 82% of the uploads were already hidden behind compute, and the GPU was busy 94% of the time. Shrinking something that's already hidden barely changes the total. Making a part faster only helps if it's the bottleneck, and the bottleneck here was the model's compute.
Before and after uint8 input
The upload bars get about four times shorter and preprocess gets lighter. The GPU compute rows barely change, because they were already the bottleneck.
To check uint8 against float I dumped every detection and compared stream 0 with stream 1, which play the same clip. Every few hundred frames one box was missing from one stream. I ruled out corrupted input (identical hashes on CPU and GPU), reran the exact batch (identical), and replayed argus's exact batch order offline (clean). The cause was my dump: argus's logger sometimes wrote a line without a trailing newline, so it got glued to the start of a detection line, and grep '^DET' skipped it. Parsing with a regex fixed it: 6,098 boxes on every run, 0 differences.
FP16
With the GPU as the bottleneck, the next step was to make the model itself cheaper. It was running in FP32. Ampere's tensor cores (special matrix units inside each SM) do FP16 math much faster, and FP16 weights and activations take half the memory traffic.
I wrote that part a long time ago and forgot why it was FP32. The builder creates the network as strongly typed (kSTRONGLY_TYPED): TensorRT uses exactly the data types written in the ONNX file and never picks a precision itself. Ultralytics exports FP32. The older "weakly typed" way, where you set BuilderFlag::kFP16 and TensorRT times FP16 and FP32 kernels per layer and keeps the faster one, is deprecated in TensorRT 10. It picks whatever is fastest, not whatever is correct, and with this model that's a problem.
To run one NMS pass that never compares boxes of different classes, add_nms.py shifts every box by class × 7680 pixels. FP16's largest number is 65,504. Class 8 lands at 61,440 and fits; class 9 lands at 69,120 and becomes infinity. Even below the limit, FP16 has about 11 bits of precision, so near 60,000 neighbouring values are 32 pixels apart, which would wreck the overlap math. A weakly typed build that picked FP16 for that block would quietly return garbage boxes.
add_nms.py --fp16 converts the model before appending NMS, by one rule: everything computed from the image runs in FP16 (the convolutions, activations and the box-decoding softmax). Everything computed from tensor shapes stays FP32, which is the anchor grid; it gets cast where it meets image data. The NMS block, the model's input and its output stay FP32, so argus itself needed no changes.
Inference capacity went from about 321 to about 660 frames per second. And the GPU wasn't the bottleneck anymore: the three encode threads were now 92 to 94% busy.
Before and after FP16
Same 250 ms, about twice as many batches. The GPU rows start to show gaps, and the encode rows fill up.
Putting NVENC and TensorRT in one context
Encode was busy, but not with CPU work: each encode thread used about a third of a core. Almost all of its time was inside FFmpeg's avcodec_send_frame, about 4 ms per frame, and the 12 NVENC streams took as long as the 4 x264 streams that encode on the CPU. NVENC should be far faster than that at 768×576.
Back in Phase 1 I gave the encoders one shared context, but it was a second context, separate from TensorRT's. The GPU's front end, which pulls work out of the hardware queues, serves one context at a time. My guess was that each NVENC frame upload, a 0.9 MB copy, sat in the encoders' context waiting for TensorRT's time slice to end. It fit one more clue: FP16 had made the inference kernels denser, and encode had slowed from about 2.4 to 4 ms per frame since the FP32 run.
Putting everything in one context took three changes:
Create FFmpeg's CUDA device with AV_CUDA_USE_PRIMARY_CONTEXT. The primary context is the one context per process that the CUDA runtime, and so TensorRT, uses automatically.
Create the two TensorRT slot streams with cudaStreamNonBlocking. Without this, sharing the context makes things worse (next box).
Call cudaSetDeviceFlags(cudaDeviceScheduleBlockingSync) before TensorRT creates the context. This is why FFmpeg refused in Phase 1 (box after next).
Every context has a special stream 0, the legacy default stream, with an old rule: work on it waits for all earlier work in every ordinary ("blocking") stream of the context, and work queued after it waits for it. It acts as a barrier across the whole context. The rule exists because early CUDA had no streams, and when streams arrived, stream 0 had to keep old code correct. FFmpeg uploads NVENC frames on stream 0. In a shared context, every 0.9 MB upload would wait for both inference batches in flight, and the next batches would wait for the upload. cudaStreamNonBlocking opts a stream out of that rule. It only affects streams in the same context, so other programs on the GPU are unaffected.
When a CPU thread waits for the GPU, it can spin: loop on a flag in memory, burning a core, and react within a microsecond. Or it can block: ask the OS to put it to sleep until the GPU raises an interrupt, which costs about 10 to 50 µs to wake up but uses no CPU while waiting. CUDA's default, "auto", spins when the machine has more cores than active contexts, so on 16 cores TensorRT's threads spun. FFmpeg only shares a context that was created with blocking sync. That's the "incompatible flags" error from Phase 1. Waking up 50 µs later on a 20 ms batch doesn't matter.
The same test with different context setups
FP16 engine, unprofiled, 3 runs each. The middle bar was my first attempt, before the blocking-sync change: FFmpeg refused to share, so each NVENC encoder made its own context again.
Funny enough, the failed first attempt was the best evidence I had that the guess was right. With 12 encoder contexts instead of 1, the test got slower, 8.1 → 10.3 s. More contexts meant more time slicing. With everything in one context, it got faster.
send frame per frame. Better, but encode is still the slowest stage.Before and after sharing one context
The difference is small by eye: a few more batches in the same 250 ms, and the inference threads now sleep while they wait instead of spinning.
Where it is now
250 ms of the final recording, every thread plus the GPU. Each inference slot finishes a batch about every 35 ms, interleaved with the other. The uploads are short bars hidden under the other slot's compute. The encode rows are the busiest CPU rows, and the NVENC uploads are the small orange ticks on the GPU rows.
The final recording, steady state
16 streams, FP16 uint8 engine, one context. Hover any bar.
In steady state the pipeline moves about 730 frames per second, against about 180 when I started. Because every stream's frames share the same batches, the budget is total frames per second, streams × frame rate:
| Source frame rate | Ceiling | With 20% headroom | At the start |
|---|---|---|---|
| 30 fps | ~24 streams | ~19 streams | ~5 streams |
| 25 fps | ~29 | ~23 | ~6 |
| 10 fps, like most public traffic cameras | ~73 | ~58 | ~14 |
The 768×576 test clip matters here. Inference doesn't care about the source resolution, since every frame is letterboxed to 640×640, but at 1080p the NVENC uploads triple, preprocess scales more pixels and decode works harder. Above 12 streams, each extra stream also encodes on the CPU.
What's still not solved
An NVENC send frame spends about 1.8 ms inside a single cuMemcpy2DAsync call, a copy that then takes 0.18 ms on the GPU. The source is already pinned and the copy engine is only 27% busy. My guess is lock contention in the driver, untested.
GeForce drivers allow 12 NVENC sessions at once here. Streams 13 and up encode with libx264 on the CPU, at about 4.7 ms of CPU per 768×576 frame.
The test clip is MS-MPEG4, which NVDEC can't decode, so decode ran on the CPU in every measurement. With H.264 input, each hardware decoder still creates its own CUDA context, the same problem Step 6 fixed for the encoders.
Inputs were files read as fast as possible. The stream counts above divide a throughput budget; a live test that adds streams until latency grows hasn't been run.
Two to four timed runs per step on a laptop in normal use. Treat differences of a few percent as noise; the big steps (pinned memory, FP16) are well outside it.
With FP16, stream 0 and stream 1 (the same clip) differ slightly on 14 of 300 frames, the same 14 every run. Every box still matches and the worst overlap is IoU 0.91. My explanation, rounding that varies with a frame's position in the batch, is a guess.
Next: keep frames on the GPU. Today a frame crosses PCIe several times. NVDEC decodes it on the GPU, it comes down for preprocess and draw, goes up as uint8 for inference, and goes up again as YUV for NVENC. Preprocess and draw as CUDA kernels would leave only the 7 KB of detections and the compressed packets crossing the bus. Then the decoders join the shared context.
Running it yourself
Make the model from an Ultralytics nms=False export, build the engine, then record a run.
# model: batched NMS, uint8 input, FP16 layers python3 tools/add_nms.py yolo11s_raw.onnx yolo11s_u8_fp16.onnx --uint8-input --fp16 ./build/argus --build --model yolo11s_u8_fp16.onnx --engine yolo11s_u8_fp16.engine --batch 16 # record: NVTX stage ranges, CUDA trace, context switches nsys profile -o argus --trace=cuda,nvtx --cpuctxsw=process-tree --sample=none \ ./build/argus --engine yolo11s_u8_fp16.engine --input clip.mp4 --input clip.mp4 --output out.mp4
Context-switch data needs kernel.perf_event_paranoid at 1 or lower; without it Nsight hides threads that do no CUDA work. Close the report in nsys-ui before recording again: while it's open, nsys can't overwrite the file.