Running vision models on your own machine: the four costs nobody mentions

Running vision models on your own machine: the four costs nobody mentions

When people compare local inference with an API call, they usually compare one number: how long the model takes to produce a result. That number is real, but it is rarely what decides whether a local tool feels usable. Four other costs show up first, and all four are predictable.

1. Disk: weights are static, and you will keep several

A vision model is a static file (or a small directory of files): weights, config, and often a tokenizer or a preprocessing description. Nothing about it changes between runs.

The consequence is that local processing starts with a download that is measured in minutes or hours, not milliseconds — and it is a one-time cost per model. The ongoing cost is disk pressure: you end up holding several models for different jobs, because a depth model, a segmentation model and an upscaler are not interchangeable.

Practical rule: put model weights on the same fast SSD as the application data, and treat the model directory as part of your working set, not as a cache you can casually delete. Tools that let you move the model directory to another disk are solving exactly this problem.

2. Memory bandwidth, not arithmetic

Inference is mostly a sequence of large matrix operations. On paper the interesting number is floating point throughput. In practice, on consumer hardware, the limit is often how fast the weights and activations can be moved between memory and the compute units (see the ONNX Runtime documentation for how execution providers schedule this).

This is why reduced precision helps so much: an int8 or fp16 weight takes half or a quarter of the bandwidth of an fp32 weight, and in many vision tasks the output quality barely moves. It is also why a GPU that is nominally much faster on paper does not always feel faster — if the data has to cross the PCIe bus often enough, the bus becomes the bottleneck.

3. VRAM is a hard boundary, not a soft one

System memory can be swapped. Video memory generally cannot. When a model plus its intermediate tensors exceed what fits in VRAM, the driver starts moving tensors over the bus, and the result is not "slightly slower" — it is often an order of magnitude slower, or an out-of-memory error.

That is why the honest specification for a local AI tool is not "requires a GPU" but "requires a GPU with at least this much VRAM for this resolution". It is also why batching has to be bounded: processing four images at once instead of one can quadruple peak memory and push a working setup over the edge.

4. Time to first result

Model load and initialization happen before the first output, every session. For a single image this can dominate everything else; for a batch of two hundred it is noise.

The fix is boring and effective: keep one warm instance of the model for the whole run instead of loading it per file. In our own tools this is a singleton process that stays alive while the queue drains. The same reasoning explains why the cold start of a local tool is a poor measure of its throughput.

What this means if you are building or buying

• Measured as a batch, not as a single file. Ask for throughput on a representative folder, not for latency on one image.
• Watch peak VRAM, not average. The failure mode is a crash, not a slowdown.
• Check where models live. Disk, permissions and removable drives all matter once weights are hundreds of megabytes.
• Separate one-time costs from per-run costs. A long first download is not a performance problem; a per-image reload is.

None of this argues against running models locally. It argues for measuring the right thing — and for treating memory, not arithmetic, as the constraint you design around.