AI in the browser: why run inference on the user's device

By default, when we talk about AI in a product, we imagine a call to a server. The user types something, the data travels across the network, a large model processes it on a remote machine, and the response comes back. This design works, scales and dominates the market. But it's not the only one.

There is a growing, and technically sound, movement to run inference on the user's own device, within the browser, without sending the data anywhere. That's what this article is about: what it is, why it matters and where it stops making sense.

I will treat the topic as engineering, not as a trend. The question is not whether this is the future, but rather in which specific cases the account closes today.

What does local inference mean in the browser

Inference is the step in which an already trained model produces an answer from an input. Training a model is expensive and cumbersome. Running inference from a small model, depending on the case, is light enough to happen on the device.

Local inference in the browser means that the model is loaded and executed on the user's machine, using the device's own CPU, GPU or NPU. The input data does not come out of there. The response is computed on the client.

This is not exotic. A text corrector that suggests the next word, a filter that identifies faces in a photo, an audio transcription that runs offline: all of this can happen without an inference server in the way.

The four reasons that justify the effort

The first is privacy. If the data does not leave the device, there is no transit of sensitive information across the network or storage on a third-party server. For certain categories of data, this ceases to be a convenience and becomes a requirement.

The second is latency. There is no back and forth over the internet. The response is practically instantaneous, limited only by local processing. For real-time interactions, like suggestions while typing, this difference defines whether the experience feels fluid or stuck.

The third is cost. Server-side inference is priced per call. Each request consumes paid GPU, adds to the bill and grows with the number of users. Local inference offloads this processing to hardware the user already has. The marginal cost of inference drops to close to zero.

The fourth is to work offline. A model loaded in the browser continues to respond even without a connection. For applications that need to operate in unstable networks or disconnected environments, this is a design advantage that is difficult to replicate on the server. This set of properties speaks directly to the local-first philosophy, which puts the device at the center.

How this works in practice

The piece that makes this scenario possible is the inference runtime that operates within the browser. The most established one today is ONNX Runtime Web. It loads models in the ONNX format and runs them on the client, using WebAssembly, WebGPU and, as it matures, WebNN to access hardware acceleration.

The conceptual flow is direct. You train or obtain a model, convert it to a format that the runtime understands, optimize the size and accuracy, and load it into the web application. From there, inference happens locally.

The trend that makes this viable is small transformers. Compact models, often distilled from larger models and then quantized to take up less memory, can fit into a reasonable download and run with acceptable performance. They don't compete with giant models in capacity, but they solve well-defined tasks with ease. To understand this compression, it is worth knowing the quantized models in the browser.

Where the approach meets its ceiling

This is where I prefer to be clear rather than optimistic. Local inference in the browser has concrete limits, and ignoring them is a recipe for frustration.

Large models do not fit. A model with billions of parameters is not viable to download or run in a common browser tab. Those who need the capacity of frontier models will continue to depend on the server. There is no trick that gets around physics and memory.

User hardware varies, a lot. A recent notebook with a dedicated NPU and an old cell phone deliver radically different experiences for the same model. You are building for a heterogeneous fleet of devices, with no control over it. What flies in one machine chokes in another.

There is also the initial download cost. Loading a model, even a small and quantized one, means downloading megabytes before the first use. This weighs on the first experience and requires caching and loading strategy.

And there is maintenance. Client-side models need to be versioned, updated and distributed. Fixing a problem in the model means pushing a new version to all devices, with all the rollout complexity that entails.

There is also a less obvious limit: observability. When inference runs on the server, you see everything, measure quality, detect degradation and adjust. When it runs on the user's device, this feedback loop becomes more opaque. You can't see the inputs or the results without collecting telemetry, and collecting telemetry about something you chose to keep local may contradict the very reason for choosing it. It's a real tension between privacy and the ability to improve the model, and it needs to be designed on purpose, not discovered later.

How to decide between local and server

The choice is not ideological, it is situational. Tasks that are small, privacy-sensitive, latency-sensitive, or require offline are natural candidates for local processing. Tasks that require large models, centralized control, or frequent updating are best done on the server.

The most mature drawing is rarely pure. Many products will combine the two: a local model for fast, private response in the common case, with escalation to the server when the task is more demanding. A well-thought-out hybrid usually wins over purism on both sides.

The important thing is to treat this as an architectural decision, with explicit criteria, and not as an adoption of fashion. AI in product is process, data governance and outcome measurement. The browser just opens one more option on the table.

A detail that often gets overlooked: local inference changes who pays the computing bill. On the server, the cost is yours and grows with the success of the product. On the device, the cost is on the user, in terms of battery and processing. This sounds great until you remember that batteries wear out and cell phones heat up. A poorly optimized model running with every touch can silently degrade the experience, and the user will feel it without knowing why. Optimization here is not a luxury, it is a requirement of experience.