AI on the device: the strategic decision between server and on-device

AI decisions quickly turn into architectural decisions, and architectural decisions turn into cost, risk and compliance. Therefore, the choice of where the inference takes place, on the server or on the user's device, should not be delegated to the end of the project. It shapes the product from the beginning.

This text is for those who decide. Technical leader, head of product, CTO. The fundamental question is practical: when does it make sense to run AI on the user's device instead of on its infrastructure, and what does this cost in return?

I'm going to treat the topic as a trade-off, because that's exactly what it is. There is no universal right side. There is a right side for a use case, an audience, and a risk profile.

The trade-off in terms of business

Running on the server gives you control. You choose the model, update it whenever you want, observe what happens, measure and adjust. You can use large, frontier models, with capacities that no user device can accommodate. The price of this freedom is twofold: you pay per inference call, and the user's data needs to travel to your machine.

Running it on the device reverses the equation. The marginal cost of inference drops to close to zero because processing uses the user's hardware. The data does not leave the device, which changes the conversation about privacy. And the application can work offline. The price here is the limitation: you are held hostage by the capacity of the device, the models that fit on it, and the complexity of maintaining distributed models.

Summarizing the central tension: server is control and capacity at the cost of money and data traffic. Device is privacy, zero inference cost, and offline at the cost of throttling and fragmentation. The technical overview of this scenario is detailed in In-Browser AI and Local Inference.

When the cost changes the bill

The financial argument deserves honest attention. Server-side inference scales with usage. The more users and the more calls, the higher the GPU bill. For a product with many active users and many AI interactions per session, this number grows aggressively.

On-device inference transfers this cost to the hardware the user has already purchased. For frequent, small tasks, this can be the difference between a unit economics that closes and one that bleeds money with each iteration.

But beware of naive reading. You exchange inference costs for engineering costs: converting, optimizing, quantizing, versioning and distributing client-side models is not free. The gain appears when the inference volume is high enough for the fixed engineering cost to be diluted. In a low-volume product, the server is usually cheaper in total.

The connection with LGPD and sensitive data

Here is, in my view, the strongest and most underrated argument for on-device. When data doesn't leave the device, much of the compliance problem simply doesn't arise.

LGPD treats categories of sensitive data with special rigor: health, biometrics, children's data, information that requires a robust legal basis and careful processing. Each time this data travels and is stored in your infrastructure, you assume responsibility, risk of leakage and protection obligation.

Processing locally reduces this surface area. If the inference happens on the device and the data is not transmitted or persisted on the server, you minimize what you collect and what you keep, a principle that is at the heart of the law itself. For industries like healthcare and government, where sensitive data is the rule rather than the exception, keeping processing on-device can be a real compliance advantage.

It's not a silver bullet. The client-side model still requires care, and there are obligations that do not disappear. But reducing the transit and storage of sensitive data is one of the most effective ways to reduce regulatory risk, because the best way to protect data is not to have it on your server.

The trust argument also applies. Being able to tell the user, with technical sincerity, that the analysis of their photo or health information will not leave the device is a powerful product message. In privacy-sensitive markets, this stops being an engineering detail and becomes a competitive differentiator and brand narrative.

The risks you take when choosing on-device

Mature strategic decision looks at the cost from below, not just the benefit from above. On-device inference carries three risks that need to be on the table.

The first is hardware fragmentation. Your audience uses devices with very different capabilities. Dedicated NPU in some, old hardware in others. The same feature delivers unequal experiences, and you don't control this park. This forces you to plan for fallback, and fallback usually means maintaining the server path as well.

The second is immature support. Most of the APIs that enable browser acceleration are still evolving. WebNN, for example, remains in preview at W3C, not yet recommended for production. Building on an unstable foundation means taking on rework.

The third is model maintenance. Updating a model on the server is trivial: you change it and that's it. Updating a model distributed across millions of devices is a rollout problem, with versions coexisting, cache to invalidate, and users in different states. This operational cost is ongoing and silent, and it is easy to underestimate in planning.

How to make the decision in practice

I use a simple, layered filter. First, the nature of the data. If it is sensitive and regulated, the on-device gains strengths for compliance. Second, the size of the model needed. If the task requires large model capacity, the server is almost mandatory. Third, the volume of inference. High volume favors the device due to economy; Low volume favors the server due to its simplicity.

The most common answer, in practice, is hybrid. Local model for the frequent, private and latency-sensitive case, with escalation to the server when the task requires more capacity. This requires an architecture that knows how to decide, at runtime, where to run each thing. It takes work, but it captures the best of both worlds without tying the product to one extreme.

What I don't recommend is deciding on a fad. AI in the product is process, data governance and outcome measurement, not buzzword racing. The question that orders everything is: which concrete problem can you best solve by changing the place where the inference takes place? If you can't answer clearly, the decision is not yet ripe.