Running AI in Environments Where Cloud Isn't an Option

| Katsuki Ono

AILLMEdge AIOn-PremiseSecurityFine-TuningEnterprise AILocal LLM

We've done hands-on post-training at the 750B-parameter scale — close to frontier-model territory — across the full pipeline of data curation, training, and evaluation.

We'll be upfront about something.

What most real-world deployments actually need isn't a massive cloud AI. It's a small, narrowly-scoped model, prepared so it can run safely inside your own environment.

We're not trying to beat frontier models

This is the part that gets misunderstood, so let's be clear.

Tuning a small model doesn't make it outperform something like GPT-5 in overall capability. Frontier models hold real advantages in breadth of knowledge, general reasoning, multilingual coverage, and complex tool use.

Having seen those strengths up close, we don't try to compete head-on.

What matters is narrowing the use case:

  • Answering questions based on internal manuals

  • Classifying inquiries that follow a fixed format

  • Summarizing field reports

  • Producing boilerplate text

  • Adapting language to a specialized domain

  • Serving as an offline assistant when there's no internet

For purposes like these, you don't need the largest AI. It's often more practical to take a small, narrowly-scoped model and adapt it to your environment.

Why run AI locally instead of in the cloud

An "edge environment" is one where processing happens close to the device, the in-house server, or the on-site equipment — not in the cloud. Running an LLM in an edge or on-premise setting has three main advantages.

Your data stays inside your environment. Business documents often contain customer information, internal know-how, contracts, or medical and welfare-related data — material that's difficult to send to external services. Running locally keeps the core of processing under your own control.

You're not dependent on the network. Cloud AI requires an internet connection. In settings where the network is slow, intermittent, or restricted, that's a problem.

Latency and cost are easier to control. Cloud costs scale with usage, and you're exposed to spec changes from the provider. Local deployment takes more upfront design, but lets you build a system with predictable cost and latency.

What you train matters more than how much

This is where insights from large-scale post-training translate directly into edge LLM tuning.

When you're tuning for edge deployment, compute is limited — GPU memory, training time, power, latency, operating cost, all of it. That's exactly why deciding what to train (and what to leave alone) matters more than how aggressively you train. It drives both efficiency and final performance.

We've worked through this empirically, not from intuition.

In one controlled study on a hybrid GDN/Attention architecture (Qwen3.5-0.8B, 84 experiments across 7 freezing conditions × 2 tasks × 6 seeds), we found that freezing 18 GDN layers — 51.6% of all parameters — performs statistically indistinguishably from freezing just 6 Attention layers (14.6%). Selective freezing also beats full SFT on knowledge QA in some cases, likely a regularization effect. And not everything is freeze-tolerant: Attention projection layers are disproportionately important for knowledge adaptation, and freezing them has a large impact.

The takeaway: not all layers are equally trainable, and the cost–performance frontier shifts substantially when you target the right ones.

In a separate survey across 43 models from 15 architecture families, residual-stream behavior during a forward pass varies by more than 500× across models, and this variation does not correlate with how robust each model is to layer pruning. Which layers can be removed, and which must be preserved, differs by model — there is no universal answer.

"Build small, train narrowly" is easy as a slogan. In practice it means examining model structure and empirically deciding what to train, model by model. That's the operating mode we bring to client engagements.

What it doesn't do matters too

For business AI, what it doesn't do matters as much as what it can:

  • Doesn't make confident claims about things it doesn't know

  • Doesn't reveal personal information unprompted

  • Doesn't answer outside the bounds of internal rules

  • Avoids language that could mislead users

  • Asks for human confirmation when needed

In practice, this kind of predictable behavior is often more valuable than versatility. Not "an AI that can answer anything," but one that operates within a defined scope, predictably, without overreach. That's where most security and governance issues in business AI actually arise — in scope and predictability, not in raw capability.

Don't build big from the start

A common failure pattern in business AI is trying to build a finished product from the start: a large model, mountains of data, a long training run, and evaluation only at the end. It eats budget and time, and when the result is bad, it's hard to tell what went wrong.

For edge and local LLMs, an incremental approach fits better:

  • Narrow the use case

  • Try it on a small dataset

  • Choose a model size that runs on your local hardware

  • Verify quality, speed, memory usage, and security requirements

  • Tune within that narrow scope, if needed

Working in stages makes it much easier to see whether local deployment is actually worth it, what level of performance is sufficient, and where to spend resources.

Where EqualFrontiers stands

Through our work on large-scale post-training, we've seen the strengths and the limits of frontier models up close. From that vantage point, we've come to a clear conclusion: many real-world problems can't be solved by bringing in the biggest AI. They can be solved by adapting a narrowly-scoped model to the constraints of the actual environment.

Some data can't go to the cloud. Some environments can't rely on a network. Not everyone has a large GPU running 24/7. And yet, there's still plenty of work AI can take off your plate.

In those situations, the practical move is to choose a narrowly-scoped model, validate it at small scale, and tune only within the range that matters. That's how we approach LLM deployment.

Typical engagements include using AI on internal documents without sending them to external services; running LLMs in on-premise or local environments; tuning small models for business-specific tasks; exploring how far you can push SFT, LoRA, and freezing strategies; designing the right split between cloud and local AI; and deploying AI under strict security requirements.

Closing

Cloud-based giants aren't the only way to deploy LLMs.

When you're working with data that can't leave the company, environments that can't rely on the network, or operations with strict security requirements, locally-run AI is often the right primary choice. Small models have limits — but if the use case is narrow enough, you can absolutely run a capable AI inside your environment.

The goal isn't to beat frontier models. It's to build AI that reliably handles the work that needs to be done, where it needs to be done, while keeping your data yours.

We channel what we've learned from large-scale post-training into the design of that "small AI."

References: Our research

‹ Back to Blog