Blog Details

For many regulated healthcare environments, a hospital network designed to be unreachable from the outside world is not hypothetical; it is an explicit design goal. This means no outbound or inbound internet access, network diagrams audited line by line, and a compliance posture focused on whether any packet ever leaves a segment. In such an environment, a clinical team sought AI-assisted documentation, including ambient note structuring, dictation cleanup, and summarization of long encounter histories. Every pattern we'd normally consider, such as a hosted LLM API, a managed inference endpoint, or even a vendor's 'private cloud' offering, assumed some connection to the internet. That connection was precisely what the network was built to prevent.

This post details our experience deploying open-source models within such isolation. We cover the split architecture used to transfer models and data without compromising network security, the cloud deployment assumptions that quietly failed, and the specific issues encountered during the initial week of running what appeared, on paper, to be a standard containerized inference stack. None of the technology was exotic. Most of the effort involved manually rebuilding the conveniences that cloud platforms typically provide for free.

Why 'Just Call the API' Was Never on the Table

The clinical network is segmented from the internet by design, not merely by a configurable firewall rule. Any outbound connection is not treated as a risk to be assessed; it is a finding, full stop, regardless of data content or business associate agreements. This distinction is crucial. It was not a data-handling problem solvable with encryption and contracts, but a connectivity problem where the absence of a route served as the primary control.

Our first instinct was to seek a vendor's on-prem or 'private cloud' LLM offering, hoping to gain hosted API convenience without internet dependency. This approach did not withstand scrutiny. Nearly all such offerings still assumed some connectivity for license validation, telemetry, model updates, or remote support tooling. Each of these touchpoints constituted a violation of the isolation requirement, irrespective of the data exposed.

• License 'phone-home' checks baked into vendor runtimes
• Telemetry and crash reporting enabled by default
• Model or security patch delivery channels requiring outbound access
• Remote support tooling assuming reachability for diagnostics
• Vendor-side logging of usage metadata off-network

The First Draft: Porting a Cloud Playbook to a Locked-Down Network

Our pragmatic starting point was to relocate our containerized inference stack, previously used in a cloud pilot, onto GPU servers within the isolated segment. This stack included an open-source serving layer, autoscaled GPU workers, and model weights pulled from object storage. On paper, this seemed like an infrastructure move, not a redesign. We anticipated a week for hardware provisioning and then completion.

The plan failed almost immediately. Container images pulled base layers from public registries during build time. Package managers accessed PyPI and apt mirrors during setup. The serving layer's default behavior was to download model weights from a public model hub on first launch. None of these were issues in the cloud pilot, as it had internet access we took for granted.

The deeper issue was not any single dependency; rather, we had quietly conflated 'runs in a container' with 'runs offline.' Containerization provides reproducibility and isolation from the host OS, but it does not guarantee that the containerized application expects to run without internet access. Every layer of cloud-native convenience we relied on turned out to harbor a hidden assumption about connectivity, and the isolated network exposed all of them simultaneously.

“Every convenience of a cloud-native stack is a hidden dependency on a live internet connection, and an air-gapped network exists precisely to sever that dependency.”

A One-Way Door: Designing the Split Architecture

Our final design split the environment into two zones. First, a connected staging environment outside the clinical network boundary allowed us to pull models, build containers, and vet dependencies with full internet access. Second, an isolated production zone, with no outbound route, hosted the actual inference. Between these zones was a one-directional transfer mechanism, reviewed and approved by the security team, which could move files in but had no path for anything to move out.

Nothing crossed that boundary informally. In staging, we packaged versioned 'bundles,' which included model weights, container images, Python wheels, checksums, and a manifest detailing the contents. We scanned these bundles for known-vulnerable packages, signed them, and pushed them through the approved transfer mechanism. Inside the network, a human verified checksums against the signed manifest before unpacking and deploying anything. This ensured the release was traceable to a specific, reviewed artifact, not a live pull from the internet.

The tradeoff was speed. Every model update, dependency bump, and configuration change had to pass through this pipeline. This transformed a five-minute cloud deployment into a multi-day, change-controlled process. While initially feeling like friction, this was the entire point for this environment: the architecture's value lay in ensuring nothing entered the network without a reviewable trail.

Choosing Models When You Can't Just Try Six of Them

In a cloud environment, model selection is often empirical: spin up candidates, run them against real prompts, and discard underperformers. This workflow is incompatible with a one-way transfer pipeline, where introducing a new model costs a change-controlled release cycle, not a simple curl command. We therefore had to conduct far more evaluation upfront, in staging, before committing anything to the actual pipeline.

This necessitated a more deliberate model selection process than simply choosing 'which one scores highest.' Licensing terms were unusually critical, as we could not renegotiate or swap a model later without connectivity for updates. Quantization support was important because the GPUs in the isolated zone were modest compared to on-demand cloud resources. Furthermore, self-contained tokenizers and chat templates were essential, as anything attempting to reach a hub for configuration at inference time would silently fail.

We ultimately selected a mid-sized, open-weight language model for note structuring and summarization, quantized to fit comfortably on the on-prem GPUs. This was paired with a smaller, open speech-to-text model for dictation cleanup. We deliberately avoided the largest, highest-benchmark models, as they would have required multi-GPU clusters that the isolated segment's hosting team could not realistically provision, monitor, or patch under the stringent audit constraints.

• Clear, permissive license that doesn't require future renegotiation
• Confirmed support for quantization at available VRAM budgets
• No runtime dependency on hub-hosted tokenizer or config files
• Reasonable quality on clinical language without a giant parameter count
• Community or documentation depth sufficient to debug offline

What Broke First: Clocks, Drivers, and Silent Failures

Once the split architecture was deployed and the first bundle landed inside the network, failures emerged. These were rarely model-related; instead, they stemmed from infrastructure plumbing that had always quietly relied on internet access. Server clocks drifted due to the lack of a public NTP pool route. This, in turn, broke TLS certificate validation across our internal service mesh. It took a day to trace the failure back to 'the clock is wrong,' not 'the cert is wrong.' GPU driver installation scripts expected to fetch the CUDA toolkit from a public repository and simply hung with unhelpful errors.

Some failures were subtler and more dangerous because they did not fail loudly. A monitoring agent, included for observability, attempted to phone home for license validation on startup. It failed silently and ran in a degraded mode, dropping most metrics. This meant we believed we had visibility into the inference servers for several days when we did not. Python packages with native extensions required a compiler toolchain absent from the hardened base image, and the errors surfaced three layers removed from the actual cause.

None of these were exotic problems. They are the types of issues invisibly resolved by any cloud deployment, as cloud images ship with working time sync, cached toolchains, and reachable package mirrors by default. Within an air-gapped network, every one of these defaults required manual rebuilding.

• TLS validation failures traced back to clock drift with no NTP access
• GPU driver installers hanging on toolkit downloads that never resolved
• A monitoring agent silently degrading after a failed license check
• Native-extension packages missing a compiler toolchain on the hardened image
• CA bundles going stale with no path to a public root store update

What We Shipped, and What It Taught Us About Regulated AI

The system stabilized once the internal NTP server, an internal package mirror within the isolated segment, a vendored and manually refreshed CA bundle, and pre-built wheels matching the exact hardware profile were all in place. Clinicians received dictation-to-structured-note assistance running entirely on local hardware, with no data ever leaving the network and no external calls in the request path. When auditors reviewed the deployment, the architecture precisely matched the network diagram, having been designed against it from day one rather than adapted afterward.

None of this is unique to healthcare. Treating 'no internet access' as a deployment detail bolted onto a cloud-first design inevitably leads to a rebuild. Much of cloud-native tooling's convenience stems from invisible internet dependencies baked into defaults, including registries, telemetry, time sync, certificate chains, and license checks. Treating isolation as a first-class constraint from the outset alters almost every technical decision, from model packaging and monitoring instrumentation to the urgency of shipping fixes.

Even organizations not literally air-gapped, such as those in finance, defense, government, or any environment with strict data-residency rules, encounter a lighter version of this problem. The discipline remains the same: version everything, sign everything, move it through a reviewable one-way path, and assume nothing about connectivity that has not been explicitly verified. A hosted API offers rented convenience. When the environment prevents renting, you must own every layer. This deliberate ownership produces a system that is ultimately more auditable than its cloud-first counterpart.