AI infrastructure
Four Vendors, Four Contracts: Why AI Infrastructure Breaks at the Seams
GPUs from one vendor, data movement from another, delivery from a third. See why split AI infrastructure breaks and what one owner changes.
A GPU can be healthy while the application around it feels broken. The model is running, the network circuit is up, and the delivery endpoint responds. Yet users wait for the first word, watch the answer stall, or abandon the request.
When compute, connectivity, data services, and delivery come from separate providers, each team may be meeting its own commitment. The gap is the journey between those commitments. Someone still has to design, measure, and operate the complete service.
That work often lands on the customer.
The user experiences one request. The infrastructure sees several services
Consider an AI application that retrieves information before generating an answer. A request enters the application, reaches a retrieval service, waits for model capacity, runs inference, and streams a response back to the user. Authentication, storage, networking, and delivery can introduce additional dependencies.
Time to first token includes more than GPU execution. Request queueing, prompt processing, and network transport all contribute. After generation begins, the application still has to deliver the stream consistently. A fast model benchmark does not describe every part of that journey.
A supplier may measure availability at its API. Another measures a network port. Another measures a cache or delivery endpoint. Those are useful measurements, but their boundaries need to match the service you are buying.
Where the seams create problems
Placement decisions happen in isolation
Cheap GPU capacity can become an expensive choice when the application repeatedly crosses regions to retrieve data or reach users. Similarly, a nearby delivery endpoint cannot remove a remote inference trip if the model still runs far away.
Compute placement, data placement, and the network route should be reviewed together. The right location depends on the workload, user geography, model requirements, and the capacity available at each site.
Available bandwidth does not guarantee a responsive stream
A circuit's headline rate tells you little about the queue a particular request encounters. When traffic exceeds an available service rate, buffers can add delay. Shared congestion, an indirect route, or an access bottleneck can affect delivery even when the GPU has already produced its answer.
A private Layer 2 service can provide a defined interconnection and a route that can be engineered around the workload. Its performance still depends on distance, provisioned capacity, equipment, and the agreed service objectives. The label alone is not a latency guarantee.
Each team has only part of the evidence
The application team sees a slow response. The compute provider sees normal utilization. The network provider sees a working link. Without correlated timestamps and request identifiers, the investigation becomes a sequence of screenshots and competing explanations.
Distributed tracing helps connect application operations across process and network boundaries. Network measurements provide a different view of the path. Both are valuable: an application trace does not automatically reveal every router queue, and a network dashboard does not explain model scheduling.
Failure handling crosses organizational boundaries
Timeouts and retries need to agree across the application, gateway, and model service. Otherwise, a request that one layer has abandoned may still consume compute while another layer submits it again. Retrying an operation also requires knowing whether repeating it is safe.
The same applies to change management. A model rollout, circuit maintenance window, routing change, or delivery configuration can affect the same user journey. Separate change calendars make coordination a job of its own.
What one accountable owner should change
A unified provider should bring the architecture and operating responsibility together. That means more than combining several invoices.
- One workload plan. Decide where the model runs, where its data lives, how traffic reaches the cluster, and where responses enter the user's access network.
- One measurement plan. Define what a successful request means and measure from locations that represent actual users. Separate serving delay from network delivery delay.
- One incident lead. Identify who coordinates diagnosis and restoration when the cause spans services. Escalation should not require the customer to establish fault before work begins.
- One change process. Coordinate maintenance, capacity changes, model deployment, and routing changes against the same service requirements.
- One written responsibility map. Record the provider's service boundaries, customer responsibilities, third-party dependencies, and escalation paths.
This approach can simplify decisions and reduce handoffs during an incident. It does not remove the underlying dependencies. Facilities, carriers, hardware, software, and the user's access network still matter. A responsible provider makes those dependencies visible and plans around them.
Measure the experience you actually need
Start with the application, then define the infrastructure requirements that support it. For interactive inference, useful measures include time to first token, response completion time, stream interruptions, and successful request rate. Inspect slower requests as well as the median.
Record the test conditions: model, input and output lengths, concurrency, serving configuration, client location, and access method. Results from an idle service in the same facility as the test client cannot establish performance for a busy production application across several regions.
Service objectives should identify the measurement points, workload conditions, thresholds, and reporting method. They should also distinguish an engineering target from a contractual service commitment. That makes performance discussions concrete before an incident occurs.
Ask these questions before adding another contract
- Who owns the complete request path, including the handoffs between services?
- Can the team show where time is spent in retrieval, serving queues, inference, and transport?
- How are compute placement and network interconnection chosen for our users?
- What happens when a request stalls but every individual service remains available?
- Who leads restoration, and how do external dependencies enter that process?
- Which performance commitments are written into the service, and under what conditions?
Multiple providers can work well when these responsibilities are explicit. The problem begins when integration and operational ownership are left for the customer to discover after deployment.
HarborOne brings compute, network, and delivery into one plan
HarborOne combines GPU compute, private networking, and AI delivery around the path between a workload and its users. Core, regional, and edge placement provide different options for where inference can run. Private interconnection and local handoff shape how its responses travel.
The goal is straightforward: give customers a clear owner for the infrastructure we provide, with the service boundaries and performance requirements defined together. The design still has to fit the model, the data, and the users.
Bring us your workload and user geography. We can discuss the compute placement, network path, and delivery requirements as one architecture.
