IBM's $0.11-per-Million-Token Benchmark Resets Self-Hosted Inference Economics

Self-hosted inference becomes a credible alternative to proprietary APIs

IBM Research reported that its open-source llm-d 1.0 stack served up to 3,000 concurrent coding agents on 544 NVIDIA H100 GPUs at roughly $0.11 per million tokens—a figure IBM claims is five to 10 times lower than equivalent commercial API pricing for context-heavy agentic workloads. The stack delivered peak throughput exceeding 6.6 million output tokens per minute with a roughly 98% prefix-cache hit rate.

The reported cost advantage is not universal—it depends on H100 utilization rates, workload characteristics and infrastructure expenses—but it provides the first concrete benchmark for enterprises modeling self-hosting economics against API spend. For buyers running high-volume agents with long, repetitive prompts, the math now favors owning the stack.

llm-d supports NVIDIA, AMD and Intel GPUs, Google TPUs, managed clouds and sovereign on-premises clusters. It integrates routing, KV-cache offloading, prefill/decode disaggregation and inference servers such as vLLM and SGLang. IBM's differentiator is portability across hardware and deployment environments rather than a single-cloud control plane, directly competing with proprietary inference platforms from AWS, Microsoft, Google and NVIDIA, as well as commercial serving layers such as Together AI, Fireworks AI and Baseten.

The tradeoff: buyers assume more responsibility for GPU capacity planning, model operations, security patching and support than with a hyperscaler-managed API. Enterprises should evaluate whether their infrastructure teams can absorb that operational load before committing to self-hosted inference at scale.

Simplismart's Series B validates latency-cost tradeoff as a procurement variable

Inference-infrastructure vendor Simplismart raised a $14 million Series B, led by Dallas Venture Capital with participation from NVIDIA, Accel, Shastra VC and Micromax. The company cited a production deployment for enterprise AI firm Ema in which its optimizer identified configurations ranging from $0.05 per million tokens at 350 ms latency to $0.12 per million tokens at 100 ms. The selected production configuration delivered $0.07 per million tokens at 180 ms.

That concrete tradeoff—7 cents per million tokens at 180 ms versus 12 cents at 100 ms—demonstrates why enterprises should procure inference on a latency/cost frontier, not on token price alone. Customer teams should require workload-specific benchmarks covering tail latency, concurrency, cache behavior and failover, not vendor-provided averages.

Simplismart competes with model-serving and GPU-orchestration platforms such as llm-d, NVIDIA's NIM ecosystem, Baseten, Modal and cloud-native offerings from AWS, Google Cloud and Microsoft Azure. The platform may reduce infrastructure complexity, but it can create another control-plane dependency. Buyers should verify exportability of routing policies, observability data and deployment configurations before standardizing on it.

Google Cloud adds OpenAI models to challenge Azure's procurement advantage

Google Cloud began rolling out OpenAI's GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna in private preview through its Model Garden, alongside more than 200 first-party, third-party and open models. Google said OpenAI usage would transact through Google Cloud Marketplace, appear on a consolidated Google Cloud invoice, draw down existing cloud commitments and count toward Flexible Savings Plans.

This directly challenges Microsoft Azure's historical advantage in offering OpenAI models and positions Google Cloud as a multi-model procurement layer. Google's pitch is model interchangeability across performance, latency and cost, while Azure retains the advantage of deeper Microsoft enterprise integration.

For organizations already committed to Google Cloud, marketplace billing could reduce procurement friction and make it easier to shift workloads among Gemini, OpenAI and open models without signing separate vendor contracts. It may also strengthen the negotiating position of buyers seeking volume discounts or committed-spend treatment.

The models were still in private preview, so buyers should not treat availability as production readiness. They should confirm regional availability, data-retention terms, service-level agreements, quota limits and whether model switching preserves API behavior and evaluation results.

What to watch

The practical trend is a move from selecting one "best" model to building an inference portfolio: proprietary APIs for difficult reasoning, open models for predictable high-volume workloads, and orchestration layers to route traffic by cost, latency, geography and data sensitivity.

IBM's reported $0.11 per million tokens, Simplismart's $0.07-per-million-token/180-ms production result, and Google Cloud's multi-model billing strategy give procurement teams concrete benchmarks for reassessing API, GPU and platform budgets. Buyers should model their current API spend against self-hosted economics, then determine whether their teams can absorb the operational responsibility. The decision is no longer about whether self-hosting is cheaper—it is about whether the organization can execute it.