Prime Intellect has officially unveiled Prime Inference, a specialized serving platform tailored for the deployment of cutting-edge open-source models. This new service offers dual deployment modes: adaptable serverless endpoints for fluctuating demand and dedicated reserved capacity, both powered by Prime Intellect's own GPU resources distributed across multiple data centers. Prior to its public release, the platform demonstrated robust performance, internally managing nearly a trillion tokens daily, generated from diverse tasks such as reinforcement learning rollouts, synthetic data creation, model evaluations, and continuous coding agents.
Revolutionizing Open Model Deployment
Prime Inference functions as the crucial serving component within Prime Intellect’s comprehensive open training ecosystem. The company already provides a suite of post-training utilities, including prime-rl, verifiers, and sandboxes. By integrating a serving layer, Prime Intellect enables a closed-loop system where models in production generate traces that can inform and refine subsequent training cycles. The platform boasts impressive performance metrics; its GLM-5.3 endpoint reportedly ranks among the fastest available on OpenRouter, accompanied by a near-zero error rate for tool calls and a flawless uptime record since its inception.
Flexible Deployment Options and Robust Infrastructure
The service architecture is built for versatility and resilience. Customers can opt for serverless endpoints, ideal for workloads with unpredictable spikes, or secure reserved capacity for consistent, high-volume operations. It features an API compatible with OpenAI’s SDK, allowing seamless integration. Reliability is paramount, with automatic failover mechanisms across data centers ensuring uninterrupted service. The underlying hardware currently utilizes NVIDIA Blackwell GPUs, with plans to integrate Vera Rubin technology. Billing is consolidated, offering team-level usage tracking, though specific per-model pricing details are still pending full publication.
Deep Dive into the Serving Stack
The serving stack is a sophisticated amalgamation of advanced technologies, including NVIDIA Dynamo, vLLM, Mooncake, and FlashInfer. Developed collaboratively with Inferact and NVIDIA, the team regularly contributes fixes and enhancements upstream to these open-source projects. Prime Inference is particularly optimized for agentic workloads, which typically involve adding approximately 6,000 tokens to a 140,000-token prompt in a single turn, benchmarked using SemiAnalysis AgentX.
Architectural Innovations for Peak Performance
- Prefill/Decode Disaggregation: Prefill and decode operations run on distinct GPU groups. Dynamo manages routing, while vLLM executes the model. Decoders retrieve computed KV caches through NIXL, leading to nearly a 40% reduction in p90 inter-token latency during internal testing.
- Cache-Aware Routing: Dynamo's intelligent router prioritizes cached prefix overlap, ensuring user sessions persist on the same decoder. Mooncake further enhances this by introducing a secondary KV tier within host DRAM.
GLM-5.3 Benchmarks on GB200 NVL72
Performance figures for GLM-5.3 running on GB200 NVL72 hardware highlight the platform’s capabilities. Aiming for an interactivity target of 100 end-to-end tokens per second per user, a 1:4 prefill/decode ratio proved most effective. This configuration accommodated 66 sessions per prefill group, each achieving 101 tokens per second per user, with 100 output tokens per second per GPU. Key optimizations include:
- DEP8 Prefill Topology: Offers approximately five times the usable prefix-cache capacity compared to TEP8 on identical hardware.
- Smaller Prefill Budget: Halving tokens per step from 8,000 to 4,000 per GPU dramatically cut median queue wait times from 550 ms to 110 ms, while also decreasing median time to first token by about 20%.
- NVFP4 KV Compression: Increases cached tokens per decoder from 1.09 million to 1.63 million by shrinking each MLA cache row.
- Native Sparse-MLA Kernel: Offers superior performance at 12.0 μs for 15 query tokens, compared to 17.7 μs staged and 13.7 μs FP8.
- BLHNC KV Layout: Reduced transfer descriptors from 19,559 to roughly 1,940, leading to a drop in mean transfer time from 146 ms to 78 ms.
Ensuring Reliable AI Tool Calls
Ensuring reliable tool calls for AI agents is another critical area of focus. Prime Intellect's engineers developed and contributed a structural-tag builder to Dynamo specifically for GLM's tool format. Subsequently, vLLM leverages xgrammar to mask tokens that do not conform to the tool schema, addressing a common source of agent failure.
Competitive Landscape and Future Outlook
While comparative pricing for GLM-5.3 on Prime Inference is not yet public, competitors like Together AI, Fireworks AI, and Baseten list prices around $1.40 for input and $4.40 for output per million tokens. Prime Inference distinguishes itself with its transparent, open-source serving stack. Future plans include the introduction of batch inference capabilities and one-click dedicated deployments, further expanding its utility for diverse AI model serving needs.
This article is a rewritten summary based on publicly available reporting. For the original story, visit the source.
Source: MarkTechPost
