Cloudflare LLM Infrastructure: Architecture at Scale

As artificial intelligence workloads surge, Cloudflare has responded with cloudflare llm infrastructure architecture that separates model processing stages, deploys a custom inference engine, and applies compression techniques. This cloudflare llm infrastructure architecture approach represents a fundamental shift in AI serving platforms engineered for production scale.

TL;DR: Cloudflare LLM infrastructure architecture uses disaggregated prefill, Infire engine, and Unweight compression for efficient AI serving.

Cloudflare LLM Infrastructure: Architecture Components

Modern large language models have grown to sizes that strain conventional infrastructure. Kimi K2.5, for example, exceeds 1 trillion parameters and requires approximately 560GB of memory just to load the model weights. This demands at least eight NVIDIA H100 GPUs before accounting for additional memory needed during actual inference processing.

Traditional AI infrastructure, built around episodic human interaction patterns, struggles with the continuous, high-volume demands of production LLM workloads. According to Cockroach Labs’ State of AI Infrastructure report, companies moving AI systems into everyday use frequently discover their existing infrastructure cannot handle the scale and reliability requirements these workloads impose. The report emphasizes that legacy systems designed for human-paced interactions simply cannot match the relentless pace of AI inference demands.

Disaggregated Prefill: Splitting the Workload

Cloudflare’s architectural innovation centers on a technique called disaggregated prefill. LLM request processing consists of two distinct stages with different resource profiles:

Prefill Stage: Processes input tokens and populates the KV (key-value) cache. This stage is compute-bound, requiring significant processing power to analyze incoming prompts and establish context.
Decode Stage: Generates output tokens sequentially. This stage is memory-bound, requiring fast access to stored context and model weights for each token generation step.

By separating these stages onto different machines optimized for their specific workloads, Cloudflare achieves better resource utilization and performance. Michelle Chen, principal product manager at Cloudflare, along with Kevin Flansburg (senior engineering manager) and Vlad Krasnov (principal systems engineer), explain in their detailed blog post that this hardware configuration improves both performance and efficiency by matching each stage to appropriate compute resources rather than forcing a one-size-fits-all approach.

The disaggregation allows Cloudflare to scale prefill and decode independently based on traffic patterns. During peak usage, more decode instances can be spun up without wasting compute resources on unnecessary prefill capacity, and vice versa.

Infire: Cloudflare’s Custom Inference Engine

Announced during Cloudflare Birthday Week 2025, Infire represents Cloudflare’s proprietary AI inference engine designed to run large language models across multiple GPUs more efficiently. The engine addresses three critical challenges in distributed LLM inference that standard frameworks like vLLM or Text Generation Inference (TGI) handle less optimally.

Pipeline Parallelism Optimization

For models split across multiple GPU stages, Infire actively load-balances all pipeline stages to prevent situations where GPUs in one stage sit idle (starving) while other stages are still executing. This ensures consistent throughput across the entire inference pipeline. The engine monitors execution times across stages and dynamically adjusts batch sizes to keep all GPUs productive.

Tensor Parallelism Efficiency

When models use tensor parallelism (splitting individual operations across GPUs), Infire optimizes cross-GPU communication pathways to minimize latency. The engine makes inter-GPU data transfers as fast as physically possible given the hardware constraints, using techniques like communication compression and overlapping computation with data transfer.

Hybrid Approach

For most production models, utilizing both pipeline parallelism and tensor parallelism in tandem provides the optimal balance of throughput and latency. Infire dynamically manages this hybrid approach based on model architecture and incoming request patterns, automatically selecting the best parallelism strategy for each workload.

According to Cloudflare’s technical deep-dive on Infire, the engine also implements advanced scheduling algorithms that prioritize latency-sensitive requests while maintaining high throughput for batch operations.

Memory Optimization: Doing More with Less

Cloudflare has optimized Infire to reduce GPU memory usage for internal processes, achieving remarkable efficiency gains that directly impact what models can run on available hardware:

Llama 4 Scout runs on just two H200 GPUs with substantial capacity remaining for context tokens, enabling longer conversation histories without requiring additional hardware
Kimi K2.5 operates on eight H100 GPUs while preserving memory for the KV cache, allowing the model to maintain context across extended interactions

These optimizations mean more memory availability for actual inference work rather than overhead, directly translating to better performance and the ability to handle longer context windows. For applications requiring extensive context (legal documents, codebases, research papers), this memory efficiency becomes a competitive advantage.

Unweight: Compression Without Accuracy Loss

Cloudflare recently introduced Unweight, a tensor compression system that reduces large language model weights by approximately 15-22% without sacrificing accuracy. This compression reduces the amount of data GPUs must load and move during inference, resulting in faster model startup times and more efficient operation during sustained workloads.

The compression technique is particularly valuable for models that must be split across multiple GPUs, as it reduces the communication overhead between devices and allows more models to fit into available memory. Unweight achieves this through sophisticated quantization-aware training techniques that preserve the numerical precision needed for accurate inference while reducing storage requirements.

Details about Unweight’s implementation are available in Cloudflare’s technical announcement, which describes the mathematical foundations and practical benefits of the compression system.

Architecture Comparison: Traditional vs. Cloudflare’s Approach

Aspect	Traditional LLM Infrastructure	Cloudflare’s Optimized Architecture
Processing Stages	Prefill and decode on same GPU	Disaggregated: separate machines for each stage
Resource Utilization	One-size-fits-all hardware	Compute-optimized for prefill, memory-optimized for decode
Inference Engine	Standard frameworks (vLLM, TGI)	Custom Infire engine with dynamic load balancing
Memory Efficiency	Standard weight loading	Unweight compression (15-22% reduction)
GPU Communication	Standard NVLink/PCIe	Optimized cross-GPU communication paths
Model Startup	Standard loading times (30-60 seconds)	Faster model initialization (10-20 seconds)
Scaling Strategy	Vertical scaling (bigger GPUs)	Horizontal scaling (more optimized nodes)

Production Deployment: Workers AI Platform

Cloudflare’s infrastructure innovations are not theoretical exercises—they power the company’s Workers AI platform, which now supports open-source models including Moonshot AI’s Kimi K2.5. Developers can access these optimized models through Cloudflare’s edge network, benefiting from the architectural improvements without managing GPU infrastructure themselves.

The platform also implements prompt caching optimizations that further reduce latency for repeated or similar queries. When multiple users ask related questions, cached intermediate results can be reused, dramatically reducing compute requirements and response times.

Industry Context: The Infrastructure Challenge

Cloudflare’s approach highlights a broader industry shift toward specialized, purpose-built infrastructure for AI workloads. The separation of concerns between prefill and decode stages represents a fundamental rethinking of LLM serving architecture that other providers are beginning to adopt.

Research from institutions like arXiv has documented similar challenges in distributed inference, validating Cloudflare’s architectural choices. As models continue to grow in size and complexity, such disaggregated approaches will likely become standard practice rather than innovative exceptions.

Organizations building AI-powered applications must understand these infrastructure layers, as the choice of inference platform directly impacts latency, cost, and the feasibility of running cutting-edge models at scale. Technical analysis from open-source communities shows growing interest in optimized inference patterns.

Looking Ahead

Cloudflare’s infrastructure innovations address immediate challenges in LLM serving while establishing patterns for future scaling. As the company continues to optimize prompt caching and expand its model catalog, the gap between research-scale models and production-ready inference continues to narrow.

The combination of architectural innovation (disaggregated prefill), software optimization (Infire engine), and compression techniques (Unweight) demonstrates that significant efficiency gains remain available through systems-level thinking rather than waiting for hardware improvements alone. This multi-layered optimization approach will likely define the next generation of AI infrastructure as the industry matures beyond early experimental deployments.

For more insights on AI infrastructure and cloud architecture, explore our previous analysis on cloud architecture patterns for modern applications.

Sources: Cloudflare blog, InfoQ analysis, Cockroach Labs State of AI Infrastructure

Discover more from Susiloharjo

Subscribe to get the latest posts sent to your email.