DiffusionGemma: Accelerating text generation by four times

Today marks the introduction of DiffusionGemma, an experimental open model that significantly enhances the speed of text generation by up to four times when using dedicated GPUs. Offered under an Apache 2.0 license, this 26B Mixture of Experts (MoE) model bypasses the typical sequential token-by-token processing used in autoregressive Large Language Models (LLMs) by generating entire text blocks at once.

DiffusionGemma is built on the foundation of the Gemma 4 family’s intelligence-per-parameter and incorporates advanced research from Gemini Diffusion. This model features a unique diffusion head tailored to optimize speed in text generation. While the autoregressive models of the Gemma 4 family continue to be the benchmark for high-quality outputs, DiffusionGemma caters to researchers and developers who value speed for tasks like in-line editing, quick iterations, and creating non-linear text structures.

Real-time interactive AI applications often encounter latency issues during local inference. DiffusionGemma addresses these concerns by offering a solution that allows for significant performance improvements on specific tasks through fine-tuning. For instance, the model was fine-tuned by Unsloth to solve Sudoku, a task that autoregressive models struggle with due to their token dependency. DiffusionGemma’s bi-directional attention allows for better handling of such tasks.

While diffusion-based text generation has been explored within the AI community, applying it to large models has posed challenges. DiffusionGemma tackles this by altering the hardware utilization approach. Traditional language models generate text like a typewriter, processing one token at a time from left to right. This method works well on cloud servers that batch multiple user requests. However, for local single-user operations, it leads to underutilization of hardware resources, as the GPU or TPU often waits idly for the next token.

DiffusionGemma resolves this inefficiency by generating a full 256-token paragraph at once, maximizing hardware utilization by providing a substantial workload for the processor. This transformation turns model inference from a sequential typewriter into a printing press capable of producing entire text blocks simultaneously.

The speed advantage of DiffusionGemma is particularly beneficial for local and low-concurrency inference settings. In cloud environments with high concurrent requests, autoregressive models can efficiently utilize computing resources, but DiffusionGemma’s parallel decoding may lead to increased serving costs. The greatest benefits appear at low-to-medium batch sizes on a single accelerator.

Similar to AI image generators that start with noise and incrementally create a clear picture, DiffusionGemma applies this iterative refinement to text. This capability allows the model to process whole paragraphs at once, enabling new behaviors like accurately finalizing complex markdown formatting and generating code in real-time.