Transformers Now Runs llama.cpp's GGUF Models

Hugging Face brings llama.cpp's quantized GGUF format into the transformers library—matching native speed on Apple Silicon.

By The ai-compute Desk

It is increasingly not. Hugging Face has added native GGUF support to the transformers library, letting developers load the same quantized checkpoints that power Ollama, LM Studio, and Jan—directly inside Python, with performance that now sits close to llama.cpp itself. [1]

GGUF is the file format developed by the llama.cpp team. It packages model weights and metadata, including tokenizer information and an optional chat template, in one file, and supports multiple quantization levels so users can trade a little precision for a much smaller memory footprint. [1] Variants such as Q4_K_M use mostly 4-bit weights while keeping sensitive tensors at higher precision—a practical middle ground that lets larger models fit on machines with limited RAM. [1]

GGUF models have been downloaded millions of times from the Hugging Face Hub, where publishers such as Unsloth, LM Studio Community, and bartowski maintain ready-to-use checkpoints across a range of quantizations. [1] Until now, loading those files inside transformers required dequantizing the weights first, which consumed extra memory and lost the performance benefits of quantized inference entirely.

The new integration sidesteps that problem by reusing llama.cpp's underlying ggml kernels through Hugging Face's kernels library. [1] When weights stay packed on Apple's Metal backend, transformers automatically loads the compatible ggml/Metal layer kernels and uses ggml-org/ggml-attn as the attention implementation. [1]

Benchmarks run on a MacBook Pro M2 Max with 32 GB unified memory show transformers is close to llama.cpp across three tested checkpoints—a small dense model, a larger dense model, and a mixture-of-experts model. [1] The comparison is not perfectly apples-to-apples: the transformers measurement includes prefill while llama-bench reports decode-only throughput. But the gap is narrow enough that the familiar Python API no longer demands a meaningful speed concession. [1]

Two changes to the generate loop are responsible for much of that headroom, and they benefit all transformers models—not just GGUF ones. [1] When a supported decoder-only input has no padding, its all-ones padding mask can be removed at the start of generation, so downstream attention code no longer needs to inspect that mask repeatedly to determine whether it can be skipped. On supported paths, generate copies the stopping decision asynchronously and consumes it on the following step, letting the CPU keep scheduling work while the GPU runs. [1] Together they reduce the synchronization overhead that quietly throttles generation speed.

The llama.cpp project—whose main goal is to enable LLM and VLM inference with minimal setup and state-of-the-art performance on a wide range of hardware—lists Apple Silicon as a first-class citizen, optimized via ARM NEON, Accelerate, and Metal frameworks. [2] Hugging Face is explicit that llama.cpp remains the recommended engine when efficient local inference is the priority. The new integration is pitched as a complement, not a replacement: a way to reach GGUF checkpoints from PyTorch workflows without switching runtimes. [1]

[1]

The real signal here is architectural, not just technical. When two previously separate ecosystems—one built for performance, one built for research—start sharing kernels, the friction between prototyping and deployment shrinks. That convergence matters more than any single benchmark number.

Current limitations are real. The packed inference path is MPS-only; padded batches cannot take the mask optimization shortcut and can have lower performance; and the packed loader currently covers the Qwen3.5 dense and MoE architectures, including compatible Qwen3.8 checkpoints. [1] Hugging Face says adding support for other architectures is relatively straightforward and will expand gradually. [1]

The bigger opportunity, as Hugging Face frames it, is using ggml kernels to accelerate models that llama.cpp does not support at all—new architectures, research variants, computer vision, audio, and multimodal models. [1] A kernel operates on tensors; it does not require the whole model to come from a GGUF file. [1] Local inference, in other words, is about to get considerably less format-dependent.

Compatibility is only useful if the model is pleasant to run.Hugging Face engineering blog