Hugging Face transformers now runs llama.cpp GGUF quants on Apple Silicon with ggml kernels

What Hugging Face announced

Hugging Face announced on 22 September 2026 that transformers can run GGUF checkpoints efficiently. The blog post is by Marc Sun, Arthur Zucker and Lysandre Debut. GGUF is the single-file format created by the llama.cpp team. It packages weights, tokenizer data and an optional chat template, and it supports quantization levels such as Q4_K_M. Developers pass the Hub repository and file name to from_pretrained through a gguf_file argument, then use the normal transformers API.

The same file can be served with transformers serve, which exposes an OpenAI-compatible endpoint that clients such as Jan or Pi can call. A dequantize option lets developers turn a GGUF checkpoint back into full-precision weights for fine-tuning.

How it works

Hugging Face is not reimplementing llama.cpp. It distributes builds of ggml's Metal kernels on the Hub through its kernels library and calls them from PyTorch. The ggml-quantization kernel reads packed quantized weights directly, so the full matrix is never expanded before each decode step. Other kernels handle normalization, flash attention and the gated delta network used in Qwen3.5 and Qwen3.8 linear-attention layers. A separate top-k kernel, written by Hugging Face, handles expert routing in mixture-of-experts models.

Two changes to generate also cut CPU and GPU synchronization, and Hugging Face says these speed up all transformers models, not only GGUF ones. In benchmarks on a MacBook Pro with an M2 Max chip and 32 GB of memory, Hugging Face reports transformers close to llama.cpp on three checkpoints. It notes the conditions differ: its own measurement includes prefill, while llama-bench reports decode only.

What changed for developers

Until now, a developer with a GGUF file from publishers such as Unsloth, LM Studio Community or bartowski needed llama.cpp or an app built on it, such as Ollama or LM Studio, to run it fast. Now the same file can sit inside a Python and PyTorch workflow. That allows hooks on activations, custom logits processors, evaluation with existing transformers scripts, and checks that a GGUF conversion matches the original weights. Hugging Face still calls llama.cpp its recommended engine for efficient local inference.

The limits are clear in the post. Packed inference works only on Apple Silicon through MPS. Padded batches run slower. Only the Qwen3.5 dense and MoE architectures and compatible Qwen3.8 checkpoints are covered. Users need transformers from the main branch and a matching kernels version until the next release.

What to watch next

Hugging Face says it will add architectures gradually and extend the work to batched generation on MPS. It also wants to apply ggml kernels to models llama.cpp does not support, including vision and audio models, without a full llama.cpp port. Support for CUDA and other backends is not stated in the announcement. More open tool releases are listed under Open Source AI.

Questions people ask

How do I load a GGUF model in transformers?

Pass the Hub repository and the GGUF file name to from_pretrained with the gguf_file argument, for example a Q4_K_M file from an Unsloth repository. Install transformers from the main branch and the kernels package first.

Does transformers GGUF support work on NVIDIA GPUs?

Packed inference is MPS-only at launch, so it runs on Apple Silicon. Other devices can still import GGUF files by dequantizing them, which uses more memory.

Is transformers now faster than llama.cpp?

Hugging Face says transformers is close to llama.cpp on its three test checkpoints, with different measurement conditions, and it still recommends llama.cpp for efficient local inference.

Sources

1. Transformers now runs llama.cpp quants (Hugging Face, 22 Sep 2026)
2. transformers repository (Hugging Face (GitHub), 22 Sep 2026)
3. llama.cpp repository (ggml-org (GitHub), 22 Sep 2026)