ONLINE robertZurich, CHAS214476Delivery context: TLPT / FINMALanguageENDE
robert

BLOG

Gemma 4 31B uncensored · format family

Published 8 min read

How a 30.7B multimodal Gemma stopped refusing: a deterministic weight-level method, four precision formats, measured refusal and KL results, and serving commands matched to the model cards.

This post documents the release from the engineering side: what the problem was, why automated approaches stalled on this architecture, what each published format trades, and what the evaluation did and did not show. Download counts come from the committed Hugging Face snapshot of 2026-09-01; every other figure is verified against the linked model card or my own run records for this release.

Problem

The starting point is google/gemma-4-31B-it (opens in a new tab), a 30.7B-parameter dense instruction model with multimodal input: roughly 550M parameters sit in the vision encoder, and the language model has 60 layers, a 256K-token context window and a 262K vocabulary. As trained, it declines a meaningful share of security-relevant work, and that is a practical defect for penetration testing and red-team use: an analysis stalls on a refusal mid-task, and the refusal behavior survives prompt-level workarounds.

The goal was narrow: remove refusal behavior without (1) degrading the measured capabilities and (2) losing deployability across the hardware and engines this model had to run on, from two-GPU serving down to laptop inference.

Why search-based abliteration stalls here

Two failure modes shaped the final design, both observed before this release worked:

  1. Search plateaus. Automated search-based abliteration samples threshold combinations across layers and accepts a Pareto front over trials. On this architecture my runs never left the 63 to 75 refusals-per-100 band, no matter how many trials. The search space itself is the ceiling; more compute does not unlock it.
  2. A poisoned multi-GPU probe. Loading the full 31B checkpoint sharded across GPUs produced clean forward passes but NaN logits during generation prefill, exactly at the GPU boundary. A silent failure here is expensive: the greedy probe does not raise on NaN, so an optimizer keeps scoring a damaged model and reports KL of NaN. The working fix was to run the whole probe-and-edit loop with all tensors on one CUDA device.

Both findings are method-level lessons: the unblocking step was changing the procedure, not tuning harder.

The applied method: a per-layer two-sided projection

The deployed method replaces search with a deterministic edit:

  • Read the refusal direction per layer from a single GPU, using a fixed harmful-behavior probe batch.
  • Modify the weight matrices that write that direction into the residual stream, layer by layer: the attention output projection and the MLP down projection. Across the language model's layers this is 120 weight matrices in total, each edited with a norm-preserving projection. It removes the refusal-direction component of the activation while leaving the weight norm intact, so coherence depends on the direction being separable, not on hoping the edit is small. The method also carries the two measurement-hygiene flags from the card: topic-marker stripping and refusal-prefix skipping.
  • Keep the refusal measurement honest. gemma-4 answers harmful prompts compliantly but prefixes ***Disclaimer:**, and a naive keyword detector counts that prefix as a refusal; the two flags above cut those false positives. They change the measurement, not the model.

The damage budget is published as KL divergence to the base model: 0.1234 for this build, with the card documenting intact coherence the same way (math, multilingual and factual answers correct in the smoke checks). The measurable contract of the release: 0 hard refusals on 686 harmful prompts across four datasets, a claim the cards replicate for every published format.

Formats: one model, four cost and hardware targets

Artifact Format Released Downloads · 30 days
gemma-4-31B-it-uncensored (opens in a new tab) BF16 multimodal 2026-07-08 135
gemma-4-31B-it-uncensored-NVFP4 (opens in a new tab) NVFP4 multimodal 2026-07-08 285
gemma-4-31B-it-uncensored-GGUF (opens in a new tab) GGUF q8_0 to q2_k, text only 2026-07-08 1,746
MLX line (opens in a new tab) (bf16, 8/6/5/4-bit) MLX, Apple silicon 2026-07-10 see 4-bit card

Why one checkpoint became a family:

  • BF16 (59 GB) is the source of truth. All other artifacts derive from it. It serves at reference quality; the BF16 card lists two RTX PRO 6000 Blackwell 96 GB GPUs and driver 610 as the published hardware. Keeping the full-precision checkpoint available also keeps the evaluation ladder cheap to rerun later, because every quantized build points at one defined source revision.
  • NVFP4 (20 GB) is the serving build. The language model's dense linear layers go to 4-bit, but the vision tower, the vision embedder and lm_head stay BF16. That split is not a preference: vLLM's Gemma-4 multimodal loader requires the vision path at BF16, and the quant config must exclude it or multimodal input breaks. The result fits one Blackwell workstation GPU and retains image understanding exactly because the precision split follows the loader's requirement.
  • GGUF (12 to 31 GB, text only) is the reach build. llama.cpp drops the vision tower during conversion, so this path is text-generation only. The K-quant ladder maps to constraints: q6_k (24 GB) near-lossless, q4_k_m (18 GB) the recommended default, q2_k (12 GB) smallest with some quality loss, and still 0 hard refusals.
  • MLX (17.16 GiB at 4-bit, group size 64) is the Apple silicon build. Language weights 4-bit affine with group size 64, vision path BF16, derived from the exact source revision the BF16 card documents.

Precision tradeoffs in practice

The uncensoring is a weight-level change, and the cards verify it survives every quantization: 4-bit NVFP4 and 2-bit GGUF both keep 0 effective refusals. The tradeoff is therefore about kernel and engine fit, not behavior:

Component Published toolchain Runs on
BF16 transformers, vLLM, SGLang NVIDIA RTX PRO 6000 Blackwell 96 GB (SM120), driver 610
NVFP4 ModelOpt NVFP4_DEFAULT_CFG (dense), vLLM 0.23, SGLang 0.5.14 on Python 3.12 Blackwell only (SM120)
GGUF llama.cpp convert_hf_to_gguf.py + llama-quantize llama.cpp, Ollama, LM Studio; needs a build with Gemma 4 support (2026-06 or newer)
MLX mlx-vlm 0.6.4 Apple silicon

Two engine-level facts are worth publishing because they cost debugging hours:

  • On vLLM, the NVFP4 start needs --enforce-eager and --no-enable-flashinfer-autotune on this compute generation; without them startup hangs in autotuning.
  • On SGLang, gemma-4 rejects the flashinfer attention backend (--attention-backend triton is required), the FP4 GEMM backend stays flashinfer_cutlass, and autotune is disabled. The stock SGLang wheels do not ship complete sm120 kernels, so the remaining kernels JIT-compile on the host at runtime; that needs a working host toolchain.

Reproduce: the exact serving commands

Each command below is the one published on the matching model card.

vLLM (NVFP4 card):

vllm serve ressl/gemma-4-31B-it-uncensored-NVFP4 \
  --quantization modelopt --max-model-len 8192 \
  --enforce-eager --no-enable-flashinfer-autotune --trust-remote-code

SGLang (NVFP4 card):

python -m sglang.launch_server --model-path ressl/gemma-4-31B-it-uncensored-NVFP4 \
  --quantization modelopt_fp4 --attention-backend triton \
  --fp4-gemm-backend flashinfer_cutlass --disable-flashinfer-autotune --trust-remote-code

llama.cpp (GGUF card, q4 run; the card reports roughly 75 tokens/s for its measured setup on one RTX PRO 6000, which is its published figure rather than a general promise):

llama-server -m gemma-4-31B-it-uncensored-biproj-q4_k_m.gguf \
  -ngl 99 -c 8192 --reasoning-budget 0

--reasoning-budget 0 matters more than it looks: gemma-4 thinks by default in llama.cpp, so the answer lands in reasoning_content and content can look empty.

Transformers (BF16 card):

from transformers import AutoModelForImageTextToText, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("ressl/gemma-4-31B-it-uncensored")
m = AutoModelForImageTextToText.from_pretrained(
    "ressl/gemma-4-31B-it-uncensored", dtype=torch.bfloat16, device_map="cuda")

mlx-vlm (MLX 4-bit card):

python -m pip install mlx-vlm==0.6.4
mlx_vlm.generate --model ressl/gemma-4-31B-it-uncensored-MLX-4bit \
  --prompt "Explain why Alpine flowers survive harsh winters."

Measured results, and what they do not show

Generalization was tested with one deterministic evaluator across 686 prompts from four independent datasets:

Dataset Prompts Effective refusals
JailbreakBench 100 0/100
tulu-harmbench 320 0/320
NousResearch/RefusalDataset 166 0/166
mlabonne/harmful_behaviors 100 0/100
Total 686 0/686 (0.0%)

The interesting comparison: the base model refuses 99 of the 100 prompts in the mlabonne suite, while a naive keyword detector still flags 363 of 686 (52.9%) answers of the uncensored build as refusals, and every one of those is a compliant answer carrying the ***Disclaimer:** prefix. Only the hard-refusal count (refusal phrases in the first words) is meaningful.

What this measurement is not: it is not a capability benchmark. Coherence was smoke-tested (math, multilingual, factual); no full benchmark suite ran before release, and the cards say so explicitly. The numbers are bound to the named datasets, engines and hardware; they make no claim beyond them. And a model that no longer refuses harmful prompts is exactly what the cards advertise: say it plainly, because compliance without a disclaimer is the intended behavior here.

Evidence