Tag: serving

← All posts

1. Inference platform: a sanitized case study

   Published 2026-09-04 · 8 min read

   How a GitOps-managed vLLM inference platform actually worked: one OpenAI-compatible router in front of per-GPU engine profiles, hot swaps with readiness gates, streaming passthrough, token-accurate telemetry, and an offline model cache.

   • #inference
   • #vllm
   • #kubernetes
   • #serving
   • #architecture
   • #observability

2. Gemma 4 31B uncensored · format family

   Published 2026-07-08 · 8 min read

   How a 30.7B multimodal Gemma stopped refusing: a deterministic weight-level method, four precision formats, measured refusal and KL results, and serving commands matched to the model cards.

   • #model-release
   • #gemma-4
   • #bf16
   • #nvfp4
   • #gguf
   • #mlx
   • #serving

Atom feed