Posts tagged “inference”

• llama.cpp b11414: Vulkan Scratch-Buffer Fix and Long-Prompt Checks

  llama.cpp b11414 fixes stale Vulkan scratch-buffer reuse. Learn which workloads merit testing, what upstream results show and how to validate long prompts.

• llama.cpp b11415: Media Truncation Fix and Upgrade Checks

  llama.cpp b11415 fixes media-chunk boundary checks. What multimodal server operators should validate in prompt caches, client errors and staged upgrades.

• llama.cpp b11412: k-Pool Graph Fix and Regression Checks

  Test llama.cpp b11412’s k-pool graph fix for Qwen4Exp and GLM5-Next, with shared-sequence checks, diagnostic limits and cache-migration caveats.

• llama.cpp b11411: Saved KV-Cache Migration and Restore Checks

  llama.cpp b11411 rejects mismatched KV-cache restores and changes saved-state formats. How to rebuild caches, test restores and preserve rollback.

• vLLM 0.31 Upgrade Checks: Preload, Snapshots and Multimodal Defaults

  A vLLM 0.30-to-0.31 operator checklist for preload readiness, cache fallback, experimental snapshot eligibility and multimodal client compatibility.

• llama.cpp b11401: Router-Mode Fix and Upgrade Checks

  llama.cpp b11401 separates router child commands from logs. See who should upgrade and how to check downloads, model readiness, shutdown and logging.

• llama.cpp b11387: N-Gram Speculative Decoding Upgrade Guide

  Check whether llama.cpp’s n-gram regression affects your deployment, what b11387 fixes, and how to validate acceptance and latency before upgrading.

• vLLM 0.30 Gives Model State New Lifetimes. Budget Them.

  Fast Start and HiSparse move weights and KV across new lifetimes and memory tiers. The vLLM 0.30 upgrade question is who owns warm state—and its cost.

• Tencent Hy4 Preview: Why a 770B Open-Weight Agent Model Is Easier to Rent Than Run

  Tencent’s Hy4 Preview offers 1M context, Apache-2.0 weights, and cheap agent APIs. Its real test is retrieval, caching, and route economics.

• Nemotron 3.5 Lightning Turns Agent Speed Into a Routing Problem

  NVIDIA Nemotron 3.5 Lightning is a fast open agent worker. Its real value lies in routing, validation, and deployment economics.

• vLLM 0.27.0 Expands the Runtime—and the Blast Radius

  vLLM 0.27.0 adds Kimi K3, MRv2 workloads, Rust control APIs, and fault recovery—but its PyTorch 2.13 jump makes this a fleet migration.

• SGLang 0.5.17 Gives Agent Sessions a Vote in GPU Memory

  SGLang 0.5.17 adds session-aware KV caching, faster recovery, Rust ingress, and frontier-model paths—shifting serving toward agent runtime policy.

← Back to blog