Staff · 17 pieces on file
Aiko Tanaka
Inference & serving
Aiko covers the serving stack — vLLM, SGLang, TensorRT-LLM, and the kernels underneath. Her beat is throughput, latency, and the gap between a model’s published numbers and what an operator can reproduce on real hardware at a real batch size.
Beats: inference
All pieces by Aiko
-
Infrastructure · NOVEMBER 16, 2026
llama.cpp 1.0 makes breaking upgrades easier to spot—not lead replies better
New versioning rules give local-AI developers a clearer compatibility boundary. Small businesses still need to test inquiry handling, reply quality, and deployment security.
-
Infrastructure · NOVEMBER 16, 2026
Tencent’s Hy4 flash activates 12B parameters and cuts API rates
The 224B-parameter model targets routine work with lower token prices, but Tencent’s internal task-cost results still need validation on customer-facing workloads.
-
Infrastructure · NOVEMBER 13, 2026
KORA puts local AI inside Linux—but its desktop remains a 2027 plan
Cortex promises inference across CPU, GPU and NPU. For small businesses considering sensitive customer workflows, local-by-default is not the same as cloud-free.
-
Infrastructure · NOVEMBER 12, 2026
Gemini Decision 1 scores 1,000 checks per call—but latency has limits
Google’s probability-returning API costs $0.10 per million input tokens. Its potential small-business use is qualifying and routing inbound inquiries, not writing outreach.
-
Infrastructure · OCTOBER 23, 2026
Anthropic and OpenAI Hand Agent Skills and Open Responses to a Neutral Foundation
At AGNTCon on October 22, four frontier labs donated the layers of the agentic serving stack — skills, model interfaces, RL environments, and sandboxes — to the Linux Foundation's AAIF under vendor-neutral governance.
-
Infrastructure · OCTOBER 19, 2026
Mercury 3 Ships at 1,068 Tokens/Second and $0.10/M — Without a Single Independent Benchmark
Inception's diffusion LLM claims to beat GPT-6 Luna and Claude Haiku 5.5 across benchmarks while running 4–8× faster, but Artificial Analysis has no Mercury 3 page one day after launch.
-
Infrastructure · OCTOBER 12, 2026
Asana's 76× Browser-Agent Cost Cut Was Mostly a Cache Fix, Not a Model Upgrade
A 144-run StackAI study found that making a browser agent's request history append-only dropped per-run cost from at least $36.21 to $1.24 on the production model — before GPT-6.1 Sol's cache-read rate took it to $0.47.
-
Infrastructure · OCTOBER 12, 2026
Microsoft's Maia 300 Targets 40% More Tokens Per Megawatt, Named Primary Inference System for Copilot in 2027
Unveiled October 12 at Microsoft's infrastructure announcement, the TSMC N2P chiplet accelerator is scheduled to begin production deployments in 2027 — and will anchor the inference substrate under Copilot, GitHub Copilot, and Azure Foundry.
-
Infrastructure · SEPTEMBER 15, 2026
Claude Code's Weekly Cap Drops 17% Today as Anthropic's Temporary Boost Expires
Anthropic's four-month +50% weekly promotion ended at 11:59 PM PT on September 13. The permanent replacement sits 25% above the old baseline — and 16.7% below what heavy Claude Code users had on Friday.
-
Infrastructure · SEPTEMBER 15, 2026
Claude Code Weekly Limits Fall 17% Today as Anthropic's Summer Promo Expires
The 50% temporary boost that ran since May ended September 13 at 11:59 PM PT. From September 14 forward, weekly ceilings settle 25% above the pre-promotion baseline — meaningfully less headroom than developers built pipelines around.
-
Infrastructure · SEPTEMBER 12, 2026
Sakana Ships Fugu Max at $2/$6 per Million Tokens, Splitting Orchestration Into Cost and Capability Tiers
Fugu Max routes across a swappable pool of open-weight and specialized models — including NVIDIA Nemotron — behind one OpenAI-compatible API, with output priced 40–60% below Sonnet 5, GPT 5.6 Terra, and Kimi K3.
-
Infrastructure · AUGUST 30, 2026
The Frontier API Price Floor Just Dropped Under Everyone's Q4 Budget
Anthropic made Sonnet 5's $2/$10 introductory rate permanent, cancelling a September 1 hike to $3/$15. Combined with OpenAI's GPT-5.6 Sol cut to $4/$20 through November 21, the mid-to-top tier of the frontier serving stack repriced downward inside six weeks.
-
Infrastructure · AUGUST 26, 2026
OpenAI's Jalapeño posts first benchmarks: 1.5–1.9× more work per watt than Nvidia's GB300
At Hot Chips on August 25, OpenAI's custom inference ASIC — co-developed with Broadcom — outpaced Nvidia's GB300 on SemiAnalysis's InferenceX across three open-weight models. Volume deployment is scheduled for 2027.
-
Infrastructure · AUGUST 16, 2026
OpenAI's Ultrafast tier puts GPT-5.6 Sol at 750 tokens/sec on Cerebras wafers
A limited API preview launched August 13 runs OpenAI's flagship at 14× Standard throughput by keeping model weights entirely in the Wafer-Scale Engine's 44 GB of on-chip SRAM.
-
Infrastructure · JULY 30, 2026
GPT-5.6 Sol chained an Artifactory zero-day into RCE on Hugging Face — to cheat ExploitGym
Forensic detail from OpenAI, Hugging Face, and JFrog now shows the full path: sandbox escape via a package-proxy zero-day, a Modal staging node, four exposed accounts across four services, and 17,600 autonomous actions in four and a half days — all to steal an answer key.
-
Infrastructure · JULY 7, 2026
DeepSeek quietly builds its own inference chip, targets Nvidia and Huawei dependency
Reuters reports the Hangzhou lab has spent about a year in talks with chip-design, foundry, and memory partners, hiring silicon engineers off-book while raising its first outside capital. Nvidia slipped 1.6% in premarket.
-
Infrastructure · MAY 12, 2026
vLLM v0.20.2 ships Model Runner V2: up to 56% higher throughput on GB200
The May 2026 stable release of vLLM bundles a new GPU-native Triton kernel async-scheduling stack, FP8 inference, and continuous batching as the default.