Tech news in 3 minutes
Google s TPUv8s for Training and Inference at Hot Chips 2026
OpenAI took the Hot Chips 2026 stage to detail Jalapeño, an in-house inference ASIC and system built with Broadcom, positioning it as the best compute platform for OpenAI’s own inference workloads with superior throughput per kilowatt and latency against NVIDIA GB200 and GB300. The chip, designed around HBM4 and a spatial programming model, moved from initial RTL to tapeout in roughly nine months, with a late 2025 tapeout and Codex running in early 2026. OpenAI frames Jalapeño as an inference platform rather than a raw accelerator, targeting state-of-the-art performance per watt at low latency for multi-chip workloads. Key metrics are time to last token (user experience) and tokens per joule (inference efficiency). Using the public InferenceX benchmark normalized to package TDP, Jalapeño at 700 watts is compared against GB200 at 1.2 kW, and GB300 and MI355X at 1.4 kW. On GPT-OSS 120B, Jalapeño shows about 1.9x higher peak mixed tokens per second per kilowatt and roughly 1.7x lower end-to-end latency. On DeepSeek R1 670B MXFP4, gains widen to 1.7x higher peak mixed tokens per kW/s and 3.6x lower latency. On the 1-trillion-parameter Kimi K2.5, Jalapeño delivers about 1.5x higher peak mixed tokens per kW/s and 3.4x lower latency. Even in single-token mode against GB300 multi-token prediction, Jalapeño leads with 1.5x higher peak mixed-token rate per kW and 2.2x lower latency. Architecturally, Jalapeño uses a single balanced chip where unused blocks are gated, avoiding idle accelerator power draw. Raw HBM4 bandwidth at 128-chip aggregate exceeds one petabyte per second, implying up to 10,000 tokens per second with speculative decoding, though real system rates are lower due to long-latency paths. OpenAI claims sub-millisecond token-to-token latency on frontier models and notes multi-token prediction could add 3x–5x latency improvement at iso-efficiency.