Ant Group, in collaboration with the vLLM community, has released Humming—a GEMM kernel library designed for the inference of quantized Large Language Models (LLMs) using Just-In-Time (JIT) compilation.
This project is released under the Apache 2.0 license via Ant Group's inclusionAI organization and is mirrored in the vLLM project. Humming addresses a common challenge in quantized serving: the need to create a specialized kernel for every combination of weight bit width, activation format, scaling scheme, dense or Mixture-of-Experts (MoE) layout structure, and GPU generation.
The library translates supported export formats into a unified quantization description, then performs JIT compilation of kernels from reusable CUDA components for a specified target shape, device, and computational options.
According to the project description, one architecture supports integer weights from INT1 to INT8 and floating-point weights from FP4 to FP8, combined with activations of FP16, BF16, FP8, FP4, INT8, or INT4, provided the formats are compatible. It supports both dense GEMM and MoE GEMM, and is targeted at NVIDIA GPUs starting from the Turing architecture.
The library dependencies are limited to PyTorch and NVCC, and the pip package size is approximately 100 KB. A significant aspect is the extension of hardware support beyond NVIDIA: Humming fully supports the T-Head Zhenwu 810E AI accelerator, which is CUDA-compatible, allowing the use of the same kernels and configuration paths on this device. T-Head is a subsidiary of Alibaba, and Zhenwu 810E was previously used in Chinese AI deployments.
Performance results presented in the joint announcement are based on proprietary microbenchmarks and team serving tests. At small batch sizes, dense W4A16 showed an improvement of approximately 3–4 times compared to cuBLAS across several devices. On B300, the W4A8 ratio to baseline DeepGEMM is estimated at about 1.6 times in some cases with small and medium batch sizes.
During end-to-end testing of DeepSeek-V4.1-Flash on eight H20 GPUs, transitioning from MoE Marlin W4A16 to Humming W4A8 increased output throughput by approximately 14%–43% and reduced the average time to first token by 25%–32%, with the largest gain observed at higher parallelism. These results have not yet undergone independent verification.
Starting with vLLM version 0.29.0, vLLM installations with CUDA include Humming as a dependency. Operators can activate it using the flags --linear-backend humming and --moe-backend humming. It should be noted that Humming is a joint project of Ant Group and the vLLM community, not exclusively an Ant Group product.