August 17, 11:58

Peking University and StepFun introduce TensorCast—a tensor management layer that reduces LLM time-to-first-token by 93.2%

For many years, optimizing inference systems for large models has focused on improving the computational efficiency of high-dimensional tensors, such as weights, activations, and KV Cache, on GPUs. However, as model scale grows, context lengths increase, and multi-step agentic applications emerge, infrastructure faces a new problem: how to flexibly manage tensor states that move between numerous system components—from terabyte-sized distributed weights during elastic scaling to gigabyte-sized KV Caches migrating between instances.

Various systems address similar challenges, but they lack a unified abstraction. To bridge this gap, Peking University, StepFun, and the Beijing University of Posts and Telecommunications jointly developed TensorCast. This is a unified, programmable abstraction for managing the lifecycle of tensors, functioning between compute engines, networks, and storage. TensorCast decouples tensor states, such as model weights and KV Cache, from specific systems, allowing workloads to utilize common management capabilities.

TensorCast achieves performance comparable to specialized systems while introducing novel cross-component interaction strategies, enabling a reduction in time-to-first-token (TTFT) by 93.2% in highly concurrent multi-agent scenarios.

Modern large model infrastructure is essentially reinventing the wheel. Separate systems exist for each task: one for model loading and weight distribution, another for KV Cache storage and migration, and a third for state recovery via checkpoints. While these solutions provide significant performance gains, they are tightly coupled to specific frameworks, networking technologies, and storage backends. Essentially, they all perform similar operations: defining tensor identity, placing tensors, moving them between nodes, transforming layouts, and materializing logical states into computable data. These capabilities reside in isolated vertical stacks, increasing development costs and hindering inter-component optimization.

Two key ideas underpin TensorCast: tensors become first-class system objects with independent identity and lifecycle, ensuring uniform management of weights, KV Cache, and intermediate states without requiring developers to know where the tensor is located; and the tensor lifecycle becomes programmable, as common operations are abstracted into composite primitives. The KV Cache migration scenario illustrates this: a session might require switching between instances due to load imbalance, which traditionally necessitates changes to the request scheduler, KV backend, and inference engine, whereas TensorCast handles this with a simple lifecycle program.

The TensorCast architecture separates control and data: a Global Store maintains cluster metadata without participating in data transfer, while Worker Nodes execute lifecycle operations using RDMA and zero-copy transfer.

Experiments confirm both performance and programmability. Integrated into vLLM and SGLang, TensorCast accelerates elastic scaling up to 228.6 times when launching an instance compared to common distributed file systems, using the Qwen3-235B-A22B model. For dynamic tensor state management, it matches the specialized Mooncake KV Cache in SGLang HiCache. By utilizing the TensorCast API, the request scheduler for multi-agent scenarios dynamically adjusts request distribution based on instance load, reducing median TTFT by 93.2%. The paper argues that as agents, multi-step reasoning, and continuous learning push systems toward greater dynamism, tensor management across nodes, components, and stages will become a central challenge for next-generation infrastructure.

pandaily.comAugust 17, 09:21

XianDao Cup Competition Attracts 2000 Students to Work with Domestic AI Cluster Sugon

The final of the seventh National Computer Systems Contest for Students, which featured the innovative design track for intelligent computing called XianDao Cup, has concluded. The competition gathered teams from 160 universities, encompassing over 500 teams and more than 1900 students, generating significant resonance in the field of artificial intelligence.

This new generation of students actively utilizes advanced domestic computing power to solve pressing issues in the real sector of the economy. XianDao Cup is the only national competition in computer systems focused on intelligent computing, boasting a history of nearly two decades.

According to Gao Xiaopeng, director of the organizing committee, the contest was initiated during a period when the transformation of education from knowledge transfer to competency development was proposed. Its goal is to form and discover talent reserves for key national software and hardware programs, creating a pool of personnel with practical skills in domestic intelligent computing.

The organizers include the National Association of Higher Education Computing, an expert group for training systemic capabilities, and initiating universities. The contest provides free computational and technical training using a standardized 'black box' evaluation system. Sugon, as a joint organizer, provided access to China's first fully domestic AI supercluster with 100,000 cards via the National Supercomputer Internet platform, presenting advanced computing resources to university students for the first time.

The contest tasks are directly oriented towards industry scenarios rather than demonstration projects. They include tasks related to large-scale modeling implementation, AI application in science, system optimization testing, engineering realization, and innovative practice.

Two tasks took center stage. The first involved optimizing the inference service for large Qwen models on domestic accelerator cards, focusing on reducing latency when handling long texts, managing GPU memory, and increasing throughput, which required deep optimization within the defined model and framework boundaries. Such students are highly valued by many domestic AI giants.

The second task concerned optimizing the deployment of a large meteorological model within the country's scientific computing ecosystem, including constraints related to low-orbit satellites. By utilizing the OneScience intelligent computing ecosystem with meteorological models for lightweight deployment, the key advantage is accelerating forecasting by over 1000 times, allowing for detailed meteorological data acquisition. Application areas include extreme weather event warnings, hydro-, wind-, and solar energy dispatching, agricultural support, port vessel operations, and atmospheric management; optimized models are available via API and for private deployment.

During the event, which lasted over three months, Sugon provided full support for domestic computing throughout the entire lifecycle, using a collaborative task definition mechanism of 1+1+1: each task was developed by one expert, one university instructor, and one technical team. The systems used by students during the contest are identical to those employed by enterprises in production processes, which contributes to data accumulation and model optimization for domestic computing, creating a snowball effect in the ecosystem. Although the habits of senior developers related to CUDA are difficult to change because domestic toolchains differ significantly, the current generation starts from scratch, avoiding historical baggage. Gao Xiaopeng noted that the value of the contest also lies in the participation of leading domestic enterprises that develop talent through competition and recruit them into their companies.

Peking University and StepFun introduce TensorCast—a tensor management layer that reduces LLM time-to-first-token by 93.2%

Similar stories

Popular

XianDao Cup Competition Attracts 2000 Students to Work with Domestic AI Cluster Sugon