How Huawei's Pangu Model Cuts Advertising AI Latency with Ascend 950

Why Advertising AI Needs to Talk Faster

In advertising, every millisecond counts. When a user loads a page, the ad tech stack has to decide which ad to show, run auction logic, and serve the creative—often in under 100 milliseconds. That means the AI models behind ad targeting, bidding, and personalization can't afford slow communication. Yet as models grow, the chatter between GPUs and accelerators becomes a bigger bottleneck than the math itself.

At AICon Shenzhen, Huawei's Li Baichao, a systems engineering expert from the 2012 Lab, walked through how the Pangu model team optimized communication on Ascend hardware. While the talk focused on large language models, the lessons map directly to advertising's real-time constraints. If a recommendation model takes 300 milliseconds to respond, the ad slot is already gone.

The Hidden Cost of Communication in MoE Models

Mixture-of-Experts (MoE) models are popular in ad ranking because they can scale parameters without exploding compute. But they come with a nasty side effect: AllToAll communication. In an MoE layer, tokens need to be shuffled across experts, and that shuffling can eat up over 30% of end-to-end time. For an ad server handling thousands of requests per second, that's not just a statistic—it's lost revenue.

Huawei's team found that the standard DeepEP approach, which works fine on some hardware, actually hurts on Ascend 910A3. So they went deeper. By adapting to the Ascend 950's network topology and using its dedicated CCU (Communication Compute Unit), they squeezed out a 10% performance gain on Pangu's EP domain AllToAll. That's not a headline number, but in a bidding war, 10% faster can be the difference between winning and losing.

When Long Contexts Stall the Show

Advertising increasingly uses long context—user history, session logs, even full-page content. But handling a million-token context means moving a huge KV cache from host memory to device. That Host-to-Device (H2D) transfer can become the new bottleneck for Time-To-First-Token (TTFT). In ad tech, TTFT is basically the time before you can even start scoring an ad. If that creeps up, users bounce.

Huawei's answer is Omni Cache, a software layer that makes H2D and D2H transfers more efficient. Combined with a dedicated H2D path on each NPU, they achieved a 10% improvement in TTFT. That might sound modest, but for a system that's already optimized to the bone, 10% is a big deal.

Hardware and Software, Pulling in the Same Direction

What stands out in Huawei's approach is the dual focus. On the hardware side, the Ascend 950 gives each NPU its own H2D channel, so transfers don't fight for bandwidth. On the software side, Omni Cache handles the KV cache offloading intelligently. The two work together to hide communication behind computation.

The team's philosophy is simple: minimize the time communication is exposed in the critical path. They're not just optimizing the communication itself—they're orchestrating it to overlap with compute. That's the kind of thinking that matters in ad serving, where you can't afford to have the model idle while data shuffles around.

What Ad Tech Can Learn from Huawei's Playbook

So what does this mean for advertising engineers? First, don't assume your communication stack is fine just because your model is fast. Profile your AllToAll and H2D times. You might be surprised.

Second, hardware affinity is a real lever. Huawei's team sacrificed portability for performance—their optimizations work on Ascend 950 but can fail on other platforms like 910A2/A3 or NVIDIA H20. That's a trade-off you have to make consciously. If you're running a dedicated ad-serving fleet, it might be worth tailoring your stack to the hardware you actually own.

Third, look beyond the model. The KV cache transfer is a perfect example of a non-obvious bottleneck. In ad tech, that could be the user embedding cache or the feature store. Anything that sits between memory and compute is a candidate for optimization.

Practical Steps to Cut Latency in Your Ad AI

Here are some actionable takeaways from the session:

• Profile your communication overhead—use tools like NCCL or your vendor's profiler to find where time goes.
• Consider topology-aware communication—if your cluster has a specific network layout, exploit it.
• Look for dedicated communication accelerators—some hardware has them, and using them can give you an easy win.
• Overlap communication and computation—use streams or async operations to hide transfer times.
• Be willing to custom-write communication operators—standard libraries might not fit your model's semantics.

These aren't just for large language models. They apply to any distributed AI system, and advertising is one of the most latency-sensitive domains out there.

The Future: Fusing and Pipelining for Even Better Performance

Huawei's team isn't stopping at 10% gains. They're looking at fused operators and multi-stream pipeline scheduling that are tailored to Ascend's architecture. The goal is to make communication completely invisible—no exposed time in the training or inference path.

For ad tech, that's the dream: a model that can respond in real time, no matter how complex the context or how many experts are in the MoE. The hardware is evolving, and so are the techniques to squeeze every last drop of performance out of it.

If you're building ad serving systems, keep an eye on these communication optimizations. They might not be flashy, but they're the kind of incremental wins that keep you ahead of the competition.