Sparse Attention Accelerator Achieves 31.1× Speedup with Predictor-Free Stage Fusion
Summarize this article with:
Attention-based artificial intelligence models continue to drive advances across many fields, but their computational demands remain a significant challenge. Huizheng Wang, Hongbin Wang, Zichuan Wang, et al. from Tsinghua University address this issue with a novel approach to sparse attention, presenting a predictor-free accelerator called PADE.
The team overcomes limitations in existing sparse attention methods, which often rely on computationally expensive predictors, by introducing a bit-serial enable stage-fusion mechanism that eliminates the need for separate prediction hardware. This innovative design, incorporating techniques for accurate pruning, efficient workload balancing, and reduced data movement, achieves substantial performance gains, delivering up to 7.43x speedup and 31.1x higher energy efficiency compared to state-of-the-art GPUs and significantly outperforming existing specialised accelerators in terms of energy savings. Sparse DNN Training and Acceleration Approaches Recent research into deep learning focuses heavily on improving efficiency and performance, particularly for demanding applications like large language models. A significant body of work explores techniques for sparse computation, quantization, and specialized hardware acceleration, aiming to reduce computational demands and energy consumption without sacrificing accuracy. Several studies investigate transformer acceleration, recognizing the growing importance of these models in natural language processing, while researchers also explore innovative computing paradigms like wafer-scale integration to build extremely powerful accelerators. A common thread throughout these investigations is the importance of co-designing hardware and software to achieve optimal results. A key trend is the exploitation of sparsity in neural networks, allowing for reduced model size and faster computation. Optimizing transformers, particularly for large language models, receives considerable attention, with researchers exploring quantization, sparsity, and architectural improvements. Hardware-software co-design emerges as a critical approach, ensuring that algorithms and hardware work seamlessly together. These combined efforts represent a significant push towards more sustainable and scalable artificial intelligence. Predictor-Free Dynamic Sparse Attention Acceleration This research introduces PADE, a novel algorithm and hardware co-design that accelerates dynamic sparse attention without relying on traditional sparsity predictors. PADE overcomes limitations of existing methods by employing a bit-serial enable stage-fusion mechanism, streamlining the process and reducing overhead. The core innovation lies in accurately identifying and pruning irrelevant tokens without a dedicated predictor, achieved through a Bit-wise Uncertainty Interval-enabled Guard Filtering strategy to assess token relevance during each bit round, accurately identifying trivial tokens without compromising accuracy. To maximize hardware utilization, the team developed Bidirectional Sparsity-based Out-of-order Execution, allowing the system to process sparse data more efficiently by dynamically adjusting the execution order based on sparsity patterns. Addressing data handling challenges, researchers introduced Interleaving-based Sparsity-tiled Attention, which reduces both input/output operations and computational complexity. Extensive experiments on 22 benchmarks demonstrate that PADE achieves a 7.43x speed up and 31.1x higher energy efficiency compared to the Nvidia H100 GPU, and significant energy savings compared to other state-of-the-art accelerators. These results highlight PADE’s effectiveness in overcoming the limitations of existing dynamic sparsity techniques and paving the way for more efficient and scalable attention-based models. PADE Accelerates Sparse Attention with High Efficiency Researchers present PADE, a novel algorithm and hardware co-design that significantly accelerates dynamic sparse attention, achieving substantial improvements in both speed and energy efficiency. Experiments demonstrate PADE achieves a 7.43x speed up and 31.1x higher energy efficiency compared to a high-performance H100 GPU, and delivers significant energy savings compared to state-of-the-art accelerators. A core innovation is the Bit-wise Uncertainty Interval-enabled Guard Filtering (BUI-GF) strategy, which accurately identifies trivial tokens during each bit round, enabling efficient pruning. By leveraging the uncertainty interval of dot products, the team confidently discards irrelevant tokens, reducing computational load. To further enhance performance, researchers developed Bidirectional sparsity-based out-of-order execution (BS-OOE), allowing processing elements to work on different bit-planes concurrently, avoiding stalls caused by memory access delays.
The team implemented a lightweight ANDer tree and a scoreboard-based partial sum buffering system to support this out-of-order execution. Addressing the challenge of tiling in sparse attention, the team introduces Interleaving-based sparsity-tiled attention (ISTA), which enables efficient pruning within tiled regions, reducing both I/O and computational complexity.
This research demonstrates that this approach requires less on-chip SRAM compared to existing solutions, reducing area inefficiency. PADE Accelerates Sparse Attention with Efficiency Gains This research presents PADE, a novel software-hardware co-design that significantly accelerates dynamic sparse attention without relying on traditional sparsity predictors.
The team overcame challenges associated with bit-level prediction and hardware under-utilisation by introducing bit-wise uncertainty interval-enabled guard filtering, bidirectional sparsity-based out-of-order execution, and interleaving-based sparsity-tiled attention. Extensive experimentation across 22 benchmarks demonstrates that PADE achieves a 7.43x speed up and a 31.1x improvement in energy efficiency compared to the H100 GPU. Furthermore, PADE outperforms state-of-the-art accelerators, delivering significant energy savings. The authors acknowledge the inherent difficulties in applying existing techniques to the demands of real-time sparse attention, and PADE addresses these limitations through a lightweight runtime pruning strategy and optimised hardware execution. This work represents a substantial advance in efficient attention mechanisms, paving the way for more powerful and energy-conscious artificial intelligence systems. 👉 More information 🗞 PADE: A Predictor-Free Sparse Attention Accelerator via Unified Execution and Stage Fusion 🧠 ArXiv: https://arxiv.org/abs/2512.14322 Tags: