VERVE: Verifiable Value Retrieval Leveraging Value Reuse in Accelerator Memory

William Taylor, Sunho Lee, Henry Wang, Noa Zilberman, and Amro Awad, "VERVE: Verifiable Value Retrieval Leveraging Value Reuse in Accelerator Memory", the 44th IEEE International Conference on Computer Design ( ICCD ), November 2026

Paper Slide

Modern data-intensive applications place unprecedented demand on memory bandwidth. However, scaling High Bandwidth Memory is approaching fundamental physical limits. This results in a growing disparity between compute and memory bandwidth, commonly referred to as the memory wall. In bandwidth-bound workloads, such as Large Language Models, compute resources are left under-utilized and performance is bottlenecked. In this work, we observe that data-intensive workloads exhibit significant data similarity, with values stored in HBM repeating across addresses and over time. We demonstrate that on average 22.4% of HBM accesses are exact duplicates, representing redundant memory traffic that wastes bandwidth and energy without contributing new data to computation. To exploit this, we propose VERVE, a novel mechanism that stores recent HBM addresses and content hashes in a dual cache architecture to verifiably determine the data returned by future accesses. Memory requests are serviced without physically accessing HBM, reducing traffic and improving effective bandwidth. VERVE eliminates on average, 21.2% of HBM read requests, improving Instructions Per Cycle by an average of 5.1% for memory-bound workloads.

One of the key security supports for neural processing units (NPUs) is the hardware-based memory protection to provide confidentiality and integrity of NPU data. However, adopting the memory encryption and integrity protection techniques developed for CPUs do not fully utilize the NPU characteristics, incurring a significant performance degradation. To address the performance challenges, this paper proposes new improvements of memory protection for NPUs based on the unique property of NPU computation. The design first proposes a context-based memory protection which imposes the hardware memory protection only for the critical memory region of NPUs. Second, it allows adjusting the counter granularity for NPU memory to reduce the overheads of common counter-mode encryption. In addition, it exploits the read-only property of machine learning parameters, and adds a trusted communication channel between the CPU and NPU. Our evaluation with a simulated NPU shows that the performance overhead of memory protection for NPUs can be significantly reduced from the state-of-the-art CPU-oriented design, improving the performance by 13.5%.