EECS Seminar: ML Workloads in AR/VR and Their Implications to the ML System Design

Abstract: Augmented and virtual reality (AR/VR) combines many machine learning (ML) models to implement complex applications. Unlike traditional ML workloads, those in AR/VR involve: (1) Multiple concurrent ML pipelines (cascaded ML models with control/data dependencies); (2) Highly heterogeneous modality and corresponding model structures; and (3) Heavy dynamic behavior based on the user context and inputs. In addition, AR/VR requires a (4) Real-time execution of those ML workloads; on (5) Energy-constrained wearable form-factor devices. All together, it creates significant challenges to the ML system design targeting for AR/VR.

In this talk, I will first demystify the ML workloads in AR/VR via a recent open benchmark, XRBench, which was developed with industry collaborators at Meta to reflect real use cases. Using the workloads, I will list the challenges and implications of the AR/VR ML workloads to ML system designs. Based on that, I will present hardware and software system design examples tailored for AR/VR ML workloads. Finally, I will discuss research opportunities in the AR/VR ML system design domain. 

Bio: Hyoukjun Kwon is an assistant professor in EECS at UC Irvine. His primary research area is computer architecture and his research focuses on AI accelerator HW/SW co-design for emerging AI workloads such as augmented and virtual reality (AR/VR) and long-context large language models. He was a research scientist at Meta Reality Labs before joining UC Irvine, and received his Ph.D. in computer science at the Georgia Institute of Technology in 2020. His research works have been recognized by IEEE Top Picks in Computer Architecture Conferences (MAERI and MAESTRO) and an honorable mention at the IEEE ACM SIGARCH/IEEE CS TCCA Outstanding Dissertation Award.​