System Support for Machine Learning
Nowadays, deep-learning (DL) computing paradigms are pervasive, increasingly diverse, and continuously evolving. Despite the significant efforts in developing one-size-fits-all DL frameworks with hand-optimized libraries, the full potential of existing high-performance GPUs could hardly be maximized, mainly due to three major challenges. First, DL workloads are increasingly diverse and featured the crucial and popular sparse-and-irregular computing patterns (e.g., Graph Neural Networks and Deep-learning Recommendation Model). These types of workloads are naturally at odds with the GPUs that excel at dense and regular computation paradigms (e.g., General Matrix Multiplication). Second, DL tasks are becoming heterogeneous and come with interacted execution dependency and interactions among each other. Third, mechanisms and strategies (e.g., NN verification and Zero-knowledge Proof) to fortify the robustness and secureness of DL models are naturally irregular in their computation (e.g., inequality-based bound computation), thus hardly benefiting from the massively parallel GPUs.
However, existing GPU-based systems follow the homogenous design (i.e. replicating computation and memory resources) in their intra-device hardware unit and inter-device scale up/out design. This creates challenges in accommodating various DL applications with different computation and communication demands. Our group endeavors to bridge those gaps through the holistic full-stack solution with an array of algorithm- and system-level innovations to unleash the full potential of massive parallel GPUs.
With the increasing popularity of deep learning, Convolutional Neural Networks (CNNs) have been widely applied in various domains, such as image classification and object detection, and achieve stunning success in terms of their high accuracy over the traditional statistical methods. To exploit the potential of CNN models, a huge amount of research and industry efforts have been devoted to optimizing CNNs.
Deep Neural Networks
Graph Neural Networks (GNNs) emerge to stand on the frontline for handling many graph-based deep learning tasks (e.g., node classification and link prediction ). Compared with standard methods for graph analytics, such as random walks and graph Laplacians, GNNs highlight themselves with the interleaved two-phase execution of both graph operations (scatter-and-gather ) at the Aggregation phase, and Neural Network (NN) operations (matrix multiplication) at the Update phase, to achieve significantly higher accuracy and better generality.
Graph Neural Networks
Over recent years, deep learning recommendation models (DLRMs) attract a lot of attention from the research and industry. DLRMs combine sparse input embedding learning with neural networks and have demonstrated significant improvement compared with the collaborative filtering-based recommendation model. DLRMs have accounted for a significant proportion of deep learning instances in the industry, including product recommendations from Amazon, personalized advertisements from Google, and e-commerce recommendations from Alibaba.
Recommendation Models
Recently, deep reinforcement learning (DRL) has raised significant interest in research and industry fields. DRL combines the conventional RL algorithms with deep neural networks and has demonstrated superhuman performance in decision making. DRL has been applied in many real-world applications in various domains, like robotics, industry control, and autonomous driving.
Deep Reinforcement Learning
Secure machine learning plays an increasingly important role in academic and industry (e.g., autonomous driving). One research direction is the adversarial learning that studies the vulnerability of machine learning models towards small perturbations (a.k.a, adversarial attacks). Robustness machine learning and neural network verification has been proposed to defend against such adversarial attacks. Another research direction is to protect the integrity and privacy of machine learning models and data using advanced cryptography techniques such as zero-knowledge proof.
Secure Machine Learning