Jinyi Zhang

Efficient Inter-Operator Scheduling for Concurrent Recommendation Model Inference on GPU

Deep learning-based recommendation systems are increasingly important in the industry. To meet strict SLA requirements, serving frameworks must efficiently handle concurrent queries. However, current serving systems fail to serve concurrent queries due to the following problems: (1) inefficient oper