Scaling LLM-Based Ranking Systems for Latency-Critical Search & Recommendation Workloads
Sundara Raman Ramachandran explains how LinkedIn scales LLM-based ranking to 1.3B members, detailing model distillation to 400M parameters, context compression, and deep Python/GPU runtime tuning.
InfoQ