📄 Download PDF

Raj Kumar Thakur

LLM inference and ML systems on AWS Trainium
Eleven years in software, the last four in AI/ML. I designed and built the disaggregated serving stack for vLLM on Trainium, started Neuron's move from PyTorch/XLA to native torch.compile, and now spend most of my time making models run faster and more accurately on Trainium.
Experience
Senior Machine Learning Engineer Dec 2022 – Present
Amazon Web Services, AWS Neuron (Annapurna ML), Cupertino, CA
Disaggregated serving
Performance and model bring-up
Native PyTorch for Neuron
Software Development Engineer II Jun 2020 – Nov 2022
Amazon Web Services (Billing, OpenSearch), Bangalore, India
Senior Software Engineer, Backend Jul 2018 – Jun 2020
Grab, Bangalore, India
Software Engineer, Yatra Online; Software Developer, Tata Consultancy Services Aug 2015 – Jul 2018
Technical Skills
LLM inference vLLM, disaggregated prefill/decode, NIXL, EFA, tensor/data/expert parallelism, MoE, multi-LoRA, Trainium2/3 Compilers & runtimes torch.compile, Dynamo, AOTAutograd, FX, StableHLO, PJRT, Neuron compiler and runtime, NKI Debugging & perf py-spy, gdb, system traces, roofline analysis, A/B benchmarking, accuracy debugging Languages & tools Python, C++, Go, Java, Docker, Kubernetes (EKS), Bazel
Education
B.Tech, Computer Science and Engineering, Jalpaiguri Government Engineering College (CGPA 8.5/10) 2015
Talks & Community