Available for collaboration

Raj Thakur - Senior ML Engineer

Senior ML Engineer at Amazon AWS

I build and tune LLM serving on AWS Trainium: disaggregated prefill/decode in vLLM, native torch.compile for Neuron, and making models like GPT-OSS run faster and more accurately on the chip.

About Raj Thakur - Senior ML Engineer

I'm a Senior Machine Learning Engineer at AWS Neuron. Eleven years in software, the last four in AI/ML, working on LLM inference for AWS Trainium. I designed and built the disaggregated serving stack for vLLM on Trainium and started Neuron's move from PyTorch/XLA to native torch.compile.


These days most of my time goes into making models run faster and more accurately on Trainium: decode performance for GPT-OSS, multimodal models like Qwen3-VL, and multi-LoRA serving. I presented our out-of-tree vLLM plugin for Trainium at PyTorch Conference North America 2026.

ML Engineering Skills & Technical Expertise

🧠

LLM Inference

vLLM, disaggregated prefill/decode, NIXL, EFA, tensor/data/expert parallelism, MoE, multi-LoRA, Trainium2/3

⚡

Compilers & Runtimes

torch.compile, Dynamo, AOTAutograd, FX, StableHLO, PJRT, Neuron compiler and runtime, NKI

🔬

Debugging & Performance

py-spy, gdb, system traces, roofline analysis, A/B benchmarking, accuracy debugging

🚀

Languages & Tools

Python, C++, Go, Java, Docker, Kubernetes (EKS), Bazel

Professional Experience - Amazon AWS & Tech Companies

Amazon Web Services (AWS)

6 yrs 3 mos • Full-time

Senior Machine Learning Engineer

Dec 2022 - Present • Calculating... Cupertino, California, United States

LLM inference on AWS Trainium: disaggregated serving in vLLM, model performance and bring-up, and native PyTorch for Neuron

  • Designed and built disaggregated inference for vLLM on Trainium. Prefill and decode run on separate nodes, the KV cache moves device to device with NIXL over EFA, and a batched runtime transfer API I added doubled KV throughput. It runs multi-node xPyD on EKS, including mixed GPU and Trainium.
  • Led the work that removed the async gap between decode steps in disaggregated serving. The runtime was rebuilding DMA rings every step because inputs kept landing at new addresses.
  • Built a multi-agent C++ code-review workflow on Claude Code that catches bugs in the team's runtime changes, including a concurrency race in async KV-offload code.
  • Brought GPT-OSS-120B decode on native PyTorch to parity with XLA by fixing two compiler inputs, one of them a small all-gather mis-tagged as FSDP.
  • Slow multi-LoRA was keeping NxDI customers from moving to vLLM Neuron. With the kernel's author, I designed the static-slot decode path for the NKI LoRA kernel, and vLLM Neuron then beat NxDI on multi-LoRA decode latency, which let us deprecate NxDI.
  • Cleared a GPT-OSS-20B accuracy regression by isolating it to one compiler lowering, and validated the workaround.
  • Brought up GPT-OSS on native PyTorch for Neuron, fixing integration bugs until BF16 passed GSM8K on Trainium2 and Trainium3. Tracked an out-of-memory failure to missing KV aliases and fixed the index bug that dropped them, bringing accuracy in line with XLA.
  • Started the native PyTorch project for Neuron inference (torch.compile lowered through torch-mlir instead of PyTorch/XLA). I proposed the native inference stack ahead of PyTorch Conference to put TorchNeuron and its ecosystem on stage there running natively, then wrote the proposal to retire PyTorch/XLA-based torch-neuronx in Neuron 2.32. All 8 required approvers signed off.
  • Built device-free torch.compile for Neuron. I wrote and merged the compiler API (FX to StableHLO to NEFF) and a compile-only mode, so inference models compile on plain CPU hosts with no Trainium attached.
  • Got the first vLLM TP8 server running on native PyTorch for Neuron, then dogfooded 6+ state-of-the-art models on it, including GPT-OSS and Qwen3-VL.
  • Earlier: owned PyTorch upgrades across 10+ Neuron packages, moved the PJRT plugin to JAX 0.7.0, and built an HLO diff tool for compiler regressions.
vLLM Disaggregated Inference torch.compile KV Cache NKI AWS Trainium

Software Development Engineer II

Jun 2020 - Nov 2022 • 2 yrs 6 mos Bengaluru, Karnataka, India

AWS Billing platform engineering and OpenSearch distributed systems

  • Built a Step Functions service that generates and delivers billing documents for thousands of enterprise payers, which drove $300M in free-cash-flow improvements. Also built payer classification and mapping, which contributed to another $100M.
  • Built a self-healing framework for OpenSearch nodes that cut customer root-cause tickets by 30%.
AWS Step Functions AWS Lambda Redshift OpenSearch Distributed Systems Java

Grab

2 yrs • Full-time

Senior Software Engineer

Jul 2018 - Jun 2020 • 2 yrs Bengaluru Area, India

Lead Engineer in Settlement Platform for Grab Financial Group

  • Rebuilt merchant settlement as an event-driven system on AWS Lambda (60% performance improvement). Designed the post-payment service in Go.
Golang Fintech Payment Systems Event-Driven Architecture

Yatra Online Pvt Ltd

9 mos • Full-time

Software Engineer, Full Stack

Nov 2017 - Jul 2018 • 9 mos Hyderabad, Telangana, India

Full-stack development for corporate travel booking

  • Designed a customized Header Service for corporate merchants using Node.js, Java, and Spring, with HAProxy and Nginx routing
Node.js Java Spring Nginx

Tata Consultancy Services

2 yrs 3 mos • Full-time

Software Engineer

Aug 2015 - Oct 2017 • 2 yrs 3 mos Hyderabad, Telangana, India

Embedded systems test frameworks and enterprise software solutions

  • Developed the TCS-Ericsson JCAT framework for testing Ericsson AXE IO embedded systems
  • Built an AI-powered web console for test recommendations, recognized with a Star Performer award
  • Customized the OpenStack console UI for the ATLAS project
Java OpenStack Embedded Systems Full-Stack Development

Technical Writing

I regularly write about machine learning infrastructure, system design, and emerging AI technologies. My articles focus on practical insights from building production ML systems.

Editor of Software System Design publication on Medium, featuring in-depth articles on scalable architecture and distributed systems.

📱
UX Design

Split Bill App Design: A Guide to Creating a Seamless Splitwise-like Experience

Comprehensive guide to designing intuitive bill-splitting applications with focus on user experience.

Read Article →
🔧
Software Engineering

DI Frameworks: Spring, Guice and Dagger

Comprehensive comparison of dependency injection frameworks for large-scale applications.

Read Article →
🏗️
System Design

System Design Interview Primer

Essential tips and strategies for tackling high-level system design questions in technical interviews.

Read Article →

Let's Connect

Interested in discussing ML innovations, system architecture, or collaboration opportunities?