21 Parts · 34 Projects · End-to-End System

From Tokensto Agents

Rebuilding the Modern LLM Stack Through 34 Engineering Projects

A systematic journey through tokenization, embeddings, attention, transformers, training, inference, long-context systems, MoE architectures, post-training, serving, agents, multimodal AI, interpretability, and complete LLM systems.

Explore The Roadmap View Projects GitHub Resume

Curriculum Parts

Engineering Projects

100M+

Parameters Trained

∞

First Principles

Interactive Systems

Live visualizations of core LLM components. These are not screenshots—they are interactive demonstrations of the concepts explored in each project.

Tokenization Pipeline

Input string as continuous characters

Transformers are powerful sequence models

Attention Heatmap

Causal self-attention pattern visualization

Transformer Architecture

Interactive decoder block diagram

Multi-Head Attention

Computes attention scores between all positions. Multiple heads attend to different aspects (syntax, semantics, position) simultaneously.

Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) V

The LLM Engineering Roadmap

Featured Engineering Case Studies

Deep dives into the most impactful projects—each representing a critical component of the modern LLM stack.

Part 1

Completed

Build a Tokenizer from Scratch

Implemented Byte-Pair Encoding (BPE) and WordPiece tokenization algorithms from first principles, including pre-tokenization, pair frequency counting, merge operations, and vocabulary construction. Built a complete tokenizer with encoding/decoding pipelines, special token handling, and unknown token management.

Byte-Pair EncodingWordPieceSubword TokenizationVocabulary Construction+3 more

3 tagsText Becomes Tensors

Read case study

Part 3

Completed

Multi-Head Attention

Extended single-head attention to multi-head attention with parallel Q/K/V projections, concatenation, and final linear projection. Analyzed how different heads specialize across syntactic, semantic, and positional tasks. Implemented head pruning to study redundancy.

Parallel Attention HeadsConcatenationHead SpecializationRedundancy+2 more

3 tagsAttention Makes Context Useful

Read case study

Part 4

Completed

Train a Mini Transformer

Trained a complete 6-layer, 8-head, 512-dim transformer from scratch on OpenWebText. Implemented distributed training with gradient accumulation, mixed precision, and learning rate scheduling. Achieved coherent text generation and analyzed scaling behavior.

Distributed TrainingGradient AccumulationMixed PrecisionLearning Rate Scheduling+2 more

3 tagsThe Transformer Block

KV Cache

Implemented KV caching for autoregressive transformer inference, reducing complexity from O(n²) to O(n) per token. Optimized cache layout for memory bandwidth, implemented cache eviction strategies, and profiled memory usage across sequence lengths.

KV CacheAutoregressive InferenceMemory BandwidthCache Layout+2 more

4 tagsKV Cache and Memory-Bound Inference

FlashAttention Comparison

Implemented FlashAttention and FlashAttention-2 from scratch, understanding the tiling, softmax rescaling, and memory access patterns that make it IO-aware. Benchmarked against standard attention across sequence lengths, batch sizes, and hardware configurations.

FlashAttentionTilingSoftmax RescalingIO-Awareness+2 more

4 tagsEfficient Attention and Hardware-Aware AI

Two-Expert Router

Implemented a Mixture of Experts layer with a learned routing mechanism, load balancing, and expert capacity management. Started with a two-expert system to understand routing dynamics, then scaled to multiple experts with top-k routing.

Mixture of ExpertsLearned RoutingLoad BalancingExpert Capacity+2 more

4 tagsMixture of Experts

RAG from Scratch

Built a complete Retrieval-Augmented Generation system from scratch: document chunking, embedding-based retrieval, reranking, and augmented generation. Implemented dense retrieval (DPR), sparse retrieval (BM25), and hybrid approaches with query rewriting.

Retrieval-Augmented GenerationDense RetrievalSparse RetrievalHybrid Search+3 more

4 tagsRetrieval, Tools, and Agents

Train, Tune, Quantize, Serve, Evaluate, and Document a Complete Language Model System

End-to-end construction of a complete LLM system from scratch: data curation, pretraining (100M parameters), supervised fine-tuning, DPO alignment, INT8 quantization, vLLM serving deployment, comprehensive evaluation, and full technical documentation. This project demonstrates mastery of the entire LLM stack.

End-to-End SystemData CurationPretrainingFine-Tuning+5 more

4 tagsFinal Capstone

Read case study

Research Interests

Areas I am actively exploring and contributing to through implementation, experimentation, and open-source work.

LLM Architecture

Designing more efficient, capable, and interpretable language model architectures. Exploring alternatives to the standard transformer stack including state-space models, linear attention, and mixture-of-experts systems.

Key Papers

Attention Is All You Need

Mamba: Linear-Time Sequence Modeling

Mixtral of Experts

Efficient Inference

Optimizing LLM inference through algorithmic improvements (FlashAttention, speculative decoding), system optimizations (continuous batching, KV cache management), and hardware-aware design.

Key Papers

FlashAttention-2

Speculative Decoding

vLLM: Easy, Fast, and Cheap LLM Serving

Model Compression

Reducing model size and inference cost through quantization, pruning, knowledge distillation, and architecture search. Focus on maintaining quality while achieving dramatic speedups.

Key Papers

GPTQ: Accurate Post-Training Quantization

AWQ: Activation-aware Weight Quantization

The Case for 4-bit Precision

Long Context Systems

Extending transformer context windows to millions of tokens through architectural innovations, memory systems, and position encoding advances. Applications in document analysis, code understanding, and multi-turn conversation.

Key Papers

Longformer

Ring Attention

YaRN: Efficient Context Window Extension

Agentic AI

Building autonomous systems that can plan, reason, use tools, and interact with environments. Focus on reliability, safety, and capability in multi-step reasoning tasks.

Key Papers

ReAct: Synergizing Reasoning and Acting

Reflexion: Self-Reflective Agents

Toolformer

Multimodal Models

Extending language models to understand and generate across vision, audio, and other modalities. Focus on efficient alignment and unified representation learning.

Key Papers

CLIP

LLaVA

Flamingo

Interpretability

Understanding the internal mechanisms of language models through mechanistic interpretability, feature visualization, and circuit tracing. Goal: making AI systems understandable and auditable.

Key Papers

A Mathematical Framework for Transformer Circuits

Sparse Autoencoders Find Highly Interpretable Features

Mechanistic Interpretability

Open Source AI

Contributing to open, reproducible, and accessible AI research. Building tools, datasets, and models that democratize access to state-of-the-art AI capabilities.

Key Papers

The Pile

OpenLM

OLMo