Tokenization

Breaking down text into tokens and understanding tokenization in language models.

What can we deduce from token embedding table after training

Understanding token representations in large language models.

Is multi-head attention really doing the job?

Alternative mechanisms to traditional attention.

Can Transformer Language Models Without Explicit Positional Encoding Learn Positional Information?

Understanding whether Transformers can recover token order when positional signals are removed.

How i trained my own GPT-2(124M) on Fineweb-edu dataset

A from-scratch walkthrough for training GPT-2 (124M) on FineWeb-Edu with practical insights.