RoPE: Rotary Position Embeddings Explained
Adding positional embeddings pollutes attention scores with cross terms. How RoPE encodes position as rotation instead, and how the rotation matrix is built.
–––
Topics / LLM
LLM
7 posts.
How large language models actually process text, from token IDs and embeddings to positional encoding and self-attention, built one idea at a time.
Self Attention
An intuition-first walk through self attention — queries, keys and values by analogy, then the implementation, multi-head attention, and transformers.
–––
Positional Embeddings
Token embeddings alone throw away word order. How position gets encoded into a sentence, and why the sinusoidal scheme is built the way it is.
–––
Token Embeddings
How tokens turn into meaningful vectors: the core idea behind embeddings, how the training data is prepared, and a working implementation.
–––
Neural Networks
Building a neural network from the ground up — what the layers actually do, how training adjusts them, and a worked example end to end.
–––
Tokenization
Why models break text into subwords, how Byte Pair Encoding builds a vocabulary from scratch, and what naive word splitting gets wrong.
–––
LLM in a nutshell
A plain-language overview of how large language models actually work — the mental model worth having before going anywhere near the internals.
–––