RoPE: Rotary Position Embeddings Explained
Adding positional embeddings pollutes attention scores with cross terms. How RoPE encodes position as rotation instead, and how the rotation matrix is built.
–––
Learnings so far.
Understand how AI actually works.
One building block at a time.
Intuition-first deep dives into how AI actually works — neural networks, embeddings, attention, and LLMs explained with diagrams and worked examples, not jargon.
Latest
All postsSelf Attention
An intuition-first walk through self attention — queries, keys and values by analogy, then the implementation, multi-head attention, and transformers.
–––
Positional Embeddings
Token embeddings alone throw away word order. How position gets encoded into a sentence, and why the sinusoidal scheme is built the way it is.
–––
Token Embeddings
How tokens turn into meaningful vectors: the core idea behind embeddings, how the training data is prepared, and a working implementation.
–––
Neural Networks
Building a neural network from the ground up — what the layers actually do, how training adjusts them, and a worked example end to end.
–––
Tokenization
Why models break text into subwords, how Byte Pair Encoding builds a vocabulary from scratch, and what naive word splitting gets wrong.
–––
LLM in a nutshell
A plain-language overview of how large language models actually work — the mental model worth having before going anywhere near the internals.
–––