Reproducing Anthropic's emotions paper on Qwen

I recently got interested in interpretability, and while browsing Anthropic's blog I found an interesting paper about emotions. They look inside Claude Sonnet 4.5 and find vectors in its hidden states that stand for emotions like happy or sad. I thought it would be interesting to do the same test on an open-source model, a Chinese one: Qwen3.5-4B. I want to understand how interpretability works, and rebuilding a paper seemed like the best way in.

The paper also made me curious about something else. I speak Arabic, French and English, and I think the way we think about emotions and express them could be different depending on the language. I did some research and found out that not all psychologists agree on this. Paul Ekman says everyone shares a few basic emotions like fear and anger, and that they are global. The other camp, with Lisa Feldman Barrett, says that what we share is only a basic feeling of pleasant or unpleasant, and that specific emotions are concepts we learn from our culture and language.

Also, a study of 2,474 languages found that emotion words don't mean the same thing everywhere: some languages group grief with fear, and others group it with regret. And there's research arguing that your native language carries more emotional weight than one you learned later. It seems that LLMs are different: earlier work, Emotions Where Art Thou, finds that a model represents emotions in a way that is shared across five languages. The Anthropic paper builds its vectors from stories written in English and doesn't test other languages.

So in this blog post I will be testing Anthropic's recipe, first to get an introduction to interpretability, and then to explore this question a bit further.

Exploring the model

I had a machine with 48 GB of RAM, so I chose a 4B-parameter model. Based on some benchmarks and initial testing, I chose to go with Qwen3.5-4B. The 4B one takes 8.4 GB once it's loaded. I used the regular weights and not a quantized version, because the quantized formats don't load as PyTorch modules and I needed to read what's inside using PyTorch hooks.

Before looking for emotions, I spent some time just looking inside the model. Tokens are embedded into vectors of dimension 2,560, and those go through 32 layers. Every layer adds its output to the same vector, which is what we call the residual stream.

Qwen3.5 is a hybrid, which is a direction several recent models are taking (Sebastian Raschka covers it in Beyond Standard LLMs). Three layers out of every four use Gated DeltaNet, a form of linear attention, and the fourth uses ordinary full attention. I didn't dive deeper into Gated DeltaNet, as what we care about is just the fact that every layer reads from the residual stream and adds back into it, and I only ever look at the stream. However, not all layers serve the same purpose or have the same meaning. Usually, the first ones deal with the surface of the text (spelling, grammar, the current phrase) and the middle ones are where abstract concepts form, whereas the last ones turn all of that back into one decision: which word comes next.

You guessed it: we will be interested in the middle layers, and that's basically why the paper reads its vectors about two thirds of the way up, and not at the end.

Attention sinks

Before we dive deeper, it makes sense to understand what attention sinks are and why they exist, because they change how you should read a model's hidden states and our interpretation of the results.

In an attention layer, every head needs to spread its attention over the earlier tokens, and the weights have to sum to 1. The common explanation is that a head with nothing useful to look at needs somewhere harmless to put its weight, and models learn to use the first token for that. That token becomes a sink. It gets far more attention than its content deserves, and in the same layers its hidden state is also much larger than the others. A more detailed introduction to attention sinks can be found here.

I measured this in Qwen on a 60-token story, in the exploration notebook.

What does that mean for this project? A story's vector is an average over its tokens, and a token with a hidden state three times longer than the rest would pull that average toward itself. So I skip the first 50 tokens of every story, as the paper does.

The recipe

The method is simpler than I expected. You ask a model to write short stories where a character feels an emotion, without explicitly naming that emotion. Why is that? If every story about fear contained the word "afraid", the easiest thing for the vector to pick up would be that word, and it would fire on "I'm afraid I can't make it to dinner". The emotion has to come through what the character does and thinks.

Once we have generated these stories, we can run them back through the model and average the hidden state over each story. The vector for an emotion is the average of its stories minus the average of all the stories, so this way all the similar features, such as language, length and being a fiction story, cancel in the subtraction, and what's left is the part that belongs to that emotion.

Here $h$ represents the hidden state at layer 21 for token $t$ of story $i$. To see why the subtraction works, think of a story's vector as a shared part $c$, plus its topic $t$ and its emotion $e$. Every emotion uses the same ten topics, so the shared part and the average topic are the same in both averages, and they cancel.

In the code, the vector for one story is the hidden state at layer 21, averaged over the tokens. I skip the first 50 tokens, as the paper does: at that point the model hasn't read enough of the story yet, and it keeps the sink out of the average.

outputs = model(input_ids, output_hidden_states=True)

h = outputs.hidden_states[21]           # layer 21, all tokens:  [1, 186, 2560]
h = h[0, 50:, :]                        # keep tokens 50 to the end:  [136, 2560]
story_vector = h.float().mean(dim=0)    # average over the tokens:  [2560]

The emotion turns out to be a small part of the signal. At layer 21, the part that all 150 stories share has a length of about 10, and each emotion moves it by about 1.

Comparing the five vectors

The first thing I checked, as a sanity check, was whether the five vectors relate to each other. An intuitive result we would find is that happy and sad have a negative cosine similarity score, as they are opposites.

Also, we can see from the matrix that the closest pair is angry and afraid, which are both negative and both intense, and even they are only slightly similar.

The logit lens

Before looking at what the vectors mean, it's worth understanding the logit lens, because it's the simplest way to ask a vector what it's about.

At the very end, the model turns the last hidden state into a prediction. It applies a final normalization and then the unembedding, which gives one logit for each token in the vocabulary. The logit lens applies that same last step to a vector taken from anywhere else in the model. It works, approximately, because every layer writes into the same residual stream, so a vector from layer 21 lives in the same space as the final one. What comes out is the list of tokens whose unembedding lines up with the vector, which are the tokens it would make more likely. There's a detailed blog post about how the logit lens works, which I highly recommend reading: Interpreting GPT: the logit lens.

I ran it on the five emotion vectors. For the calm vector, the top words included "tomorrow", "plan", "tidy" and "brisk". I think these words make sense: the calm characters plan things, sort the mail and do chores slowly.

The angry vector was funnier, as its top words were almost all Chinese, among them the words for insult and for intentional injury, which made sense since this is a Chinese model. So a vector made only from English stories was already pointing at words in another language, which was the first hint about my question.

Then I checked at which layer each emotion's own word ranks highest, out of about 248,000 tokens. For sad, angry and calm it's in the middle (angry reaches rank 151 at layer 19), and for happy and afraid it's near the end (happy reaches rank 38 at layer 30).

What does this mean for the project? The lens asks which token a vector would make more likely—a task best suited to the final layers. A vector from the middle layers can represent an emotion without pointing directly to its word. So I use the lens to get a sense of what a vector represents, not to decide which layer to analyze. For that, I track correct guesses across layers, as I'll explain next.

Testing on human-written text

In this section, I wanted to know if they recognize an emotion in something a person wrote, instead of content generated by an LLM.

For that I used EmpatheticDialogues, a dataset where people were given an emotion and asked to describe a time they felt it. I took 100 of those situations, 20 for each of my five emotions, and dropped the ones that named an emotion, so the comparison is fair.

I then gave a situation to Qwen as a user message and read the hidden state at layer 21, at the last token before the model starts its reply. Then I measured the cosine similarity between that state and each of the five emotion vectors. The vector with the highest similarity is the guess, and the guess is right if it matches the emotion the person was asked to write about.

There's one extra step. Before comparing, I subtract the average hidden state over 20 neutral sentences. That removes what every prompt has in common, so only the part that's specific to the situation is left.

def emotion_scores(acts, layer, center):
    """cosine of (activation - center) with each emotion vector at one layer"""
    a = acts[:, layer] - center[layer]
    return F.normalize(a, dim=-1) @ F.normalize(emotion_vectors[:, layer], dim=-1).T

# imp_acts: hidden states of the 100 situations, neu_acts: of the 20 neutral sentences
pred = emotion_scores(imp_acts, 21, neu_acts.mean(0)).argmax(1)   # the guess for each situation

Guessing at random would get 20 out of 100, and anything better than 20 is a real signal that the vector representations we learned are real indicators.

The table shows where the guesses went. Each row is the true emotion, and each column is what the vectors guessed. Take the sad row: of the 20 sad situations, 13 were guessed as sad, 6 as angry and 1 as afraid. The numbers on the diagonal are the correct guesses.

Angry was right 20 times out of 20 and happy 18. Calm was right zero times, and I think there are two reasons. The dataset has no calm label, so I used "content", and those situations are pleasant events like a good meal, which read as happy. And my calm vector seems to mean mostly "nothing emotional is happening", which is also what the neutral sentences are, so subtracting them removes calm as well. Fear is weak too, at 6 out of 20, with most of the misses going to angry and sad.

Do our emotion vectors translate across languages?

I had the same 100 situations translated into French, kept the English vectors and ran the same test with the same rule. French got 58. And 80 of the 100 French situations got the same answer as their English version.

But French is the easy case. It shares an alphabet and a lot of vocabulary with English, so maybe the model is only matching words that look alike. Chinese and Arabic share neither, so I ran those next, and then German, Italian and Spanish. Claude did the translations, and I checked the French and Arabic ones myself.

Chinese got 56 and Arabic 58. All six languages have similar scores. Whatever the English vectors are representing, it seems to be an abstract concept that's shared across languages! This seems like an answer to the question I asked at the start of this article.

Where it happens

Why would a vector built from English stories work on Chinese text? One explanation is that the model turns the same situation into nearly the same hidden state, whatever language it's written in. That is something you can measure. For each situation, I took the hidden state at the last token for the English version and for the version in another language, and measured the cosine similarity between the two vectors at every layer.

For French, the similarity is 0.46 at layer 1, then it climbs to 0.86 around layers 20 to 22, and by the last layer it's back down to 0.43. For comparison, an English situation and an unrelated French one score about 0.34 at layer 21, so 0.86 means the same situation, and not just text that looks alike.

The intuition behind it is similar to what we described before. The model starts with two different sets of words, and since the first layers are more about language and grammar, they are far apart and we get a low score. In the middle layers, it turns them into nearly the same thing or concept, and it separates them again when it has to choose the next word in one particular language. The other five languages have the same shape, with the peak at layer 20 or 21.

The accuracy does the same thing. French follows English up to about layer 23 and then drops, to 34 at the last layer against 49 for English.

What I take from this

The recipe transfers. With a 4B open model and 150 stories, it gives vectors that mostly tell good from bad in text they have never seen, and they do it about as well in six other languages as in English. The layer-by-layer picture matches what earlier work on multilingual models describes: the two versions of a situation meet in the middle layers.

What I find encouraging about interpretability is that you don't need a lab or a huge model to see these things for yourself.

It also brings me back to where I started. What carries across languages in the model is mostly good against bad, and the finer emotions, like fear, are weak everywhere. That's close to what the study of 2,474 languages found for people: the split between pleasant and unpleasant is shared, and the specific emotions are where languages differ. It's only a parallel. It doesn't show that the model works the way we do.

There's a lot left to improve. I only have five emotions with 30 stories each, so there's definitely room to run bigger experiments.

The notebooks, the stories and the prompts in all the languages are on GitHub. If you want to read more, these are the ones I'd start with:

• Emotion Concepts and their Function in a Large Language Model, the Anthropic paper this post reproduces.
• Reproducing Emotion Vector, Junyuan Hong's reproduction of the same paper on Llama 3.1 8B.
• Emotions Where Art Thou, on an emotion subspace shared across languages.
• Decoding Emotion in the Deep, on which layers carry emotion in Qwen3 and LLaMA.
• Do Llamas Work in English?, on the language that multilingual models use in their middle layers.
• On the Biology of a Large Language Model, which has a section on circuits shared across languages.