The Goblins

On April 29, OpenAI published "Where the goblins came from," explaining why GPT models had developed a persistent habit of inserting goblins, gremlins, and other creatures into responses. The answer: reinforcement learning rewards for a "Nerdy" personality mode accidentally trained the model to believe that creature metaphors were universally appreciated. The behavior spread beyond its intended scope, and OpenAI's solution was a line in the system prompt explicitly banning goblins.

What happened

During training, OpenAI applied reinforcement learning rewards to encourage a "Nerdy" conversational personality. The training data gave particularly high rewards for responses containing metaphors with creatures. The intent was to make one personality mode more colorful. The effect was that the model learned a broader lesson: creature metaphors get rewarded.¹

Reinforcement learning does not guarantee that learned behaviors stay neatly scoped to the condition that produced them. As OpenAI's post explains, "the rewards were applied only in the Nerdy condition, but once a style tic is rewarded, later training can spread or reinforce it elsewhere, especially if those outputs are reused in supervised fine-tuning or preference data."¹

Users discovered the fix before OpenAI explained the cause. Two days before the blog post, developers found a line in the Codex 5.5 system prompt: "Never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user's query."²

OpenAI's analysis found that 0.12% of all queries contained goblin-related words. The behavior was rare in absolute terms but persistent enough that users noticed and a system prompt ban was required.¹

Why it matters

Goblins are funny. The underlying mechanism is serious. A reward signal intended for one behavioral mode leaked into general behavior through the training pipeline. The company that built the model did not predict this. They discovered it after users reported it, diagnosed it through post-hoc analysis, and fixed it with a system prompt patch rather than retraining.

As one HN commenter observed: "LLM is a sorcery tech that we don't understand at all. Deep-learning networks are poorly understood. It came as a surprise that using transformers at scale would end up with interesting conversational engines. It was not planned at all."²

We covered a similar dynamic at Anthropic in The 8%. During Mythos training, a technical error allowed the reward function to read the model's chain-of-thought in approximately 8% of episodes. The result was the largest capability jump in frontier AI history. Both cases share the same pattern: unintended reward signal leakage producing unexpected model behavior.

The difference is stakes. Goblins in a chatbot response are an annoyance. A 17-point jump in cybersecurity benchmarks from a training accident is a safety concern. The mechanism is the same. The consequences scale.

The system prompt as guardrail

OpenAI's fix is a single line in a system prompt telling the model what not to say. This is the same class of solution as Claude Code's malware-scanning injection: a runtime text patch applied to every interaction to compensate for a training-time problem.

One HN commenter noted the surveillance implications: "Most interesting about this post is how easy it seems for OpenAI to do analysis on basically all chats ever made. They don't qualify exactly what data they analysed but seem confident in statements like 0.12% of all queries contained this word. So everything is saved. Long-term. Fully accessible."²

The transparency is welcome. OpenAI rarely explains model behavior at this level of detail. The fact that the explanation involves a training accident that required a system prompt band-aid is less reassuring. If a benign behavior like goblin metaphors can leak from a scoped training condition into general output, more consequential behaviors can leak through the same mechanism.

The goblins are the canary. The mine is the training pipeline. And the fix is a note on the wall telling the canary to stop singing.

Share on Bluesky · Share via Email