Along my journey to create intelligent NPCs, I’m often asked what I’m trying to build. The best comparison I can offer, especially for those less versed in game AI, is Star Trek’s holodeck, a simulated world where characters respond to you as if they were real. Now, there won’t be force field holograms or matter replication any time soon, but real-time interaction with characters who think for themselves is exactly what I’m aiming for.
Intell(m)igence
Of course, I have to work within the bounds of today’s technology, and large language models were the natural choice for the NPC brain. Thoughtful, non-deterministic behavior is my goal, and LLMs, which are essentially context-aware random number generators, fit the bill. But context is the key to a rational, valid response, and building it brings certain challenges: converting raw game state into text, and keeping that context focused over a long play session. Then there’s inference time. An LLM can take several seconds to respond, which is an eternity in a real-time game.
After tackling these problems over multiple iterations, I arrived at an event-driven behavior system modeled on the stimulus-response pattern from behavioral psychology. Figure 1 shows how the pieces fit together at a high level.
A Stimulating Experience
An NPC needs a reason to act, and that reason comes from the stimulus pipeline. Observable data is collected over successive game ticks and fed to the NPC’s perception system, which uses heuristics to decide whether something interesting has happened. If so, it sends a signal to the brain, kicking off a thinking turn. That signal, the stimulus, becomes the latest development the NPC weighs alongside everything else it knows.
A stimulus can be as simple as spotting another character for the first time. Intuition is the catch-all for everything else: signals that come from analyzing what the NPC sees and hears. Someone sprinting past, for instance, can be picked up from position and speed alone.
The Brain
Once the signal arrives, the LLM-backed brain runs a round of inference to produce a short-term plan. The prompt structure is loosely modeled on common cognitive architectures: a long-term memory holding everything the character came into the world knowing, alongside a working memory built from the current session’s raw game state and events. Everything the model sees, from an entity’s properties to the events themselves, is translated out of engine data into terse, plainspoken text. The prompt is rebuilt from scratch every turn as a single message, so nothing carries over except what I put back in.
Real-time performance lives or dies on response time, and the first lever is prompt size. To that end, the prompt is built in YAML, which is compact and readable, and it carries only the entities the NPC has noticed recently. Events accumulate in a running log, and once that log grows long enough, the entries are folded into a summary, leaving only what has happened since. The NPC keeps a sense of its history without carrying every detail.
Response Ability
The thinking turn ends with a plan. The plan is the NPC’s response to the stimulus that set it off, a handful of actions to be carried out in order. An action is the smallest unit of behavior the NPC works with, though not as small as a footstep or a single swing. It is closer to a few seconds of intent: moving to a nearby landmark, or speaking a line of dialogue. Once the plan arrives, the NPC follows it through to the end unless a new stimulus cuts it short, which in a real-time world happens constantly. Whatever was left of the old plan is discarded, and the new one takes over.
Because the plan only covers the near term, it rarely runs more than three to five actions. That keeps the output small, and most inference rounds finish in under two seconds. The plan is also streamed, so actions are pulled off as they arrive instead of waiting for the whole response to land. The NPC is already moving while the rest of its plan is still being written. A short plan and a running start are what stand between a character that reacts and one that leaves you waiting. Those actions land back in the world, where they become observable data for anyone watching, and the cycle begins again.
Seeing It in Action
The clip below is actual gameplay. A robot guards a winding road up to a secret base, and the player is on his way up it. The clip covers two thinking turns. The first stimulus is the robot spotting the player, and the plan that follows is cut short when the player sprints past.
The Road Ahead
Plenty is still unfinished. Memory is the clearest gap. Long-term memory covers who the character is, working memory covers what is happening now, and there is nothing in between. The running log keeps the NPC current, and folding it into a summary keeps it small, but every summary is built on the one before it, and details wear away with each pass. Over a long interaction, a detail from the start of the session can quietly disappear. What the NPC needs is a way to promote what matters out of the session and hold on to it deliberately.
The other limit is that the NPC’s interactivity is bounded by two sets: the stimuli it can notice and the actions it can take. If nothing is watching for a sprint past a checkpoint, that is simply not something the NPC perceives. If there is no action for it, it is not something the NPC can do. Everything a character is capable of lives in the product of those two sets, and both could stand to grow. But bigger is not automatically better. There are only eight notes in a scale, and nobody has run out of melodies yet. The small sets have produced more interesting play than their size suggests, and the actual work is in the balance, enough range to be believable without drowning out the melody.
Real time was the constraint that shaped every decision on this quest. Getting an NPC to think for itself is one problem. Getting it to think fast enough that you never notice it thinking is another. The holodeck is still a long way off, but this is a step in that direction.
Leave a Reply