Always-on attention

Beyond the wake word

A voice assistant can wait for a trigger because, between requests, it has almost no body to account for. An embodied robot does not have that luxury. It is still situated while waiting at a desk, monitoring a room, moving toward a goal, yielding a passage, or recovering from a failed observation. Its head has an orientation. Its sensors have a field of view. Its silence takes time. People can see all of this and form an interpretation before the first word is spoken.

This makes the apparent gaps between interactions part of the interaction. A neutral pose may read as available, absent, cautious, or stuck. A delayed turn may preserve a conversation, miss an approaching visitor, or avoid an unnecessary interruption. Even when the system decides to keep still, that decision consumes a current estimate of the scene: who is present, which commitment remains open, what evidence is becoming stale, and whether a new event deserves acknowledgment.

Always-on therefore does not mean always-maximal. Continuously running every model, tracking every candidate at full rate, and animating every internal transition would be wasteful and socially noisy. The useful meaning is narrower: the allocation process remains alive. During idle monitoring it may spend little. During ambiguity it may turn, increase sensing, retrieve memory, or request slower semantic analysis. During a committed exchange it may suppress distractions rather than increase visible motion.

A robot is never between moments. It is always allocating some minimum of observation, memory, restraint, or readiness.

The timeline film below is an explanatory model of that claim, not a telemetry trace. Its aligned bands show life state, attention target, and resource intensity so the invisible allocation problem can be inspected. The profile is intentionally qualitative. It states that interaction, action, waiting, and apparent idleness contain no cognitively blank interval; it does not state how much compute a deployed robot used at a particular second.

What is real today

Most of this page is an argument about how attention should be built. This section is the part that is already standing, and it is placed here rather than at the end so that a reader can weigh everything that follows against it.

“Implemented” means code and host tests exist for the named scope. “Connected” means the software path is wired across its components while its physical operating characteristics remain unverified. “Staged” means explanatory media whose value is inspectability rather than autonomous performance evidence. The distinction matters: a green host suite can establish arbitration, fallback, and record contracts without establishing robot motion, network timing, model stability, or human response. A convincing film can clarify intended behavior without validating a policy.

Our definition

We define a robot attention engine as a meta-controller. It does not replace perception, planning, dialogue, or motor control. It decides which of those capabilities should receive focus and budget next. Its inputs are what the robot currently has reason to think, what it is trying to do, and what it is not allowed to do. Its output is a unified attention intent rather than an immediate motor pose.

Belief is what the robot has reason to think, not a copy of the world: target continuity, estimated location, role or need when available, uncertainty, and the history of an attention episode. The goal supplies context, because greeting a visitor differs from navigating past one. Constraints remove unsafe or unavailable options before preference is considered. Joint limits, control ownership, target validity, stale data, and minimum safety-loop rates are not rewards that a learned policy may trade away.

The resulting intent allocates four coupled budgets, and it carries the reason for acting so that each executor can preserve it.

• Gaze: where the embodied sensor is physically directed.
• Sensing: which evidence process receives rate, region, resolution, and time.
• Cognition: which internal question receives memory or model computation.
• Expression: how the current focus and commitment become visible to another person.

This division is why gaze cannot serve as a synonym for attention. A robot may keep its camera on one speaker while using compute to interpret another audio event. It may hold its head still while increasing the tracking rate. It may look at someone mainly to signal that their turn remains active even though no new visual evidence is needed. Gaze is an actuator and a signal of attention. The attention system is the policy that decides why, whether, and for how long that actuator should be used alongside other scarce resources.

The abstraction also preserves ownership. Task policy still decides how to navigate, speak, grasp, or hand over an object. Low-level control still solves inverse kinematics, smooths trajectories, and respects hardware limits. Keeping these responsibilities separate lets the high-level intent survive a change of camera, model, head mechanism, or embodiment without pretending that all robots share the same motor vocabulary.

Four questions, one loop

The architecture becomes easier to read when expressed as four questions. The first pair forms cognition. “What does it mean?” asks about identity, category, role, language, affect, relationship, and the significance of the situation. “What can happen, and what can I do?” asks about location, motion, reachability, occlusion, collision, and the possible consequences of action. The distinction is functional, inspired by separate visual pathways for perception and action; it is not a claim that the robot reproduces the biological brain.

These two streams must meet before a useful choice is possible. Geometry may show that a person is approaching, while semantic evidence suggests that the person is a visitor, a colleague, or a passer-by. Meaning without action geometry cannot tell the robot whether a glance is feasible. Geometry without meaning cannot tell it whether an interruption is warranted. Their outputs update a shared belief with explicit uncertainty rather than forcing one large, synchronized observation frame.

The second pair selects behavior. “What should I observe or think next?” is an epistemic question: which act is worth its cost because it can reduce consequential uncertainty? “What should I do or express next?” is pragmatic: which act changes the world, advances the task, or coordinates with another person? Speaking, waiting, yielding, switching attention, and preserving a commitment can all be pragmatic.

Cognition precedes action in the logical account, but the running system is recurrent rather than a four-stage waterfall. A gaze action changes the next observation. An expression changes what another person does, which changes the scene. A task action can make a hypothesis easier or harder to test. New evidence updates belief, and the loop asks the four questions again at different rates. A single act may carry both purposes: looking toward a person can improve the image and communicate that the robot is listening.

First form the best available belief. Then decide whether the next scarce unit should reduce uncertainty or change the world.

Gaze changes two worlds

Gaze has an observational effect. Moving a head-mounted camera changes field of view, target size in pixels, angle, occlusion, and the evidence a model can obtain. The same physical scene produces different evidence under different camera poses and sensing configurations. A gaze shift may therefore be valuable even when it performs no external task, because the new view can resolve identity, intent, or motion uncertainty.

Gaze also has a human-facing effect. People use orientation, duration, repetition, and timing to infer whether a robot noticed them, is listening, remains committed, or is about to change turns. Prior studies in human-robot interaction support the narrower claim that robot gaze changes perceived attention and can shape mutual-looking behavior. They do not provide a universal timing recipe. Group size, task, culture, motion design, and embodiment all affect how a gaze act is read.

One act can change both worlds at once. Turning toward a waiting visitor may place their face in a better sensing region and signal that they have been seen. Holding gaze on the current speaker may add little visual information yet protect the social commitment. A short glance toward a familiar passer-by may acknowledge them without surrendering the active exchange. The policy must therefore weigh observational value and communicative value together, while charging for motion, interruption, latency, and possible confusion.

This dual role creates a demanding standard for visible behavior. Expression should be derived from a real focus, uncertainty, or wait state rather than attached as decorative animation. At the same time, internal attention need not be displayed literally: a robot can retrieve memory without looking away. The design problem is alignment, not pantomime. People should receive enough evidence to predict the robot’s focus and commitments, while sensor and cognitive allocation remain free to serve the task.

A ceiling tuning could not move

An early version of our social selector had a defect worth describing, because the shape of the defect is more instructive than the fix. The person the robot was engaged with contributed a score that eventually stopped growing. Any other visible person contributed a score that grew with the time they had gone unattended. Given two eligible people and enough elapsed time, one curve was guaranteed to cross the other.

The consequence was that the robot would abandon a conversation on a schedule. Not because the exchange had finished, not because the other person wanted anything, but because a clock had run. Raising the cost of switching, or the minimum time before a switch was allowed, only moved the crossing point further out. It could not remove it, because the crossing was a property of the state representation rather than of the coefficients.

It was not a tuning problem. A bounded commitment cannot win against a quantity that grows without bound.

The replacement changes what the state means. Need is raised by evidence and by events—someone arrives, someone hesitates, mutual gaze occurs, a known person appears—and it is cleared when an appropriate response has been given. It does not accumulate merely because a timer is running. Time without robot gaze stopped being treated as a debt the robot owes.

Evaluation changes with the representation. Absolute gaze duration is a poor universal target: twelve seconds is excessive for a five-second exchange and insufficient for a longer one. Why attention ended is more diagnostic than how long it lasted. Attention should end because a commitment completed, the person left, evidence failed beyond a bounded limit, or a higher-priority need legitimately won. Expiry by timer should disappear as a cause. The research question becomes whether persistence follows the structure of the interaction without making the robot socially unresponsive.

How it is built

The work began with a narrower question: how can a robot keep looking at a moving person? Detection was the first layer, and it was immediately insufficient. A detector output is neither an identity nor a durable target. Measurements arrive at uneven rates, disappear under occlusion, and become stale. Treating every absent detection as absence in the world produces brittle behavior, so detection had to become evidence inside a temporal system rather than a direct command.

Above that sits continuity, which preserves where a target is between slower recognition updates, and then the episode: a beginning, a maintained commitment, a policy for temporary loss, and an end with a reason. An episode can survive a short perception gap without declaring the person gone, yet it must eventually release when evidence is exhausted. That history matters socially, because switching away during an open question is not the same act as switching after the exchange has closed.

Selection sits above the episode. Once several people are present, the system compares what each situation asks for against what responding to it would disrupt, after hard constraints have already removed anything unsafe or unavailable. Above that again, slow semantic work runs off the fast loop entirely, so a model call that takes a second cannot stall a control loop that must not miss a frame.

• Detection and estimation: obtain fallible evidence.
• Continuity: preserve a target between unequal observation rates.
• Binding and episode: maintain a bounded commitment through brief loss.
• Selection: compare what is owed against what would be disrupted.
• Cognitive scheduling: spend slow compute without blocking fast control.
• Record: retain the alternatives that were compared, not only the choice.

Three clocks, and what happens when one stops

Embodied attention runs on at least three clocks: a fast sensorimotor loop that maintains target evidence and feeds motion, a middle loop that assembles belief and chooses, and slow semantic work that is variable by nature. These should not be forced into one synchronized super-frame. A late semantic result must not freeze optical continuity or servo updates, so each source keeps its own time, confidence, and validity horizon, and the policy reads a coherent snapshot without pretending everything was observed at once.

Failure behavior follows from the same separation. High-level attention influence expires by design. If the deciding service becomes slow, stale, or unreachable, that influence lapses and the fast loop returns to its own autonomous policy rather than continuing to obey an old instruction. This is a bounded loss of high-level judgment, not a hazard: hardware limits and safety rates remain under their independent owners throughout.

Recording the alternatives, not only the choice

A chosen action alone teaches very little. Each decision is recorded with the alternatives that were available and how they compared, along with the version of the policy that made the call, so that a later policy can be evaluated against logged history before it is allowed to move a robot. Initial weights remain explicit while evidence accumulates, and hard constraints stay outside whatever is learned. A learned preference cannot spend control ownership, joint limits, safety rates, or target validity for a higher social score.

One boundary is worth stating plainly, because it limits what the current record can prove. Motor execution evidence is not yet joined to the attention record, so the dataset can show that a decision was made and published but not that the body carried it out. Motor success is not social success, and neither is currently a substitute for the other.

A reception in four beats

A reception sequence makes the policy problem concrete. In the first beat, arrival, a visitor enters while the robot is attending to someone else. The arrival is evidence, not an automatic switch. The system must preserve target continuity, register the new candidate, and estimate what the event actually asks for. Immediate motion would be easy to script and often wrong.

In the second beat, waiting, the visitor stops and remains oriented toward the robot. Time now has meaning only because it accompanies behavior: approach ended, presence continued, and the active exchange has not yet closed. A legible robot can acknowledge that tension without pretending it has already transferred the conversational turn.

In the third beat the first person finishes and turns away. That ending lowers the cost of transfer, and the robot can move to the waiting visitor because the prior episode closed rather than because it timed out. In the fourth beat a passer-by crosses the scene and nothing happens: nothing about the event exceeds the cost of disruption. Restraint is a positive output of the attention policy, not a missing animation.

The film at the top of this page stages those four beats, and it must be read with a strict boundary. Its robot mesh and head kinematics come from the actual stack, but the people’s movement, scene timing, and attention switches are scripted. The attention engine was not driving the sequence. The film shows what a decision looks like once expressed by a body; it does not show an autonomous engine making that decision.

• Arrival: detect a new candidate without surrendering the current commitment.
• Waiting: accumulate evidence of what is needed, not generic elapsed time.
• Completion: move because the current episode has a legitimate end.
• Passer-by: preserve stillness when nothing warrants interruption.

Research agenda

The architecture is a research hypothesis assembled from supported components, not a claim that a unified social attention engine has already been validated. Its value depends on comparisons that can fail. We therefore frame the next stage as five testable questions spanning information efficiency, continuity, legibility, transfer, and learning.

Each question needs a baseline and an attributable measure. Fixed model cadence can anchor call efficiency. The previous selector and explicit end reasons can anchor persistence. Human judgments can test whether expression improves focus prediction. Embodiment adapters can expose what transfers. Logged alternatives can support replay evaluation before a learned policy is allowed to affect the robot.

No single metric is sufficient. Fewer model calls are useful only at matched confidence or task result. Longer persistence is useful only when warranted by the interaction. Legibility should not require exaggerated motion. Transfer should preserve intent rather than identical trajectories. Learning is acceptable only when hard safety and social constraints remain enforceable and the resulting decisions stay attributable.

1. Can dynamic attention reach the same confidence with fewer model calls than a fixed-rate baseline?
2. Does persistence reduce meaningless switching without making the robot socially unresponsive?
3. Can people predict the robot’s focus more accurately when internal attention and visible expression are aligned?
4. Does the same high-level attention intent transfer across embodiments without being rewritten in motor coordinates?
5. Can logged alternatives improve the policy while hard safety and social constraints stay outside the bargain?

Sources

External primary research supports the narrower claims on this page about functional visual pathways, information-gathering actions, dynamic compute allocation, and the human interpretation of robot gaze. None of that work validates the complete system proposed here, and we do not present it as doing so.

Repository-state statements are current as of 2026-08-23. Internal design records and field reports are named as a single group rather than enumerated, and are available to research and industrial partners under a confidentiality agreement.