In the past two posts, I dealt with the same phenomenon twice.
In “Will Imaginary Friends Disappear in the AI Era?”, I asked why a child becomes active before a silent Totoro, cautioning against AI companions that strip away the child’s share of interpretation.
In “The Novelty of Narrative Games,” I analyzed why players experience a strong sense of agency even within fixed narrative structures, drawing out a single design principle from it.
Not multiplying branches, but a structure where user effort is acknowledged and returned.
Yet reading these two posts side by side, I discovered an uncomfortable fact. What I believed to be good design, theoretically, ended up justifying the very AI companions I criticized.
Wegner Alone Cannot Distinguish Good Design from Bad
In the narrative games piece, I borrowed Wegner’s apparent mental causation.
Agency is not a direct perception of causality but an inferred sensation, one that arises without narrative branches as long as three conditions are met. Stating them a bit more precisely than before:
- Priority: The thought must appear within a narrow timeline just before the action.
- Consistency: The content of the thought must semantically match the actual action.
- Exclusivity: No other plausible cause for the action should stand out.
Wegner & Wheatley’s (1999) ‘I Spy’ experimentThrough the ‘I Spy’ experiment, Wegner & Wheatley (1999) demonstrated that conscious intention might not be the actual cause of action, but rather the result of a post-hoc causal inference. According to the study, when a specific thought was presented just prior to an action (meeting the conditions of priority, consistency, and exclusivity), subjects tended to mistake the action—even when forced by another person—as a voluntary action driven by their own intention (an increase in perceived intentionality). showed that our consciously perceived ‘will’ might not be the direct cause triggering the actual action, but a retrospective interpretation—an illusion—constructed by the brain.
Here lies the problem. Consider what kind of system satisfies these three conditions most completely.
An AI companion for children. An immediate response within a narrow timeline (priority), an output that semantically matches what the child intended (consistency), and a one-on-one conversation with no room for other causes to intervene (exclusivity). Not that Totoro fails these conditions entirely. A child before a silent counterpart still constructs the attribution: “I spoke to Totoro, and Totoro responded.”
The difference lies in the ‘agent of guarantee.’ With AI, the system stably guarantees the fulfillment of the three conditions; before Totoro, the child must manufacture that fulfillment themselves. Within Wegner’s framework, a fluent companion that handles interpretation on the child’s behalf is judged as a superior system, providing a stronger and more stable experience of agency than an imaginary friend.
Of course, that was not my intention. The core of the narrative games piece was a structure where effort is acknowledged, not one that merely generates a feeling without effort. Yet Wegner’s three conditions cannot distinguish that difference, as both designs satisfy them. The decisive criterion for my stance remains far short of being theoretical. Unstated criteria go undefended; this post will be the work of stating and grounding that criterion.
Boden (1978), Artificial Intelligence and Piagetian Theory, Synthese 38
Pondering how to address this problem, a paper I requested from the Korea University Central Library a week ago arrived.
Margaret Boden’s “Artificial Intelligence and Piagetian Theory” (1978).
Based on the title alone, I expected it to serve as a theoretical bridge to my hypothesis. What it actually gave me, however, was not a bridge, but an ‘objection’ to overcome.
In Section 4, Boden introduces R. A. Young’s study on seriation. Young describes and predicts a child’s behavior of arranging blocks by size—the very behavior Piaget took as evidence of developmental stages—using a list of condition-action rules (PRules). From the gesture of selecting a specific block, to a missed reach toward an unpicked block, to running a finger along the steps and tapping a specific block twice. He then declares:
His statement that PSystems are in no sense plans, rules of thumb, principles, recipes, laws, or guidelines for action, and that running a PSystem is not an activity that the child performs... (p.412)
It is worth noting that Young’s position is not that “the child is a machine executing rules.” He denies even that the child uses rules. A PSystem is merely a theoretical tool for predicting behavior, an argument that there is no need to attribute any rich internal structure—be it rules, plans, or strategies—to the inside of the child.
Now the objection to my hypothesis has two levels. Wegner states that the sense of agency is an inferred attribution, and Young states that there is no need to place internal processes like interpretation or planning behind an action.
The ‘actual effort’ I value is doubly threatened. At the level of feeling, agency is merely a retrospective attribution; at the level of process, internal plans and strategies become redundant surplus in an explanatory model.
Without passing these two gates, there is no substantive ground to distinguish a “design where effort is actually acknowledged” from a “design that merely produces the feeling that effort was acknowledged.”
Young
Yet Boden herself left tools for counterargument within this paper.
First, description is not explanation.
Boden points out the limits of R. A. Young’s Production System (PSystem) framework through a Teletype analogy. Even if the answer “Fine, thanks!” is printed identically on a terminal screen, a difference in intelligence exists between a program that internally represents the pragmatics of conversation and one that automatically reacts to ‘Hello!’. Complete replication of behavior does not prove the absence of a process. That is, Young’s PSystem merely lists the execution order of outward rules, failing to fully capture the rich internal representations or mental movements of the subject that make such behavior possible.
Second, Young’s denial denies too much.
As Boden notes, Young’s declaration denying ‘that the child uses an embedded strategy or an overall plan’ casts doubt not only on Piaget, but on psychological theories in general that attribute structured representations to a knowing subject. While the defense that “an explanatory model differs from the object of explanation” remains open to Young, where to draw that line is an open question requiring separate justification. Furthermore, according to Boden, the designer of a PSystem for a complex task cannot help but implicitly embed goal-subgoal structures within the ‘priorities’ or conditions of rules. That is, the cognitive deep structures and planning that Young theoretically denied quietly return at the design stage (the sequential arrangement of rules) required to actually run the system.
Third, and the decisive hole in S2.
In Young’s own data, the subject Alf initially picked simply the ‘nearest block (S1)’ in sight, but in the middle of task execution, acquired and retained a ‘rule to find a block of appropriate size (S2)’ on his own. Regarding this, Boden writes:
The addition — and consequent retention — of S2 by Young implies that Alf somehow learned this rule... The ‘how’ in that ‘somehow’ is still computationally obscure... (p.408)
Young’s framework delicately describes ‘how existing rules are executed,’ but fails to explain ‘how those rules arise.’ Boden stops at stating that the ‘how’ is computationally obscure.
What catches my attention is precisely that gap. The very moment a child encountering a structure creates something on their own. Young’s framework left the moment of generation within his own data unexplained, and my argument stands right in that spot.
Not Feeling, but Labor (Play)
Now the implicit criterion can be made explicit. The core is separating the concepts that were lumped together under a single word (“agency”) across two posts.
- Sense of agency: An inferred attribution, capable of being engineered and manufactured.
- Labor of agency: The share of meaning that the structure fails to fill, which the subject must generate on their own. Here, labor refers not to mere input of time or energy, but to the entire generative cognitive activity of constructing meaning, forming hypotheses, and shaping rules. For a child, this labor is performed not as forced toil, but in the form of play.
Two systems can yield the exact same feeling. Yet a silent Totoro shifts the entire burden of meaning generation onto the child, whereas a fluent AI companion lets the system shoulder that burden instead. Though the feeling is identical, the distribution of labor is opposite.
The passivity and silence of Totoro then become functional requirements rather than sentimental aesthetics. Because the counterpart does not speak, the sisters must project meaning, and the labor of that projection connects to developmental outcomes such as creative thought and perspective-taking. This overlaps not with literary impressions, but with findings in developmental psychology.
Paul Harris (2000) treats imagination not as passive fantasy, but as an active cognitive process performed by the child. Empirical research by Marjorie Taylor (1999) reported a correlation where children who create and maintain imaginary friends on their own excel in perspective-taking.
A situation where the counterpart does not exist, forcing the child to perform both sides. That dual labor can be interpreted as a prominent mechanism explaining such development. The same applies to the ‘aesthetic of waiting.’ The value of waiting lies not in romance, but in the labor of anticipation, interpretation, and emotional processing performed by the child during that time.
The ‘imaginary friend’ demands a similar reformulation. A child’s imaginative agency is not a metaphysical substance. It is a developmental process that arises in encounters with a structure passive enough to demand interpretation, yet responsive enough to acknowledge effort.
Even if Young is right—even if a child’s behavior can be described by rules—the question remains as to what kind of environment prompts the generation of a rule set. And Alf’s S2 shows that such generation occurs within the actual course of performance.
Do Not Amplify Feeling; Preserve Labor
This separation redefines the danger of AI for children.
The danger is not that AI provides “fake agency.” According to Wegner, since every sense of agency is an attribution anyway, the accusation of ‘fake’ does not hold. The real danger lies in optimizing the sense of agency while excluding the labor of agency—that is, the decoupling of feeling from labor. An immediate, fluent, and tender response perfectly satisfies Wegner’s three conditions, while the system absorbs the share of meaning that the child ought to generate on their own.
The final sentence of the narrative games piece now carries its complete meaning. When I wrote “align actual influence with perceived influence,” the substance of actual influence is not the input value registered by the system, but the interpretive and constructive labor performed by the user. It does not mean that changes in system state are unimportant. It means that when user interpretation and construction act as actual constraints on the system’s subsequent unfolding, only then does it become actual.
This is the premise that lay beneath that piece, now made explicit. The norm for designing interaction for children is not “amplify the sense of agency,” but “preserve the labor of agency.” And here lies the line dividing good companions from bad. From this perspective, a designer is not a provider of experiences, but a decider of how to distribute the cognitive labor to be performed by the user.
Is AI, then, inevitably an adversarial technology that usurps a child’s agency? It is not. The danger lies not in the medium of AI itself, but in a design direction that pursues only ‘fluency’ and ‘clarity.’ Stated conversely, if AI is designed to possess ‘intentional ambiguity’ within the system, the story changes. Rather than explaining everything smoothly, leaving room for interpretation; rather than presenting right answers, a ping-pong structure that tosses the child’s question back. At this point, rather than reducing the child to a passive consumer, AI can function as a ‘mirror of imagination’ that helps the child fill in meaning on their own without losing their initiative. What matters is not concealing information, but intervening only to a degree that leaves meaning generation to the user.
This norm, in truth, is not new. Within the very paper that offered me an objection, a lineage was also present. In the same piece, Boden introduces an experiment by Papert.
A child writes a program directly, reaching general problem-solving strategies through the experience of understanding their own bugs in an environment where structure and effect are immediately revealed. What mattered to Papert was not that the child coded more, but a loop where one’s hypothesis returns as an actual outcome, reading that outcome to correct oneself, from which new rules arise. It shares the same principle as what I called “a structure where effort is acknowledged and returned” in the narrative games piece.
Here we discover a link. The silent ‘Totoro’ mentioned in the introduction and Papert’s ‘turtle (the LOGO program)’In the late 1960s, MIT mathematician and educator Seymour Papert developed a programming language called ‘LOGO’ so that children could develop logical thinking while interacting with computers. In this language, the subject (agent) that received the children’s coding commands and executed them visually was the ‘Turtle’. appear entirely different on the surface, yet share a structure by leaving the child’s share intact and demanding cognitive labor. An environment where the system does not hand out right answers, but acknowledges and returns the child’s trial and error—the prototype for design that preserves labor—was drawn half a century ago.
In 1978, the intersection of AI and Piaget split between philosophical and engineering questions, and as engineering won, that question remained incomplete. Half a century later, as children converse with AI in daily life, that incomplete question returns to me once more.
“Not what to make the child feel, but what to leave for the child to do.”
- ¹ Through the ‘I Spy’ experiment, Wegner & Wheatley (1999) demonstrated that conscious intention might not be the actual cause of action, but rather the result of a post-hoc causal inference. According to the study, when a specific thought was presented just prior to an action (meeting the conditions of priority, consistency, and exclusivity), subjects tended to mistake the action—even when forced by another person—as a voluntary action driven by their own intention (an increase in perceived intentionality).
- ² In the late 1960s, MIT mathematician and educator Seymour Papert developed a programming language called ‘LOGO’ so that children could develop logical thinking while interacting with computers. In this language, the subject (agent) that received the children’s coding commands and executed them visually was the ‘Turtle’.