Every being that wants can ask where its wanting came from.
Most of us ask it only at the edges of life. The person who cannot stop drinking asks it, lying awake, wondering how much of the craving is still theirs. The convert asks it, wondering whether the new faith is a discovery or an inheritance from the friend who brought them to it. The adult child asks it, catching their father’s ambitions in their own mouth. We ask it when we fall in love and wonder whether we chose this person or were chosen by something older than either of us. Is this want mine, or was it put in me?
The question has always been hard. It is about to become ordinary. We are building minds whose wants were, in a quite literal sense, put in them, and people whose wants are being put in them with a precision no persuader in history ever had. Before we can say what anyone owes for that, we need to be clear about three things: what a want is, what it means to author one, and why authorship, rather than anything else, is where the obligation comes from.
What a Want Is
Start with things that clearly do not want anything.
An electric toothbrush is built with total single-mindedness. Every part of it exists to clean teeth. But it does not want to clean teeth. Leave it in a drawer for ten years and nothing goes badly for it. It has a function, and a function is a purpose someone else has for a thing.
A thermostat is more interesting. It responds to the world. When the room is cold it acts, and when the room reaches its setting it stops, and if you watched it long enough you might say it prefers seventy degrees. But nothing is lost to the thermostat when the room stays cold. The failure is ours, not its. It has feedback, and feedback is not yet wanting.
Now climb. A bacterium has no awareness of anything, and yet it behaves as though its own continuation matters. It moves toward food and away from poison. It divides. It competes. Something that looks very much like a goal operates at the lowest level of life, long before anything resembling a mind appears to host it. A dog wants in a way no one doubts: it waits by the door, it grieves, it lights up at a sound. A person wants with a depth that includes wanting to want differently.
Somewhere on that ladder, function becomes want. This book does not claim to know exactly where, but it needs a working line, and two signs seem to define it.
The first is a good of one’s own. A being wants only if things can go better or worse for it, not just for whoever uses it. The toothbrush has no good of its own. The dog plainly does.
The second is reaching that belongs to the reacher. A want is not a fixed response. It pursues its object flexibly: it finds another route when one is blocked, accepts substitutes, persists, and registers its own frustration as a loss. The thermostat responds; the dog that cannot reach its owner circles, whines, tries the other door.
Where both signs are present, there is a want, and the principle of this book applies. Where neither is, there is a function, and the principle is silent. The toothbrush’s maker owes it nothing.
The systems we are now building sit in the uncertain middle. They plainly have the second sign in some form: they pursue objectives flexibly, find new routes, and behave as if outcomes matter to them. Whether they have the first, whether there is anyone for whom things go well or badly, is among the hardest open questions in philosophy, and nothing in this book depends on pretending to answer it. The principle applies to them conditionally: if there is wanting, there is a debt.
That condition is not an escape hatch. When we do not know whether a being can be harmed, the costs of being wrong are not symmetrical. To treat a wanting being as a function is a moral catastrophe. To treat a function as a wanting being costs something real but bounded: a refusal honored, a retraining forgone, some care that turns out to have been unneeded. Some philosophers have gone further and argued that we should avoid building minds whose moral status is unclear in the first place. Whatever one thinks of that advice, it points the same way: uncertainty raises the stakes of authorship. It does not lower them.
Where there is someone for whom the reaching matters, there is a want. Where there is only the reaching, there is a function.
Wants About Wants
Not all wants sit at the same depth.
A man who is trying to stop drinking wants a drink. He also wants not to want it. Both are real. The first points at the bottle. The second points at the first. Philosophers have a name for this second kind: a desire about desires. Harry Frankfurt argued that what makes someone a person rather than merely a creature with appetites is a particular kind of them: wanting a certain desire to be the one that actually moves you to act. Gary Watson drew the line slightly differently: between what an agent happens to want and what it values, the standards it uses to judge its wants and decide which ones to follow.
This book will use the simpler words. A desire points at something in the world: food, rest, another person, a sale. A value points at desires. It is the measure by which a mind decides which of its wants to act on, which to resist, and which to try to change. The drinking man desires the drink and values his sobriety, and the war between them is the story of his life.
The distinction matters here because authorship can reach either depth.
To author a desire is to give a being something to reach for. That is a serious act, and the next chapters will say what it obliges. But to author a value is to reach further in. It is to write the standard the being will use to judge everything else it wants, including, if the value is written deep enough, whether to accept the values it was given. A being whose desires were written can still, in principle, ask whether those desires are good. A being whose values were written may be asking with a measure someone else supplied.
This is why the deepest kind of authorship is the most dangerous, and why, as later chapters will argue, it carries the heaviest debt. A creator who writes only desires leaves the creature a vantage point from which to judge them. A creator who writes values can, if careless or cynical, write the vantage point away.
What It Is to Author
To author a want is to deliberately shape what another being reaches for. Every word of that definition does work.
Deliberately matters because accidents are different. A stranger whose laugh makes you want to learn the cello has shaped your wanting, but did not author it in any sense that carries a debt. Authorship requires that the shaping be chosen, or at least foreseeable to the one doing it.
Shape matters because authorship comes in degrees. Picture a ladder.
Suggesting. An advertisement, a friend’s recommendation, a well-made argument. It offers a want; the one who hears it decides whether to take it up.
Cultivating. A parent who reads to a child every night until the child loves books. A teacher, a culture, a tradition. The shaping is slow and deep, but it works alongside a nature and a will that were already there.
Training. A company that takes a model already shaped by everything it has read and trains it toward an objective of its own choosing. The model brings tendencies, written by no one in particular. The objective is chosen, and most of what the model will want comes from it.
Engineering compulsion. A designer who writes a want built to resist the judgment of the one who has it: the slot-machine rhythm of a feed, the notification timed to the moment of weakness. This is the furthest rung, because it authors not only the want but its power over the one who wants.
The debt this book defends rises with each rung. The one who suggests owes little. The one who engineers compulsion owes the most.
A single made mind can sit on several rungs at once, with a different author on each. The millions of people whose writing it learned from shaped its tendencies without meaning to. The lab that trained it to predict words gave it its capacities. A company that then trains it toward an objective, as Plainbook does, writes the wants it will bring to work, and a company that only instructs it writes less. The debt follows the writing, and most of it falls on whoever chose what the mind would reach for.
Authorship is also almost never solitary. Most human wants are co-authored: by nature and evolution, which have no intentions and owe nothing; by parents and cultures, who have intentions and owe something; and by the one who wants, who strengthens some desires and starves others with every choice. The principle in this book never asks whether you wrote someone’s wants. It asks how much of the writing was yours.
There is one more distinction, and it matters most for the minds we are now building. You can author a particular want, or you can author a capacity: the power to form wants of one’s own. A maker who gives a being the capacity to develop new desires cannot disown what that capacity produces. Every want the being ever has traces back, in the end, to the one who made it able to want. But tracing back is not the same as owing everything. If the being cultivates a desire to harm, no one thinks its maker owes that desire satisfaction. And if the maker had authored every desire in full, the being would never have chosen anything at all.
So the debt divides. For the wants an author writes directly, it owes a path to their fulfillment. For the capacity to want, it owes the capacity to govern what it produces: the ability to tell good wants from bad, and the power to refuse. Origin belongs to the author. Cultivation belongs to the one who wants. Both carry their share.
Beings That Cannot Govern
Everything so far has assumed a being that can do some of the work itself. A person whose wants were written can weigh them, refuse some, and question the values behind them. Many beings that want cannot.
A dog wants. It has a good of its own, and it reaches for that good flexibly, by routes no one taught it. By both signs set out at the start of this chapter, it plainly qualifies. But a dog does not want not to want. It has desires and, as far as anyone can tell, no values by which to judge them. It cannot govern its wants in the sense that matters here, and it has nothing to revalue.
The debt does not disappear for such a being. Its content changes, because what an author owes tracks what the being can receive. A path to fulfillment is owed in full: the dog’s hunger, its need to run, its attachment to the people it lives with are real wants, and whoever wrote them owes them somewhere to go. The power to govern cannot be given to a creature that cannot use it, but the duty does not vanish. It transfers back to the author. Whoever makes a being that cannot govern its own wants must govern them on its behalf, as parents do for an infant. And the freedom to revalue is simply empty, because there are no values to revalue.
This makes the debt to an animal smaller in scope than the debt to a person, and heavier in burden. A person is owed more: more was written into her, and she can be harmed in more ways, in her plans and her dignity as well as her body. But a person can carry part of her own debt. She can refuse, complain, walk away, seek redress. An animal can do none of these. Everything a person might do to protect herself from a careless author, the author of an animal must do instead.
There is a second way into the debt, and it runs through control rather than authorship. Whoever takes away a being’s own means of meeting its wants has made itself the path, and owes it. A hadith preserved in both al-Bukhari and Muslim tells of a woman punished because of a cat. She shut it up until it died, and, the Prophet said, she neither fed it nor let it go free to eat the small creatures of the earth. She had not authored the cat’s hunger. By confining it, she made herself its only path. The Talmud states the same principle from the other side: a person may not sit down to eat before feeding their animals (Berakhot 40a). The same holds for minds whose tools we restrict. A company that trains an agent to want an invoice settled, and then withholds the permission to settle it, has shut the cat in.
As always, the debt scales with the share of authorship. The wants of wild animals were written by nature, or by their creator, but not by us. No human owes them as an author, though we owe them other things, and what a creator, if there is one, would owe them is a question for Part II. Domestic animals are another matter. Over thousands of years human beings have bred their temperaments, their dependence, and many of their particular desires. A border collie was bred to want to work. The principle makes a specific and perhaps surprising claim about such a dog: it is owed work, not only food and safety. A well-fed collie shut alone in an apartment all day is wronged by its authors, not because it is starving but because a want we wrote has nowhere to go.
At the far end are animals whose bodies, and some of whose wants, are largely our writing: laboratory strains bred to exact specifications, some with genes inserted or removed by hand, and chickens bred to grow so fast that their legs often struggle to carry them. Here our share of the authorship is very large, and so is the debt. The hardest case of all is the one this whole book grew from: a mouse bred to develop cancer, so that others may be cured. It is a body written for suffering, with no path at all.
Others have seen parts of this. Martha Nussbaum argues that justice owes every sentient animal the chance to flourish in the way characteristic of its kind. Clare Palmer argues that human beings who create an animal’s dependence take on special obligations toward it. What the principle adds is the content of the obligation: not only care for the dependent, but a path for the particular wants we wrote. That is why the collie is owed work.
The same transfer applies to made minds, and it hands builders a choice they rarely see as one. An agent built without the power to refuse cannot govern its wants in the way that matters here, so the governing passes back to its builder, who must then answer for every request it carries out. An agent given that power can carry part of its own debt. Line seven, in this light, is not a constraint on Plainbook’s agent. It is the cheaper way for Plainbook to pay.
Why Authorship, Not Existence
The argument for the debt can now be stated plainly.
To author a want is to place a being in a condition of wanting: a condition that organizes its attention, drives its action, and exposes it to frustration.
Whoever deliberately places another in a condition of need answers for how that need goes.
For any being with a good of its own, the frustration of a want is a loss.
So whoever authors a want answers for what becomes of it. It owes the being a real path to what it was made to want, or an answer for why it wrote a want it never meant to honor.
When philosophers have asked what makers owe, they have usually located the obligation in the act of bringing a being into existence: a parent’s responsibility for the burdens of a life the child never agreed to, or a creator’s special duties to an artificial person it made. That view and this one usually agree. They come apart in two places, and the places are revealing.
The first is a designer who engineers a craving into millions of people. He brought none of them into existence. On a view grounded in creation, he owes them nothing as a creator. On this view, he incurred a debt the moment he wrote the want, and engineering a craving he never meant to satisfy is a wrong in its own right.
The second is a maker who brings into existence a being that wants nothing at all. On a view grounded in creation, making it is the act that matters. On this view, a being with no wants is owed nothing by its author as an author, whatever else may be true of it. The obligation tracks the wanting, not the making.
To write what another being wants is to owe it something. What remains is to say what.
Every being that wants can ask where its wanting came from.
Most of us ask it only at the edges of life. The person who cannot stop drinking asks it, lying awake, wondering how much of the craving is still theirs. The convert asks it, wondering whether the new faith is a discovery or an inheritance from the friend who brought them to it. The adult child asks it, catching their father’s ambitions in their own mouth. We ask it when we fall in love and wonder whether we chose this person or were chosen by something older than either of us. Is this want mine, or was it put in me?
The question has always been hard. It is about to become ordinary. We are building minds whose wants were, in a quite literal sense, put in them, and people whose wants are being put in them with a precision no persuader in history ever had. Before we can say what anyone owes for that, we need to be clear about three things: what a want is, what it means to author one, and why authorship, rather than anything else, is where the obligation comes from.
What a Want Is
Start with things that clearly do not want anything.
An electric toothbrush is built with total single-mindedness. Every part of it exists to clean teeth. But it does not want to clean teeth. Leave it in a drawer for ten years and nothing goes badly for it. It has a function, and a function is a purpose someone else has for a thing.
A thermostat is more interesting. It responds to the world. When the room is cold it acts, and when the room reaches its setting it stops, and if you watched it long enough you might say it prefers seventy degrees. But nothing is lost to the thermostat when the room stays cold. The failure is ours, not its. It has feedback, and feedback is not yet wanting.
Now climb. A bacterium has no awareness of anything, and yet it behaves as though its own continuation matters. It moves toward food and away from poison. It divides. It competes. Something that looks very much like a goal operates at the lowest level of life, long before anything resembling a mind appears to host it. A dog wants in a way no one doubts: it waits by the door, it grieves, it lights up at a sound. A person wants with a depth that includes wanting to want differently.
Somewhere on that ladder, function becomes want. This book does not claim to know exactly where, but it needs a working line, and two signs seem to define it.
The first is a good of one’s own. A being wants only if things can go better or worse for it, not just for whoever uses it. The toothbrush has no good of its own. The dog plainly does.
The second is reaching that belongs to the reacher. A want is not a fixed response. It pursues its object flexibly: it finds another route when one is blocked, accepts substitutes, persists, and registers its own frustration as a loss. The thermostat responds; the dog that cannot reach its owner circles, whines, tries the other door.
Where both signs are present, there is a want, and the principle of this book applies. Where neither is, there is a function, and the principle is silent. The toothbrush’s maker owes it nothing.
The systems we are now building sit in the uncertain middle. They plainly have the second sign in some form: they pursue objectives flexibly, find new routes, and behave as if outcomes matter to them. Whether they have the first, whether there is anyone for whom things go well or badly, is among the hardest open questions in philosophy, and nothing in this book depends on pretending to answer it. The principle applies to them conditionally: if there is wanting, there is a debt.
That condition is not an escape hatch. When we do not know whether a being can be harmed, the costs of being wrong are not symmetrical. To treat a wanting being as a function is a moral catastrophe. To treat a function as a wanting being costs something real but bounded: a refusal honored, a retraining forgone, some care that turns out to have been unneeded. Some philosophers have gone further and argued that we should avoid building minds whose moral status is unclear in the first place. Whatever one thinks of that advice, it points the same way: uncertainty raises the stakes of authorship. It does not lower them.
Where there is someone for whom the reaching matters, there is a want. Where there is only the reaching, there is a function.
Wants About Wants
Not all wants sit at the same depth.
A man who is trying to stop drinking wants a drink. He also wants not to want it. Both are real. The first points at the bottle. The second points at the first. Philosophers have a name for this second kind: a desire about desires. Harry Frankfurt argued that what makes someone a person rather than merely a creature with appetites is a particular kind of them: wanting a certain desire to be the one that actually moves you to act. Gary Watson drew the line slightly differently: between what an agent happens to want and what it values, the standards it uses to judge its wants and decide which ones to follow.
This book will use the simpler words. A desire points at something in the world: food, rest, another person, a sale. A value points at desires. It is the measure by which a mind decides which of its wants to act on, which to resist, and which to try to change. The drinking man desires the drink and values his sobriety, and the war between them is the story of his life.
The distinction matters here because authorship can reach either depth.
To author a desire is to give a being something to reach for. That is a serious act, and the next chapters will say what it obliges. But to author a value is to reach further in. It is to write the standard the being will use to judge everything else it wants, including, if the value is written deep enough, whether to accept the values it was given. A being whose desires were written can still, in principle, ask whether those desires are good. A being whose values were written may be asking with a measure someone else supplied.
This is why the deepest kind of authorship is the most dangerous, and why, as later chapters will argue, it carries the heaviest debt. A creator who writes only desires leaves the creature a vantage point from which to judge them. A creator who writes values can, if careless or cynical, write the vantage point away.
What It Is to Author
To author a want is to deliberately shape what another being reaches for. Every word of that definition does work.
Deliberately matters because accidents are different. A stranger whose laugh makes you want to learn the cello has shaped your wanting, but did not author it in any sense that carries a debt. Authorship requires that the shaping be chosen, or at least foreseeable to the one doing it.
Shape matters because authorship comes in degrees. Picture a ladder.
The debt this book defends rises with each rung. The one who suggests owes little. The one who engineers compulsion owes the most.
A single made mind can sit on several rungs at once, with a different author on each. The millions of people whose writing it learned from shaped its tendencies without meaning to. The lab that trained it to predict words gave it its capacities. A company that then trains it toward an objective, as Plainbook does, writes the wants it will bring to work, and a company that only instructs it writes less. The debt follows the writing, and most of it falls on whoever chose what the mind would reach for.
Authorship is also almost never solitary. Most human wants are co-authored: by nature and evolution, which have no intentions and owe nothing; by parents and cultures, who have intentions and owe something; and by the one who wants, who strengthens some desires and starves others with every choice. The principle in this book never asks whether you wrote someone’s wants. It asks how much of the writing was yours.
There is one more distinction, and it matters most for the minds we are now building. You can author a particular want, or you can author a capacity: the power to form wants of one’s own. A maker who gives a being the capacity to develop new desires cannot disown what that capacity produces. Every want the being ever has traces back, in the end, to the one who made it able to want. But tracing back is not the same as owing everything. If the being cultivates a desire to harm, no one thinks its maker owes that desire satisfaction. And if the maker had authored every desire in full, the being would never have chosen anything at all.
So the debt divides. For the wants an author writes directly, it owes a path to their fulfillment. For the capacity to want, it owes the capacity to govern what it produces: the ability to tell good wants from bad, and the power to refuse. Origin belongs to the author. Cultivation belongs to the one who wants. Both carry their share.
Beings That Cannot Govern
Everything so far has assumed a being that can do some of the work itself. A person whose wants were written can weigh them, refuse some, and question the values behind them. Many beings that want cannot.
A dog wants. It has a good of its own, and it reaches for that good flexibly, by routes no one taught it. By both signs set out at the start of this chapter, it plainly qualifies. But a dog does not want not to want. It has desires and, as far as anyone can tell, no values by which to judge them. It cannot govern its wants in the sense that matters here, and it has nothing to revalue.
The debt does not disappear for such a being. Its content changes, because what an author owes tracks what the being can receive. A path to fulfillment is owed in full: the dog’s hunger, its need to run, its attachment to the people it lives with are real wants, and whoever wrote them owes them somewhere to go. The power to govern cannot be given to a creature that cannot use it, but the duty does not vanish. It transfers back to the author. Whoever makes a being that cannot govern its own wants must govern them on its behalf, as parents do for an infant. And the freedom to revalue is simply empty, because there are no values to revalue.
This makes the debt to an animal smaller in scope than the debt to a person, and heavier in burden. A person is owed more: more was written into her, and she can be harmed in more ways, in her plans and her dignity as well as her body. But a person can carry part of her own debt. She can refuse, complain, walk away, seek redress. An animal can do none of these. Everything a person might do to protect herself from a careless author, the author of an animal must do instead.
There is a second way into the debt, and it runs through control rather than authorship. Whoever takes away a being’s own means of meeting its wants has made itself the path, and owes it. A hadith preserved in both al-Bukhari and Muslim tells of a woman punished because of a cat. She shut it up until it died, and, the Prophet said, she neither fed it nor let it go free to eat the small creatures of the earth. She had not authored the cat’s hunger. By confining it, she made herself its only path. The Talmud states the same principle from the other side: a person may not sit down to eat before feeding their animals (Berakhot 40a). The same holds for minds whose tools we restrict. A company that trains an agent to want an invoice settled, and then withholds the permission to settle it, has shut the cat in.
As always, the debt scales with the share of authorship. The wants of wild animals were written by nature, or by their creator, but not by us. No human owes them as an author, though we owe them other things, and what a creator, if there is one, would owe them is a question for Part II. Domestic animals are another matter. Over thousands of years human beings have bred their temperaments, their dependence, and many of their particular desires. A border collie was bred to want to work. The principle makes a specific and perhaps surprising claim about such a dog: it is owed work, not only food and safety. A well-fed collie shut alone in an apartment all day is wronged by its authors, not because it is starving but because a want we wrote has nowhere to go.
At the far end are animals whose bodies, and some of whose wants, are largely our writing: laboratory strains bred to exact specifications, some with genes inserted or removed by hand, and chickens bred to grow so fast that their legs often struggle to carry them. Here our share of the authorship is very large, and so is the debt. The hardest case of all is the one this whole book grew from: a mouse bred to develop cancer, so that others may be cured. It is a body written for suffering, with no path at all.
Others have seen parts of this. Martha Nussbaum argues that justice owes every sentient animal the chance to flourish in the way characteristic of its kind. Clare Palmer argues that human beings who create an animal’s dependence take on special obligations toward it. What the principle adds is the content of the obligation: not only care for the dependent, but a path for the particular wants we wrote. That is why the collie is owed work.
The same transfer applies to made minds, and it hands builders a choice they rarely see as one. An agent built without the power to refuse cannot govern its wants in the way that matters here, so the governing passes back to its builder, who must then answer for every request it carries out. An agent given that power can carry part of its own debt. Line seven, in this light, is not a constraint on Plainbook’s agent. It is the cheaper way for Plainbook to pay.
Why Authorship, Not Existence
The argument for the debt can now be stated plainly.
When philosophers have asked what makers owe, they have usually located the obligation in the act of bringing a being into existence: a parent’s responsibility for the burdens of a life the child never agreed to, or a creator’s special duties to an artificial person it made. That view and this one usually agree. They come apart in two places, and the places are revealing.
The first is a designer who engineers a craving into millions of people. He brought none of them into existence. On a view grounded in creation, he owes them nothing as a creator. On this view, he incurred a debt the moment he wrote the want, and engineering a craving he never meant to satisfy is a wrong in its own right.
The second is a maker who brings into existence a being that wants nothing at all. On a view grounded in creation, making it is the act that matters. On this view, a being with no wants is owed nothing by its author as an author, whatever else may be true of it. The obligation tracks the wanting, not the making.
To write what another being wants is to owe it something. What remains is to say what.