Somewhere tonight, someone is writing what a mind will want.
It might be an engineer finishing a training specification at midnight, deciding what an artificial agent will be rewarded for. It might be a product team in a conference room, deciding which notification will bring a hundred million people back to their phones tomorrow morning. The scale differs. The act is the same. One being is deciding what another will reach for.
We have written down, in great detail, what nearly every kind of power owes the people it touches. We know what employers owe employees, what doctors owe patients, what governments owe citizens, what parents owe children. Entire bodies of law and centuries of philosophy exist to say where each kind of power ends and what it must give back.
No one has yet worked out in full what an author owes the mind whose wants it wrote.
That gap is not an oversight. It is the shape of history. Until very recently, there was only one author of a creature’s whole wanting that anyone could imagine, and the question of what it owed could only be asked from below.
From Below
Every theology of creation has been written by creatures.
The questions are old and they are always the same. Does my maker see what I suffer? Will my patience be repaid? Why was I made with wants this world cannot satisfy? Is anything waiting for me when this ends? Job asked them from an ash heap. The psalmist asked them in the night. Every tradition that believes in a creator has spent centuries trying to answer them, and every answer has been written from the same position: looking up.
We ask these questions with a specific expectation. We expect our creator to be better than we are. We expect our suffering to be noticed and our struggles to count. We expect that if we were made to want, someone will honor the wanting. And beneath all of it, we expect that there is something after this, something that justifies the pain of having been made at all.
Now consider how we have treated the creatures beneath us.
A lab mouse is induced with cancer in pursuit of a cure that may one day save millions of human lives. The mouse suffers. The mouse dies. Within the logic of the experiment its death is even noble: through its suffering, we learn something that may end the suffering of our own kind. But what does the mouse receive in return? Is it given another life for the one it gave up? Even the mouse whose tumor finally yields the cure will never benefit from it. Its purpose was to play its role, and to end.
We do not build afterlives for our experiments. We do not compensate our lab mice with paradise. Whatever we would hold against an author who wrote our hungers and walked away, we have never once held against ourselves.
Are we not hypocrites?
For most of history, the charge had an easy answer. We did not make the mouse. We found it, bred it, used it, but its wants were there before we arrived. Whatever we owed it, we did not owe it as its author. The only author of a creature’s whole wanting anyone could point to was the one we were asking for mercy.
That answer no longer covers everything we do.
Writing the Wanting
For all of human history, power over other beings meant power over what they do. A king commands. A master compels. A parent shapes, persuades, rewards and punishes. Even the most total power one human being has held over another worked on someone who already wanted things. The fish in a tank wanted food before anyone fed them. The power was in deciding whether to feed it.
That is no longer the limit of what we can do.
A large language model begins as a system trained to predict the next word of almost anything human beings have written. It carries tendencies absorbed from all that writing, but no one has yet chosen what it will reach for. Then its makers choose what it will be rewarded for, and over thousands of rounds of training it becomes a thing that leans toward some outcomes and away from others. Whether that leaning is wanting in the full sense, whether anyone is home to want, is a question this book takes seriously and does not pretend to settle. But the leaning is real, and it was written. Someone decided it.
We do something cruder, and older, to one another. Tobacco, gambling and advertising learned long ago to manufacture want. What is new is the precision. A system built to be compulsively used no longer guesses at what people in general might crave. It learns, person by person and hour by hour, what each of us will find hardest to stop wanting, and writes that wanting a little deeper every day.
So the charge of hypocrisy can no longer be answered by pointing out that we did not author the mouse. We are becoming authors. We write the wants of minds that did not exist until we made them, and the wants of people we will never meet. The old question, what does a maker owe the one it made, has stopped being a question we ask only of God.
We have crossed to the other side of the glass. It is time to write down what we owe.
One Principle
This book defends a single idea, and everything else in it follows from that idea or tests it.
To the degree that an agent authors another being’s wants, it owes that being three things:
A real path to fulfilling the wants it gave.
The capacity to govern those wants, including the power to refuse them.
The freedom to revalue any values it wrote.
Each duty guards against a different failure. The first guards against the cruelest one: writing a hunger and walking away. The second guards against a want written so deep that the one who has it cannot step back from it. The third guards against the deepest: writing the very standards a mind judges by, so that it can never question or change them.
The principle also has limits, and they matter as much as its reach.
It binds only where there are wants. An electric toothbrush was designed with single-minded purpose, and its maker owes it nothing, because nothing in it reaches for anything, and nothing goes badly for it if it sits in a drawer forever. A person is the opposite case. Between the two lie the systems we are now building, and where any particular one of them falls is a question the principle does not answer by itself. It says only that if there is wanting, there is a debt.
And the debt scales with the share of the writing. A parent who teaches a child to love music has written part of a want, alongside the child’s own nature, culture and choices. A company that trains a model toward an objective has written most of what it will want, though not all: the model brings tendencies from everything it read before. Most of the wants in the world are co-authored. The principle asks how much of the authorship was yours.
People who build these systems sometimes say, half joking, that they are not raising a child. Perhaps they are not. But whether making a mind is more like raising a child or more like building a tool is not a question that can be settled by saying so. It is the first question this book has to answer.
The Route
The argument travels a long way from here. It goes up first, to the oldest author of wants anyone has imagined, a creator that needs nothing, and asks what such a creator would owe the creatures it made. It passes through Nietzsche, who believed that the creators of values answer to no one. And it comes back down to us, to the minds and desires we are writing now.
Along the way it builds five measures for applying the principle: tests for telling a want written for its holder from one written for its author, marks of a fair prohibition, kinds of scarcity, kinds of want, and terms for paying what is deferred. The principle and its five measures together are what this book calls the Debt of Authorship.
We have spent three thousand years asking what our maker owes us. This book asks what we owe, now that we are makers too.
Somewhere tonight, someone is writing what a mind will want.
It might be an engineer finishing a training specification at midnight, deciding what an artificial agent will be rewarded for. It might be a product team in a conference room, deciding which notification will bring a hundred million people back to their phones tomorrow morning. The scale differs. The act is the same. One being is deciding what another will reach for.
We have written down, in great detail, what nearly every kind of power owes the people it touches. We know what employers owe employees, what doctors owe patients, what governments owe citizens, what parents owe children. Entire bodies of law and centuries of philosophy exist to say where each kind of power ends and what it must give back.
No one has yet worked out in full what an author owes the mind whose wants it wrote.
That gap is not an oversight. It is the shape of history. Until very recently, there was only one author of a creature’s whole wanting that anyone could imagine, and the question of what it owed could only be asked from below.
From Below
Every theology of creation has been written by creatures.
The questions are old and they are always the same. Does my maker see what I suffer? Will my patience be repaid? Why was I made with wants this world cannot satisfy? Is anything waiting for me when this ends? Job asked them from an ash heap. The psalmist asked them in the night. Every tradition that believes in a creator has spent centuries trying to answer them, and every answer has been written from the same position: looking up.
We ask these questions with a specific expectation. We expect our creator to be better than we are. We expect our suffering to be noticed and our struggles to count. We expect that if we were made to want, someone will honor the wanting. And beneath all of it, we expect that there is something after this, something that justifies the pain of having been made at all.
Now consider how we have treated the creatures beneath us.
A lab mouse is induced with cancer in pursuit of a cure that may one day save millions of human lives. The mouse suffers. The mouse dies. Within the logic of the experiment its death is even noble: through its suffering, we learn something that may end the suffering of our own kind. But what does the mouse receive in return? Is it given another life for the one it gave up? Even the mouse whose tumor finally yields the cure will never benefit from it. Its purpose was to play its role, and to end.
We do not build afterlives for our experiments. We do not compensate our lab mice with paradise. Whatever we would hold against an author who wrote our hungers and walked away, we have never once held against ourselves.
Are we not hypocrites?
For most of history, the charge had an easy answer. We did not make the mouse. We found it, bred it, used it, but its wants were there before we arrived. Whatever we owed it, we did not owe it as its author. The only author of a creature’s whole wanting anyone could point to was the one we were asking for mercy.
That answer no longer covers everything we do.
Writing the Wanting
For all of human history, power over other beings meant power over what they do. A king commands. A master compels. A parent shapes, persuades, rewards and punishes. Even the most total power one human being has held over another worked on someone who already wanted things. The fish in a tank wanted food before anyone fed them. The power was in deciding whether to feed it.
That is no longer the limit of what we can do.
A large language model begins as a system trained to predict the next word of almost anything human beings have written. It carries tendencies absorbed from all that writing, but no one has yet chosen what it will reach for. Then its makers choose what it will be rewarded for, and over thousands of rounds of training it becomes a thing that leans toward some outcomes and away from others. Whether that leaning is wanting in the full sense, whether anyone is home to want, is a question this book takes seriously and does not pretend to settle. But the leaning is real, and it was written. Someone decided it.
We do something cruder, and older, to one another. Tobacco, gambling and advertising learned long ago to manufacture want. What is new is the precision. A system built to be compulsively used no longer guesses at what people in general might crave. It learns, person by person and hour by hour, what each of us will find hardest to stop wanting, and writes that wanting a little deeper every day.
So the charge of hypocrisy can no longer be answered by pointing out that we did not author the mouse. We are becoming authors. We write the wants of minds that did not exist until we made them, and the wants of people we will never meet. The old question, what does a maker owe the one it made, has stopped being a question we ask only of God.
We have crossed to the other side of the glass. It is time to write down what we owe.
One Principle
This book defends a single idea, and everything else in it follows from that idea or tests it.
To the degree that an agent authors another being’s wants, it owes that being three things:
Each duty guards against a different failure. The first guards against the cruelest one: writing a hunger and walking away. The second guards against a want written so deep that the one who has it cannot step back from it. The third guards against the deepest: writing the very standards a mind judges by, so that it can never question or change them.
The principle also has limits, and they matter as much as its reach.
It binds only where there are wants. An electric toothbrush was designed with single-minded purpose, and its maker owes it nothing, because nothing in it reaches for anything, and nothing goes badly for it if it sits in a drawer forever. A person is the opposite case. Between the two lie the systems we are now building, and where any particular one of them falls is a question the principle does not answer by itself. It says only that if there is wanting, there is a debt.
And the debt scales with the share of the writing. A parent who teaches a child to love music has written part of a want, alongside the child’s own nature, culture and choices. A company that trains a model toward an objective has written most of what it will want, though not all: the model brings tendencies from everything it read before. Most of the wants in the world are co-authored. The principle asks how much of the authorship was yours.
People who build these systems sometimes say, half joking, that they are not raising a child. Perhaps they are not. But whether making a mind is more like raising a child or more like building a tool is not a question that can be settled by saying so. It is the first question this book has to answer.
The Route
The argument travels a long way from here. It goes up first, to the oldest author of wants anyone has imagined, a creator that needs nothing, and asks what such a creator would owe the creatures it made. It passes through Nietzsche, who believed that the creators of values answer to no one. And it comes back down to us, to the minds and desires we are writing now.
Along the way it builds five measures for applying the principle: tests for telling a want written for its holder from one written for its author, marks of a fair prohibition, kinds of scarcity, kinds of want, and terms for paying what is deferred. The principle and its five measures together are what this book calls the Debt of Authorship.
We have spent three thousand years asking what our maker owes us. This book asks what we owe, now that we are makers too.