Boris Cherny at the Computer History Museum

I ended my previous post about Joanna Stern's book talk with a small resolution: I should use these nearby opportunities more often. Living close to the Computer History Museum only means something if I actually go there. So when I saw that Boris Cherny, the creator of Claude Code, would be speaking, I decided to go back. On September 23, I was at the museum again, this time for his conversation with CHM curator David C. Brock.

There was a YouTube livestream. I could have stayed home and heard the same answers, probably with fewer interruptions to my evening. But I wanted the part that the video could not quite give me: being in a room with other people who had also decided this was worth coming out for. I wanted that feeling of sharing a space with people trying to understand something that I was trying to understand too.

Much of the evening was a general discussion of Claude Code, agents, and the future of software development. A few questions stayed with me longer than the overview. One came from Donald Knuth. Brock had collected questions from several CHM Fellows and relayed Knuth's question about memory: does Claude Code remember its solutions to problems, perhaps even from one user to another, or does it start from scratch every time?

My first reaction was surprise. This was Donald Knuth, one of the people whose work helped define how we think about programming. (TAOCP!) The question sounded to me as though the distinction between a model and an agent was not entirely clear. Cherny's description of an agent had been straightforward: a chatbot that can use tools. The model generates responses. The surrounding system gives it ways to read files, run programs, and save information for later. An agent remembering a useful detail does not necessarily mean the underlying model has learned something new. Still, one question relayed by a moderator is not enough to tell me what Knuth understood or intended. Knuth was also asking whether something solved once could help solve another problem later.

Cherny answered in terms of memory tools. The difficulty, he said, was getting the agent to remember the right things. In an early version, asking for one button to be blue could become a lasting rule that buttons should always be blue. Saving information was easy, but deciding what lesson to carry forward was harder. Cherny's example made the distinction concrete. Hearing this question from Knuth surprised me. It made me wonder how uneven familiarity with these tools might be, even among people who understand computing deeply.

Andrej Karpathy's recent description of an AI capability “funnel” gave me another way to think about that gap. He described people barely encountering AI, people occasionally using free chatbots, professionals using frontier tools for substantial coding or mathematics, and people with access to systems inside the labs. Their impressions can differ because their tools, tasks, and access differ. To me, that gap also shapes judgments about how important AI is, how much it will change our lives, and how seriously we should take its risks. What seems like a minor convenience to one person may look capable of transforming work, or causing significant harm, to another. We can end up arguing about AI's future while imagining very different systems.

Cherny had described another part of this gap as model overhang. A model may already be able to do more than the product allows its users to discover. His example was a model capable of writing a whole function or file being presented through an interface that mainly completed the next line of code. Giving it access to files and a way to run the result changes what someone can do with it, even before the model itself improves. For a builder, that is a useful and slightly uncomfortable thought. Some limitations may belong to the way we have decided to use the tool, and we may need to experiment to find out which ones.

That led to a comparison I liked. Brock brought up Butler Lampson's suggestion that computer science might become more like biology, and Cherny agreed. Building with models involves observing behavior, trying things, and revising assumptions. You can build the system without being able to predict everything it will do. Experience with conventional software does not automatically tell you which approach will work. There is something appealing to me about that empirical process, even if it makes the work less comfortable.

Since the talk, I have also been following the discussion around Terence Tao's Math 2.0 lecture. In one thought experiment, he imagined an AI-designed cancer treatment that passed a phase-three clinical trial, then asked whether some human understanding of the treatment or its mathematical model would provide reassurance. The analogy drew criticism, including the argument that a treatment can have demonstrated value before its mechanism is fully understood. Tao's concern was more careful than saying an unexplained cure would be worthless: he was also asking whether an AI could exploit weaknesses in the test or model. But the disagreement helped me put my own reaction into words.

I am comfortable with useful results arriving before a complete explanation. If something produces a benefit we can measure convincingly, the remaining mystery does not make it meaningless. (I love empirical science, and Michael Faraday is my favorite scientist.) I am drawn to the way observation can challenge what we think we understand. Control, though, is a separate concern. Seeing a system perform well in one setting does not tell us how it will behave with more freedom or in a different environment. I can accept an incomplete explanation and still believe we should be able to set limits on what a system is allowed to do.

The questions from Leslie Lamport and Alan Kay brought the conversation to those harder concerns. Lamport's question, as Brock relayed it, was whether this technology could destroy civilization. Cherny explained why he takes the risks seriously. Kay's question was more specific: if an AI product is used for hacking, how should responsibility be divided between the user and the company that made it? That was the question to which Cherny said he did not have a good answer and would want lawyers and ethicists involved. (Come on, this is something every engineer should be thinking about.)

He did describe practical measures: training models to act as intended, detecting suspicious behavior, restricting access, and using another model to review proposed actions. Those are real attempts to reduce risk. Yet I still found myself dissatisfied with how often alignment becomes the destination of the discussion. In his explanation, it included getting the model to do what the user intended. But whose intentions should matter when they conflict, and who takes responsibility for the consequences? Explaining the safeguards does not settle those questions.

I submitted a question that was not selected. I wanted to ask whether Cherny could imagine an ideal state of AI: what would the world look like when we had developed it enough? Was there an end state he hoped we would reach, something more definite than a more capable model? It was the question I kept returning to afterward.

Of course there are objectives. People want useful products, help with difficult work, and systems that behave safely. Cherny was clear about wanting to make AI's development go better. What I am less sure about is whether we have a shared picture of enough. A particular task has an end: the program works, the problem is solved. The larger project of making AI more capable seems much harder to finish. I would like to hear more about the conditions under which we would decide that another increase in capability was unnecessary, or not worth what came with it.

I do not know whether a doomsday scenario is close, or whether it will happen at all. What worries me is the possibility that we could keep moving toward the day without ever making that choice explicitly. Each step might have an understandable purpose. Each new capability might come with another repair to the safeguards. If we have no idea what would count as enough, though, I do not see what would make us stop before the consequences forced the decision.

Meanwhile, people still need to make decisions for next semester. The final audience question asked what a second-year computer science student should do: keep studying, or switch departments? Cherny first advised curiosity and learning to use the tools, including agents. Then he added that the student should probably get a minor. He was not telling them to abandon computer science. He was suggesting that they learn something beyond it.

That was a modest answer after an evening that had reached questions about civilization. We may have little confidence in the long-term picture, but we can learn the tools and broaden what we know now. For now, getting a minor felt like an answer a student could act on. I understood why he gave it, and I would probably give something similar. I still wished we had a clearer picture of the future that the younger generation was preparing for.