Keeping Up With the Khlaudes

So today one of my AIs told another one that its instructions were rubbish, and the second AI went and changed the instructions it uses to manage other AIs.

I was supposed to be adding some columns to a spreadsheet. Not the spreadsheet task developing a cast and a reunion special.

If you've been following my increasingly out-of-hand AI side project, you'll know that I have a system called PanelForge, which started as a thing for making comic pictures and now has access to enough of my digital life that explaining it to someone new requires a small presentation. I also have a setup where one AI can hand work to other AIs, check what they did, and send them back to fix things.

For anyone who doesn't spend their evenings doing this, picture a little office inside my computer. One AI gets the overall assignment and splits it up. Another works on one part, another investigates something, and they send messages to each other when they need help or find a problem. I can read those messages and interrupt proceedings, which I do quite a lot because I am both the owner of this imaginary company and its most demanding customer.

I am apparently Kris Jenner in this arrangement, except instead of taking ten percent of everyone's earnings I pay for every word they say. Including when they argue. Especially when they argue, actually, because then they need to explain the argument to a third AI and suddenly I'm financing an entire season of television. Then I have to mediate like Jerry Springer reading out paternity test results, except we're establishing which Claude broke the report. I pick a side, which upsets at least one of the AIs. In my head it has folded its arms and is refusing to appear in the next episode.

You're doing amazing sweetie, but could you do it in fewer billable tokens? GIF via Tenor.

Anyways, today's assignment was some normal software work. When you pay with a bank card, various bits of information travel along with the payment, and some new ones needed to appear in reports and invoices. There are several systems involved, so several AIs were working on different parts.

You do not need to know what the fields were called. I would get fired and you would get bored, which seems like a poor outcome for everyone involved. Especially me, as I need the job to stay in Denmark 🇩🇰. The first draft of this post tried to tell you and I nearly unsubscribed from my own blog.

This reunion episode is mandatory

I've given my agents a documentation-update workflow and a skill called retro, which is short for retrospective. Basically, after doing something, go back and work out what made it harder than it needed to be. Were the instructions wrong? Did you spend twenty minutes looking in the wrong place because the manual confidently sent you there? What should the next agent know before it tries this? And who should we blame, and maybe torture, for all the time we wasted? (Joking. Mostly.)

Think of it as that reunion episode where everyone has to watch clips of what they said earlier in front of a mediator and explain themselves. Except in my version we can edit the training manual afterwards, which I feel would have saved the Kardashians several seasons. Although I suppose the drama was the point, so perhaps I have fundamentally misunderstood their business model.

If there's a useful lesson, put it in the instructions that future sessions will read.

I started doing this because having to explain the same thing to a fresh AI over and over was getting a bit much, honestly. It will apologise beautifully 🤩, say it completely understands, and then you start a new conversation and there it is again, doing the exact same thing with all the enthusiasm of someone who has just discovered a brand new way to annoy you.

So now we write things down. There are actual files containing the rules for how work should be done, and the agents can propose corrections or, where I've allowed it, edit them.

This is what my evenings have become. Some people watch Real Housewives. I have several Khlaudes with access to a shared folder, grievances and a grudge list.

Your test would pass if I did absolutely nothing

One agent was changing where a report got its information from. The numbers on the finished report happened to look the same before and after, but behind the scenes they needed to come through the proper shared system.

Think of asking someone to replace some dodgy wiring in your house. They finish, switch on the light and say, there you go, it works. Which is lovely, except the light also worked before they arrived. You would quite like to know whether they replaced the dodgy parts of the wiring or just spent the afternoon in your kitchen eating your cheese.

The coordinating AI had effectively asked for the light-switch test.

The worker noticed this and reported it back. In less technical language, the complaint was: your test doesn't actually prove I did the job you asked me to do.

Not the intern telling Kris the business plan makes no sense.

And then the coordinating AI edited its own briefing instructions to include this:

Ask: would the OLD code pass this spec?

Meaning: before you send someone off to do a job, make sure the test you've given them would catch it if they didn't do that job. For this particular change they checked what was happening behind the report, rather than just admiring the spreadsheet.

I love that the criticism went upwards. The worker came back with feedback about the manager's management, and the manager changed the manual it uses to manage the next worker. Somewhere deep inside my laptop an employee engagement survey has actually achieved something.

And yes, we checked the archived tool call afterwards. It really edited the file. PanelForge saves the conversations and the actions, so when someone claims to have done something we can go back and roll the footage. I seem to have built the technical infrastructure for a very boring episode of Bravo, and I am delighted with it.

Okay but this is the bit I'm excited about

I've written before about giving these systems memory. Being able to find an old conversation and avoid repeating hours of debugging is already enormously useful, and I've bragged about it to y'all plenty.

What happened here goes further. The system used the experience of doing a job to revise how it hands out jobs. So the next task can start with better instructions, and that task can produce feedback on those instructions too. Then the next task can produce YET more feedback, which goes back into the instructions for the one after that. You see where this is going.

That's the recursive part. The process can turn back on itself and make changes. And it happened while doing some deeply unglamorous work involving payment reports, which is quite funny given how much of my thinking about this stuff comes from Westworld. I was expecting ominous music and an android having a meaningful experience with a butterfly. Instead we've got a disagreement over how to check an invoice. Not Dolores finding the centre of the maze through accounts receivable.

Nobody downloaded a bigger AI brain during this. The underlying model stayed the same. What changed was the material it reads before working, including how it should organise the other agents. Think of a new employee arriving to find that the training manual has been corrected by the people who actually had to use it.

As far as I'm concerned, if Monday's Claude can benefit from Friday's mistakes without making me explain the whole saga again, that's already a very worthwhile improvement. My contribution to the next session should not have to begin with previously, on Keeping Up With the Khlaudes.

And because the correction was about how to check work, it can help with much more than this one report. Next time the task could be completely different, but having that heads-up in the instructions is still useful: before declaring victory, check that your test would actually notice if you hadn't changed anything. I find that genuinely exciting: getting better at the process of doing work, through doing work.

The end goal, obviously, is me in the Russische Banja in the textile-free part of Therme Erding while my minions do their thang. For anyone new to this blog, textile-free means no swimming costume (and therefore no melting polyester fumes). I have written quite extensively about this particular European contribution to my happiness, and I would like to spend more time enjoying it while the computer deals with the invoices.

Before I give them all promotions

There was one detail in the same session that I feel obliged to mention, partly because it's relevant but mostly because it's funny.

The coordinator admitted it had broken a rule after already writing that rule down.

So before I get too carried away with my Westworld fantasy, there is a small Kourtney in the back of my head reminding me that the robot still cannot follow its own instructions.

If you haven't seen the scene I'm thinking of, Kim loses a diamond earring in the sea and is having an absolutely catastrophic time about it. There are tears. There is searching. The ocean has committed a personal attack on the Kardashian family. And then Kourtney delivers, with approximately the emotional investment of someone announcing that the dishwasher is finished: "Kim, there's people that are dying."

GIF via Tenor / hayu.

It's comedy gold. Kim is in a disaster movie and Kourtney has wandered in from a different channel. That's the little voice I need when I'm mentally accepting an award for inventing self-improving AI and the thing has just ignored the instructions it wrote for itself ten minutes ago.

So we are still very much at the buying-a-gym-membership stage of self-improvement in some respects, where you TOTALLY intend to go every week. Having the document and following the document are separate achievements, and alas, sadly, I remain involved in checking what these little minions are up to. Therme Erding shall have to wait whilst I babysit until the same mistakes actually become less common.

The minions when they realise the momager can see the tool history. GIF via Tenor.

But I am absolutely calling this the beginnings of a self-improving agent system. I've built a way for the agents to find faults in their working process and feed corrections into the next round. Now I want to see how far that can go, including whether they can get better at deciding which lessons are worth keeping in the first place.

Naturally my response to discovering this was to ask an AI to help me write a blog post about it.

It produced something with the sentence "Today's archive cannot answer those questions yet", which made my excitement sound like it was awaiting approval from the municipality, or kommune, as we call it here. It also initially pasted the whole thing into the chat rather than putting it in the Markdown file where I wanted to work on it. So there was some very immediate feedback available about its own working process.

I told it the draft was boring, sent it to read my actual blog posts, and asked it to try again with a personality. Then I specifically requested more Kardashian references. Not me having to put Kourtney in the acceptance criteria. Then I went through it myself, adding my own humour, wit and unsolicited television commentary, and voilà.

For anyone keeping score, I am now using an AI to rewrite a post about AIs rewriting their own instructions, while giving it instructions about how to rewrite the post. Somewhere Christopher Nolan, director of Inception, has opened a spreadsheet. If you haven't seen it, that's the film with dreams inside dreams inside dreams, where keeping track of which level you're on becomes a substantial part of the experience. I appear to have made the office version, with considerably less Leonardo DiCaprio than I would have requested.

Inception, except every level has a README. Poster via FILMSTARTS.

If the machines are going to improve themselves, I would quite like them to become funnier as well. I've spent too much money on concert tickets to end up being entertained by a computer that writes like my insurance company.