Well I’m back into the world of work now - which is both excellent (compelling problems to solve, brilliant people to get to work with) and exhausting, and I’m trying to still find time to push on with my side-quests. Which is hard.
One of the tools I’ve turned to is, of course, AI, and I wanted to spend some time reflecting on how that has changed the process with regard to both the coding projects (btreeny) and the research projects (shnon - my work on semantic hashing).
The uncomfortable truth is that I’m able to cover a lot more ground now than I ever thought possible, but in doing so I’m losing sight of what is my creativity vs what is brute-force enumeration of options.
This shouldn’t feel new - it’s a well-studied result that those using AI feel more productive without necessarily becoming so in the long term (and demonstrably poorer understanding of the things they have built). The question for me is “does that matter?”
Let’s take ICTK, for example. I wanted a tidy React component that could be plumbed into existing projects and make displaying ancillary information on map markers super easy. Claude let me bash this out in a weekend to a standard I was willing to bring into other (personal, not commercial!) products. With guidance it built a reasonably ergonomic library with a clean internal structure, Playwright tests (something I’ve never written myself), and a pretty swish little demo. There’s only so far you need to push such a simple icon component (I’m sure that statement could start an argument in the right meeting room), but I’m pretty proud of what a weekend’s work achieved.
The flip side is my recent disconcerting experience on this semantic hashing work. It’s much more research-oriented, which means hypothesis generation, experiment design, evaluation and ideation.
The first turn of the handle was (mostly) my own brain, applying mathematics and concepts I’d read in papers to code. The “phase 2” has seen my role move almost to an academic advisor, suggesting experiments to run, providing conceptual framing, and pushing back on the incessant need of Claude to add layers of complexity. In the same way that an academic understands the work of others through reading papers, with only a very few actually attempting to reproduce results (a fundamental but often necessary failing of science, one might argue - not everyone can have their own CERN/SKA), my understanding has become “ask for a write up, and read the ‘published’ code”. A bit unfair since many of the original ideas to implement came from my understanding, but many of the evaluations were designed, executed and reported with minimal input from yours truly.
Perhaps I can summarise using a recent interaction. I wanted to better understand how a Prolly Tree works, as a data structure allowing efficient change detection and resolution for distributed systems. The article linked is a brilliant walkthrough, and I felt the missing piece was a simple non-performant implementation. I set up a uv project and the obvious thought occurred: “I could ask Claude to write this and then review it, to save some time”. The result, of course, is that I have a somewhat useless implementation of a prolly tree that I did not build, and only understand at a descriptive describe-over-a-pint-level.
I’ve started using a phrase a lot lately when it comes to working with AI - “Trust, but verify”. The concept being that even if we are comfortable to outsource details of the internals of a codebase to AI, we must still take responsibility for the correctness of the contract that codebase makes with those that depend on it, and put the hard work in (possibly using AI, carefully) to build the verification harnesses that instil confidence.
There’s perhaps another quote that I think deserves its place in my vernacular - “What I cannot create, I do not understand.” For the things we wish to understand (research and learning-focused work), who does the AI-empowered shortcut serve?