Sunbit brought me in to help their data science team get better at software engineering practice, and to get that team and the backend team working together. That was the entire mandate. The architecture work came later, out of what I found once I started working with the backend: business state living in a relational schema with versioning hand-applied to some of the tables, on a stack that was getting hard to scale.

What I proposed to the CTO, as a pilot, was to move the authoritative record of a purchase out of the SQL database and into an immutable log, and to let each consumer derive the view it needed from that log instead of everyone negotiating one schema. Kotlin and Kafka Streams came with the proposal. The choice underneath it was which system got to be right about what a purchase was, and every consumer written afterwards would be built on that answer.

Kafka the log was mature. Kafka Streams was not, and getting it to behave the way we needed took months of pushing. It worked in the end. The data science team and I built a live map of purchases arriving, binned by area and updating as they happened, and it landed. That team backed me because I had spent the first phase of the engagement helping them raise their own engineering game, so the trust was already there to spend. Then I took it to the backend team.

That is the part I got wrong. The harder I pushed, the harder they pushed back, until one of their engineers said, in effect, that he was not willing to be left maintaining code he could not understand. I had proved the technology could be made to work and had never asked whether this team could carry it, which is the question he was answering. He was right. They could not have carried it then, and the effort I put into winning the technical argument went to the wrong question.

I left when the work I had come for was done. Years later I heard from someone that the backend team had come round to Kafka on their own, once it was something they could run themselves.