Data Integrity Service

Context

On a platform where thousands of people work daily and services publish events to each other, a single unintended write does not stay in one place. It propagates, it surfaces days later as something that looks unrelated, and it arrives at support as a story rather than a stack trace. I led the project that turned that into something the platform could detect itself.

The challenge

Detection is the easy half to describe and the hard half to get right. The service had to recognise states that should not be possible, tell an actual fault apart from a legitimate edge case, and then repair without becoming the next source of corruption. It also had to fit an existing event driven architecture rather than sit beside it and guess.

What I did

• Researched the problem before building, which meant reading how the corruption actually arose in practice and what the platform already knew about it, not just what a good integrity check looks like in general.
• Interviewed the people who carried the cost, support and the engineers who kept getting the tickets, so the rules encoded the real failure classes and not the theoretical ones.
• Built the service in Python, checking invariants across services and flagging states the platform should never be able to reach.
• Read the event store to find the cause, following the chain of behaviour back from a bad state to the action that created it, which is what makes a repair safe to automate.
• Automated the repair, for the failure classes where the correct end state is unambiguous, and left the rest visible rather than guessed at.

The outcome

• Data corruption detected and repaired by the platform itself.
• Support load down, because the class of ticket that generated it was being closed at source.
• A platform that reports its own faults, which is a different conversation with a customer than one that waits to be told.

Technologies used

• Python: The integrity service itself.
• Event sourcing: The event store the service reads to trace cause.
• Microservices: The architecture it checks invariants across.
• GraphQL: The API layer over the platform.
• OAuth 2.0: Authorisation on the platform APIs the service is reached through.
• Docker: How the service is packaged and run.
• Jest: Automated tests around the checks and repairs.

If your support queue keeps returning the same shape of ticket, that is usually a data problem and I will say so plainly.