Address
Utrecht, Veenendaal

Work Hours
Monday to Friday: 9am to 5pm
Weekend: 10am to 5pm

Eventstore Development

A new event store for the Qfact SaaS platform, built with the software architect. It persists business logic as events and gives the platform an audit trail that meets government compliance requirements.

  • ClientQfact, a SaaS platform for complex analysis
  • Period2021
  • RoleEngineer, with the software architect
  • Way of workingSCRUM, inside the platform team
  • What changed, when it changed and what caused it, visible for every record on the platform.
  • Designed, built, migrated and integrated into a platform that stayed up throughout.
  • Cassandra for the event log, MongoDB for the read models, RabbitMQ carrying the events between services.

Context

Qfact runs a microservices SaaS platform used every day by thousands of people for complex analysis work. Its customers include organisations that have to account for their data to a regulator, which makes what a record looked like last March, and what changed it, a functional requirement rather than a nice to have.

The challenge

A platform that stores current state can tell you what is true. It cannot tell you what was true, or which action made it so, and no amount of logging bolted on afterwards recovers that honestly. The store also had to go into a live platform: existing data had to move into it, existing services had to start reading from it, and neither could take the platform down.

What I did

  • Designed the store with the software architect, from the event model outward. What counts as an event, what a read model may assume, and how a service subscribes without coupling itself to the internals of another one.
  • Built it on Cassandra, for the append only event log, where write throughput and horizontal growth are what matter and rewriting history is not a feature anyone wants.
  • Projected read models into MongoDB, so the services above could query the shape they needed without replaying the log every time.
  • Wired it through RabbitMQ, so events reach the services that care about them and nothing polls.
  • Ran the migration and the integration, moving existing data in and switching services over in steps, delivered through SCRUM so each step went live on its own.

The outcome

  • Audit capability that meets government compliance requirements, with full visibility of what changed, when, and what caused it.
  • A storage layer that scales on the write path, which is where an event log actually gets tested.
  • Integrated into the existing platform without disrupting operations or uptime.

Technologies used

  • Node.js: The event store services.
  • TypeScript: The language across those services.
  • Event sourcing: The pattern the whole store is built on.
  • Cassandra: The append only event log.
  • MongoDB: The read models projected from that log.
  • RabbitMQ: Event delivery between services.
  • GraphQL: The API layer over the platform.
  • OAuth 2.0: Authorisation on the platform APIs the store is reached through.
  • Jest: Automated tests around the store and its projections.
  • SCRUM: How design, build, migration and integration were delivered.

Event sourcing pays for itself exactly where an audit trail is a requirement, and I am happy to say where I think it does not.