Reporting and Analysis Engine

Context

Every reporting request that reaches a developer costs a sprint slot and arrives again three months later in a slightly different shape. Qfact had a steady stream of them, and at the same time the existing analysis methods were getting slower as the data they ran over grew. Both problems had the same answer.

The challenge

A configurable reporting tool is easy to build badly. Give users too little and every real question still needs a developer. Give them a query builder and you have handed them a way to take the platform down at eleven on a Tuesday. The engine had to be open enough to cover the requests that kept arriving and closed enough to stay fast on data that was still growing.

What I did

• Led it from functional design to release, including the part where you work out which requests are actually the same request wearing different clothes.
• Designed the architecture with the software architect and a senior developer, as a new service inside the existing microservices landscape rather than another responsibility bolted onto an existing one.
• Built composable views, with filters, column selection and aggregations, so a use case is assembled by the person who wants it.
• Generated the MySQL queries dynamically, from that configuration, which is what turns a fixed report into 25 or more of them without 25 implementations.
• Kept purpose built reporting tables in sync, through scheduled jobs, so a complex report loads from data already shaped for it rather than from a join computed while the user waits.

The outcome

• Users serve 25 or more frequently requested use cases themselves.
• Complex reports load quickly, on data volumes that were previously the problem.
• Reporting requests stopped consuming developer sprint capacity.

Technologies used

• Node.js: The reporting service.
• TypeScript: The language across the service.
• Python: The data work behind the report generation.
• MySQL: Report storage and the dynamically generated queries over it.
• MongoDB: Configuration and supporting data.
• Microservices: The architecture the service was built into.
• GraphQL: The API layer over the platform.
• OAuth 2.0: Authorisation on the platform APIs the reporting service is reached through.
• Vue.js: The interface where a report is configured.
• Jest: Automated tests around the engine.

Self serve reporting is worth building when the same question keeps arriving, and I can tell you fairly quickly whether yours qualifies.