Address
Utrecht, Veenendaal

Work Hours
Monday to Friday: 9am to 5pm
Weekend: 10am to 5pm

Document Indexing and Search Platform

The document indexing foundation of QIntelligence at Qfact: user documents indexed, full text searchable and available to every service above them. I led the initiative from the research through the architecture to delivery.

  • ClientQfact, a SaaS platform for complex analysis
  • Period2023
  • RoleLed the initiative, research to delivery
  • Part ofQIntelligence, phase one of two
  • Architected to carry a second phase that did not exist yet, and it did. The automated analysis system was built straight onto it.
  • Elasticsearch underneath it, at the volumes the platform was growing into.
  • Delivered as a service inside the existing microservices landscape, not beside it.

Context

Qfact customers were already uploading the documents their work produces. That content sat in storage as files, which means it was kept but not usable: nothing could search it, and nothing could be built on it. QIntelligence started here, with making that content addressable.

The challenge

The requirement was not "add search". It was to choose a foundation without knowing everything that would be built on it, because the analysis work that became phase two was still an idea at this point. Getting that wrong is expensive twice: once when you rebuild it, and again in everything that was written against the first version.

What I did

  • Led the initiative, including the part before any code, which was working out with stakeholders what document information would actually be used for.
  • Researched the options properly, storage and retrieval approaches weighed against the volumes and the query shapes the platform would realistically ask for.
  • Designed the architecture for a second phase it did not have yet, which is the decision the rest of QIntelligence rests on.
  • Built the indexing service in Python, handling upload, extraction, indexing and retrieval as a service in the existing microservices landscape.
  • Put Elasticsearch underneath it, for full text search across the indexed content at the volumes the platform was growing into.

The outcome

  • Uploaded documents became searchable content instead of stored files.
  • A foundation that carried phase two without being rebuilt for it.
  • Document upload, indexing and full text search available to the platform as ordinary platform capability.

Technologies used

  • Python: The document indexing service.
  • Elasticsearch: Storage and retrieval of indexed document content.
  • Microservices: The architecture the service was built into.
  • GraphQL: The API layer over the platform.
  • OAuth 2.0: Authorisation on the platform APIs the indexing service is reached through.
  • Docker: How the service is packaged and run.
  • Playwright: End to end coverage of the upload and search flows.
  • Jest: Automated tests around the indexing service.

Read the second phase in the AI document analysis system, which is what this foundation was built to carry.