Query Engine II

The Query Engine has a simple concept - it presents a web endpoint that is able to return data from a datastore.

This sounds so simple as to make it redundant, surely any system can present its own data? What makes the Query Engine worth its existence is that it sits in a sweet spot between a fully graphical reporting solution, and having to manually code all feeds from an application.

Features of the Query Engine include:

• Queries for the Query Engine can be written, and deployed, independently of your application.

• The Query Engine is designed to support multi-tenant (SAAS) applications - dynamically selecting a datasource is built in.

• Data is streamed, it is not (usually) necessary for all of the data to be held by the Query Engine process (sorting the stream is an obvious exception, sort in the datastore whenever possible).

• Dynamic fields are supported directly by Query Engine, separate runs of a query can produce a different structure (depending on the data).

• Data can be output in various formats, including CSV, Json, XML, HTML, Excel (XLSX) workbooks (all streamed).

• Authorization is governed by validated JWTs.

• Query Engine is intended to be used via "path-hijacking", providing seamless integration with your application, but can also be accessed directly.

• Query Engine can be easily embedded within a host application.

• Full support for OpenAPI documentation.

• User interface plugin and API for running queries.

• User interface for designing queries.

Naming

This is a clean-room re-implementation of the first version of Query Engine, hence this is Query Engine 2. However, the first version was closed source so no-one outside of my employer will ever see it (nor would you want to, QE2 is better).

In these documents the term Query Engine II will be used in headings and titles, in the body the term will be just Query Engine.

Deployment

The Query Engine is provided as a container image and it is expected that it be deployed via docker compose (including swarm) or kubernetes.

Query Deployment

The queries run by the Query Engine are specified in files (one file per query). Any directory hierarchy use in the storage of the files will be reflected in the paths for running the queries.

The Query Engine itself does not attempt to control how files are managed or made available to it, but it is expected that the files will all exist in a git repository that has some kind of CI pipeline (or other auto-update mechanism) to update files as they are changed.

Query Development

By default the Query Engine cannot modify queries - for it to be able to do so would be a major security breach.

The Query Engine can be deployed in "Design Mode" which breaks that rule and allows the UI to save designed queries. In order to prevent accidental deployment of Design Mode it is an entirely separate image.

• Documentation

Full documentation for Query Engine (apart from this file) can be accessed from a running instance via the user interface. To access the documentation via GitHub start here: https://github.com/Yaytay/query-engine/blob/main/query-engine/docs/Introduction.adoc