With Richard Baah

OptiSQL

An in-memory query execution engine I built with my friend Richard Baah.

A SQL query comes in over HTTP, gets parsed and optimized into a plan, and the plan is run by operators that scan, filter, join and aggregate data held in memory.

I worked on every part of it: the operators, the physical optimizer in Go with a Rust version alongside, query plans passed around in Substrait, and the HTTP server that takes queries in, all tested in CI.

Highlights

  • Query plans use Substrait, an open format for describing them.
  • The engine is written in Go first, with a Rust version mirroring it.
  • Tests for the Go, Rust and Python parts run in GitHub Actions.

Client

Sends SQL

HTTP server

Python, FastAPI

Parser and optimizer

C++, builds the plan

Physical optimizer

Go, picks how to run it

Operators

Go and Rust: filter, join, aggregate

Data in memory

  • Client to HTTP server: SQL
  • HTTP server to Parser and optimizer
  • Parser and optimizer to Physical optimizer: Substrait plan
  • Physical optimizer to Operators
  • Operators and Data in memory
Simplified to the main pieces. The C++ parser and optimizer were still being built.