SQLDB is an open-source Rust SQL database with MariaDB-compatible client connections and a TypeScript API. It combines transactional catalog snapshots with Arrow columnar batches, Zstandard compression, and DataFusion's parallel, vectorized query execution for metrics aggregation. It runs its own engine; it does not require a MariaDB or ClickHouse server.

This initial 0.1 implementation supports a defined subset of MariaDB's SQL and wire protocol. It is not a drop-in replacement for the complete MariaDB server. The compatibility and capacity boundaries below are part of the API.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Build and run

The initial package is developed in this repository. Build the TypeScript module and native binaries before running the examples or tests:

pnpm install --frozen-lockfile
pnpm build
pnpm test
pnpm test:rust

Use Rust 1.94 or newer and the package-manager version in package.json. Cross-building Linux ARM64 on x86 requires aarch64-linux-gnu-gcc. The same @git.zone/tsrust and @push.rocks/smartrust setup used by NoSQLDB and ObjectStorage builds static dist_rust/rustsqldb_linux_amd64 and dist_rust/rustsqldb_linux_arm64 executables. The package's files list includes these artifacts. macOS binaries are not included in this initial build configuration.

import { SqlDb } from './ts/index.js'; // use @lossless.org/sqldb after publication
import * as path from 'node:path';

const password = process.env.SQLDB_PASSWORD;
if (!password) throw new Error('SQLDB_PASSWORD is required');

const db = new SqlDb({
  storage: { mode: 'file', path: path.resolve('metrics.redb') },
  username: 'app',
  password,
  database: 'metrics',
});

await db.start();
try {
  await db.query(`CREATE TABLE IF NOT EXISTS samples
    (id BIGINT PRIMARY KEY, service VARCHAR(64) NOT NULL, latency DOUBLE)`);
  await db.query('INSERT INTO samples VALUES (?, ?, ?)', [1, 'api', 12.5]);
  const result = await db.query(`SELECT service, COUNT(*) AS requests,
    AVG(latency) AS mean_latency FROM samples GROUP BY service`);
  console.log(result.columns, result.rows);
  console.log(await db.stats());
} finally {
  await db.stop();
}

Choose { mode: 'memory' } explicitly for disposable data. File mode requires an absolute path, an existing parent directory, and exclusive ownership of the database file. There is no fallback from file storage to memory. Keep the configured database name unchanged when reopening a file. To copy a database for backup, stop its owner first; copying an open file is not a supported backup procedure.

start() and stop() are serialized and idempotent. stop() waits for admitted operations to drain within shutdownTimeoutMs (default 120 seconds). A successful stop closes connections, releases storage ownership, and confirms that the Rust process has exited. If that wait expires, stop rejects with an error and confirms termination using SIGTERM, followed by SIGKILL after five seconds if necessary. This failure does not report a successful drain: an unacknowledged commit may have persisted, and file storage must be reopened to determine its outcome. Increase the shutdown deadline for large or slow durable writes. address reports the actual host, assigned port, and database; port zero selects an available local port. isRunning becomes false when the child exits. A stopped memory instance starts empty.

query(sql, parameters) executes one autocommit statement. Parameters are strings, finite numbers, booleans, or null, bound structurally through the parsed SQL plan. Use strings with an explicit CAST(? AS BIGINT) for integers outside JavaScript's safe range. DDL cannot contain parameters. Transaction commands require a MariaDB connection. The IPC response wait is queryTimeoutMs + 30,000 milliseconds, separately from native execution and durable commit. A response timeout does not cancel an already admitted commit; inspect the resulting state before retrying a write.

Results contain columns: { name, type, nullable }[], positional rows, and affectedRows. Integer values with 64-bit types and decimal values are returned as strings to avoid precision loss. Smaller integers, finite floating-point values, and booleans use their JSON primitives. Temporal and extended analytic values use strings; binary values use 0x followed by hexadecimal bytes. SQL NULL is null.

MariaDB clients and transactions

The official mariadb Node connector is used in integration tests, including its binary prepared-statement protocol:

import * as mariadb from 'mariadb';

const connection = await mariadb.createConnection({
  ...db.address,
  user: 'app',
  password,
});
try {
  await connection.beginTransaction();
  await connection.execute('INSERT INTO samples VALUES (?, ?, ?)', [2n, 'worker', 18.5]);
  await connection.commit();
} catch (error) {
  await connection.rollback();
  throw error;
} finally {
  await connection.end();
}

Supported commands include connection authentication with mysql_native_password, database selection, ping, text queries, prepare/execute/close/reset, bounded long parameter data, and connection reset. Prepared results preserve integer and binary values. Temporal results are strings so Arrow precision is not silently reduced by a wire representation. The handshake version identifies SQLDB as 5.7.0-SQLDB-0.1.0; it is a protocol compatibility identifier, not a claim that this is MySQL 5.7.

BEGIN, START TRANSACTION, COMMIT, ROLLBACK, and SET autocommit = 0/1 support repeatable snapshot reads. Writes remain private until commit. A failed statement preserves earlier staged writes. Disconnect or reset rolls back uncommitted work. Result packets expose the actual transaction/autocommit status.

Commit uses an optimistic check against the database's catalog revision. If any other writer committed since a transaction's snapshot, its write commit fails with error 1213 and SQLSTATE 40001; roll back and retry the complete application transaction. This check currently covers the whole database, including unrelated tables. DDL implicitly commits earlier staged writes before executing. Nested transactions and savepoints are unsupported.

Each acknowledged file-mode commit atomically persists the catalog and its referenced compressed Arrow batches through redb before publishing the new in-memory snapshot. Unchanged batches retain their durable IDs. An I/O error during commit makes the engine reject further queries until reopened, because commit durability may be indeterminate. Tests cover constraint atomicity, competing transactions, restart, and recovery after SIGKILL following an acknowledged commit.

SQL and metrics

Supported operations include basic CREATE TABLE, DROP TABLE, INSERT VALUES, INSERT SELECT, UPDATE, DELETE, SELECT, joins, CTEs, grouping, ordering, and window queries. Immediate primary/unique constraints, NULL restrictions, defaults, VARCHAR(n) character limits, and the 65,535-byte BLOB limit are enforced before commit. Text comparisons use Arrow's case-sensitive semantics; MariaDB's default case-insensitive collations are not emulated.

Common numeric types, DECIMAL, TEXT, VARCHAR, BLOB, DATE, and TIMESTAMP map to Arrow storage. Unsupported types or modifiers must return an error. DATETIME, fixed-width character padding, custom collations, foreign keys, CHECK constraints, AUTO_INCREMENT, ALTER TABLE, views, triggers, stored routines, replication, and secondary indexes are not implemented. Use explicit aliases for projected expressions that would otherwise have duplicate names. SQL functions and casts follow DataFusion semantics where they differ from MariaDB; clients must not assume server-wide MariaDB function parity.

The metrics engine provides column projection, parallel batch execution, exact aggregations, approximate percentiles, time bucketing, and windows:

SELECT service,
       COUNT(*) AS requests,
       AVG(latency) AS mean_latency,
       APPROX_PERCENTILE_CONT(latency, 0.95) AS p95_latency
FROM samples
GROUP BY service;

-- For a table with a TIMESTAMP column named recorded:
SELECT date_trunc('hour', recorded) AS hour, COUNT(*) AS samples
FROM readings GROUP BY hour ORDER BY hour;

These are ClickHouse-inspired columnar analytics capabilities. SQLDB does not implement ClickHouse's protocol, MergeTree, distributed execution, or its full SQL dialect. Committed tables are loaded into memory on startup; this version is intended for datasets that fit the configured capacity. Constraint validation scans candidate keys, and updates/deletes can rebuild batches. No secondary-index or sustained-ingestion throughput guarantee is made.

Run pnpm benchmark for a reproducible one-million-row workload with 100 metric series. It validates counts, sums, means, and percentile bounds before reporting one warmup and five measured grouped queries. Timings include IPC and response encoding. It reports the CPU, architecture, runtime, ingestion time, and stored columnar bytes. Results describe that workload and machine, not comparative performance against MariaDB or ClickHouse.

Limits and deployment boundary

The listener currently requires 127.0.0.1 or ::1. It requires an explicit username and password of 8–1024 UTF-8 bytes. It does not support TLS, public binding, multiple users, privileges, or multiple databases per instance. SQL cannot read arbitrary files, register external data sources, write exports, or change DataFusion's engine configuration. Run the engine with OS permissions appropriate for its database file.

Option	Default	Meaning
`maxConnections`	64	Concurrent sockets, including authentication
`maxMemoryBytes`	512 MiB	Committed/candidate catalog capacity and, separately, shared query scratch budget
`maxResultRows`	100,000	Rows returned by one query
`maxResultBytes`	8 MiB	Arrow result and encoded response capacity
`queryTimeoutMs`	30,000	Statement planning/execution deadline; durable commit is outside this deadline
`shutdownTimeoutMs`	120,000	Maximum wait for shutdown drain before a failed stop terminates the child

The memory setting is not a process RSS limit. Retained snapshots, simultaneous transactions, buffers, and engine overhead can consume additional memory. Query spilling to disk is disabled. Apply process/container limits when needed. There are also fixed limits of 64 KiB SQL, 256 parameters, 1 MiB wire packets and long data per connection, 128 prepared statements per connection, and 16 management queries in flight. Authentication has a five-second deadline, idle connections five minutes, and response writes ten seconds.

stats() returns the catalog revision as a string, table count, row count, approximate Arrow buffer bytes, and active socket count. binaryPath or SQLDB_RUST_BINARY explicitly selects a native executable; ordinary discovery uses packaged platform binaries and dist_rust. It does not search the system PATH.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in license.md. Third-party dependency licenses and notices are retained in third_party_licenses/; their terms apply to the corresponding components.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

readme.md Unescape Escape

@lossless.org/sqldb