Architecture

Zatabase is a hybrid data and compute platform built in Rust. It combines relational SQL, NoSQL document storage, vector similarity search, and a job orchestrator in a single binary. This page describes the internal architecture and how the major components fit together.

High-Level Overview

                    ┌─────────────┐
                    │   Clients   │
                    └──────┬──────┘
                           │
              ┌────────────┼────────────┐
              │            │            │
        ┌─────▼─────┐ ┌───▼────┐ ┌─────▼─────┐
        │  HTTP API  │ │PG Wire │ │  WebRTC   │
        │  (Axum)    │ │Protocol│ │ Real-time │
        │ Port 8686  │ │Port 5432│ │           │
        └─────┬──────┘ └───┬────┘ └─────┬─────┘
              │            │            │
              └────────────┼────────────┘
                           │
                    ┌──────▼──────┐
                    │   zserver   │
                    │  (Router)   │
                    └──────┬──────┘
                           │
         ┌─────────────────┼─────────────────┐
         │                 │                 │
    ┌────▼─────┐    ┌─────▼──────┐   ┌──────▼──────┐
    │  zauth   │    │  ZQL       │   │ zpayments   │
    │  (Auth)  │    │  (Engine)  │   │ (Credits)   │
    └────┬─────┘    └─────┬──────┘   └─────────────┘
         │                │
    ┌────▼─────┐    ┌─────▼──────┐
    │zpermis-  │    │    ZQL     │
    │ sions    │    │  Patent-   │
    │ (RBAC)   │    │  Pending   │
    └──────────┘    └─────┬──────┘
                          │
                    ┌─────▼──────┐
                    │  Storage   │
                    └────────────┘

ZQL Engine (Patent-Pending)

At the core of Zatabase is ZQL (Z Query Language), a patent-pending database engine that provides fast similarity search, exact matching, and SQL compatibility. ZQL is a separate, private codebase that Zatabase embeds as a library.

Key Capabilities

• O(1) exact match: The = operator provides instant exact matching
• Prefix scans: The STARTS_WITH operator efficiently finds records by text prefix
• Similarity search: The ~ operator with a threshold parameter finds semantically related records
• SQL compatibility: Standard SQL is auto-detected and executed through ZQL
• Dense f32 vectors: Standard floating-point vectors (any dimension) for ML embeddings, with L2, cosine, and inner product distance metrics

Crate Map

Zatabase is organized as a Rust workspace with feature-gated crates:

Core Crates (Always Built)

Crate	Purpose
zplatform-storage	KV API backed by the ZQL engine
zcore-catalog	Table and column metadata management
zauth	JWT authentication, OAuth, session management
zpermissions	Role-based access control (RBAC)
zpayments	ZATA ledger, escrow, and billing primitives
zserver	Axum HTTP server, routes, streaming ingest
zpg-protocol	PostgreSQL wire protocol implementation
zobservability	Metrics, health checks, tracing
zcommon	Shared types and utilities

Feature-Gated Crates

Crate	Feature Flag	Purpose
zevents	events	Event system with webhooks and triggers
zorganizations	organizations	Multi-tenant organization management
zteams	organizations	Team collaboration and membership
zfunctions	functions	Serverless function execution
zdomains	domains	Custom domain management
zprojects	projects	Project lifecycle management
znotifications	notifications	Alert and notification delivery
zsdk-generator	sdk-generator	Client SDK code generation
zmesh-network	mesh	Peer-to-peer mesh networking
zconsensus	consensus	Raft consensus for replication
ztransaction	transactions	Distributed transactions and sagas
zcloud-deploy	cloud	Cloud deployment automation
zcloud-hetzner	cloud	Hetzner Cloud provider integration
zcloud-cloudflare	cloud	Cloudflare integration
zidentity	identity	Verifiable credentials and DID

Storage Architecture

All data in Zatabase flows through the ZQL engine. The platform storage layer (zplatform-storage) provides a key-value API on top of ZQL for metadata and configuration:

Application Code
       │
       ▼
zplatform-storage (KV API)
  put_json / get_json / scan_prefix_json / delete
       │
       ▼
ZQL Engine (patent-pending)
  query / insert / exact_lookup / prefix_scan / execute_sql
       │
       ▼
Disk (data_dir/<org_id>/<store_name>/)

Each organization gets isolated storage directories. Within an organization, data is partitioned by store (table) name.

Security Model

Authentication

Zatabase uses a layered authentication system:

1. JWT tokens: Short-lived access tokens (15 min default) + long-lived refresh tokens (7 days)
2. SCRAM-SHA-256: For PostgreSQL wire protocol connections
3. Dev tokens: HMAC-signed tokens for development (blocked in production)
4. OAuth: Delegated authentication via Google, GitHub, Discord, Apple

Authorization (RBAC)

Permissions are enforced at API boundaries using org-scoped role-based access control:

• Principals: Users, Roles, Groups, Labels
• Resources: Collections, Tables, Indexes, Jobs, Files, Permissions
• Actions: Create, Read, Update, Delete
• Built-in roles: admin (full access), operator (read/write data), reader (read-only)

Permission checks are applied by every handler before executing operations:

fn ensure_permission(
    org_id: u64,
    principals: &[Principal],
    resource: ResourceKind,
    action: Crud,
) -> Result<(), PermissionError>

Multi-Tenancy

• Every request is scoped to an organization via the JWT token
• Storage is physically isolated per org (separate directories)
• Permissions are evaluated per-org
• Environment isolation (live/sandbox) provides logical separation within an org

Real-Time Communication

WebSocket

Zatabase supports WebSocket connections for real-time event streaming:

• Topic-based pub/sub (subscribe to job updates, data changes)
• Bounded message queues with backpressure (256 message capacity)
• Concurrent connection handling with graceful cleanup

WebRTC

For low-latency data channel communication:

• Room-based signaling with ICE trickle support
• Query execution over data channels (SQL and QueryBuilder)
• Real-time metrics, logs, and event forwarding
• Database trigger notifications

Consensus and Replication

When the consensus feature is enabled, Zatabase uses Raft consensus to replicate write operations across cluster nodes:

• Write operations are proposed to the Raft leader
• The leader replicates to a majority before committing
• Read operations go directly to the local engine
• A CompositeStateMachine routes operations to the appropriate subsystem

Job Orchestration

Zatabase includes a built-in job orchestrator (zbrain + zworker) for compute tasks:

• Submit job specifications via REST API
• Jobs execute as native processes (local mode) or containers (Docker mode)
• Real-time log streaming via WebSocket
• Artifact management with content-addressed filesystem storage
• Cancellation support with configurable timeouts

Build Configuration

Zatabase uses feature flags to control which subsystems are compiled:

# Core only (minimal binary)
cargo build -p zserver --no-default-features --features core

# V1 build (production release)
cargo build -p zserver --features v1

# Full build (all features including post-V1)
cargo build -p zserver --features full

Feature Profiles

v1 (production release): core, websocket, webrtc, events, organizations, consensus, tls, security-hardening, cypher, pg-tunnel, mesh, cloud, functions, projects.

full (all features): Everything in v1 plus identity, transactions, domains, notifications, sdk-generator, integrations.

Individual Feature Flags

Flag	Description
core	Database essentials (always included)
websocket	WebSocket real-time event streaming
webrtc	WebRTC data channels for low-latency queries
events	Event system with webhooks and triggers
organizations	Multi-tenant org and team management
consensus	Raft consensus for replication
tls	TLS termination for HTTP API
security-hardening	Argon2id key management, X.509 cert gen, compliance
cypher	Cypher graph query language support
pg-tunnel	PostgreSQL wire protocol with TLS tunnel
mesh	Peer-to-peer mesh networking
cloud	Cloud deployment (Hetzner, Cloudflare)
functions	Serverless function execution
projects	Project lifecycle management
identity	Verifiable credentials and DID (post-V1)
transactions	Distributed transactions and sagas (post-V1)
domains	Custom domain management (post-V1)
notifications	Alert and notification delivery (post-V1)
sdk-generator	Client SDK code generation (post-V1)
integrations	Cross-service routing (post-V1)
s3-storage	S3-compatible file storage backend