HeliosDB Lite Deployment Modes Guide¶

Overview¶

HeliosDB Lite supports 4 distinct deployment modes, each optimized for different use cases. This guide helps you choose the right mode for your application and configure it for optimal performance.

Quick Comparison¶

Feature	Embedded	Memory-Only	Server	Hybrid ⭐
Network Access	❌	❌	✅	✅
Persistent Storage	✅	❌	✅	✅
Memory Cache	⚠️ Basic	✅ Full	⚠️ Basic	✅ Configurable
Multi-Client	❌	❌	✅	✅
PostgreSQL Protocol	❌	❌	✅	✅
Performance	Good	Excellent	Good	Excellent
Durability	✅	❌	✅	✅
Best For	Single-process apps	Testing, CI/CD	Basic server	Production

Mode 1: Embedded Mode¶

Description¶

SQLite-style in-process database with persistent storage. The database runs within your application's process.

Architecture¶

┌─────────────────────────────┐
│   Your Application          │
│  ┌────────────────────────┐ │
│  │ HeliosDB Lite (Embed)  │ │
│  │  ┌──────────────────┐  │ │
│  │  │  Storage Engine  │  │ │
│  │  └────────┬─────────┘  │ │
│  └───────────┼─────────────┘ │
└──────────────┼───────────────┘
               │
               ▼
        ┌──────────────┐
        │  Disk Files  │
        │ (RocksDB)    │
        └──────────────┘

Configuration¶

Rust API:

use heliosdb_lite::EmbeddedDatabase;

// Basic usage
let db = EmbeddedDatabase::new("./data.helio")?;

// With custom configuration
let config = Config {
    storage: StorageConfig {
        path: Some("./mydb".into()),
        memory_only: false,
        cache_size: 512 * 1024 * 1024, // 512 MB
        compression: CompressionType::Zstd,
        ..Default::default()
    },
    ..Default::default()
};
let db = EmbeddedDatabase::with_config("./mydb", config)?;

CLI:

# REPL with persistent storage
heliosdb-lite repl -d ./mydb

Use Cases¶

✅ Desktop applications (Electron, Tauri)
✅ Mobile apps (via FFI bindings)
✅ Edge devices and IoT
✅ Single-process microservices
✅ Command-line tools
✅ Embedded systems

Pros¶

Simple deployment (single process)
Zero network overhead
Low latency
No port conflicts
Easy to backup (just copy files)

Cons¶

Single-process access only
Cannot scale horizontally
No network access for monitoring tools

Performance Characteristics¶

Latency: 0.1-1ms (no network overhead)
Throughput: 10,000-50,000 ops/sec (depends on disk)
Memory Usage: Low (minimal caching)

Mode 2: Memory-Only Mode¶

Description¶

ACID-compliant RAM-only storage with no persistence. Data is lost when the process terminates.

Architecture¶

┌─────────────────────────────┐
│   Your Application          │
│  ┌────────────────────────┐ │
│  │ HeliosDB Lite (Memory) │ │
│  │  ┌──────────────────┐  │ │
│  │  │   RAM Storage    │  │ │
│  │  │  (Temporary Dir) │  │ │
│  │  └──────────────────┘  │ │
│  └────────────────────────┘ │
└─────────────────────────────┘
         (No disk writes)

Configuration¶

Rust API:

use heliosdb_lite::EmbeddedDatabase;

// Memory-only database
let db = EmbeddedDatabase::new_in_memory()?;

// Or with Config
let config = Config::in_memory();
let db = EmbeddedDatabase::with_config_in_memory(config)?;

CLI:

# REPL in memory-only mode
heliosdb-lite repl --memory

Use Cases¶

✅ Testing and CI/CD pipelines
✅ Temporary data processing
✅ Fast analytics on ephemeral data
✅ Development and debugging
✅ Session storage (short-lived)
✅ Cache with SQL interface

Pros¶

Extremely fast (no disk I/O)
No cleanup required
Reproducible test environments
Zero disk space usage

Cons¶

Data lost on shutdown
Limited by RAM size
Not suitable for production data

Performance Characteristics¶

Latency: 0.01-0.1ms (RAM-only)
Throughput: 100,000+ ops/sec
Memory Usage: High (all data in RAM)

Mode 3: Server Mode¶

Description¶

PostgreSQL-compatible network server with persistent storage. Enables multi-client access via PostgreSQL wire protocol.

Architecture¶

┌──────────┐  ┌──────────┐  ┌──────────┐
│  Client  │  │  Client  │  │  Client  │
│  (psql)  │  │  (App)   │  │  (pgAdmin)│
└────┬─────┘  └────┬─────┘  └────┬─────┘
     │             │              │
     └─────────────┼──────────────┘
                   │ PostgreSQL Wire Protocol
              ┌────▼─────────────────┐
              │  HeliosDB Server     │
              │  ┌────────────────┐  │
              │  │ Storage Engine │  │
              │  └────────┬───────┘  │
              └───────────┼──────────┘
                          │
                          ▼
                   ┌──────────────┐
                   │  Disk Files  │
                   └──────────────┘

Configuration¶

Config file (heliosdb.toml):

[storage]
path = "./heliosdb-data"
memory_only = false
cache_size = 134217728  # 128 MB (basic cache)

[server]
listen_addr = "127.0.0.1"
port = 5432
max_connections = 100
tls_enabled = false

CLI:

# Initialize database
heliosdb-lite init ./mydb

# Start server
heliosdb-lite start --port 5432 --data ./mydb

# Or with config file
heliosdb-lite start --config heliosdb.toml

Connect with clients:

# psql
psql postgresql://localhost:5432/mydb

# pgAdmin, DBeaver, etc.
# Connection string: postgresql://localhost:5432/mydb

Use Cases¶

✅ Web applications (multi-client)
✅ Microservices architecture
✅ PostgreSQL migration (95%+ compatible)
✅ Multi-language applications
✅ Remote database access
✅ Monitoring and admin tools

Pros¶

Multi-client support
PostgreSQL wire protocol compatibility
Works with existing PostgreSQL tools
Network-accessible
Familiar server model

Cons¶

Network latency overhead
Port management required
More complex deployment
Basic caching (not optimized)

Performance Characteristics¶

Latency: 1-10ms (network overhead)
Throughput: 5,000-20,000 ops/sec (network-bound)
Memory Usage: Moderate

Mode 4: Hybrid Mode (Server + Memory Cache) ⭐¶

Description¶

Recommended for production. Combines server mode with persistent storage AND configurable memory caching for optimal performance.

Architecture¶

┌──────────┐  ┌──────────┐  ┌──────────┐
│  Client  │  │  Client  │  │  Client  │
└────┬─────┘  └────┬─────┘  └────┬─────┘
     │             │              │
     └─────────────┼──────────────┘
                   │ PostgreSQL Wire Protocol
              ┌────▼──────────────────────┐
              │  HeliosDB Server (Hybrid) │
              │  ┌──────────────────────┐ │
              │  │   Memory Cache       │ │ ⭐
              │  │   (512MB - 4GB)      │ │
              │  ├──────────────────────┤ │
              │  │   Storage Engine     │ │
              │  └──────────┬───────────┘ │
              └─────────────┼─────────────┘
                            │
                            ▼
                     ┌──────────────┐
                     │  Disk Files  │
                     │  (RocksDB)   │
                     └──────────────┘

Why Hybrid Mode?¶

Hybrid mode provides: 1. Network Access: Multi-client support via PostgreSQL protocol 2. Persistent Storage: Data durability with RocksDB 3. Memory Cache: Configurable in-memory cache for hot data 4. Performance: Best of both worlds

This is the only mode that combines all these benefits.

Configuration¶

Recommended config (heliosdb.toml):

[storage]
# Persistent storage path
path = "./heliosdb-data"

# NOT memory-only (enable disk persistence)
memory_only = false

# Memory cache size (key configuration!)
# Adjust based on available RAM:
cache_size = 536870912      # 512 MB (for 4-8 GB RAM systems)
# cache_size = 1073741824   # 1 GB (for 8-16 GB RAM systems)
# cache_size = 2147483648   # 2 GB (for 16-32 GB RAM systems)
# cache_size = 4294967296   # 4 GB (for 32+ GB RAM systems)

# Enable compression for better cache efficiency
compression = "zstd"        # or "lz4" for faster compression

# Write-ahead log for crash recovery
wal_enabled = true

[server]
listen_addr = "127.0.0.1"   # Use "0.0.0.0" for external access
port = 5432
max_connections = 100

# Optional: Enable TLS for production
tls_enabled = false
# tls_cert_path = "/path/to/cert.pem"
# tls_key_path = "/path/to/key.pem"

[performance]
# Use all CPU cores
worker_threads = 8          # Adjust based on CPU count

# Enable SIMD acceleration
simd_enabled = true

# Enable parallel query execution
parallel_query = true

# Query timeout (seconds)
query_timeout_secs = 300

[encryption]
# Optional: Enable encryption at rest
enabled = false
algorithm = "Aes256Gcm"
key_source = { Environment = "HELIOSDB_ENCRYPTION_KEY" }

Start server:

# Using config file (recommended)
heliosdb-lite start --config heliosdb.toml

# Or with CLI flags
heliosdb-lite start \
  --port 5432 \
  --data ./heliosdb-data \
  --cache-size 536870912 \
  --max-connections 100

Cache Size Tuning¶

How to choose cache_size:

System RAM	Recommended cache_size	Percentage
4 GB	512 MB (536870912)	~12%
8 GB	1 GB (1073741824)	~12%
16 GB	2 GB (2147483648)	~12%
32 GB	4 GB (4294967296)	~12%
64 GB+	8 GB+ (8589934592)	~12-25%

Rule of thumb: Allocate 10-25% of available RAM to cache, leaving room for OS and application memory.

Monitoring cache efficiency:

-- Check cache hit rate (planned feature)
SELECT cache_hit_rate FROM system_stats;

Use Cases¶

✅ Production web applications (high traffic)
✅ High-performance microservices
✅ Multi-tenant SaaS applications
✅ Real-time analytics
✅ API backends with hot data
✅ Content management systems
✅ E-commerce platforms

Pros¶

✅ Best performance (memory cache + persistence)
✅ Multi-client support (network server)
✅ Data durability (persistent storage)
✅ Configurable (tune for your workload)
✅ Production-ready (all features enabled)
✅ Scalable (within single node)

Cons¶

⚠️ Higher memory usage (but configurable)
⚠️ More complex configuration (but flexible)
⚠️ Requires tuning for optimal performance

Performance Characteristics¶

Latency:
Cache hit: 0.5-2ms (fast)
Cache miss: 5-15ms (disk read + network)
Throughput: 20,000-100,000 ops/sec (depends on cache hit rate)
Memory Usage: Configurable (512MB - 8GB+)
Cache Hit Rate: 80-95% (for typical workloads)

Performance Optimization¶

1. Monitor your workload:

-- Identify hot tables/queries
EXPLAIN ANALYZE SELECT * FROM users WHERE active = true;

2. Tune cache size: - Start with 512 MB (conservative) - Monitor cache hit rate - Increase if hit rate < 80% - Decrease if memory pressure

3. Use compression:

# Zstd: Best compression (slower)
compression = "zstd"

# LZ4: Faster compression (larger cache footprint)
compression = "lz4"

4. Enable all performance features:

[performance]
simd_enabled = true        # SIMD acceleration
parallel_query = true      # Parallel execution
worker_threads = 8         # Use all cores

Deployment Comparison Matrix¶

By Use Case¶

Use Case	Recommended Mode	Why?
Desktop app	Embedded	Single-process, simple
Mobile app	Embedded	Offline-first, local storage
Testing/CI	Memory-Only	Fast, reproducible
Development	Memory-Only or Embedded	Quick iteration
Web app (low traffic)	Server	Multi-client, basic needs
Web app (production)	Hybrid ⭐	Performance + durability
Microservices	Hybrid ⭐	Multi-client, high performance
Analytics	Hybrid or Memory-Only	Large cache or all-in-memory
Edge computing	Embedded	Local-first, offline
PostgreSQL migration	Server or Hybrid	Wire protocol compatible

By Performance Needs¶

Requirement	Mode	Rationale
Lowest latency	Memory-Only	No disk I/O
Best throughput (single-client)	Memory-Only	RAM-only
Best throughput (multi-client)	Hybrid ⭐	Cache + parallel
Highest durability	Embedded or Hybrid	Persistent storage
Network access	Server or Hybrid	Wire protocol
Production-ready	Hybrid ⭐	All features

By Resource Constraints¶

Constraint	Mode	Notes
Low memory (< 2GB)	Embedded	Minimal caching
No disk space	Memory-Only	Temporary only
No network access	Embedded or Memory-Only	Local only
High memory (16GB+)	Hybrid ⭐	Large cache
Multi-core CPU	Hybrid ⭐	Parallel queries

Migration Between Modes¶

Embedded → Hybrid¶

Why migrate: - Need multi-client access - Want better performance - Scaling beyond single process

Steps: 1. Create config file:

[storage]
path = "./existing-data"  # Point to existing data
memory_only = false
cache_size = 1073741824   # 1 GB cache

[server]
port = 5432
max_connections = 100

Start server:

heliosdb-lite start --config heliosdb.toml

Update application:

// Old: Embedded
// let db = EmbeddedDatabase::new("./data")?;

// New: Connect via client
use postgres::Client;
let mut client = Client::connect(
    "postgresql://localhost:5432/mydb",
    NoTls
)?;

Memory-Only → Hybrid¶

Why migrate: - Need persistence - Moving from development to production

Steps: 1. Export schema and data:

-- In memory-only instance
\d  -- Note all tables
-- Export data (manually or via pg_dump equivalent)

Set up Hybrid mode:

# Create config (as shown above)
heliosdb-lite start --config heliosdb.toml

Import data:

-- Recreate schema
CREATE TABLE ...;

-- Import data
INSERT INTO ...;

Server → Hybrid¶

Already there! Just tune the cache:

[storage]
# Increase cache size
cache_size = 2147483648  # 2 GB (was 128 MB)

Restart server and you're in optimized Hybrid mode.

Configuration Best Practices¶

Development¶

# Quick iteration, minimal resources
[storage]
path = "./dev-data"
memory_only = false
cache_size = 134217728    # 128 MB (small cache)
compression = "lz4"       # Fast compression

[server]
listen_addr = "127.0.0.1" # Localhost only
port = 5432
max_connections = 10      # Few connections

Staging¶

# Production-like, but smaller
[storage]
path = "./staging-data"
memory_only = false
cache_size = 536870912    # 512 MB
compression = "zstd"      # Production compression

[server]
listen_addr = "0.0.0.0"   # Network access
port = 5432
max_connections = 50

[encryption]
enabled = true            # Test encryption

Production (Hybrid) ⭐¶

# Optimized for performance and durability
[storage]
path = "/var/lib/heliosdb"
memory_only = false
cache_size = 2147483648   # 2 GB (tune based on workload)
compression = "zstd"
wal_enabled = true

[server]
listen_addr = "0.0.0.0"
port = 5432
max_connections = 100
tls_enabled = true
tls_cert_path = "/etc/heliosdb/cert.pem"
tls_key_path = "/etc/heliosdb/key.pem"

[performance]
worker_threads = 16       # Use all cores
simd_enabled = true
parallel_query = true
query_timeout_secs = 300

[encryption]
enabled = true
algorithm = "Aes256Gcm"
key_source = { Environment = "HELIOSDB_ENCRYPTION_KEY" }

[audit]
enabled = true
log_path = "/var/log/heliosdb/audit.log"

Monitoring and Diagnostics¶

Check Current Mode¶

From logs:

# Server logs show mode at startup
heliosdb-lite start --config heliosdb.toml

# Output:
# [INFO] Starting HeliosDB Lite in Hybrid Mode
# [INFO] Cache size: 512 MB
# [INFO] Persistent storage: /var/lib/heliosdb
# [INFO] Server listening on 0.0.0.0:5432

From configuration:

-- Query configuration (planned feature)
SELECT * FROM system_config;

Performance Metrics¶

Key metrics to monitor: - Cache hit rate (target: > 80%) - Query latency (p50, p95, p99) - Disk I/O (lower is better) - Memory usage (should be < cache_size + overhead) - Connection count (should be < max_connections)

Summary¶

Mode	Key Feature	Best For
Embedded	Simple, local	Desktop apps, edge devices
Memory-Only	Fast, ephemeral	Testing, temporary data
Server	Network, persistent	Basic multi-client needs
Hybrid ⭐	All features	Production applications

Recommendation for most users: Start with Hybrid mode for production. It provides the best balance of performance, durability, and scalability.

Additional Resources¶

Questions?¶

GitHub Issues: https://github.com/heliosdb/heliosdb/issues
Discussions: https://github.com/heliosdb/heliosdb/discussions
Discord: https://discord.gg/heliosdb (coming soon)