HeliosDB Nano Production Deployment Guide

HeliosDB Nano Production Deployment Guide

---

Table of Contents

1. Architecture
2. System Requirements
3. Installation
4. Configuration
5. Docker Deployment
6. Kubernetes Deployment
7. Performance Tuning
8. High Availability
9. Monitoring & Operations
10. Troubleshooting
11. Upgrade Procedures

---

Architecture

Embedded Database Architecture

HeliosDB Nano is an embedded database that runs within your application process:

┌─────────────────────────────────────────┐
│          Your Application               │
│  ┌────────────────────────────────────┐ │
│  │    HeliosDB Nano (Embedded)       │ │
│  │  ┌──────────────────────────────┐ │ │
│  │  │  SQL Query Processor         │ │ │
│  │  ├──────────────────────────────┤ │ │
│  │  │  Storage Engine              │ │ │
│  │  ├──────────────────────────────┤ │ │
│  │  │  Vector Index (HNSW)         │ │ │
│  │  ├──────────────────────────────┤ │ │
│  │  │  Compression (ALP/FSST)      │ │ │
│  │  ├──────────────────────────────┤ │ │
│  │  │  MVCC Transaction Manager    │ │ │
│  │  └──────────────────────────────┘ │ │
│  └────────────────────────────────────┘ │
└─────────────────────────────────────────┘
        ↓
    Data Files
    (./data/)

Key Characteristics

• In-Process: No separate server process
• Zero Network Overhead: Direct memory access
• Shared Lifecycle: Database lifetime = application lifetime
• Single Writer: One application instance manages database
• Multiple Readers: Read-only replicas supported (via snapshots)

---

System Requirements

Minimum Requirements

Component	Requirement
CPU	2 cores minimum
Memory	4 GB minimum
Disk	10 GB minimum
OS	Linux, macOS, Windows
Rust	1.70+ (for building from source)

Recommended Production Setup

Component	Recommendation
CPU	8+ cores
Memory	16+ GB
Disk	100+ GB SSD
OS	Ubuntu 22.04 LTS / CentOS 8+
Filesystem	ext4 or XFS with noatime
Network	Gigabit Ethernet

Disk Space Calculation

Disk Space = Data Size × Compression Factor + WAL Log Space
           = (1 TB data × 0.3 compression) + 50 GB WAL
           = 350 GB

---

Installation

1. From Binary Release

# Download latest release
VERSION=3.0.0
wget https://github.com/dimensigon/HDB-HeliosDB-Nano/releases/download/v${VERSION}/heliosdb-nano-${VERSION}-x86_64-linux.tar.gz

# Extract
tar xzf heliosdb-nano-${VERSION}-x86_64-linux.tar.gz
sudo mv heliosdb-nano /usr/local/bin/

# Verify
heliosdb-nano --version
# Output: heliosdb-nano 3.0.0

2. From Rust Crates

cargo install heliosdb-nano --version 3.0.0

3. From Docker

docker pull heliosdb/heliosdb-nano:latest

4. From Source

git clone https://github.com/dimensigon/HDB-HeliosDB-Nano.git
cd heliosdb-nano

cargo build --release
sudo cp target/release/heliosdb-nano /usr/local/bin/

---

Configuration

1. Environment Variables

# Data directory
export HELIOSDB_DATA_DIR=/data/heliosdb
export HELIOSDB_LOG_LEVEL=info
export HELIOSDB_MAX_CONNECTIONS=1000
export HELIOSDB_QUERY_TIMEOUT=30000

# Performance tuning
export HELIOSDB_VECTOR_CACHE_SIZE=1000000
export HELIOSDB_COMPRESSION_ENABLED=true
export HELIOSDB_WAL_ENABLED=true

2. Configuration File

config.yaml

database:
  data_dir: /data/heliosdb
  log_level: info

performance:
  max_connections: 1000
  query_timeout_ms: 30000
  vector_cache_size: 1000000
  compression_enabled: true
  compression_algorithm: alp  # or fsst

storage:
  wal_enabled: true
  wal_sync_mode: fsync  # or async
  checkpoint_interval_secs: 3600
  snapshot_interval_secs: 600

api:
  rate_limit_max_requests: 10000
  rate_limit_window_secs: 60
  rate_limit_burst_capacity: 100

security:
  tls_enabled: true
  tls_cert: /etc/heliosdb/cert.pem
  tls_key: /etc/heliosdb/key.pem

3. Runtime Configuration

use heliosdb_nano::{HeliosDB, Config};
use std::time::Duration;

let mut config = Config::default();
config.data_dir = "/data/heliosdb".into();
config.max_connections = 1000;
config.query_timeout = Duration::from_secs(30);
config.compression_enabled = true;
config.wal_enabled = true;

let db = HeliosDB::new_with_config(config)?;

---

Docker Deployment

1. Basic Docker Setup

# Dockerfile
FROM heliosdb/heliosdb-nano:3.0.0

RUN apt-get update && apt-get install -y \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Create non-root user
RUN useradd -m -u 1000 heliosdb
WORKDIR /app

# Copy application code
COPY . .

# Build application
RUN cargo build --release

# Switch to non-root user
USER heliosdb:heliosdb

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8080/health || exit 1

EXPOSE 8080
CMD ["./target/release/app"]

2. Docker Compose

version: '3.8'

services:
  heliosdb-app:
    build: .
    container_name: heliosdb-prod
    ports:
      - "8080:8080"
    volumes:
      - heliosdb_data:/data/heliosdb
      - ./config.yaml:/etc/heliosdb/config.yaml:ro
    environment:
      - HELIOSDB_DATA_DIR=/data/heliosdb
      - HELIOSDB_LOG_LEVEL=info
    restart: unless-stopped
    networks:
      - heliosdb_network

  # Optional: Backup sidecar
  backup:
    image: heliosdb/heliosdb-nano:3.0.0
    container_name: heliosdb-backup
    volumes:
      - heliosdb_data:/data/heliosdb:ro
      - ./backups:/backups
    environment:
      - BACKUP_SCHEDULE="0 2 * * *"  # Daily at 2 AM
    restart: unless-stopped
    networks:
      - heliosdb_network

volumes:
  heliosdb_data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/data/heliosdb

networks:
  heliosdb_network:
    driver: bridge

---

Kubernetes Deployment

1. StatefulSet Deployment

apiVersion: v1
kind: ConfigMap
metadata:
  name: heliosdb-config
  namespace: default
data:
  config.yaml: |
    database:
      data_dir: /data/heliosdb
      log_level: info
    performance:
      max_connections: 1000

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: heliosdb-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 100Gi

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: heliosdb
  namespace: default
spec:
  serviceName: heliosdb
  replicas: 1
  selector:
    matchLabels:
      app: heliosdb
  template:
    metadata:
      labels:
        app: heliosdb
    spec:
      serviceAccountName: heliosdb
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
      - name: heliosdb
        image: heliosdb/heliosdb-nano:3.0.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: api
          protocol: TCP
        env:
        - name: HELIOSDB_DATA_DIR
          value: /data/heliosdb
        - name: HELIOSDB_LOG_LEVEL
          value: info
        volumeMounts:
        - name: data
          mountPath: /data/heliosdb
        - name: config
          mountPath: /etc/heliosdb
          readOnly: true
        resources:
          requests:
            cpu: 2
            memory: 4Gi
          limits:
            cpu: 8
            memory: 16Gi
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
      volumes:
      - name: config
        configMap:
          name: heliosdb-config
      - name: data
        persistentVolumeClaim:
          claimName: heliosdb-pvc

---
apiVersion: v1
kind: Service
metadata:
  name: heliosdb
  namespace: default
spec:
  clusterIP: None
  selector:
    app: heliosdb
  ports:
  - port: 8080
    targetPort: 8080
    name: api

2. Horizontal Pod Autoscaling (Read Replicas)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: heliosdb-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: heliosdb-reader  # Read replicas only
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

---

Performance Tuning

1. Connection Pool Sizing

// Calculate optimal pool size
let cpu_cores = num_cpus::get();
let max_pool_size = cpu_cores * 4;  // 4x cores rule of thumb

let db = HeliosDB::new_with_config(Config {
    max_connections: max_pool_size,
    ...
})?;

2. Memory Configuration

# JVM-like tuning for vector indexing
performance:
  # Cache for vector searches
  vector_cache_size: 10000000  # 10M vectors max

  # Buffer pool for storage
  buffer_pool_size_mb: 4096    # 4GB buffer

  # Query optimizer budget
  max_query_complexity: 1000

3. Compression Tuning

// Benchmark compression trade-off
let compression_config = CompressionConfig {
    enabled: true,
    algorithm: CompressionAlgorithm::ALP,
    compression_level: 6,  // 1-9, higher = better ratio
    chunk_size: 65536,      // bytes
};

4. Storage Engine Tuning

# Filesystem optimization for database
# Add to /etc/fstab
/dev/nvme0n1p1 /data/heliosdb ext4 noatime,nobarrier,data=writeback 0 0

# Mount with performance options
mount -t ext4 -o noatime,nobarrier,data=writeback /dev/nvme0n1p1 /data/heliosdb

---

High Availability

1. Backup Strategy

#!/bin/bash
# Daily backup script
BACKUP_DIR=/backups/heliosdb
RETENTION_DAYS=30
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Create backup
tar czf $BACKUP_DIR/heliosdb_$TIMESTAMP.tar.gz /data/heliosdb/

# Encrypt backup
gpg --symmetric --cipher-algo AES256 $BACKUP_DIR/heliosdb_$TIMESTAMP.tar.gz

# Remove old backups
find $BACKUP_DIR -name "heliosdb_*.tar.gz.gpg" -mtime +$RETENTION_DAYS -delete

# Verify backup integrity
tar tzf $BACKUP_DIR/heliosdb_$TIMESTAMP.tar.gz > /dev/null && echo "Backup OK"

2. Replication (Read Replicas)

// Primary instance (write)
let primary_db = HeliosDB::new("/data/heliosdb/primary")?;

// Create read-only snapshot for replica
let snapshot = primary_db.create_snapshot()?;
snapshot.save_to("/backups/snapshot-latest")?;

// Replica instance (read-only)
let replica_db = HeliosDB::open_readonly("/backups/snapshot-latest")?;

3. Disaster Recovery

# RTO: Recovery Time Objective = 5 minutes
# RPO: Recovery Point Objective = 1 hour

recovery_procedure:
  step1:
    description: "Detect failure"
    alert: "heartbeat_missed"
    timeout: 30s

  step2:
    description: "Failover to replica"
    procedure: "promote_replica_read_write"
    timeout: 120s

  step3:
    description: "Restore from backup"
    location: "/backups/snapshot-latest"
    timeout: 300s

  rto: 300s  # 5 minutes
  rpo: 3600s # 1 hour

---

Monitoring & Operations

1. Key Metrics to Monitor

metrics:
  availability:
    - database_up: bool (0=down, 1=up)
    - api_health: bool (0=unhealthy, 1=healthy)

  performance:
    - query_latency_p50: milliseconds
    - query_latency_p99: milliseconds
    - queries_per_second: rate
    - connection_count: gauge

  storage:
    - disk_usage_bytes: gauge
    - disk_free_bytes: gauge
    - wal_size_bytes: gauge

  errors:
    - error_rate: rate
    - auth_failures: counter
    - rate_limit_exceeded: counter

2. Prometheus Scrape Config

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'heliosdb'
    static_configs:
      - targets: ['localhost:8080']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

3. Alerting Rules

groups:
- name: heliosdb_alerts
  rules:
  - alert: HeliosDBDown
    expr: database_up == 0
    for: 1m
    annotations:
      summary: "HeliosDB is down"

  - alert: HighErrorRate
    expr: error_rate > 0.05
    for: 5m
    annotations:
      summary: "High error rate: {{ $value }}"

  - alert: LowDiskSpace
    expr: disk_free_bytes < 10737418240  # 10GB
    for: 5m
    annotations:
      summary: "Low disk space: {{ $value }} bytes remaining"

---

Troubleshooting

Issue: High Memory Usage

# Check memory usage
ps aux | grep heliosdb

# Solution: Reduce vector_cache_size
export HELIOSDB_VECTOR_CACHE_SIZE=5000000

# Monitor memory
watch -n 1 'ps aux | grep heliosdb'

Issue: Slow Queries

-- Enable query logging
SELECT * FROM system.query_log ORDER BY execution_time DESC LIMIT 10;

-- Check index usage
SELECT table_name, index_name, usage_count FROM system.index_stats;

-- Create missing indexes
CREATE INDEX idx_user_id ON users(user_id);

Issue: Disk Space Growing Rapidly

# Check WAL size
du -sh /data/heliosdb/wal/

# Solution: Force checkpoint
heliosdb-nano --checkpoint /data/heliosdb

# Or restart to trigger checkpoint
systemctl restart heliosdb-app

---

Upgrade Procedures

1. In-Place Upgrade

# 1. Backup current database
cp -r /data/heliosdb /data/heliosdb-backup-3.0.0

# 2. Stop application
systemctl stop heliosdb-app

# 3. Update binary
wget https://github.com/dimensigon/HDB-HeliosDB-Nano/releases/download/v3.0.1/heliosdb-nano-x86_64-linux.tar.gz
tar xzf heliosdb-nano-x86_64-linux.tar.gz
sudo mv heliosdb-nano /usr/local/bin/

# 4. Verify version
heliosdb-nano --version

# 5. Start application
systemctl start heliosdb-app

# 6. Verify functionality
curl http://localhost:8080/health

2. Blue-Green Deployment

# Blue (current)
- deployment: heliosdb-blue
  version: 3.0.0
  replicas: 3

# Green (new)
- deployment: heliosdb-green
  version: 3.0.1
  replicas: 3

# Switch traffic
kubectl patch service heliosdb -p '{"spec":{"selector":{"version":"3.0.1"}}}'

3. Canary Deployment

- deployment: heliosdb-stable
  version: 3.0.0
  replicas: 9

- deployment: heliosdb-canary
  version: 3.0.1
  replicas: 1

# Monitor canary metrics
# If stable, promote: replicas 8 + 2
# If issues: rollback

---

Summary

Aspect	Recommendation
Deployment	Kubernetes StatefulSet or Docker
Storage	SSD with 3.5x data overhead
Backup	Daily encrypted backups, 30-day retention
Monitoring	Prometheus + Grafana
Alerting	CPU >80%, Memory >90%, Disk <10%
Upgrades	Blue-green or canary deployments
RTO	5 minutes (failover + restore)
RPO	1 hour (snapshot frequency)