Skip to content
HeliosDB HeliosDB Nano

HeliosDB Nano Advanced Feature User Guide

HeliosDB Nano Advanced Feature User Guide

Table of Contents

  1. Introduction
  2. Product Quantization for Vectors
  3. Quantized HNSW Index
  4. Database Branching
  5. Time-Travel Queries
  6. Materialized Views
  7. System Views
  8. Performance Tuning
  9. Best Practices
  10. Troubleshooting

Introduction

HeliosDB Nano includes advanced features for vector compression, zero-downtime workflows, point-in-time access, and materialized view acceleration:

  • 8-16x vector compression with Product Quantization
  • Git-like database branching for zero-downtime workflows
  • Time-travel queries for point-in-time data access
  • Intelligent materialized views with auto-refresh

Feature Map

FeatureUsageDescription
Product QuantizationRust API and vector indexes8-16x vector memory compression
Quantized HNSWHNSW indexes with PQMemory-efficient vector search
SQL BranchingCREATE, DROP, and MERGE DATABASE BRANCHCopy-on-write database workflows
Time-TravelAS OF TIMESTAMP, AS OF TRANSACTION, AS OF SCNPoint-in-time queries
Materialized ViewsCREATE, REFRESH, and DROP MATERIALIZED VIEWStored query results with refresh controls

Product Quantization

Overview

Product Quantization (PQ) compresses high-dimensional vectors by 8-16x with minimal accuracy loss (95-98% recall@10).

How It Works:

  1. Split vector into M sub-vectors
  2. Quantize each sub-vector independently
  3. Store codes instead of full values

Memory Savings:

  • 768-dim vector: 3,072 bytes → 8 bytes (384x compression)
  • 512-dim vector: 2,048 bytes → 8 bytes (256x compression)
  • 256-dim vector: 1,024 bytes → 4 bytes (256x compression)

Quick Start

use heliosdb_nano::vector::quantization::{
    ProductQuantizer,
    ProductQuantizerConfig,
};


// 1. Configure PQ for your vector dimension
let config = ProductQuantizerConfig::default_for_dimension(768)?;


// 2. Train on your data (10K+ vectors recommended)
let pq = ProductQuantizer::train(config, &training_vectors)?;


// 3. Encode vectors for storage
let quantized = pq.encode(&vector)?;
println!("Compressed from 3KB to {} bytes!", quantized.memory_size());


// 4. Search efficiently
let query = &my_query_vector;
let distance_table = pq.precompute_distance_table(query)?;
let distance = pq.compute_distance_with_table(&distance_table, &quantized)?;

Configuration Options

pub struct ProductQuantizerConfig {
    /// Number of sub-quantizers (M)
    /// Default: 8 for 768-dim, 4 for 256-dim
    pub num_subquantizers: usize,


    /// Centroids per sub-quantizer (K)
    /// Default: 256 (fits in u8)
    pub num_centroids: usize,


    /// Vector dimension (D)
    /// Must be divisible by num_subquantizers
    pub dimension: usize,


    /// K-means iterations for training
    /// Default: 25
    pub training_iterations: usize,


    /// Minimum training samples
    /// Default: 10,000
    pub min_training_samples: usize,
}

Performance Tips

Training:

  • Use 10,000+ vectors for best quality
  • More iterations = better codebook (diminishing returns after 25)
  • Training is one-time cost, amortize over all searches

Search:

  • Always use precompute_distance_table() for batch queries
  • Precompute once, search many times (100-1000x faster)
  • For single query, direct compute_distance() is fine

Memory:

  • Codebook size: M × K × (D/M) × 4 bytes
    • Example (768-dim): 8 × 256 × 96 × 4 = 768KB
  • Per-vector: M bytes
    • Example (M=8): 8 bytes

Quantized HNSW Index

Overview

Combines HNSW graph structure with Product Quantization for memory-efficient approximate nearest neighbor search.

Benefits:

  • 8-16x memory reduction vs standard HNSW
  • 95-98% search accuracy maintained
  • Fast insertion and search

Quick Start

use heliosdb_nano::vector::{
    QuantizedHnswIndex,
    QuantizedHnswConfig,
};


// 1. Configure index
let config = QuantizedHnswConfig::default_for_dimension(768)?;


// 2. Train index (trains PQ internally)
let index = QuantizedHnswIndex::train(config, &training_vectors)?;


// 3. Insert vectors
for (id, vector) in vectors.iter().enumerate() {
    index.insert(id as u64, vector)?;
}


// 4. Search for nearest neighbors
let results = index.search(&query, 10)?;  // Top 10
for (id, distance) in results {
    println!("Vector {} is {} away", id, distance);
}


// 5. Check memory savings
let stats = index.memory_stats();
println!("Compression: {:.1}x", stats.compression_ratio);
println!("{}", stats.format());

Configuration Options

pub struct QuantizedHnswConfig {
    /// HNSW graph connectivity
    pub max_connections: usize,      // Default: 16


    /// Build quality (larger = better accuracy)
    pub ef_construction: usize,      // Default: 200


    /// Search quality (larger = better recall)
    pub ef_search: usize,            // Default: 200


    /// Vector dimension
    pub dimension: usize,


    /// Distance metric (L2, Cosine, etc.)
    pub distance_metric: DistanceMetric,


    /// Product Quantization config
    pub pq_config: ProductQuantizerConfig,


    /// Use PQ for storage (vs keeping originals)
    pub use_pq_storage: bool,        // Default: true
}

Memory Usage Example

Scenario: 100,000 vectors (768-dim)


Without PQ (standard HNSW):
- Vectors: 100K × 768 × 4 = 307 MB
- Graph: ~50 MB (estimate)
- Total: ~357 MB


With PQ (Quantized HNSW):
- Quantized vectors: 100K × 8 = 800 KB
- Codebook: 768 KB
- Graph: ~50 MB
- Total: ~51.6 MB


Compression: 6.9x

Database Branching

Overview

Create isolated database branches for testing, development, or rollback scenarios.

SQL Syntax

Create Branch

-- Create branch from current state
CREATE DATABASE BRANCH staging FROM CURRENT AS OF NOW;


-- Create branch from specific point in time
CREATE DATABASE BRANCH test FROM main
  AS OF TIMESTAMP '2025-11-15 06:00:00';


-- Create branch from transaction
CREATE DATABASE BRANCH backup FROM CURRENT
  AS OF TRANSACTION 987654;


-- With options
CREATE DATABASE BRANCH prod_copy FROM main AS OF NOW
WITH (
    replication_factor = 3,
    region = 'us-west-2'
);

Drop Branch

-- Drop branch (fails if doesn't exist)
DROP DATABASE BRANCH staging;


-- Drop if exists
DROP DATABASE BRANCH IF EXISTS staging;

Merge Branch

-- Merge with conflict resolution
MERGE DATABASE BRANCH staging INTO main
WITH (
    conflict_resolution = 'branch_wins',
    delete_branch_after = true
);


-- Conflict resolution options:
--   'branch_wins' - Source branch changes win
--   'target_wins' - Target branch changes win
--   'fail' - Fail on any conflict

Use Cases

Development Workflow:

-- 1. Create dev branch
CREATE DATABASE BRANCH dev FROM main AS OF NOW;


-- 2. Make changes in dev branch
-- ... development work ...


-- 3. Test thoroughly
-- ... testing ...


-- 4. Merge back to main
MERGE DATABASE BRANCH dev INTO main
WITH (conflict_resolution = 'branch_wins');


-- 5. Clean up
DROP DATABASE BRANCH dev;

Blue-Green Deployment:

-- 1. Create green deployment
CREATE DATABASE BRANCH green FROM blue AS OF NOW;


-- 2. Deploy new version to green
-- ... deployment ...


-- 3. Switch traffic to green (application-level)


-- 4. If success, make green the new blue
MERGE DATABASE BRANCH green INTO blue;


-- 5. If rollback needed, just switch back to blue

Time-Travel Queries

Overview

Query historical data as it existed at a specific point in time.

SQL Syntax

AS OF TIMESTAMP

-- Query data as of specific timestamp
SELECT * FROM orders
AS OF TIMESTAMP '2025-11-15 06:00:00'
WHERE user_id = 123;


-- Compare current vs historical
SELECT
    current.total as current_total,
    historical.total as yesterday_total
FROM orders current
JOIN orders AS OF TIMESTAMP '2025-11-17 00:00:00' historical
    ON current.order_id = historical.order_id;

AS OF TRANSACTION

-- Query as of transaction ID
SELECT * FROM orders
AS OF TRANSACTION 987654
WHERE status = 'pending';

AS OF SCN

-- Query as of System Change Number
SELECT * FROM orders
AS OF SCN 123456789;

Use Cases

Audit and Compliance:

-- Show what user saw at time of complaint
SELECT * FROM product_catalog
AS OF TIMESTAMP '2025-11-15 14:23:00'
WHERE product_id = 'WIDGET-123';

Data Recovery:

-- Find accidentally deleted rows
SELECT * FROM orders
AS OF TIMESTAMP '2025-11-18 00:00:00'
WHERE order_id NOT IN (SELECT order_id FROM orders);

Debugging:

-- Compare data before/after bug
SELECT
    before.*,
    after.*
FROM orders AS OF TIMESTAMP '2025-11-17 23:59:59' before
FULL OUTER JOIN orders after USING (order_id)
WHERE before.total != after.total;

Materialized Views

Overview

Precomputed query results that can auto-refresh with CPU awareness.

SQL Syntax

Create Materialized View

CREATE MATERIALIZED VIEW user_stats AS
SELECT
    user_id,
    COUNT(*) as order_count,
    SUM(total) as total_revenue,
    AVG(total) as avg_order_value
FROM orders
GROUP BY user_id
WITH (
    auto_refresh = true,              -- Enable auto-refresh
    max_cpu_percent = 15,              -- CPU limit (0-100)
    threshold_table_size = '1GB',      -- Min base table size
    threshold_dml_rate = 100,          -- DMLs/sec trigger
    lazy_update = true,                -- Defer during high load
    lazy_catchup_window = '1 hour'    -- Max staleness allowed
);

Refresh Materialized View

-- Manual refresh
REFRESH MATERIALIZED VIEW user_stats;


-- Concurrent refresh (non-blocking)
REFRESH MATERIALIZED VIEW CONCURRENTLY user_stats;

Drop Materialized View

-- Drop view
DROP MATERIALIZED VIEW user_stats;


-- Drop if exists
DROP MATERIALIZED VIEW IF EXISTS user_stats;

Configuration Options

OptionTypeDefaultDescription
auto_refreshboolfalseEnable background auto-refresh
max_cpu_percentfloat15.0Maximum CPU usage (0-100)
threshold_table_sizestring’0’Minimum base table size for refresh
threshold_dml_rateint0DMLs/sec to trigger refresh
lazy_updateboolfalseDefer refresh during high load
lazy_catchup_windowstring’1h’Maximum staleness allowed

Use Cases

Analytics Dashboard:

CREATE MATERIALIZED VIEW daily_metrics AS
SELECT
    DATE(created_at) as date,
    COUNT(*) as orders,
    SUM(total) as revenue
FROM orders
GROUP BY DATE(created_at)
WITH (
    auto_refresh = true,
    max_cpu_percent = 10,
    threshold_dml_rate = 50
);

Real-Time Aggregations:

CREATE MATERIALIZED VIEW product_popularity AS
SELECT
    product_id,
    COUNT(*) as view_count,
    COUNT(DISTINCT user_id) as unique_viewers
FROM product_views
GROUP BY product_id
WITH (
    auto_refresh = true,
    max_cpu_percent = 20,
    lazy_catchup_window = '5 minutes'
);

System Views

Overview

Query system metadata and statistics.

Available System Views

pg_database_branches()

-- List all branches
SELECT * FROM pg_database_branches();


-- Output columns:
--   branch_name, parent_branch, created_at, size_bytes

pg_compare_branches()

-- Compare two branches
SELECT * FROM pg_compare_branches('main', 'staging');


-- Output columns:
--   table_name, rows_in_main, rows_in_staging, difference

pg_mv_staleness()

-- Check materialized view freshness
SELECT * FROM pg_mv_staleness();


-- Output columns:
--   view_name, last_refresh, staleness_sec, auto_refresh

pg_mv_cpu_usage()

-- Monitor MV CPU usage
SELECT * FROM pg_mv_cpu_usage();


-- Output columns:
--   view_name, max_cpu_percent, current_cpu_percent, status

pg_vector_index_stats()

-- Vector index statistics
SELECT * FROM pg_vector_index_stats();


-- Output columns:
--   index_name, num_vectors, memory_bytes, compression_ratio

Performance Tuning

Product Quantization Tuning

Accuracy vs Compression Tradeoff:

// Higher accuracy, less compression
let config = ProductQuantizerConfig {
    num_subquantizers: 4,   // Fewer = better accuracy
    num_centroids: 256,
    dimension: 768,
    ..Default::default()
};


// Higher compression, lower accuracy
let config = ProductQuantizerConfig {
    num_subquantizers: 16,  // More = better compression
    num_centroids: 256,
    dimension: 768,
    ..Default::default()
};

Training Quality:

  • More training samples = better codebook (10K minimum, 100K+ ideal)
  • More iterations = better convergence (25 recommended, 50 for critical applications)

HNSW Tuning

Build Quality:

let config = QuantizedHnswConfig {
    ef_construction: 200,   // Default
    // For better accuracy: 400-800
    // For faster build: 100-150
    ..Default::default()
};

Search Quality:

let config = QuantizedHnswConfig {
    ef_search: 200,         // Default
    // For better recall: 400-1000
    // For faster search: 50-100
    ..Default::default()
};

Materialized View Tuning

CPU Budget:

-- Conservative (low impact)
WITH (max_cpu_percent = 10)


-- Balanced
WITH (max_cpu_percent = 15)


-- Aggressive (faster refresh)
WITH (max_cpu_percent = 30)

Refresh Triggers:

-- Refresh frequently
WITH (
    threshold_dml_rate = 10,          -- Low threshold
    lazy_catchup_window = '1 minute'  -- Short window
)


-- Refresh infrequently
WITH (
    threshold_dml_rate = 1000,        -- High threshold
    lazy_catchup_window = '1 hour'    -- Long window
)

Best Practices

Vector Workloads

  1. Train on Representative Data

    • Use production-like vectors for training
    • Include edge cases and outliers
    • Retrain periodically as data distribution changes

  2. Batch Operations

    • Always use precompute_distance_table() for multiple searches
    • Encode vectors in batches for better throughput
    • Amortize training cost over many vectors

  3. Monitor Accuracy

    • Periodically check recall@k on validation set
    • Alert if accuracy drops below threshold
    • Consider retraining if drift detected

Database Branching

  1. Branch Naming Convention

    <purpose>-<timestamp>
    Examples:
    - dev-20251118
    - staging-v2.0
    - backup-before-migration

  2. Branch Lifecycle

    • Create → Work → Test → Merge → Delete
    • Don’t accumulate stale branches
    • Document branch purpose in metadata

  3. Merge Conflicts

    • Use branch_wins for development → production
    • Use target_wins for rollbacks
    • Use fail for manual conflict resolution

Materialized Views

  1. Start Conservative

    -- Begin with safe settings
    WITH (
        max_cpu_percent = 10,
        lazy_update = true
    )
    

    -- Gradually increase if needed

  2. Monitor Performance

    -- Check regularly
    SELECT * FROM pg_mv_staleness();
    SELECT * FROM pg_mv_cpu_usage();

  3. Index the MV

    -- Create indexes on frequently queried columns
    CREATE INDEX idx_user_stats_revenue
    ON user_stats(total_revenue DESC);

Troubleshooting

Product Quantization Issues

Problem: Low search accuracy (recall <90%)

Solutions:

  • Use more training samples (100K+ recommended)
  • Reduce number of sub-quantizers (try M=4 instead of M=8)
  • Increase training iterations (try 50 instead of 25)
  • Check training data quality (outliers, distribution)

Problem: Slow training

Solutions:

  • Reduce training set size (10K is minimum)
  • Reduce training iterations
  • Use smaller num_centroids (128 instead of 256)

Problem: Out of memory during training

Solutions:

  • Train in batches
  • Reduce num_centroids
  • Use dimensionality reduction first

HNSW Issues

Problem: Slow insertion

Solutions:

  • Reduce ef_construction
  • Increase max_connections (paradoxically can be faster)
  • Batch insertions

Problem: Low search recall

Solutions:

  • Increase ef_search
  • Retrain with more representative data
  • Check PQ configuration

Materialized View Issues

Problem: MV is stale

Solutions:

-- Check staleness
SELECT * FROM pg_mv_staleness()
WHERE view_name = 'my_view';


-- Manual refresh if needed
REFRESH MATERIALIZED VIEW my_view;


-- Adjust auto-refresh settings
ALTER MATERIALIZED VIEW my_view
SET (threshold_dml_rate = 50);  -- Lower threshold

Problem: High CPU usage

Solutions:

-- Lower CPU budget
ALTER MATERIALIZED VIEW my_view
SET (max_cpu_percent = 10);


-- Enable lazy updates
ALTER MATERIALIZED VIEW my_view
SET (lazy_update = true);

Migration Guide

From v1.x to v2.0

Vector Indexes:

-- Old (v1.x): Standard HNSW
CREATE INDEX vec_idx ON docs USING hnsw (embedding);


-- New (v2.0): Quantized HNSW
CREATE INDEX vec_idx ON docs
USING hnsw (embedding vector_cosine_ops)
WITH (quantization = 'product', pq_subquantizers = 8);

No Breaking Changes:

  • All v1.x queries continue to work
  • Vector indexes auto-upgraded to use PQ
  • Existing data preserved

Getting Help

Documentation:

  • Full API docs: docs/api/
  • Examples: examples/
  • Benchmarks: benches/

Support:

Contributing:

  • Contribution guide: CONTRIBUTING.md
  • Code of conduct: CODE_OF_CONDUCT.md

User Guide Version: 1.0 Applies to: HeliosDB Nano v2.0.0-beta and later