Skip to content
HeliosDB HeliosDB Full

LSM Tree Tuning Guide for HeliosDB Storage Engine

LSM Tree Tuning Guide for HeliosDB Storage Engine

Overview

The HeliosDB storage engine uses an LSM (Log-Structured Merge) tree architecture with adaptive tuning capabilities. This guide explains how to configure and optimize the storage engine for different workload patterns.

Table of Contents

  1. Architecture Overview
  2. Adaptive Tuning
  3. Configuration Parameters
  4. Workload-Specific Tuning
  5. Performance Metrics
  6. Best Practices

Architecture Overview

LSM Tree Components

Write Path:
  Client → Commit Log → Memtable → SSTable (L0) → ... → SSTable (Ln)


Read Path:
  Client → Memtable → Immutable Memtables → SSTables (newest → oldest)

Key Components:

  1. Commit Log (WAL): Durable write-ahead log for crash recovery
  2. Memtable: In-memory sorted data structure (Skip List)
  3. Immutable Memtables: Memtables being flushed to disk
  4. SSTables: Sorted String Tables on disk, organized in levels
  5. Bloom Filters: Probabilistic data structure for fast negative lookups
  6. Block Cache: LRU cache for frequently accessed data blocks

Compaction Strategies

1. Leveled Compaction (LCS)

  • Best for: Read-heavy workloads, point lookups
  • Write amplification: ~10x
  • Space amplification: ~1.1x
  • Read amplification: ~1x per level

2. Size-Tiered Compaction (STCS)

  • Best for: Write-heavy workloads, time-series data
  • Write amplification: ~2-4x
  • Space amplification: ~2x
  • Read amplification: Higher (scans more SSTables)

3. Universal Compaction

  • Best for: Time-series, append-only workloads
  • Write amplification: ~2x
  • Space amplification: ~2x
  • Optimized for sequential writes

Adaptive Tuning

HeliosDB features automatic workload detection and configuration tuning.

How It Works

  1. Workload Detection: The system continuously monitors:

    • Write/read ratio
    • Point lookup vs range scan ratio
    • Operation latencies
    • Amplification factors

  2. Pattern Classification: The workload is classified as write-heavy, read-heavy, scan-dominated, point-lookup-dominated, time-series (append-only), or balanced.

  3. Auto-Configuration: When pattern changes, LSM parameters are automatically adjusted.

Configuration Parameters

Memtable Configuration

Guidelines:

  • Write-heavy: 128-256 MB, 6-8 buffers
  • Read-heavy: 32-64 MB, 2-4 buffers
  • Balanced: 64-128 MB, 4 buffers
  • Time-series: 256-512 MB, 8+ buffers

Level 0 Triggers

Recommendations:

WorkloadTriggerSlowdownStop
Write-heavy83050
Read-heavy21020
Balanced42036
Time-series84064

Bloom Filter Configuration

False Positive Rate:

  • 10 bits/key ≈ 1% FPR
  • 14 bits/key ≈ 0.1% FPR
  • 16 bits/key ≈ 0.05% FPR

Compression Configuration

Compression Trade-offs:

AlgorithmRatioSpeedCPUBest For
None1.0xFastestMinimalL0, L1
Snappy2-3xFastLowL2-L4
Zstd3-5xMediumMediumL5+
Lz42-2.5xVery FastLowAlternative to Snappy

Block Cache Configuration

Guidelines:

  • Read-heavy: 1-2 GB (larger is better)
  • Write-heavy: 256-512 MB
  • Balanced: 512 MB - 1 GB
  • Memory-constrained: 128-256 MB

Workload-Specific Tuning

1. Write-Heavy Workloads

Characteristics:

  • High insert/update rate
  • Infrequent reads
  • Examples: Log ingestion, metrics collection

Expected Performance:

  • Write throughput: 50,000+ writes/sec
  • Write latency: <1ms p99
  • Write amplification: 2-3x

2. Read-Heavy Workloads

Characteristics:

  • High query rate
  • Mostly point lookups
  • Examples: User profiles, session stores

Expected Performance:

  • Read throughput: 100,000+ reads/sec
  • Read latency: <500μs p99
  • Read amplification: 1-2x

3. Time-Series Workloads

Characteristics:

  • Append-only writes
  • Time-based queries
  • Examples: Metrics, events, logs

Expected Performance:

  • Write throughput: 100,000+ writes/sec
  • Write latency: <500μs p99
  • Compression ratio: 3-5x

4. Mixed OLTP Workloads

Characteristics:

  • Balanced read/write ratio
  • Transactions with updates
  • Examples: E-commerce, booking systems

Expected Performance:

  • Total throughput: 50,000+ ops/sec
  • Mixed latency: <1ms p99
  • Balanced amplification

Performance Metrics

Key Metrics to Monitor

1. Write Amplification

Write Amplification = Bytes Written to Disk / Bytes Written by User
  • Target: <5x for leveled, <3x for size-tiered
  • High values indicate excessive compaction

2. Read Amplification

Read Amplification = Number of SSTables Checked per Read
  • Target: <5 SSTables per read
  • Reduced by bloom filters and compaction

3. Space Amplification

Space Amplification = Disk Space Used / Logical Data Size
  • Target: <2x
  • Affected by tombstones, duplicates, compression

4. Operation Latencies

  • Write latency p99: <2ms
  • Read latency p99: <1ms
  • Scan latency: Depends on range size

Best Practices

1. Enable Adaptive Tuning

Benefits:

  • Automatic optimization for workload changes
  • Reduced operational overhead
  • Better resource utilization

2. Size Memtables Appropriately

Rule of Thumb:

Memtable Size = (Write Rate MB/s) × (Target Flush Interval seconds)

Example:

  • Write rate: 10 MB/s
  • Target flush interval: 10 seconds
  • Memtable size: 100 MB

3. Configure Bloom Filters Per-Level

  • Use larger bloom filters (14-16 bits) for L0-L2 (hot data)
  • Use smaller bloom filters (6-8 bits) for L5+ (cold data)
  • Saves memory while maintaining read performance

4. Balance Compression vs. CPU

  • Use no compression for L0-L1 (written frequently)
  • Use Snappy for L2-L4 (good balance)
  • Use Zstd for L5+ (rarely read, maximize space savings)

5. Tune L0 Compaction Triggers

For Write-Heavy:

  • Higher triggers (8-16 files) to batch compactions
  • Reduces write amplification
  • May increase read latency temporarily

For Read-Heavy:

  • Lower triggers (2-4 files) to minimize L0 files
  • Reduces read amplification
  • Maintains consistent read performance

6. Monitor and Alert

Set up monitoring for:

  • Write amplification >10x
  • Read amplification >10x
  • Space amplification >3x
  • P99 latency >10ms
  • L0 file count approaching stop trigger

7. Use I/O Throttling

Benefits:

  • Prevents compaction from overwhelming I/O
  • Maintains consistent foreground performance
  • Better multi-tenant resource sharing

Troubleshooting

High Write Latency

Symptoms:

  • P99 write latency >10ms
  • Writes blocked due to L0 files

Solutions:

  1. Increase level0_slowdown_trigger and level0_stop_trigger
  2. Increase memtable_size_mb to reduce flush frequency
  3. Increase write_buffer_count for more concurrency
  4. Use size-tiered or universal compaction

High Read Latency

Symptoms:

  • P99 read latency >5ms
  • Too many SSTables to check

Solutions:

  1. Decrease level0_file_trigger for faster compaction
  2. Increase bloom_bits_per_key for better filtering
  3. Increase block_cache_mb for more caching
  4. Use leveled compaction strategy

High Write Amplification

Symptoms:

  • Write amplification >10x
  • Excessive I/O utilization

Solutions:

  1. Use size-tiered or universal compaction
  2. Increase target_file_size_base for larger SSTables
  3. Reduce level0_file_trigger to reduce compactions
  4. Enable early tombstone deletion

High Space Usage

Symptoms:

  • Space amplification >3x
  • Disk usage growing faster than expected

Solutions:

  1. Enable compression (Zstd for maximum compression)
  2. Reduce gc_grace_seconds for faster tombstone removal
  3. Trigger manual compaction to remove duplicates
  4. Use leveled compaction for better space efficiency

Performance Targets

Production Targets (per node)

Indicative figures, measured on representative fixtures; reproduce on your own hardware.

MetricWrite-HeavyRead-HeavyBalancedTime-Series
Write Throughput50K ops/s10K ops/s25K ops/s100K ops/s
Read Throughput10K ops/s100K ops/s25K ops/s10K ops/s
Write Latency (p99)2ms5ms2ms1ms
Read Latency (p99)5ms1ms2ms5ms
Write Amplification2-3x8-10x5x2x
Read Amplification5-10x1-2x3-5x10-20x
Space Amplification2x1.1x1.5x1.8x

Conclusion

The HeliosDB LSM storage engine provides powerful tuning capabilities with adaptive optimization. By understanding your workload pattern and applying the appropriate configuration, you can achieve:

  • High throughput: 50,000+ operations per second
  • Low latency: Sub-millisecond p99 latencies
  • Efficient resource usage: Low amplification factors
  • Automatic optimization: Adapts to workload changes

Start with the defaults and enable adaptive tuning. Monitor metrics and fine-tune as needed for your specific workload.

For questions or issues, refer to the HeliosDB documentation or contact support@heliosdb.com.