HTAP Dual-Format Storage

Storage Modes

Per-Column Storage Modes

HeliosDB provides four storage modes that can be assigned independently to each column in a table. This per-column granularity — rather than per-table — lets you optimize the physical layout for the exact access pattern each column experiences. Mix row-oriented, columnar, dictionary-encoded, and content-addressed storage within a single table.

• Default (row-oriented) — The standard row store, optimal for OLTP point lookups and transactional updates. Each row is stored as a contiguous tuple. This is the default when no STORAGE clause is specified.
• Columnar (batched analytics) — Values are stored in 1,024-row batches organized by column, enabling sequential reads and SIMD-accelerated aggregation. Ideal for columns frequently used in GROUP BY, SUM, AVG, and full-table scans.
• Dictionary (low-cardinality) — Values are replaced by compact integer codes that reference a shared dictionary. Best for columns with few distinct values such as status, category, or country — delivers both compression and fast equality predicates.
• ContentAddressed (deduplication) — Values are stored by content hash and deduplicated automatically. Ideal for large text payloads, JSON documents, or any column where many rows share identical values.

Use ALTER TABLE ... ALTER COLUMN ... SET STORAGE to migrate columns between storage modes at any time without rewriting the entire table. The EXPLAIN (STORAGE) command verifies the current layout.

-- Per-column storage modes in CREATE TABLE
CREATE TABLE events (
    id         INT PRIMARY KEY,              -- Default (row store)
    user_id    INT,                          -- Default (row store)
    category   TEXT STORAGE DICTIONARY,      -- Dictionary encoding
    amount     FLOAT8 STORAGE COLUMNAR,      -- Columnar batches
    timestamp  INT8 STORAGE COLUMNAR,        -- Columnar batches
    payload    TEXT STORAGE CONTENT_ADDRESSED -- Deduplication
);

-- Migrate columns between storage modes
ALTER TABLE events
  ALTER COLUMN user_id SET STORAGE COLUMNAR;

-- Verify storage layout
EXPLAIN (STORAGE) SELECT * FROM events;

Query Execution

Columnar Scan Detection

When all columns in a scanned table use STORAGE COLUMNAR, the EXPLAIN output shows “Columnar Scan” instead of the standard “Seq Scan”. This confirms that the query engine is using the optimized batch columnar scan operator — the ColumnarScanOperator — which reads column batches sequentially for maximum analytics throughput.

• Automatic detection — No hints or configuration required. The planner inspects column storage modes and selects the columnar scan operator when all referenced columns are columnar.
• Batch processing — The columnar scan reads data in 1,024-row batches, enabling SIMD-accelerated predicate evaluation and vectorized aggregation.
• I/O efficiency — Sequential column reads minimize random I/O. Only the columns referenced in the query are read from storage, reducing bandwidth by up to 90% for wide tables.
• Mixed-mode fallback — If any referenced column uses a non-columnar storage mode, the standard scan operator is used with transparent ColumnarRef resolution. The query always returns correct results regardless of storage layout.

-- All columns use STORAGE COLUMNAR
CREATE TABLE metrics (
    ts    INT8 STORAGE COLUMNAR,
    value FLOAT8 STORAGE COLUMNAR
);

helios> EXPLAIN ANALYZE SELECT * FROM metrics;
-- Shows: → Columnar Scan metrics
-- Instead of: → Scan metrics

-- The ColumnarScanOperator reads
-- column batches sequentially for
-- maximum analytics throughput.
--
-- Aggregations on columnar tables
-- are 10-50x faster than row scans:

helios> EXPLAIN ANALYZE
        SELECT AVG(value) FROM metrics
        WHERE ts > 1704067200;
-- → Columnar Scan metrics
--      Filter: ts > 1704067200
--      Rows: 50000, Time: 1.2ms

System Views

HTAP Layout Statistics

Two system views provide real-time HTAP monitoring. The pg_htap_layout_stats view shows per-table storage layout information, giving you a high-level overview of how data is distributed between the row store and column store across all tables in the database.

Column	Type	Description
table_name	TEXT	Table name
row_store_size	INT8	Estimated row store size in bytes
column_store_size	INT8	Estimated column store size in bytes
row_store_rows	INT8	Number of rows in row store
column_store_rows	INT8	Number of rows in columnar batches
sync_status	TEXT	row_only, columnar_only, dual_format, or mixed
implemented	BOOLEAN	Whether HTAP is active

SELECT * FROM pg_htap_layout_stats;

-- Example output:
--  table_name | row_store_size | column_store_size
-- -----------+----------------+------------------
--  orders     |          64000 |             48000
--  users      |          12800 |                 0
--
--  row_store_rows | column_store_rows | sync_status
-- ---------------+-------------------+------------
--           1000 |              1000 | dual_format
--            200 |                 0 | row_only
--
--  implemented
-- -------------
--  true
--  true

-- Filter for dual-format tables only
SELECT table_name,
       row_store_rows,
       column_store_rows,
       sync_status
FROM pg_htap_layout_stats
WHERE sync_status = 'dual_format';

-- Monitor storage distribution
SELECT table_name,
       row_store_size AS row_bytes,
       column_store_size AS col_bytes,
       ROUND(column_store_size::float
         / (row_store_size + column_store_size)
         * 100, 1) AS columnar_pct
FROM pg_htap_layout_stats
WHERE implemented = true;

Resource Management

Resource Isolation

The HTAP resource manager maintains separate CPU, memory, and I/O budgets for OLTP and OLAP workloads. This isolation prevents analytical queries — which may scan millions of rows — from consuming resources needed by latency-sensitive transactional operations. The default allocation is 60% OLTP / 40% OLAP, fully configurable via heliosdb.toml.

• CPU isolation — OLTP and OLAP workloads are scheduled on separate thread pools with configurable CPU ratios. Analytical queries cannot starve transactional operations of compute time.
• Memory budgets — Each workload type has a dedicated memory allocation. The buffer pool, sort buffers, and hash tables for OLAP queries draw from the OLAP budget, leaving the OLTP budget untouched.
• Connection limits — Separate connection pools for OLTP (default: 1,000) and OLAP (default: 16) prevent connection exhaustion. A long-running analytical query cannot block incoming transactional connections.
• I/O prioritization — OLTP reads and writes are prioritized at the I/O scheduler level. OLAP full-table scans are submitted at lower priority to maintain transactional responsiveness.
• Automatic classification — The WorkloadClassifier assigns each query to the OLTP or OLAP resource pool automatically. No application-level changes required — just configure the ratios and let the system manage isolation.

# heliosdb.toml - Resource isolation settings
[htap]
oltp_cpu_ratio = 0.6      # 60% CPU for OLTP
olap_cpu_ratio = 0.4      # 40% CPU for OLAP
oltp_memory_ratio = 0.6   # 60% memory for OLTP
olap_memory_ratio = 0.4   # 40% memory for OLAP
oltp_max_connections = 1000
olap_max_connections = 16

# I/O priority settings
[htap.io]
oltp_io_priority = "high"
olap_io_priority = "normal"
olap_scan_throttle = false

# Workload classifier tuning
[htap.classifier]
enabled = true
oltp_threshold = 0.6
olap_threshold = 0.6
# Queries scoring below both thresholds
# are classified as HTAP (mixed) and
# can draw from either resource pool.

# Example resource allocation on a
# 16-core, 64GB server:
#   OLTP: 10 cores, 38GB RAM, 1000 conns
#   OLAP:  6 cores, 26GB RAM,   16 conns