Skip to content
HeliosDB HeliosDB Full

Complex Event Processing (CEP) Patterns Guide

Complex Event Processing (CEP) Patterns Guide

Overview

Complex Event Processing enables pattern detection in event streams using MATCH_RECOGNIZE SQL syntax and NFA (Non-deterministic Finite Automaton) matching. HeliosDB Streaming provides comprehensive CEP capabilities for detecting complex patterns across time-ordered events.

Pattern Types

1. Sequence Patterns

Detect ordered sequences of events.

Use Case: User journey tracking (view → add to cart → purchase)

SQL Example:

SELECT *
FROM events
MATCH_RECOGNIZE (
    PARTITION BY user_id
    ORDER BY event_time
    MEASURES
        FIRST(A.event_time) AS journey_start,
        LAST(C.event_time) AS journey_end,
        (C.event_time - A.event_time) AS conversion_time
    PATTERN (A B C)
    DEFINE
        A AS A.event_type = 'page_view',
        B AS B.event_type = 'add_to_cart',
        C AS C.event_type = 'purchase'
)

2. Missing Event Patterns

Detect absence of expected events within a time window.

Use Case: Abandoned cart detection

SQL Example:

SELECT *
FROM events
MATCH_RECOGNIZE (
    PARTITION BY user_id
    ORDER BY event_time
    MEASURES
        A.cart_id AS abandoned_cart,
        A.event_time AS cart_time,
        CURRENT_TIMESTAMP AS detected_at
    PATTERN (A B* NOT C)
    WITHIN INTERVAL '1' HOUR
    DEFINE
        A AS A.event_type = 'add_to_cart',
        B AS B.event_type != 'purchase' AND B.event_type != 'cart_clear',
        C AS C.event_type = 'purchase'
)

3. Looping Patterns

Detect repeated occurrences of events.

Use Case: Failed login detection (3+ failed attempts)

SQL Example:

SELECT *
FROM events
MATCH_RECOGNIZE (
    PARTITION BY user_id
    ORDER BY event_time
    MEASURES
        COUNT(A.*) AS failed_attempts,
        FIRST(A.event_time) AS first_failure,
        LAST(A.event_time) AS last_failure
    PATTERN (A{3,})
    WITHIN INTERVAL '5' MINUTE
    DEFINE
        A AS A.event_type = 'login_failed'
)

4. Alternation Patterns

Multiple possible event sequences.

Use Case: Payment method selection

SQL Example:

SELECT *
FROM events
MATCH_RECOGNIZE (
    PARTITION BY user_id
    ORDER BY event_time
    MEASURES
        CASE
            WHEN MATCH_NUMBER(B) = 1 THEN 'credit_card'
            WHEN MATCH_NUMBER(C) = 1 THEN 'paypal'
            ELSE 'unknown'
        END AS payment_method
    PATTERN ((A B) | (A C))
    DEFINE
        A AS A.event_type = 'checkout_started',
        B AS B.event_type = 'credit_card_selected',
        C AS C.event_type = 'paypal_selected'
)

5. Conditional Patterns

Patterns with conditional logic.

Use Case: High-value transaction anomaly detection

SQL Example:

SELECT *
FROM events
MATCH_RECOGNIZE (
    PARTITION BY account_id
    ORDER BY event_time
    MEASURES
        A.amount AS suspicious_amount,
        AVG(B.amount) AS avg_normal_amount,
        (A.amount / AVG(B.amount)) AS anomaly_ratio
    PATTERN (B{5,} A)
    WITHIN INTERVAL '1' DAY
    DEFINE
        B AS B.amount BETWEEN 10 AND 100,
        A AS A.amount > AVG(B.amount) * 5
)

Performance Considerations

1. NFA State Explosion

Complex patterns can create large state machines. Mitigation:

  • Limit pattern complexity (max 5-7 operators per pattern)
  • Use time constraints to bound state lifetime
  • Partition by key to distribute load

2. Memory Usage

Each active pattern instance maintains state.

3. Latency Optimization

  • Early Termination: Use NOT patterns for fast rejection
  • Pattern Ordering: Most selective conditions first
  • Partition Pruning: Partition by high-cardinality keys

Troubleshooting

High Memory Usage

Symptom: OOM errors or high GC pressure

Solution:

  1. Reduce max_pattern_instances
  2. Decrease time windows
  3. Add more aggressive state cleanup

Slow Pattern Matching

Symptom: High latency in pattern matching

Solution:

  1. Simplify pattern complexity
  2. Increase parallelism
  3. Optimize event predicates (avoid expensive operations)

Missing Matches

Symptom: Expected patterns not detected

Solution:

  1. Check event ordering (ORDER BY in SQL)
  2. Verify partition keys
  3. Increase time windows
  4. Enable debug logging

References

Version: 1.0