Redis-based FIX storage

FIX services that support persistent FIX sessions (FIX Drop Copy and FIX Store Gateway) support ability to store their state into Redis Cluster. This article describes configuration required.

The following areas are stored:

• Session State (senderSeqNum, targetSeqNum, sessionStartTime, etc)
• Outbound FIX Messages (for the current FIX session only)

To enable Redis storage backend you will need to add the following configuration stanza to ember.conf:

redisSessionStorage {
  factory = "deltix.efix.endpoint.redis.JedisClusterClientFactory"
  settings {
     namespace = ember
     nodes = [ node1, node2, node3 ]
     password = null  # use reference to env variable or Ember ability to reference Secret storage

     connectionTimeout = 2s
     socketTimeout = 2s
     maxAttempts = 5

     pool {
       maxTotal = 8
       maxIdle = 8
       minIdle = 0
       maxWait = -1
     }

     # Only one dispatcher (per namespace/gateway) may hold this lock at a time, preventing two instances
     # of the same service from writing to the same Redis-backed session state concurrently. The lock is
     # acquired only after all sessions/drop-copies are opened (opening a session only reads from Redis,
     # never writes), so lockTtl only needs to cover the normal renewal cadence, not startup time.
     lockTtl = 30s
     lockRenewalInterval = 10s

     # Max messages read from the journal per dispatcher work cycle. Session-state/message-store writes are
     # not fenced by the lock (only the lock key itself is protected), so if the dispatcher thread stalls
     # (GC pause, slow I/O) past lockTtl, another instance could acquire the lock and start writing while
     # the stalled instance is still mid-flight. A smaller batch bounds how much a stalled cycle can enqueue
     # for session threads to write before the next lease-renewal check, shrinking (not eliminating) that
     # exposure window. Should stay well above normal per-cycle message volume.
     dispatchBatchSize = 256
  }
}

Parameters

Parameter	Description
namespace	Prefix shared by every Redis key written by this service instance. Lets multiple independent environments (e.g. UAT, DEV) or deployments share a single Redis cluster without key collisions.
nodes	List of host:port addresses of Redis Cluster nodes used as initial contact points for topology discovery.
password	Redis auth password. Normally supplied via an environment variable reference or Ember's Secret storage rather than in plain text.
connectionTimeout	Timeout for establishing a connection to a Redis node.
socketTimeout	Timeout for a single socket read/write operation.
maxAttempts	Max retries/redirect follows (e.g. on MOVED/ASK cluster redirections) for a single command.
pool.maxTotal / pool.maxIdle / pool.minIdle / pool.maxWait	Standard Apache Commons Pool settings for the underlying connection pool. One pool serves every session/drop-copy of the service, so it should be sized for their total number.
lockTtl	Lease duration of the single-active-dispatcher lock (see below).
lockRenewalInterval	How often the lock lease is renewed; must be comfortably shorter than lockTtl.
dispatchBatchSize	Max journal messages read per dispatcher work cycle, bounding how much can be enqueued for session threads to write to Redis between two lock-renewal checks.

The namespace/gateway/name triple identifying each FIX session (see Key layout below) comes from the namespace setting above, plus the gateway kind (fullfix for FIX Store Gateway, ember for Drop Copy) and the session/drop-copy name from the FIX session configuration itself — they are not set directly under redisSessionStorage.

Implementation details

• Ember uses Jedis Java client API to Redis Cluster
• Redis version 7.2, 7.4, 8.0, 8.2 and 8.4 should be supported.

Performance

Predominant operation is storing each outbound FIX message into Redis-cluster backed storage. Our limited performance study shows that Redis storage adds about 27 microseconds to outbound FIX message processing (lowering overall FIX message rate to ~35K messages/second).

Key layout

Each FIX session/drop-copy gets its own set of keys, all sharing the hash tag {namespace/gateway/name} so Redis Cluster co-locates them on the same shard (required since session state and message chunks are updated together):

Key	Redis type	Contents
{namespace/gateway/name}/state	Hash	Session state fields: senderSeqNum, targetSeqNum, sessionStartTime, connectionStartTime, lastSentSeqNum
{namespace/gateway/name}/store/<chunk>	Hash	Outbound messages with sequence numbers chunk*1024 .. chunk*1024+1023, one hash field per message

• namespace — the configured namespace (e.g. ember, fullfix, or an environment name like UAT)
• gateway — identifies the gateway kind, e.g. fsg (FIX Store Gateway) or dc (Drop Copy)
• name — the FIX session or drop-copy name as it is defined in ember config, e.g. CLIENT2

Session state hash

Field names are plain ASCII strings; values are fixed-width binary integers (little-endian, 4 bytes for int, 8 bytes for long) written via HSET:

HSET {fullfix/CLIVNDDEF1}/state senderSeqNum <4-byte int> targetSeqNum <4-byte int> \
     sessionStartTime <8-byte long> connectionStartTime <8-byte long> lastSentSeqNum <4-byte int>

A missing field is treated as "not yet set" and falls back to its default (sequence numbers default to 1, lastSentSeqNum to 0, timestamps to "unset").

Message store chunks

Outbound messages are grouped into fixed-size chunks of 1024 sequence numbers each — sequence number seqNum lives in chunk seqNum / 1024, under hash field seqNum (encoded as a 4-byte little-endian int). This bounds the size of any single hash and lets old chunks be dropped wholesale once no longer needed.

Each hash field's value is a small self-describing binary record:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Record Length                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                   Message Sequence Number                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                         Sending Time                          +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Message Type Length                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Message Body Length                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                Message Type ... Message Body                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

All integers are little-endian. Sending Time is nanoseconds since epoch. Message Type (FIX tag 35 value, e.g. D for New Order Single) and Message Body (the rest of the FIX message body) are stored back-to-back as raw bytes right after the fixed header. Record Length and Message Sequence Number are redundant with the hash key/value sizing and are validated on read as a consistency check.

For example, message with seqNum=7 in a FIX Store Gateway session named CLIVNDDEF1 is stored as field 7 (4 bytes: 07 00 00 00) of hash key {fullfix/CLIVNDDEF1}/store/0 (chunk 7 / 1024 = 0).

Estimating Redis memory requirements

Session state (5 fixed-width fields per session) is negligible — memory usage is dominated by the message store. Each stored message costs roughly its FIX wire size plus the ~24-byte fixed record header (record length, sequence number, sending time, type/body length fields). Assuming an average FIX message size of 750 bytes, that's about 800 bytes per stored message once header and Redis's own per-field overhead are included.

Given:

• n — orders per day
• m — FIX messages generated per order (typically 1–16, depending on how many execution reports/acks a single order produces)
• X — days messages accumulate before the session resets and old chunks are dropped (1 for a daily FIX session, 7 for a weekly one)
• G — number of gateway services storing the same order flow into Redis (1 if only FIX Store Gateway or only Drop Copy is used, 2 if the same messages pass through both and are stored independently by each)

Redis memory (bytes) ≈ n × m × X × G × 800

Treat this as a same-order-of-magnitude estimate, not an exact figure — actual FIX message sizes vary by message type and tag content. When sizing a cluster, also account for:

• Redis Cluster replication factor (e.g. ×2 or ×3 for replicas), on top of the formula above, which only estimates primary data size.
• 20–30% headroom for Redis memory fragmentation and non-data overhead.

Example

50,000 orders/day, 5 FIX messages/order, weekly FIX session (X = 7), stored in both FIX Store Gateway and Drop Copy (G = 2):

50,000 × 5 × 7 × 2 × 800 bytes ≈ 2.8 GB

Appendix: how to inspect this data in Redis

Introspect business-level keys. Here we can see that we have session stores and states for Drop Copy client sessions of CLIENT1 and CLIENT2:

kubectl exec -n redis redis-cluster-0 -- sh -c '
  export REDISCLI_AUTH="$(cat /opt/bitnami/redis/secrets/redis-password)"
  redis-cli --scan 2>&1
'

{ember/dc/CLIENT1}/state
{ember/dc/CLIENT1}/store/0
{ember/dc/CLIENT2}/state
{ember/dc/CLIENT2}/store/0

Introspect FIX session state of a specific Drop Copy client:

kubectl exec -n redis redis-cluster-0 -- sh -c '
  export REDISCLI_AUTH="$(cat /opt/bitnami/redis/secrets/redis-password)"
  key="{ember/dc/CLIENT1}/state"

  for field in $(redis-cli -c --raw HKEYS "$key"); do
    printf "%s: " "$field"
    redis-cli -c HSTRLEN "$key" "$field"
  done
'
senderSeqNum: 4
targetSeqNum: 4
sessionStartTime: 8
connectionStartTime: 8
lastSentSeqNum: 4

Introspect FIX message store of a specific Drop Copy client:

kubectl exec -n redis redis-cluster-0 -- sh -c '
  export REDISCLI_AUTH="$(cat /opt/bitnami/redis/secrets/redis-password)"
  key="{ember/dc/CLIENT2}/store/0"

  for field in $(redis-cli -c --raw HKEYS "$key"); do
    printf "%s: " "$field"
    redis-cli -c HSTRLEN "$key" "$field"
  done
'
: 0
: 0
: 0
: 0