Redis-based FIX storage
FIX services that support persistent FIX sessions (FIX Drop Copy and FIX Store Gateway) support ability to store their state into Redis Cluster. This article describes configuration required.
The following areas are stored:
- Session State (senderSeqNum, targetSeqNum, sessionStartTime, etc)
- Outbound FIX Messages (for the current FIX session only)
To enable Redis storage backend you will need to add the following configuration stanza to ember.conf:
redisSessionStorage {
factory = "deltix.ember.fix.JedisClusterClientFactory"
settings {
namespace = ember
nodes = [ node1, node2, node3 ]
password = null # use reference to env variable or Ember ability to reference Secret storage
connectionTimeout = 2s
socketTimeout = 2s
maxAttempts = 5
pool {
maxTotal = 8
maxIdle = 8
minIdle = 0
maxWait = -1
}
# Only one dispatcher (per namespace/gateway) may hold this lock at a time, preventing two instances
# of the same service from writing to the same Redis-backed session state concurrently. The lock is
# acquired only after all sessions/drop-copies are opened (opening a session only reads from Redis,
# never writes), so lockTtl only needs to cover the normal renewal cadence, not startup time.
lockTtl = 30s
lockRenewalInterval = 10s
# Max messages read from the journal per dispatcher work cycle. Session-state/message-store writes are
# not fenced by the lock (only the lock key itself is protected), so if the dispatcher thread stalls
# (GC pause, slow I/O) past lockTtl, another instance could acquire the lock and start writing while
# the stalled instance is still mid-flight. A smaller batch bounds how much a stalled cycle can enqueue
# for session threads to write before the next lease-renewal check, shrinking (not eliminating) that
# exposure window. Should stay well above normal per-cycle message volume.
dispatchBatchSize = 256
}
}
Parameters
| Parameter | Description |
|---|---|
namespace | Prefix shared by every Redis key written by this service instance. Lets multiple independent environments (e.g. UAT, DEV) or deployments share a single Redis cluster without key collisions. |
nodes | List of host:port addresses of Redis Cluster nodes used as initial contact points for topology discovery. |
password | Redis auth password. Normally supplied via an environment variable reference or Ember's Secret storage rather than in plain text. |
connectionTimeout | Timeout for establishing a connection to a Redis node. |
socketTimeout | Timeout for a single socket read/write operation. |
maxAttempts | Max retries/redirect follows (e.g. on MOVED/ASK cluster redirections) for a single command. |
pool.maxTotal / pool.maxIdle / pool.minIdle / pool.maxWait | Standard Apache Commons Pool settings for the underlying connection pool. One pool serves every session/drop-copy of the service, so it should be sized for their total number. |
lockTtl | Lease duration of the single-active-dispatcher lock (see below). |
lockRenewalInterval | How often the lock lease is renewed; must be comfortably shorter than lockTtl. |
dispatchBatchSize | Max journal messages read per dispatcher work cycle, bounding how much can be enqueued for session threads to write to Redis between two lock-renewal checks. |
The namespace/gateway/name triple identifying each FIX session (see Key layout below) comes from the namespace setting above, plus the gateway kind (fullfix for FIX Store Gateway, ember for Drop Copy) and the session/drop-copy name from the FIX session configuration itself — they are not set directly under redisSessionStorage.
Implementation details
- Ember uses Jedis Java client API to Redis Cluster
- Redis version 7.2, 7.4, 8.0, 8.2 and 8.4 should be supported.
Key layout
Each FIX session/drop-copy gets its own set of keys, all sharing the hash tag
{namespace/gateway/name} so Redis Cluster co-locates them on the same shard
(required since session state and message chunks are updated together):
| Key | Redis type | Contents |
|---|---|---|
{namespace/gateway/name}/state | Hash | Session state fields: senderSeqNum, targetSeqNum, sessionStartTime, connectionStartTime, lastSentSeqNum |
{namespace/gateway/name}/store/<chunk> | Hash | Outbound messages with sequence numbers chunk*1024 .. chunk*1024+1023, one hash field per message |
namespace— the configured namespace (e.g.ember,fullfix, or an environment name likeUAT)gateway— identifies the gateway kind, e.g.fullfix(FIX Store Gateway) orember(Drop Copy)name— the FIX session or drop-copy name, e.g.CLIENT2
Session state hash
Field names are plain ASCII strings; values are fixed-width binary integers
(little-endian, 4 bytes for int, 8 bytes for long) written via HSET:
HSET {fullfix/CLIVNDDEF1}/state senderSeqNum <4-byte int> targetSeqNum <4-byte int> \
sessionStartTime <8-byte long> connectionStartTime <8-byte long> lastSentSeqNum <4-byte int>
A missing field is treated as "not yet set" and falls back to its default
(sequence numbers default to 1, lastSentSeqNum to 0, timestamps to
"unset").
Message store chunks
Outbound messages are grouped into fixed-size chunks of 1024 sequence numbers
each — sequence number seqNum lives in chunk seqNum / 1024, under hash
field seqNum (encoded as a 4-byte little-endian int). This bounds the size
of any single hash and lets old chunks be dropped wholesale once no longer
needed.
Each hash field's value is a small self-describing binary record:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Record Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Sending Time +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Message Type Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Message Body Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Message Type ... Message Body |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
All integers are little-endian. Sending Time is nanoseconds since epoch.
Message Type (FIX tag 35 value, e.g. D for New Order Single) and
Message Body (the rest of the FIX message body) are stored back-to-back as
raw bytes right after the fixed header. Record Length and Message Sequence Number are redundant with the hash key/value sizing and are
validated on read as a consistency check.
For example, message with seqNum=7 in a FIX Store Gateway session named
CLIVNDDEF1 is stored as field 7 (4 bytes: 07 00 00 00) of hash key
{fullfix/CLIVNDDEF1}/store/0 (chunk 7 / 1024 = 0).
Appendix: how to inspect this data in Redis
kubectl exec -n redis redis-cluster-0 -- sh -c '
export REDISCLI_AUTH="$(cat /opt/bitnami/redis/secrets/redis-password)"
redis-cli --scan 2>&1
'
{fullfix/CLIVNDDEF1}/state
{ember/CLIENT2}/state
{ember/CLIENT2}/store/0
{fullfix/CFQADEF1}/state
{fullfix/CLIENT2}/state
{fullfix/CLIENT2}/store/0
kubectl exec -n redis redis-cluster-0 -- sh -c '
export REDISCLI_AUTH="$(cat /opt/bitnami/redis/secrets/redis-password)"
key="{fullfix/CLIVNDDEF1}/state"
for field in $(redis-cli -c --raw HKEYS "$key"); do
printf "%s: " "$field"
redis-cli -c HSTRLEN "$key" "$field"
done
'
senderSeqNum: 4
targetSeqNum: 4
sessionStartTime: 8
connectionStartTime: 8
lastSentSeqNum: 4
kubectl exec -n redis redis-cluster-0 -- sh -c '
export REDISCLI_AUTH="$(cat /opt/bitnami/redis/secrets/redis-password)"
key="{fullfix/CLIENT2}/store/0"
for field in $(redis-cli -c --raw HKEYS "$key"); do
printf "%s: " "$field"
redis-cli -c HSTRLEN "$key" "$field"
done
'
: 0
: 0
: 0
: 0