Order State Synchronization after disconnect

This article describes the procedure for order state synchronization between Ember OMS and an exchange.

note

For every order request Ember expects to receive some kind of order event that communicates ACK/NACK.

For example, OrderNewRequest should result in OrderNewEvent/OrderRejectRequest (OrderTradeEvent can also serve as ACK, and OrderCancelEvent can be also sometimes used for NACK).

Similarly, OrderReplaceRequest should result in OrderReplaceEvent/OrderReplaceRejectRequest.

Trades should be communicated using OrderTradeReportEvent, and so on.

Order Events is normal order state delivery mechanism. This article focuses on state synchronizatoin after disconnect.

Basic Procedure

This procedure is performed for all orders that Ember OMS considers active on startup and each time the system reconnects to an exchange after a disconnect.

• Ember OMS: For each active order, Ember OMS issues an OrderStatusRequest to the order destination. If the order has pending replace requests, multiple status requests may be sent (discussed later).

OrderStatusRequest contains the following fields:

• orderId — identifies the order in question
• sourceId — identifies the source of the order
• orderCorrelationId — identifies the first request in the replace chain (or is identical to orderId if the order has not been modified)
• externalOrderId — exchange-assigned order identifier, if known

• Exchange (via trade connector): The exchange is expected to respond with an OrderStatusEvent reflecting the current order state. An OrderStatusEvent is expected for each OrderStatusRequest (with the exception of active replacement chains, or lossless APIs discussed later). When reporting the current order state, special attention should be given to the following fields:

  orderStatus should reflect the current state of the order (NEW / CANCELLED / COMPLETELY_FILLED, etc.)
  cumulativeQuantity and averagePrice should reflect the cumulative traded quantity and price
  remainingQuantity, if provided, should be consistent with the reported status

If the order state reported during synchronization differs from the last known order state, the OMS translates the difference into a series of synthetic events (for example, an OrderTradeReportEvent followed by an OrderCancelEvent). This allows downstream OMS clients to resynchronize their state using the standard event vocabulary.

note

Ember uses write-ahead logging (WAL) for all trading requests and events. Under normal operation, this guarantees that Ember maintains a persistent record of every order request sent to the exchange.

The WAL persistence mechanism does not force a disk flush for each individual request, as doing so would severely impact latency. If a higher level of durability is required, use an Ember cluster.

Examples

• If a previously active order is no longer active and no fills were observed, respond with
  OrderStatusEvent(orderId=..., state=CANCELLED).
  You may also report remainingQuantity=0 and cumulativeQuantity=0, but this is optional.
  A cancellation reason may be provided via the optional reason field.

• If a previously active order was completely filled, respond with
  OrderStatusEvent(orderId=..., state=COMPLETELY_FILLED, cumulativeQuantity=..., averagePrice=...).
  You may also report remainingQuantity=0, since the order is no longer working on the exchange.

• If a previously active order was partially filled and then cancelled, respond with
  OrderStatusEvent(orderId=..., state=CANCELLED, cumulativeQuantity=..., averagePrice=...).
  You may also report remainingQuantity=0, since the order is no longer working on the exchange.
  A cancellation reason may be supplied if available.

Variations of Synchronization Procedure

Lossless Exchange APIs

Some exchange APIs provide event sequencing and support fetching missed events (for example, FIX). In these cases, the exchange connector can report missed events immediately upon connect, or as soon as gap-resend messages arrive from the exchange. If the connector can guarantee that no exchange updates were lost during the disconnect, explicit order status requests may be skipped.

Stateful Exchange Connectors

As part of an Ember restart, all historical requests for active orders are replayed to the connector during the system BOOTSTRAP stage (when the Ember journal is read). During this stage, Ember also replays order events originally emitted by each exchange connector back to the same connector. This allows the connector to reconstruct its internal state machine and recover the active order state as last reported to Ember.

See the Trade Connector and Algorithm Developer Guide for examples of reusing Ember’s OrderProcessor state machine to simplify this process.

If a connector supports full state recovery through this bootstrap mechanism, it may also ignore OMS OrderStatusRequest calls and implement a custom synchronization flow. The replayed order state then serves as the authoritative reference for the last known state of each order.

Exchange Order Identity

Some exchanges assign their own order identifiers, which are required for all subsequent requests related to a given order.

Ember OMS captures the exchange-assigned order identifier from order events (field externalOrderId). If an exchange-assigned order ID is available, it will be included in the OrderStatusRequest during state synchronization procedure.

Cancel on Disconnect

Many trading venues do not provide direct support for OrderStatusRequest (for example, only a subset of FIX-based exchanges support message 35=H). When post-disconnect order state synchronization becomes complex or fragile, consider enabling the Cancel-on-Disconnect feature offered by some venues.

Order Is Unknown

Special caution should be exercised when responding with OrderStatusEvent.IsOrderUnknown (signaling to the Ember OMS that the requested order is not known to the exchange). This will lead to immediate order cancellation on the Ember side.

Some market venues only cache active orders or a short window of recently completed orders. If a previously filled order falls outside of the exchange cache scope, the exchange may respond that the order is unknown even if it had partial or full fills.

In questionable cases, it may be safer to keep the order open on the Ember side. This will prompt the operations team to investigate and prevent premature loss of order state.

Order Replace Chain State Synchronization

Imagine we have order A that has been modified many times (for example price amendments). Last order replacement confirmed by exchange is R42 and we had pending replacement R43 prior to disconnect.

Prior to disconnect:

[A]  ──▶  [R1]  ──▶ ... ──▶  (R42)  ──▶  {R43}
                               │
                               └── Working on exchange

1. OrderNewRequest(orderId=A) ...
2. OrderReplaceRequest(orderId=R42, originalOrderId=R41)
3. OrderReplaceRequest(orderId=R43, originalOrderId=R42)

Assume that at this point we experience a disconnect from the exchange.

Upon reconnection, Ember’s state-recovery logic sends a series of OrderStatusRequest messages to synchronize order state.
For orders that have pending replace requests, Ember sends one status request for each active order ID:

3. OrderStatusRequest(orderId=R42, correlationOrderId=A)
4. OrderStatusRequest(orderId=R43, correlationOrderId=A)

Sending a status request for each active order ID enables stateless trading-connector logic (not always possible, but often helpful).

Let us examine how different states of the replacement chain should be communicated back to the Ember OMS.

During state recovery, exchange connectors typically respond with OrderStatusEvent (ExecType = Status) reflecting the current highest-precedence OrdStatus and aggregate quantities. This works well for most cases, but some tricky situations it may be better to respond with OrderReplaceEvent or OrderReplaceRejectEvent (as shown below). Typically single OrderStatusEvent is sufficient, but in some cases at exchange connector discretion a series of events may be used (for example, a sequence of unique OrderTradeReportEvents followed by OrderCancelEvent).

State recovery scenario 1: Order is gone

Order no longer active on the exchange. If the order becomes inactive while Ember was disconnected, we expect to receive at least one OrderStatusEvent signaling the final order state.

• OrderStatusEvent may reference either order ID (R42 or R43), ideally referencing the ID of the last order accepted by the exchange (if this information is available).
• The event should report the final order status (CANCELLED | COMPLETELY_FILLED).
• Any fills that occurred should be reflected in cumulativeTradeQuantity and averageTradePrice fields.
• If the remainingOrderQuantity field is reported, it must be set to zero (no part of order remains active on the).

Current state:

[A]  ──▶  [R1]  ──▶ ... ──▶  [R42]  ──▶  [R43]

State recovery scenario 2: Replace went throught

Order is still active, R42 has been successfully replaced by R43. OMS expects:

• OrderStatusEvent(orderId=R43, status = NEW | PARTIALLY_FILLED)
  (i.e., not PENDING_REPLACE)

NOTE: Do not send OrderStatusEvent(orderId=R42, status=CANCELLED) — this will incorrectly cancel the whole order chain. This would be incorrect since we specifically consider this in context of scenario when exchange communicates state of a still active order. Do not confuse state of order amendment request with the state of entire order (chain).

Best practice in this scenario is to send an explicit OrderReplaceEvent(R42 -> R43) rather than a generic OrderStatusEvent. If the order sitting on exchange is partially filled you can follow up with OrderTradeReported event communicating cumulative filled quantity and average price.

Current state:

[A]  ──▶  [R1]  ──▶ ... ──▶  [R42]  ──▶  (R43)
                                           │
                                           └── Working on exchange

---

State recovery scenario 3: Replace was rejected

Order is still active, R42 is working on the exchange, but replacement R43 is rejected. OMS expects:

• OrderStatusEvent(orderId=R42, status = NEW | PARTIALLY_FILLED)
  (i.e., not PENDING_REPLACE)

Best practice is to respond with an explicit OrderReplaceRejectEvent(R43) rather than a generic OrderStatusEvent.
Any missed trades should be reported using OrderTradeReportEvent.

Current state:

[A]  ──▶  [R1]  ──▶ ... ──▶  (R42)  ──▶  [R43]
                                │
                                └── Working on exchange

State recovery scenario 3A: Replace is lost

Order is still active, R42 is working on the exchange, but replacement R43 is unknown to the exchange (lost in transmission). This case is similar to Scenario 3, except B was lost in transmission (e.g., the TCP session dropped while B was being sent), rather than explicitly rejected. OMS expects:

• OrderStatusEvent(orderId=R42, status = NEW | PARTIALLY_FILLED)

As in Scenario 3, you may emit an explicit OrderReplaceRejectEvent(R43) instead.

State recovery scenario 4: Replace still pending

Order is still active, R42 is working version of the order, R43 is still pending exchange confirmation. OMS expects:

• OrderStatusEvent(orderId=R43, status = PENDING_REPLACE).

In the spirit of FIX, you the most recent active order ID (R43).

This situatoin is somewhat uncommon since exchanges usually instantly ACK/NACK orders and if replacement didn't went through during our absence it is likely never made it to exchange. Still some execution brokers have this situation. For example, broker that is running our TWAP order may be still in process of modifying ongoing algo behavior.

Current state:

[A]  ──▶  [R1]  ──▶ ... ──▶  (R42)  ──▶  {R43}
                                │
                                └── Working on exchange