Skip to main content

SIMULATOR Algorithm

Overview​

The simulator algorithm simulates order execution on an exchange using prices from a real exchange using the Deltix QuoteFlow library.

Features:

  • The simulator tries to reconstruct a Level3 order book (market by order) from Level1 or Level2 feeds, coming from a simulated exchange.
  • The simulator can produce a market data feed that shows the prices of passive simulated orders and trades.
note

The simulator does not perform matching/crossing of orders submitted to it. It simulates execution using prices from an exchange order book. Use matching engines provided with Ember if you want internal order crossing market.

The algorithm can consume data from an external liquidity provider and generate executions based on this feed. It has two use cases:

For venues without a useful public book (dark markets), set darkMarketMode to use trade-print simulation instead of QuoteFlow L2/L3 book reconstruction.

Strategy-Backed Testing​

Strategy-backed testing is useful when you have historical data and want to apply certain strategies to be as precise as possible for all executions by influencing the exact distribution of orders. Heuristic algorithms enrich what external liquidity providers send and try to restore an optimal order structure inside of price levels.

In a typical First In First Out (FIFO) сase, the orders placed on a given price level could have various distributions, and the placement of your order in the distribution can affect the speed with which it passes.

In the example below, where your order has a size=3 (highlighted yellow), your order will pass less quickly in Stage A than in Stage B because there is less liquidity before your order in Stage B.

order-structure-comparison-image

The heuristic algorithm tries to build a Level3 order book from a Level2 feed by guessing what orders are on the market based on information about price levels.

Level2 to Level3 Conversion​

There are three messages that can occur based on the actions the liquidity provider (LP) takes.

  • L2EntryNew: Occurs when the LP sends a new price level. In this case, the algorithm can't build a distribution because there is no historical data. Therefore, the distribution is equal to level size.
  • L2EntryUpdate (order size increase): Occurs when you receive an entry update increasing the order size on a specific price level. The algorithm adds extra quantity to the existing order at the end of the price level by placing a subsequent order.
  • L2EntryUpdate (order size decrease): Occurs when you receive an entry update decreasing the order size on a specific price level. The algorithm considers two possible scenarios:
    • When the trade already happened, the algorithm reduces the quantity of the first order to 6 and subtracts 2 from the second order. This is caused by the FIFO rules of execution the algorithm uses.
    • When there is an order cancellation on the level, the algorithm assumes that the order at the end of the queue was cancelled, and reduces the size from the end of the level.

In the case of an order size decrease, to figure out which of the two possible scenarios occurred, the algorithm tries to match the L2 feed to market data with a feed with trades. Technically, these feeds are stored in the same timebase stream, but logically, they are two independent streams.

The size window parameters sizeOfWindowBeforeInTime and sizeOfWindowAfterInTime tell the system, that if there was an update at time X, it will find the trade between (X - sizeOfWindowBeforeInTime) (in milliseconds) before the message and (X + sizeOfWindowAfterInTime) (in milliseconds) after the message. If it finds the trade that matches the updated entry, it reduces order size from the beginningof the order. If it does not find a matching trade, it assumes there was a cancellation that happened from the end of the price level.

You can also specify size window parameters sizeOfWindowBeforeInMessages & sizeOfWindowAfterInMessages that are measured by the number of TimeBase messages before and after a given point.

Example Scenario​

This section describes an example scenario inside the simulated order book.

simulated-order-book-example

  1. Initial Scenario: At $11.15, the order book has a total size of 7. The Simulator deconstructs the total size into orders of [5] and [2].

  2. Step A: The user places an order with size=3 at $11.15. This information is registered by the simulated order book, however it is not registered by the exchange's order book. The simulated order book at this price level becomes [3] [5] [2], with a total size of 10.

  3. Step B: A market data update occurs, and the exchange tells us that there are 17 contracts at the $11.15 price level. With this update, and including the order placed in Step A, the simulated order book becomes [10] [3] [5] [2].

  4. Step C: On the exchange, we see a trade of size=3 at $11.15 and the total size updated from 17 to 14. The simulated order book at this price level becomes [10] [3] [4].

Dark Market Mode​

When a dark market mode is enabled, SIMULATOR does not rebuild an exchange L2/L3 book. It fills resting orders from the live trade print stream.

Fill price equals the triggering trade price. QuoteFlow book settings (window sizes, executionMode, L2 output feed, and so on) are unused in these modes.

Select the variant with darkMarketMode:

ValueDescription
PROBABLE_FRACTIONStochastic dark matching: eligible resting orders share accessible trade quantity in FIFO order of acceptance, with participation probability and accessible fraction. A marketable order may also fill immediately against the last trade as a proxy for latent contra liquidity.
LAST_TRADENaive complete fill: a marketable order executes at the price of the last received trade (may fill immediately on accept/replace).
NEXT_TRADENaive complete fill: the order stays pending until the next trade arrives and executes at that trade's price (full order size, regardless of the print size).

Legacy darkMarket = true is equivalent to darkMarketMode = PROBABLE_FRACTION. If both are set, darkMarketMode wins.

note

For PROBABLE_FRACTION, parameters participationProbability and accessibleFraction are rough default values, not numbers tuned to match any real venue. Because a dark venue's real order book is hidden, the simulator can't reproduce exactly what would have happened to your order — it produces one realistic, randomized guess at a fill outcome, not the guaranteed true outcome.

Probable fraction​

For each reported trade, the simulator decides fills in two steps:

  1. How much of the trade is fillable at all — controlled by accessibleFraction. Since a dark venue's real order book isn't visible, we don't know how much of a print's quantity was resting liquidity versus, say, one counterparty absorbing the whole size. accessibleFraction estimates the portion of each print that plausibly came from other resting orders like yours, e.g. accessibleFraction = 0.1 means only 10% of each print's quantity is treated as up for grabs; the rest is assumed to have gone to liquidity outside the model.
  2. Which resting orders get a piece of that fillable amount — orders eligible for the print (matching side/price) are checked in FIFO (acceptance-time) order. For each one, participationProbability is the probability that this specific order participates in this specific print at all, modeling the odds that your resting order happened to be matched against this particular piece of flow rather than some other outstanding interest at the venue. If an order "wins" the coin flip, it takes what it can from the remaining accessible quantity (in FIFO priority) and the pool shrinks accordingly for the next candidate; if it "loses," it gets nothing from this print and waits for the next one.

Both parameters default to 0.1 and are uncalibrated placeholders — deliberately conservative, low-fill-rate defaults chosen so that a newly configured dark market doesn't over-fill by default. Tune them per-venue against real fill-rate observations if you have them.

Fills only happen when a trade print arrives, so illiquid or rarely-traded instruments fill rarely, regardless of participationProbability and accessibleFraction. An order resting with no prints at all simply waits — there's no timeout or synthetic fill. The one exception is at order accept/replace: if a print has already occurred at some point in the past, the new order immediately gets one shot to fill against that last print (still gated by both parameters) before settling in to wait for the next one. Since low activity means few chances to draw a winning coin flip, expect resting orders on thin instruments to sit unfilled for long stretches even with default settings; increase participationProbability and/or accessibleFraction if you need faster turnover in this scenario for testing purposes.

Example:

algorithms {
  SDARK: ${template.algorithm.SIMULATOR} {
    subscription {
      streams = [ SOME_FEED ]
    }
    settings {
      darkMarket = true
      # or darkMarketMode = PROBABLE_FRACTION
      participationProbability = 0.1
      accessibleFraction = 0.1
    }
  }
}

Last / Next trade​

Use when you want a simple market-order simulation on a trade-only feed:

  • Decision to send can be based on the current/last trade price.
  • Execution is at the last or the next trade price — full size, complete fill.
  • Advanced simulation with partial fills from a series of trade sizes is out of scope for these modes.

Example:

algorithms {
  SDARK_NEXT: ${template.algorithm.SIMULATOR} {
    subscription {
      streams = [ SOME_FEED ]
    }
    settings {
      darkMarketMode = NEXT_TRADE
    }
  }
}

Configuration​

To deploy SIMULATOR for a specific exchange to Ember, add the following stanza to ember.conf. Let's imagine we want to deploy a simulator of the KRAKEN exchange:

KRAKEN: ${template.algorithm.SIMULATOR} {
  subscription {
    streams = [KRAKEN]
  }
}

Now orders that have KRAKEN set as their destination are routed to this simulated market.

note

SIMULATOR requires market data to have exchange Id field in each market data entry. For the sample above, SIMULATOR expects all market data entries (bids/asks/trades) to have exchangeId set to KRAKEN.

Advanced Settings​

There are a number of settings that can affect SIMULATOR behavior:

ParameterDefault ValueTypeDescription
bookSnapshotInterval60sDurationInterval of market data snapshots.
copiedCustomAttributesnullStringSpecifies the attributes copied from an order request to order events. Comma-separated list.
Example: "1000,1001"
eligibleOrderParametersnullStringOptional. Allows limiting supported order types or time-in-force conditions. Formatted as a comma separated list of pairs containing two values. The first value = order type (e.g., limit, market, stop, stop limit). The second value = order time in force (e.g., immediate or cancel, good till date, good till cancel).
exchangeMappingStringCan be used to map exchange code in market data to exchange code in orders.
executionModePESSIMISTICExecutionModeDetermines whether orders placed close to the other side of the market have a higher/lower chance of execution.
Possible values: OPTIMISTIC, PESSIMISTIC.
If set to PESSIMISTIC, the algorithm executes orders only when they are crossing the market.
If set to OPTIMISTIC, when sending orders inside of a spread, the order executes with a probability depending on how far the order is from the opposite side of the market.
feedMultiDestinationCastAddressStringRelated to timebase connectivity.
feedStreamKeynullStringOptional. When configured, simulator will generate own market data feed into TimeBase.
feedStreamMaxBufferSizeintRelated to timebase connectivity.
feedTopicKeyStringRelated to timebase connectivity.
generateExecutionForLegstruebooleanDetermines how multileg instruments are treated - whether the algorithm generates only one trade from the main instrument of the multileg symbol, or whether it adds different executions for any leg of the multileg instrument. If set to false (default), the algorithm does not generate executions per leg. If set to true, the algorithm generates executions per leg. Keep the default value.
latencies : {
    minValue:
    maxValue:
}		Example:
latencies : {
    minValue: 0.005
    maxValue: 0.05
}		Introduce random latency between events in a minimum & maximum range. In seconds.
localtruebooleanDetermines the security metadata that commissions are taken from. If set to false, Commissions, OrderPricePrecision, and OrderQuantityPrecision are looked up from the Central Security Master API. If set to true, they are taken from the security stream.
maxSnapshotDepthintDetermines market depth of the L2 data feed.
processHistoricalDatafalsebooleanDetermines whether orders placed in SIMULATOR are resumed or canceled after system restart. If set to false, orders are canceled. If set to true, orders are resumed.
sizeOfWindowAfterInMessages100intSize of window in messages for matching trades with messages after trades. Measured in Milliseconds. Default should be good for most cases.
sizeOfWindowAfterInTime100longSize of window in time for matching trades with messages after trades. Measured in Milliseconds. Default should be good for most cases.
sizeOfWindowBeforeInMessages100intSize of window in messages for matching trades with messages after trades. Measured in number of messages coming from exchange. Default should be good for most cases.
sizeOfWindowBeforeInTime100longSize of window in time for matching trades with messages before trades. Measured in number of messages coming from exchange. Default should be good for most cases.
stopOrdersExecutionModeEXECUTE_BY_OPPOSITE_
SIDE_PRICE		StopOrdersExecutionModeDetermines what triggers an order stop. Possible values: EXECUTE_BY_TRADE_PRICE, EXECUTE_BY_OPPOSITE_SIDE_PRICE, EXECUTE_BY_SAME_SIDE_PRICE, EXECUTE_BY_SAME_SIDE_OR_TRADE_PRICE
supportL1DataFormatfalsebooleanConfigures simulator to expect L1 (top of the book) market data only (by default SIMULATOR expects Level 2 - Market By Order data format).
supportLegacyMarketDataFormatfalsebooleanConfigures simulator to read Legacy (Classic) Market data format (by default SIMULATOR expects market data in newer Universal Market Data format).
symbolsLimit100intOptional. Safeguard parameter that limits the total number of symbols the simulator uses to track the order book in one instance. (For some crypto exchanges you may need up to 20Mb/symbol. For traditional markets that have visible order book of 5-10 price levels this estimate is about 100Kb/symbol).
useClassicSimulatorfalsebooleanLightweight simulation mode from the "classic" simulator algorithm from QuantOffice.
darkMarketModenullDarkMarketModeSelects trade-print simulation instead of QuoteFlow L2/L3 book reconstruction. See Dark Market Mode. Values: LAST_TRADE, NEXT_TRADE, PROBABLE_FRACTION.
darkMarketfalsebooleanLegacy. When true and darkMarketMode is unset, equivalent to darkMarketMode = PROBABLE_FRACTION.
participationProbability0.1doublePROBABLE_FRACTION only. Probability that a given resting order is matched against a given eligible trade print, checked independently per order in FIFO order. Higher values mean your resting orders get filled more often. Range [0, 1], uncalibrated placeholder.
accessibleFraction0.1doublePROBABLE_FRACTION only. Fraction of each trade print's quantity treated as available to resting simulated orders (the rest is assumed to have gone to liquidity outside the model). Higher values mean more size is fillable per print. Range [0, 1], uncalibrated placeholder.

All these parameters can be placed into settings {} section of algorithm config:

algorithms {
  SBINANCE: ${template.algorithm.SIMULATOR} {
    subscription {
      streams=[ BINANCE ]
    }
    settings {
      symbolsLimit = 100
      processHistoricalData = false
    }
  }
}

Exchange​

By default QuoteFlow expects market data feed to identify exchange of each market data entry. Grouping market data entries by originating exchange ID enables accurate per-exchange order book tracking.

However, in some cases market data feed coming from single market may not identify exchange or mis-identify exchange. Startiung from QuoteFlow 3.8.32 there is a special flag to ignore exchangeId:

Example:

  SGEMINI: ${template.algorithm.SIMULATOR} {
    subscription {
      streams=["gemini.data"]
    }
    settings {
      symbolsLimit = 100
      processHistoricalData = false
      ignoreExchangeId = true        # <---------- override market data entry exchange with "SIM"
    }

How it will work:

• All market data coming to simulator will seem as market data from exchange with id "SIM". • All orders coming to the SGEMINI destination will seem as orders with exchangeId = SIM, no matter what is specified in exchangeId field • Following these, all events and all market data generated by this simulator will have exchangeId = SIM

Common pitfalls​

Orders are rejected with "Symbol X is not in ALGO market data subscription"​

This reject means the simulator has an explicit subscription.symbols list and subscriptions added dynamically that does not include the symbol the order is targeting.

Fix: Wait for an order book for subscribed symbol to be created OR add the missing symbol to subscription.symbols in the simulator's algorithm config:

SBITSTAMP: ${template.algorithm.SIMULATOR} {
  subscription {
    streams = [BITSTAMP]
    symbols = [BTCUSD, BTCUSDT, SOLEUR, LTCEUR]   # <-- add missing symbol here
  }
}

Market orders are rejected with "no active orders" reason​

Checklist:

  • Make sure order specifies correct exchange (matches exchange in market data)
  • Make sure SIMULATOR subscribed to correct market data streams
  • Check nature of your market data (if you use L1 feed, make sure to configure supportL1DataFormat flag; for classic L1 format set supportLegacyMarketDataFormat flag)
  • Make sure you are not trading during market closed hours (simulator might have seen empty books at the market close)