Skip to main content

Market Data Overview

This document describes the format used by Deltix products to transmit and record financial market data. More specifically, we will focus on Best Bid Offer, Trades, and Market Depth messages that transmit the state of exchange order book. If you are not familiar with these concepts, we suggest reading the following Investopedia articles:

The Deltix Market Data Format (also called universal format in this document) is influenced by FIX Protocol and native APIs of various trading systems that we have previously worked with. It is designed to provide a high-level, normalized format, suitable to capture data from the majority of very different trading venues.

This document is focused on Market Data API messages (prices). If you are interested in the Order Entry API (trading), please refer to the Trading Data model page.

The universal format does not provide data about the status and definition of a security, such as: security tick size, market depth, etc. This information is provided by means of other data messages which are out of scope for this document.

Introduction​

The following diagram illustrates a Central Limited Order Book.

Order Book diagram The ordrer book demonstrates a First In First Out (FIFO) Order Queue, which is a collection of orders at the same price level. Orders advance in the queue when orders that arrived in front of them are cancelled or matched (new arrivals from the opposite side of the market).

The universal format got its name from the fact that it can represent different market depth using similar data entries:

  • Level1: Top of the Book (also called Best Bid Offer or BBO)
  • Level2: Market By Level (an order book that aggregates same-priced bids and offers)
  • Level3: Market By Order (streaming information about individual orders standing on the exchange)

Streams​

Market Data flows in streams. Streams typically combine messages of the same source. For example, we can imagine a stream of equities prices from the NYSE exchange. In some cases, (ticker plants), a stream can contain data from multiple exchanges (e.g., Forex spot prices).

The special field exchangeId identifies the information source (source exchange) of each stream data entry.

One stream can provide data of different levels of details even within one exchange (e.g. both Best Bid Offer messages and Market By Level). Each stream consumer can decide which level of details they need.

Typically, streams correspond to TimeBase streams, but the model described below can be used separately.

Data Packages​

Packages are used to group data entries together. Usually, all data entries within a package are a part of the same exchange transaction and are reported using the same timestamp.

Snapshots vs. Updates​

Snapshots​

From time to time, exchanges publish a snapshot of an entire order book. For example, for Level2 market data, the snapshot will contain all price levels for bids and offers. For exchanges that report very deep order books, these snapshot packages can be very large (hundreds or even thousands of data entries).

Updates​

To reduce network bandwidth, most of the time, exchanges transmit incremental updates. These packages report only what has changed on the exchange.

For example (Level2):

  • Insertion of new price level (somebody sent a new order at a different price)
  • Change of available size on a certain price level (somebody increased or decreased their order, or cancelled it)
  • Removal of new price level (somebody cancelled the last order standing at this price)
note

An incremental update package can contain messages from multiple exchanges.
However, each snapshot package must contain data from a single exchange only.

Data Package​

Each package instance contains two fields: a package type and a list of data entries.

Package Type​

The package type is an enumeration of:

  • VENDOR_SNAPSHOT – a snapshot that comes directly from a data vendor (e.g., an exchange).
  • PERIODICAL_SNAPSHOT - a synthetic snapshot generated by Deltix Market Data Aggregation software to simplify data validation and allow late joiners (consumers that appear in the middle of trading session).
  • INCREMENTAL_UPDATE – an incremental update, a list of insert/update/delete data entries.

This format does not support snapshot and increment messages mixed in one package. Trade entries can be easily combined with increments.

It is important to differentiate one type of snapshot from another. When you process a market data stream, a PERIODIC_SNAPSHOT can be skipped, whereas a VENDOR_SNAPSHOT must be processed in one way or another. A periodical snapshot does not provide a state change. It contains data to initialize a state for new subscribers or to initialize a back-testing run starting from a random point in time. However, a vendor snapshot, as a rule, contains state changes and has been sent on the initiative of the vendor or because some logic requested it (for example, because of a possible error in the current state).

Each snapshot reports entire state of the book. (A snapshot of both sides of the market must be reported in one package).

API:

  • Deltix.Timebase.Api.Messages.Universal.PackageType (C#)
  • deltix.timebase.api.messages.universal.PackageType (Java)

Data Entries​

As a main payload, each package has a list of package entries. Packages cannot be empty.

Package entries can be of the following types:

The hierarchy of entries is presented below:

Market Data Package Entries

Note that:

  • BBO data is supplied as L1Entry.
  • L2 data (FIX IncrementalRefresh/FullRefresh, CME.SBE, FAST, etc.) is converted into:
    • L2EntryNew for new price levels
    • L2EntryUpdate for updating or deleting price levels
  • L3 data (MBO.FIX, NASDAQ ITCH, BATS.PITCH) is converted into:
    • L3EntryNew for new price levels
    • L3EntryUpdate for updating or deleting price levels
  • Market trades are presented as the TradeEntry type.
  • The Book Reset Entry type can communicate that the resulting order book should be cleared.

Level1: BBO message (Bid, Ask, or both)​

This type may represent both exchange-local top of the book (BBO), as well as National Best Bid Offer (NBBO). You can use the property isNational to filter out NBBO messages. This is always a one sided quote, unlike the old BestBidOfferMessage, which is two-sided (with nullable properties).

note

The unique key for Level1 data entry is a combination of symbol, exchange, and side fields.

Level2: New or Update​

This type reports incremental Level2-updates, such as the insertion, deletion, or update of one line in the order book, either on the ask or bid sides. It can also encode Level2-snapshot entries. A snapshot is a message that reports the full order book state (snapshot) at once. Note that Level2 is a level-oriented depth-of-the-book format and should be used whenever the price or price-level index is used to locate book changes.

note

The unique key Level2 data entry is a combination of symbol, exchange, side, and price-level indexes.

Level2 also supports individual quote books of arbitrary depth. However, if the Quote ID is used to identify changes, the Level3 data type should be used instead.

Level3: New or Update​

This type reports incremental updates in Market-By-Order feeds, such as the addition, cancelation, modification, and replacement of one quote in the order book, either on the ask or offer sides. It also can encode Level3-snapshot entries. Note that Level3 is a quote oriented depth-of-the-book format and should be used whenever the Quote ID is used to locate book changes.

note

The unique key of Level3 data entry is a combination of symbol, exchange, side, and quote IDs.

Trade Entry​

The Trade Entry type communicates a trade (bid and offer match) that has happened on the market. Trades are not specific to any level - they can be sent within Level1, Level2 or Level3 packages. Note that Level3 trades usually provide more information (for example, Quote IDs from each counterparty).

Book Reset Entry​

Book Reset is a special type of entry that communicates that the market data provider wants you to clear all entries in an accumulated order book.

The Book Reset can only be included as a single entry within snapshot packages (and cannot appear in incremental update packages).

note

When you receive BookResetEntry you should clear the book and be ready to process next incremental update or snapshot message.

Statistics Entry​

Statistics Entry provide additional information about market conditions. Each entry can communicate one of:

  • Opening or Closing price
  • Settlement Price
  • Session VWAP Price
  • Open Interest
  • Trade Volume
  • Imbalance
  • Session High or Low price
  • etc.

Statistics Entry can appear in both Snapshot (e.g. trading volume so far) and incremental Update messages (e.g. Session High price changed). Statistics Entries must be supported by upstream data feed API, and require special support in each Deltix data connector.