Streaming Orders and Trading History into TimeBase

Overview

This document explains how to configure and use TimeBase as data warehouse for the Execution Server.

TimeBase is a highly efficient time-series data storage that can store Ember trading history.

Install and Configure TimeBase

The Ember installation usually already includes a local installation of TimeBase. For more information, see the TimeBase documentation.

Configure the Execution Server

TimeBase Streams

As of Ember version 1.0, you can set up a warehouse service to export data into TimeBase.

This service creates two TimeBase streams:

• Orders Stream: Records the final state of each order that Ember processed. This stream contains one message per order.
• Messages Stream: Encompasses all trading flow messages, including trading requests (order submissions, cancellations, modifications) and trading events (order ACKs/NACKs, cancellation ACKs/NACKs, fills, etc.). You can configure a filter to reduce this stream to trade events only (fills). This significantly reduces stream size.

Data Export Modes

The Execution Server can export data into TimeBase in two modes:

• Live Mode: A special daemon service exports data in near real-time.
• Batch Mode: A periodic process exports all recently accumulated data in batches.

Configuration

To configure the Execution Server, add the following section to $EMBER_HOME/ember.conf.

Note that:

• The data export mode is controlled by the live parameter.
• You can change the names of the TimeBase output streams.

warehouse {
  timebase {            # unit id, you will use it when you run the app, it might be any
    live = true         # the process does not wait for new data and exits if false
    
    messages = [                               
      ${template.warehouse.timebase.messages} { # loader which loads order messages
        filter.settings.inclusion = null        # EVENTS or TRADES, null means no filter
        
        loader.settings {                 # default                        # description
          url = "dxtick://localhost:8011" # ${timebase.settings.url}       # server url
          username = "john"               # ${timebase.settings.username}  # username
          password = "********"           # ${timebase.settings.password}  # password
          stream = "messages"             # messages # stream name
          clear = false                   # false    # clears stream on startup      
          purgePeriod = 1d                # 0        # purges stream repeatedly at period                                                    
          purgeWindow = 2d                # 0        # purges stream up to (now - purge window)
        }
      }
    ]

    orders = [                               
      ${template.warehouse.timebase.orders} {   # loader which loads closed orders
         loader.settings {                # default                        # description
          url = "dxtick://localhost:8011" # ${timebase.settings.url}       # server url
          username = "john"               # ${timebase.settings.username}  # username
          password = "********"           # ${timebase.settings.password}  # password
          stream = "orders"               # orders    # stream name
          clear = false                   # false     # clears stream on startup      
          purgePeriod = 1d                # 0         # purges stream repeatedly at period                                                    
          purgeWindow = 2d                # 0         # purges stream up to (now - purge window)
        }
      }
    ]
  }
}

Start Exporting Data

To begin streaming Execution Server data into TimeBase, use the data-warehouse service. This service reads the Ember Journal and loads all trading messages and completed orders into the Messages and Orders TimeBase streams.

note

Active orders are not exported until they are complete (completely filled, cancelled, or rejected). However, events concerning active orders (for example, trades) appear in the Messages stream almost immediately.

Run Ember's data-warehouse script with single argument that specifies the TimeBase data warehouse:

export EMBER_HOME=/deltix/emberhome
/deltix/ember/bin/data-warehouse timebase

In Batch Mode, when the live parameter is set to false, this script exits as soon as all recent data is exported. If you re-run the script, it appends any new data that the Execution Server accumulated since the last export.

View Data

You can view data in the TimeBase Administrator GUI, available for both desktop and web platforms.

Here is a screenshot of the desktop version:

Data Format

This section covers the TimeBase-specific aspects of the data model.

Data stored in Ember's TimeBase data warehouse is described in detail in the Trading Data Model document.

Message Identity and Gap Filling

Monotonic Message Sequence Numbers

Ember uses a monotonic sequence number to uniquely identify each message. With this approach, if the data warehousing pipeline is restarted, a gap in stored messages can be identified and filled.

The synthetic message identifier is more reliable than a message timestamp since Ember's API allows clients to set inbound request timestamps, and clients may have unsynchronized clocks.

Under the hood, the message sequence number defines the offset of an encoded message in the Ember Journal (a binary sequence of bytes). As time goes by, the journal grows and special a special journal compaction mechanism deletes historical journal data partitions. However, message sequence numbers continue to grow as if the journal has an indefinite size (INT64). Journal compaction never deletes data that hasn't been processed by all data warehouse pipelines configured in the system.

For more information, visit the Ember Journal documentation.

Journal Term

On special occasions, clients may delete the Ember Journal. When this happens, the entire trading history on which Ember OMS operates, including message sequence numbers, resets. This presents a problem for existing data warehouse pipelines, where the sequence numbers of stored messages might seem to emerge from the future.

To remedy this situation, Ember introduces the concept of a journal term, which increases each time the journal is created. In the near future, the TimeBase data warehouse will be improved to take the journal term into account when determining the last stored message.

If the journal term doesn’t match with the term used by the last stored message, the data warehouse fails to start. System operators must delete or move the data warehouse storage produced from previous term. For the TimeBase data warehouse, operators must delete or rename the TimeBase streams used for storing Ember data.

For TimeBase data warehouse term is stored in description field of orders or messages stream. Users are strongly advised against manually editing this field.

Timestamps

TimeBase is a time-series database. It is used by Ember in an append-only mode, meaning that Ember doesn't update previously warehoused messages. In each TimeBase stream, messages are strictly sorted in chronological order according to their timestamp.

When Ember stores data into TimeBase, it takes special care to prevent timestamps from going backwards. For example, if an order request uses a timestamp that appears to be from the past relative to a previously stored message, the problematic timestamp is overridden. In other words, Ember's data warehouse pipeline strives to preserve a strict chronological order of timestamps by overriding "late" timestamps with previously stored timestamps.

In addition to a normal timestamp, some messages like trade evens may include the originalTimestamp field. This field is recorded in TimeBase as-is, without any modifications.

Orders Stream

The Orders stream captures the final state of each order and contains one message per order.

Column	Type	Example	Description
SourceId	ALPHANUMERIC(10) String	CLIENT52	Order source, ALPHANUMERIC(10)
DestinationId	ALPHANUMERIC(10) String	TWAP	Order destination, ALPHANUMERIC(10)
OrderId	String	ICAP1983EE	Identifies each order for Execution Server, unique per-source. OrderID is assigned by order source.
ParentSourceId	ALPHANUMERIC(10) String	CONTROL	Identifies source of parent order (optional)
ParentOrderId	String	ICAP321XX1	Identifies parent order (optional)
ExternalOrderId	String	ZZ132131	Optional order identifier assigned to the order by execution venue. For example, if we send order to execution venue, like CME they assign their own order identifier. This identifier can be subsequently used to locate this order on CME.
Account	String	Gold	Identifies order account
Trader	String	jdoe	Identifies trader who submitted this order
Symbol	String	EUR/USD	Order symbol (in symbology configured inside Deltix system)
InstrumentType	Enum	FX	Instrument type
Exchange	ALPHANUMERIC(10) String	HOTSPOT	Destination exchange (if available)
Currency	ALPHANUMERIC(10) String	USD	Order currency. Optional. Usually used only for orders that use term currency (rather than base currency).
Side	Enum	BUY	Order side
TimeInForce	Enum	DAY	Order time In force
ExpireTime	Long (Linux Epoch Time)	2019-02-27 17:00:00.000	Order expiration time (only for GOOD_TILL_DATE orders)
OrderStatus	Enum	CANCELLED	Final state of the order
OpenTime	Long (Linux Epoch Time)	2019-02-27 16:51:48.002	Order submission time
CloseTime	Long (Linux Epoch Time)	2019-02-27 16:51:48.120	Order completion time
OrderType	Enum	PEG_TO_MIDPOINT	Order type
LimitPrice	DECIMAL64 as Long	1.33	Limit price. Can be specified for LIMIT, STOP_LIMIT, PEGGED, or CUSTOM order types
StopPrice	DECIMAL64 as Long	1.20	Stop price. Can be specified for STOP and STOP_LIMIT order types.
Quantity	DECIMAL64 as Long	10000	Order quantity
DisplayQuantity	DECIMAL64 as Long	1000	Order display quantity (some time described as “max show quantity” or “max floor quantity”), where applicable.
MinQuantity	DECIMAL64 as Long	1000	Minimum fill quantity (where applicable).
CumulativeQuantity	DECIMAL64 as Long	1500.50	Cumulative filled quantity
AveragePrice	DECIMAL64 as Long	132.56	Average fill price
VendorRejectCode	Int	1003	Vendor specific reject code. For example, CME’s. Since Ember 1.4.
DeltixRejectCode	Int	120	Reject code in Deltix classification. Since Ember 1.4.
Reason	String	“Cancelled by user request”	For cancelled or rejected orders this field contains textual reason.

Messages Stream

The Messages stream contains all the real-time activity related to orders.

This stream records:

• Order requests, including original submission, cancellation, and order modification requests.
• Order events, including order acknowledgements, cancellation confirmations, trades, etc.

This TimeBase stream is polymorphic, meaning that each message type is represented as a dedicated message class. For example, trade messages are filtered for the OrderTradeReportedEvent type.

For a better understanding of trading workflows in the Execution Server, refer to the beginning of this section.

Common Attribute	Type	Example	Description
Timestamp	Timestamp	2019-02-27 16:51:48.123	Timestamp provided by request originator or event producer. See Timestamps section for more details.
OriginalTimestamp	Timestamp	2019-02-27 16:51:48.123	Timestamp reported by execution venue (optional).
SourceId	ALPHANUMERIC(10) String	CLIENT52	Order source, ALPHANUMERIC(10)
DestinationId	ALPHANUMERIC(10) String	TWAP	Order destination, ALPHANUMERIC(10)
OrderId	String	ICAP1983EE23	Identifies order for Execution Server, unique per-source.
OriginalOrderId	String	ICAP1983EE22	For order replacement request, as well as events that relate to cancel replace workflow (such as PendingReplace, ReplaceReject, and Replace ACK) this field identifies original order in cancel-replace chain.
CorrelationOrderId	String	ICAP1983EE00	Identity of the first order in cancel-replace chain. Same as OrderID for orders that do not (yet) participate in cancel-replace workflow.
ParentSourceId	ALPHANUMERIC(10) String	CONTROL	Identifies source of parent order (optional)
ParentOrderId	String	ICAP321XX1	Identifies parent order (optional)
RequestId	String	XCL#554	For order cancel request, as well as cancel ACK and cancel NACK events this field identifies specific cancel request.
ExternalOrderId	String	ZZ132131	Optional order identifier assigned to the order by execution venue
EventId	String	AAAT31231	Optional attribute available for events coming from some venues. This attribute allows identifying duplicate events. May have different uniqueness scope, but must be unique at least in the context of single order. NOTE: OMS is responsible for filtering out duplicate events before they reach data warehouse or other downstream consumers.
ReferenceEventId	String		Used by trade correction and cancellation events to identify previously communicated event that has to be corrected or cancelled.
OrderStatus	Enum	PARTIALLY_FILLED	Order status (available for order events only).
Symbol	String	EUR/USD	Order symbol (in symbology configured inside Deltix system)
InstrumentType	Enum	FX	Instrument type
Currency	CHAR(10)	USD	Order currency. Optional. Usually used only for orders that use term currency (rather than base currency).
Exchange	ALPHANUMERIC(10) String	HOTSPOT	Destination exchange (if available) for outbound messages and source exchange for inbound messages. For example, fills will report their exchange in this field.
Account	String	Gold	Identifies order account
Trader	String	jdoe	Identifies trader who submitted this order
Side	Enum	BUY	Order side
TimeInForce	Enum	GOOD_TILL_CANCEL	Order time in force condition
ExpireTime	Long (Linux Epoch Time)	2019-02-27 17:00:00.000	Order expiration time (only for GOOD_TILL_DATE orders)
Quantity	FLOAT8	100.50	Order quantity
MinQuantity	FLOAT8	10	Minimum quantity to execute (optional order request attribute)
DisplayQuantity	FLOAT8	5	Minimum quantity to display on exchange floor (optional order request attribute)
OrderType	Enum	LIMIT	Order Type
LimitPrice	FLOAT8	1.33	Limit price. Can be specified for LIMIT, STOP_LIMIT, PEGGED, or CUSTOM order types
StopPrice	FLOAT8	1.20	Stop price. Can be specified for STOP and STOP_LIMIT order types.
PegDifference	FLOAT8	0.03	Peg offset, in order money. Optional attribute for PEGGED order types.
AveragePrice	FLOAT8	1.325	Average execution price (order events only).
CumulativeQuantity	FLOAT8	25	Cumulative executed quantity  (order events only).
RemainingQuantity	FLOAT8	75.50	Remaining order quantity (part of original order quantity that is still working on the market)
TradePrice	FLOAT8	1.321	Trade events only: price of individual trade described by this event. Not to be confused with average price of all trade events reported so far for an order (AveragePrice field).
TradeQuantity	FLOAT8	5	Trade events only: size of individual trade described by this event. Not to be confused with total executed size reported so far by all trade events of an order (CumulativeQuantity field).
Commission	FLOAT8	0.0001	Trade commission (when known)
CommissionCurrency	ALPHANUMERIC(10) String	USD	Trade commission currency (when known, by default assume order currency)
CounterPartySourceId	ALPHANUMERIC(10) String	JOHN	Identifies source of other side of the trade (when reported)
CounterPartyOrderId	String	FED76123155	Identifies other side of the trade (when reported)
SettlementDate	Long (Linux Epoch Time)		Trade settlement date (when reported)
TradeDate	Long (Linux Epoch Time)		Trade date (when reported)
Reason	String	Market is closed	Reason communicated for cancel or reject events.
VendorRejectCode	Int	1003	Vendor specific reject code. For example CME’s.
DeltixRejectCode	Int	120	Reject code in Deltix classification
MultiLegReportingType	Enum		Used for trade reports when order instrument is exchange traded-synthetic. Identifies single-leg trade or whole contract trade of multi-legged security.
AggressorSide	Enum		Reports our side as passive or aggressive role in this trade.
OrderUnknown	Boolean		Flag used by order Cancel Reject events.
CancelType	Enum		Enum used by Cancel events.
ExecRestatementReason	Enum		Used by Order Restate Events to classify restate type
Flags	Int	3	Order flags. Bitmask containing various order flags. For example, bit 0 marks manual order.
UserData	String	Foo152	User-provided order tag
Attributes	List	[{"key":6001,"value":"4h"},{"key":6002,"value":"FAST"}]	Custom order attribute key (numeric, corresponds to custom FIX tags specified during order submission)

Updating TimeBase Stream Schema Format

On rare occasions, the data format of the TimeBase streams used by the Ember data warehouse changes. In most of these cases, the Ember schema either introduces new fields or deprecates old fields. You do not have to migrate existing streams, as TimeBase can write data using the old schema.

However, if you want to keep your warehouse up-to-date, you can convert the stream format using QQL.

Update Options

There are two options for updating the stream format with QQL:

• Use the TimeBase Administrator GUI and run a query there.
• If you do not have access to the GUI, use a TimeBase CLI called tickdb shell, located in the QuantServer bin/ directory.

The following command line connects to a TimeBase server running on localhost at port 8011 and executes the upgrade script e:\orders-1.7.qql:

tickdb -db dxtick://localhost:8011 -exec open -exec exec e:\orders-1.7.qql -exec quit

Sample

Here is a sample from the actual schema upgrade script files that Deltix provides with documentation:

MODIFY STREAM "orders" (
    CLASS "deltix.timebase.api.messages.common.ApiMessage" (
        "destinationId" 'Destination Id' VARCHAR ALPHANUMERIC (10) COMMENT 'Identifies message final destination (trading connector). Can be null.',
        "originalTimestamp" 'Original Timestamp' TIMESTAMP COMMENT 'In contrast to Timestamp field that captures moment of time this message is recorded into TimeBase,',
        "sourceId" 'Source Id' VARCHAR ALPHANUMERIC (10) COMMENT 'Identifies message source (e.g. FIX Sender Comp ID or trading connector). For internal orders source identifies source Anvil component (e.g. "TWAP" algo).Source is used for ID uniqueness.'
    )
        NOT INSTANTIABLE
        COMMENT 'Base class for all trade messages (requests and events)';
    CLASS "deltix.timebase.api.messages.trade.CustomAttribute" 'CustomAttribute' (
        "key" 'Key' INTEGER SIGNED (32) COMMENT 'UNDOCUMENTED',
        "value" 'Value' VARCHAR COMMENT 'UNDOCUMENTED'
    )
        COMMENT 'UNDOCUMENTED';
    ENUM "deltix.timebase.api.messages.trade.OrderStatus" 'OrderStatus' (
        "PENDING_NEW" = 0,
        "NEW" = 1,
        "REJECTED" = 2,
        "PENDING_CANCEL" = 3,
        "CANCELED" = 4,
        "PENDING_REPLACE" = 5,
        "REPLACED" = 6,
        "PARTIALLY_FILLED" = 7,
        "COMPLETELY_FILLED" = 8,
        "EXPIRED" = 9,
        "SUSPENDED" = 10
    );
    ENUM "deltix.timebase.api.messages.trade.OrderType" 'OrderType' (
        "CUSTOM" = 0,
        "MARKET" = 1,
        "LIMIT" = 2,
        "STOP" = 3,
        "STOP_LIMIT" = 4,
        "PEG_TO_MARKET" = 5,
        "PEG_TO_MIDPOINT" = 6,
        "PEG_TO_PRIMARY" = 7,
        "MARKET_ON_CLOSE" = 8,
        "LIMIT_ON_CLOSE" = 9,
        "LIMIT_OR_BETTER" = 10,
        "PREVIOUSLY_QUOTED" = 11
    );
    ENUM "deltix.timebase.api.messages.trade.Side" 'Side' (
        "BUY" = 0,
        "SELL" = 1,
        "SELL_SHORT" = 2,
        "SELL_SHORT_EXEMPT" = 3
    );
    ENUM "deltix.timebase.api.messages.trade.TimeInForce" 'TimeInForce' (
        "DAY" = 0,
        "GOOD_TILL_CANCEL" = 1,
        "AT_THE_OPENING" = 2,
        "IMMEDIATE_OR_CANCEL" = 3,
        "FILL_OR_KILL" = 4,
        "GOOD_TILL_CROSSING" = 5,
        "GOOD_TILL_DATE" = 6,
        "AT_THE_CLOSE" = 7
    );
    CLASS "deltix.timebase.api.messages.tradestate.Order" 'Order' UNDER "deltix.timebase.api.messages.common.ApiMessage" (
        "account" 'Account' VARCHAR COMMENT 'Order Trading Account',
        "attributes" 'Attributes' ARRAY(OBJECT("deltix.timebase.api.messages.trade.CustomAttribute") NOT NULL) COMMENT 'Custom attributes',
        "averagePrice" 'Average Price' FLOAT DECIMAL64 COMMENT 'Average fill price for currently executed shares for chain of orders. Corresponds to FIX tag AvgPx(6).This field is undefined if and only if cumulative quantity is undefined.(Since some instruments are executed with negative prices or sometimes even with price equal to zero thereis no way to represent ''undefined'' value other than binding this field to <tt>Cumulative Quantity</tt>).',
        "clearingAccount" 'Clearing Account' VARCHAR COMMENT 'Order Clearing Account',
        "closeSequence" 'Close Sequence' INTEGER COMMENT 'Order completion sequence number. Can be used as unique synthetic timestamp of each message. Unique per Partition/server.',
        "closeTime" 'Close Time' TIMESTAMP COMMENT 'UNDOCUMENTED',
        "commissionCurrency" 'Commission Currency' VARCHAR ALPHANUMERIC (10) COMMENT 'Identifies currency used for commission in alphanumeric encoding. Wherever possible Deltix uses ISO 4217 currency codes.Crypto currencies use Centralized Security Master codes (for example, Bitcoin is "BTC" rather than "XBT").',
        "cumulativeQuantity" 'Cumulative Quantity' FLOAT DECIMAL64 COMMENT 'Currently executed number of contracts for chain of orders. Corresponds to FIX tag CumQty(14).',
        "currency" 'Currency' VARCHAR ALPHANUMERIC (10) COMMENT 'Identifies currency used for prices in alphanumeric encoding. Wherever possible Deltix uses ISO 4217 currency codes.Crypto currencies use Centralized Security Master codes (for example, Bitcoin is "BTC" rather than "XBT").',
        "deltixRejectCode" 'Deltix Reject Code' INTEGER SIGNED (32) COMMENT 'Deltix-spec rejection code. See DeltixRejectCodes.',
        "displayQuantity" 'Display Quantity' FLOAT DECIMAL64 COMMENT 'Can be used for iceberg orders. Set to TypeConstants.DECIMAL64_NULL when not defined."MaxShow/DisplaySize" order quantity in fixed-point representation.',
        "exchangeId" 'Exchange Id' VARCHAR ALPHANUMERIC (10) COMMENT 'Order Exchange or ECN. Required for destinations that act as a gateway to final execution venue.This field stores textual exchange code as INT64 using ALPHANUMERIC(10) encoding.Use ExchangeCodec helper to encode/decode text to numeric.',
        "expireTime" 'Expire Time' TIMESTAMP COMMENT 'Order Expiration time. Required for TimeInForce.GOOD_TILL_DATE orders.',
        "externalOrderId" 'External Order Id' VARCHAR COMMENT 'External Order ID assigned to the order by execution venue.Similar to tag OrderID(37) in FIX ExecutionReport.',
        "flags" 'Flags' INTEGER SIGNED (32) COMMENT 'Order Flags bitmask (See OrderFlagsHelper and OrderFlags enum)',
        "limitPrice" 'Limit Price' FLOAT DECIMAL64 COMMENT 'Limit price. Required for LIMIT or LIMIT_STOP orders (and some custom orders). Set to TypeConstants.DECIMAL64_NULL when not defined.',
        "minQuantity" 'Min Quantity' FLOAT DECIMAL64 COMMENT 'Optional attribute supported by some execution venues. Set to TypeConstants.DECIMAL64_NULL when not defined.Minimum quantity allowed for execution, in fixed-point representation.',
        "moduleKey" 'Module Key' VARCHAR COMMENT 'Used by QuantOffice orders to identify strategy that placed this order. Other clients can use this as custom field for order tagging and position tracking.',
        "openSequence" 'Open Sequence' INTEGER COMMENT 'Order submission sequence number. Can be used as unique synthetic timestamp of each message. Unique per Partition/server.',
        "openTime" 'Open Time' TIMESTAMP COMMENT 'UNDOCUMENTED',
        "orderId" 'Order Id' VARCHAR COMMENT 'Unique identifier for Order as assigned by  (broker, exchange, ECN). Uniqueness must be guaranteed for each order source.Firms which accept multi-day orders should consider embedding a date within the OrderID field to assure uniqueness across days.Note that in case of cancel-replace chains (where each node has its own identifier) this field stores ID of the most recent node',
        "orderStatus" 'Order Status' "deltix.timebase.api.messages.trade.OrderStatus" COMMENT 'Describes current state of order, if known. Similar to tag OrderStatus(39) in FIX ExecutionReport.',
        "orderType" 'Order Type' "deltix.timebase.api.messages.trade.OrderType" COMMENT 'UNDOCUMENTED',
        "parentOrderId" 'Parent Order Id' VARCHAR COMMENT 'UNDOCUMENTED',
        "parentSourceId" 'Parent Source Id' VARCHAR ALPHANUMERIC (10) COMMENT 'UNDOCUMENTED',
        "partitionId" 'Partition Id' VARCHAR COMMENT 'Identifies "partition" server that produces sequence numbers',
        "pegDifference" 'Peg Difference' FLOAT DECIMAL64 COMMENT 'Peg Difference for Pegged order types. Optional - implied value is zero. Amount (signed) added to the price of the peg for a pegged order.',
        "portfolioKey" 'Portfolio Key' VARCHAR COMMENT 'Used by QuantOffice orders to identify portfolio this order belongs to. Other clients can use this as custom field for order tagging and position tracking.',
        "quantity" 'Quantity' FLOAT DECIMAL64 COMMENT 'Order quantity in fixed-point representation.Order Quantity is required for any order processing service like Deltix Execution Server.',
        "quoteId" 'Quote Id' VARCHAR COMMENT 'Identifies quote/rate we want to deal with. May be required for some FX trading destinations. Quote ID usually comes from Bid/Ask Quote IDs of market data feed.',
        "reason" 'Reason' VARCHAR COMMENT 'Textual Order cancellation or rejection reason.',
        "side" 'Side' "deltix.timebase.api.messages.trade.Side" COMMENT 'Order side.',
        "stopPrice" 'Stop Price' FLOAT DECIMAL64 COMMENT 'Order stop price. Required for STOP and LIMIT_STOP orders (and some custom orders).  Set to TypeConstants.DECIMAL64_NULL when not defined.',
        "timeInForce" 'Time In Force' "deltix.timebase.api.messages.trade.TimeInForce" COMMENT 'Usually (but not always) required attribute that specifies how long the order remains in effect',
        "totalCommission" 'Total Commission' FLOAT DECIMAL64 COMMENT 'UNDOCUMENTED',
        "traderGroup" 'Trader Group' VARCHAR COMMENT 'Assigned by Deltix Trading Node and identifies single group that order''s Trader belongs to.Helps to filter out orders available in order reports.',
        "traderId" 'Trader Id' VARCHAR COMMENT 'User who submitted this order (manually or automatically)',
        "userData" 'User Data' VARCHAR COMMENT 'User-defined cookie that will be reported back in order events. For example, order reason. Sometimes custom order attributes are sometimes too heavy.',
        "vendorRejectCode" 'Vendor Reject Code' INTEGER SIGNED (32) COMMENT 'Vendor-spec rejection reason (for example https://www.cmegroup.com/confluence/display/EPICSANDBOX/iLink+Reject+Codes ).This field must be filled if reject comes from execution venue (vendor) otherwise it will be set to MIN_INTEGER (Integer.MIN_VALUE).If vendor does not provide reject codes this field will be set to zero.'
    )
        COMMENT 'UNDOCUMENTED';
)
OPTIONS (POLYMORPHIC; PERIODICITY = 'IRREGULAR'; DF = 1; HIGHAVAILABILITY = FALSE)
CONFIRM CONVERT_DATA

Reading the TimeBase Data Warehouse with Java

The following code sample shows how to read trades from a data warehouse Messages stream using Java:

private static void consumeLiveStream(DXTickDB timebase, String streamKey) throws ParseException {

    System.out.println("Connected to TimeBase. Reading stream " + streamKey);

    //String [] messageTypes = null; // null = all
    String [] messageTypes = {
        OrderTradeReportEvent.class.getName()
    };

    // Sample will start reading data recorded after this time stamp
    final long startTime = parseTimestamp ("2020-12-01 12:00:00.000");

    SelectionOptions options = new SelectionOptions (false, true); // live mode = will wait for new data to arrive
    options.typeLoader = ParentIgnoringTypeLoader.INSTANCE; // Suppress some warnings
    try (TickCursor cursor = timebase.select(startTime, options, messageTypes, getStream(timebase, streamKey))) {
        consume (cursor);
    }
}

private static void consume(TickCursor cursor) {
    while (cursor.next()) {
        InstrumentMessage message = cursor.getMessage();
        if (message instanceof OrderTradeReportEvent) {
            processTradeReport((OrderTradeReportEvent) message);
        }
    }
}

private static void processTradeReport(OrderTradeReportEvent tradeReport) {
    // This code shows how to convert fields into more conventional data types

    Date timestamp = new Date(tradeReport.getTimeStampMs());
    long originalTimestamp = tradeReport.getOriginalTimestamp(); // when message is recorded into timebase


    String sourceId = AlphanumericCodec.decode(tradeReport.getDestinationId());
    String orderId = CharSequenceUtil.toString(tradeReport.getOrderId());

    Side side = tradeReport.getSide();
    String symbol = tradeReport.getSymbol().toString();
    double tradeQuantity = Decimal64Utils.toDouble(tradeReport.getTradeQuantity());
    double tradePrice = Decimal64Utils.toDouble(tradeReport.getTradePrice());

    String connectorId = AlphanumericCodec.decode(tradeReport.getSourceId());
    String exchange = AlphanumericCodec.decode(tradeReport.getExchangeId());
    String account = CharSequenceUtil.toString(tradeReport.getAccount());
    String traderId = CharSequenceUtil.toString(tradeReport.getTraderId());
    String tradeExecId = CharSequenceUtil.toString(tradeReport.getEventId());

    String counterPartySourceId = AlphanumericCodec.decode(tradeReport.getCounterPartySourceId());
    String counterPartyOrderId = CharSequenceUtil.toString(tradeReport.getCounterPartyOrderId());

    System.out.printf("%tT Order %s:%s traded - %s %f %s @ %f (ExecID:%s)%n",
            timestamp, sourceId, orderId, side, tradeQuantity, symbol, tradePrice, tradeExecId);
}