oms-debugger

Overview

This document describes the OMS Debugger, which is part of the Ember diagnostic tools. This tool is designed to troubleshoot order transitions recorded in the system journal. It works by replaying journal messages from the beginning and applying them to orders. The tool provides a command-line interface (CLI) that allows users to step through individual messages as they are replayed and to query and examine the state of orders at specific breakpoints.

Setup

To use the OMS Debugger, follow these steps:

1. Open a console window.

2. Set the EMBER_HOME environment variable to the location of your Ember home. If your working files are located on a fast drive, you can also set the EMBER_WORK environment variable to a directory containing system log files (both journal.data and journal.metadata files will be needed).

   set EMBER_HOME=/deltix/emberhome

3. Launch the tool:

   <ember-deploy-dir>/bin/oms-debugger

How it works

The OMS Debugger reads and replays the journal, providing the CLI that allows users to step through individual messages and set breakpoints to inspect the Order Management System (OMS) state at specific points.

Step 1

After starting, the OMS debugger stops at Step 1 and displays information about the first message in the journal and the corresponding order, prompting for a command:

~> oms-debugger
[Step 1]    (step>=1)
Message: {"$type":"OrderNewRequest","timeInForce":"DAY", "currency":"USD","orderId":"1570047149342","side":"SELL","quantity":4,"orderType":"MARKET", "exchangeId":"FILL","flags":0,"sequence":3,"symbol":"BTCUSD","instrumentType":"FX","timestamp":"2019-10-02T20:12:37.600Z", "sourceId":"TRADER-CLI", "destinationId":"SIM"}
Order: (1570047149342) SELL 4 BTCUSD @ MARKET UNACKNOWLEGED (TRADER-CLI)=>(SIM), Exchange: FILL, TimeInForce: DAY, Filled: 0, Remaining: 4
Total number of active orders: 1
==> 

help

The help command (or h) provides a list of all available commands and their usage information:

==> help
Usage:
	(s | step) [<number_of_steps>]
	(c | continue) [-o <order_id>] [-p <price>] [-s <source_id>] [-d <destination_id>] [-i <symbol>] [-n <sequence>] [-t <timestamp>] [<number_of_steps>]
	(b | break) [([-o <order_id>] [-s <source_id>] [-d <destination_id>] [-i <symbol>] [-t <timestamp>][-n <sequence>][-p <price>]) | -l | -r (<breakpoint_index> | all)]
	(l | list) [-a] [-o <order_id>] [-s <source_id>] [-d <destination_id>] [-e <side>] [-i <symbol>][-p <price>][-l <limit>][-t <timestamp>][-m <number_of_minutes>]
	(i | info) [-s <source_id> -o <order_id>]
	(q | quit)
	(h | help) [<command>]

The help command followed by a command name displays detailed information about that command and its parameters.

break

To halt the journal replay at specific points, users can set breakpoints using break (or b) command. This command allows setting breakpoints based on message timestamp, source, destination or order ID. For instance, to stop at messages related to a specific order, users can add a breakpoint for the corresponding order ID:

==> b -o 1570129867583
Added breakpoint: (1) Break at order_id=1570129867583

To add a breakpoint based on the timestamp string value in ISO-8601 format, use:

==> b -t 2019-10-03T19:11:09.070Z
Added breakpoint: (2) Break at timestamp>=2019-10-03T19:21:09.070Z

step & continue

To continue reading messages, users can use the step (or s) and/or continue (or c) commands. To continue reading until the next breakpoint, enter c at the prompt.

With the two breakpoints specified, the debugger reads the journal and stops either at the first message related to the specified order or when the message timestamp reaches the breakpoint timestamp. The debugger displaus the condition that caused the break next to the step number at the top:

==> c
[Step 23]    (order_id=1570129867583)
Message: {"$type":"OrderNewRequest","timeInForce":"DAY", "currency":"USD","orderId":"1570129867583","side":"SELL","quantity":4,"orderType":"MARKET", "exchangeId":"FILL","flags":0,"sequence":39,"symbol":"BTCUSD","instrumentType":"FX","timestamp":"2019-10-03T19:11:09.058Z", "sourceId":"TRADER-CLI", "destinationId":"SIM"}
Order: (1570129867583) SELL 4 BTCUSD @ MARKET UNACKNOWLEGED (TRADER-CLI)=>(SIM), Exchange: FILL, TimeInForce: DAY, Filled: 0, Remaining: 4
Total number of active orders: 1
==> 

list -a

To see all other active orders available at a specific point, use the list -a or l -a command:

==> l -a
Active orders:
      (1570129867583) SELL 4 BTCUSD @ MARKET UNACKNOWLEGED (TRADER-CLI)=>(SIM), Exchange: FILL, TimeInForce: DAY, Filled: 0, Remaining: 4
Total number of orders found: 1
==>

The returned order list can be filtered by symbol, side, source, destination or order submission time.

To continue reading the journal until the next message from this order, enter the continue command again.

To stop at each message, use the step command. This command takes the optional number_of_steps parameter.

The break -l command displays the breakpoints that have been set:

==> b -l
Breakpoints: 
	(1) Break at order_id=1570129867583
	(2) Break at timestamp>=2019-10-03T19:21:09.070Z

To clear a breakpoint, the index in the parentheses can be passed with the -r (remove) option:

==> b -r 1
Removed breakpoint: (1) Break at order_id=1570129867583
==> b -r 2
Removed breakpoint: (2) Break at timestamp>=2019-10-03T19:11:09.070Z

Last Step

When the debugger reads all the messages in the journal, it stops after the last step and displays *** End of Journal ***:

==> c
[Step 26]
Message: {"$type":"OrderRejectEvent","reason":"Risk[8] Symbol[BTCUSD]:MaxPositionLong: Order cannot be validated - 'reference' market price is unknown for instrument \"BTCUSD\"","orderId":"1570577307417","correlationOrderId":"1570577307417","orderStatus":"REJECTED","eventId":"1570577281206","side":"SELL","quantity":4,"orderType":"MARKET", "exchangeId":"FILL","cumulativeQuantity":0,"averagePrice":0,"remainingQuantity":0,"timeInForce":"DAY","sequence":44,"symbol":"BTCUSD","instrumentType":"FX","timestamp":"2019-10-08T23:28:30.004Z", "sourceId":"SIM", "destinationId":"TRADER-CLI"}
Order: (1570577307417) SELL 4 BTCUSD @ MARKET REJECTED (TRADER-CLI)=>(SIM), Exchange: FILL, TimeInForce: DAY, Filled: 0, Remaining: 0
Total number of active orders: 0
*** End of Journal ***
==> 

Users can still query active and inactive order information at this point. If the Ember Server is running and writing more messages to the journal, the debugger can continue reading them with the step or continue commands.

quit

To exit the debugger, use the quit command (or just q).

List of Supported Сommands

Here is the description of all supported commands and their parameters:

==> help step
usage: (s | step) [<number_of_steps>]
Read next message or specified number of messages

==> help continue
usage: (c | continue) [-o <order_id>] [-p <price>] [-s <source_id>] [-d
          <destination_id>] [-i <symbol>] [-n <sequence>] [-t <timestamp>]
          [<number_of_steps>]
Continue reading messages until the next breakpoint
 -d,--destination <destination_id>   Break for message sent to destination
 -i,--symbol <symbol>                Break for messages with symbol
 -n,--sequence <sequence>            Break at message sequence number
 -o,--order <order_id>               Break for order message
 -p,--price <price>                  Break for new order requests with
                                     specified price
 -s,--source <source_id>             Break for message from source
 -t,--timestamp <timestamp>          Break at timestamp specified in
                                     ISO-8601 format
                                     
==> help break
usage: (b | break) [([-o <order_id>] [-s <source_id>] [-d
          <destination_id>] [-i <symbol>] [-t <timestamp>][-n
          <sequence>][-p <price>]) | -l | -r (<breakpoint_index> | all)]
Add, remove or list breakpoints
 -d,--destination <destination_id>   Add breakpoint for messages sent to
                                     destination
 -i,--symbol <symbol>                Add breakpoint for messages with
                                     symbol
 -l,--list                           List breakpoints
 -n,--sequence <sequence>            Add breakpoint for message with
                                     sequence
 -o,--order <order_id>               Add breakpoint for order messages
 -p,--price <price>                  Add breakpoint for orders with
                                     specified limit price
 -r,--remove <breakpoint>            Remove all breakpoints or breakpoint
                                     with specified index
 -s,--source <source_id>             Add breakpoint for messages from
                                     source
 -t,--timestamp <timestamp>          Add breakpoint at timestamp specified
                                     in ISO-8601 format

==> help list
usage: (l | list) [-a] [-o <order_id>] [-s <source_id>] [-d
          <destination_id>] [-e <side>] [-i <symbol>][-p <price>][-l
          <limit>][-t <timestamp>][-m <number_of_minutes>]
List orders matching specified selection attributes
 -a,--active                         Select from active orders only
 -d,--destination <destination_id>   Select orders with specified
                                     destination ID
 -e,--side <side>                    Show orders with specified side
 -i,--instrument <symbol>            Select orders with specified symbol
 -l,--limit <limit price>            Show orders with specified limit
                                     price or better
 -m,--minutes <minutes>              Show orders older than specified
                                     number of minutes ago
 -o,--order <order_id>               Show order by ID
 -p,--price <price>                  Show orders with specified limit
                                     price
 -s,--source <source_id>             Select orders with specified source
                                     ID
 -t,--timestamp <timestamp>          Show orders older than specified
                                     timestamp in ISO-8601 format
                                                                          
==> help info
usage: (i | info) [-s <source_id> -o <order_id>]
Display information about specified order or the last message
 -o,--order <order_id>     Order ID
 -s,--source <source_id>   Source ID

==> help quit
usage: (q | quit)
Quit debugger

==> help help
usage: (h | help) [<command>]
List commands or display command help

Here is the same list in a table:

Command	Description
step (s)	Reads the next message from the journal.
step (s) with <number_of_steps>	Reads the specified number of messages from the journal.
continue (c)	Continues reading messages from the journal until the next breakpoint is reached.
continue (c) with <number_of_steps>	Continues reading messages from the journal until the specified number of steps or the next breakpoint is reached.
continue (c) with options -o, -s, -d, -i, -t, -n, -p	Continues reading messages from the journal until the next breakpoint is reached, filtered by order ID, source ID, destination ID, symbol, or timestamp, respectively.
break (b)	Sets a breakpoint at a specific message.
break (b) with options -o, -s, -d, -i, -t, -n, -p	Sets a breakpoint based on the order ID, source ID, destination ID, symbol, or timestamp, respectively.
break (b) with -l	Lists all the set breakpoints.
break (b) with -r <breakpoint_index>	Removes the breakpoint at the specified index.
break (b) with -r all	Removes all breakpoints.
list (l)	Lists all active orders at the current point in the journal.
list (l) with options -a, -o, -s, -d, -e, -i, -p, -l, -t, -m	Filters the list of active orders by all, order ID, source ID, destination ID, side, symbol, price, price limit or order submission time.
info (i) with options -s, -o	Displays detailed information about a specific source ID or order ID.
quit (q)	Exits the OMS Debugger.
help (h)	Displays usage information for all commands or for a specific command.
help (h) with <command>	Displays detailed usage information for a specific command.

Example of running on host OS

When ember is running under docker compose the following example can be used to run the tool using ember container (you may need to adjust version number and path to EMBER_HOME and EMBER_WORK directories):

docker run --entrypoint "/opt/deltix/ember/bin/oms-debugger" -it -v "/home/r8fin/ember/config:/var/lib/emberhome" -v "/home/r8fin/ember/work:/var/lib/emberwork" "registry.deltixhub.com/deltix.docker/anvil/deltix-ember:1.14.149"

Example of troubleshooting session

Goal: find out what happened in custom matching engine running inside Ember around 12:48:32 UTC on August 6, 2024.

First we launch OMS debugger using command above.

Then we set a breakpoint at the time we want to investigate and execute continue command:

b -t 2024-08-06T12:48:32Z
c

Once we reach given time in historical part of Ember journal (assuming it wasn't compressed), Ember will display the first message that appears on or after given timestamp:

[Step 17517108] [2024-08-06T12:48:32.092320372Z]    (step>=17517108)
Message: {"$type":"OrderCancelRequest","requestId":"00BBRjFPgsK001","correlationOrderId":"00BBRjFPgsK000","externalOrderId":"6632614329","orderId":"00BBRjFPgsK000","side":"BUY","quantity":2,"orderType":"LIMIT","account":"0","traderId":"jdoe","originalTimestamp":"2024-08-06T12:48:32.092000000Z","sequence":6632683721,"symbol":"912810UA4","timestamp":"2024-08-06T12:48:32.092320372Z", "sourceId":"HIDEME_OE"}
Order: (00BBRjFPgsK000) BUY 2 912810UA4 @ 108.6484375 OPEN (HIDEME_OE)=>(null), TimeInForce: DAY, Filled: 0, Remaining: 2, Canceling!
Total number of active orders: 1878

Now we remove this breakpoint and execute journal messages step by step:

b -r 1
s
s
s

After each step command OMS debugger displays the message that was replayed and associated order state.

In this particular investigation OMS debugger confirmed 5 second a gap in message timestamps. Your scenario may be different.

Please share with us ideas what additional features you want to see in OMS debugger.