journal-transform

Overview

This document describes how to use the journal-transform tool. This tool can be used to edit the Execution Server journal on a customer site in the case of various special situations.

Disclaimer

The Ember system uses a replicated state machine approach, also known as "event sourcing". The trading State is stored in the journal as a sequence of trading requests and events that have occur within the system. When utilizing the journal-transform tool to modify the journal, special care must be taken to ensure consistency and avoid disrupting the trading state.

caution

The tool operator bears full responsibility for any actions that may lead to an inconsistent trading state.

Examples of potentially harmful actions:

• Deletion of a OrderNewRequest that corresponds to a still active order. This could result in the system not recognizing any subsequent events for this order.
• Deletion of any messages may cause a shift in message sequence numbers. As a result, trading connectors using message sequence to generate unique order IDs will be affected.

Prerequisites

Before using the journal-transform tool, ensure you meet the following requirements:

• Familiarize yourself with the Ember Trading Model. This document describes the format of messages the tool can transform.
• Starting from Ember version 1.14 this tool runs Rhino JavaScript Engine with most ES2015 and many ES6/ES2016 features supported. Sidenote: In earlier versions this tool used the Nashorn JavaScript engine.
• Make sure you have enough disk space to accommodate the transformed journal. Measure the current size of the journal directory, as you will likely need a similar amount of space for the transformed journal.

tip

As the journal-transform tool rewrites the Ember Journal (which is normally owned by Ember), make sure to stop Ember's main and satellite processes, such as the journal compactor, before running this tool.

Quick Start

To get started with the journal-transform tool, create a JavaScript file that contains your transformation function. This tool reads the provided file and invokes the transform function for every message in the journal.

Let's create a file transform.js with the following content:

var skippedSource = Number(Packages.deltix.anvil.util.codec.AlphanumericCodec.encode('CLIENT1'))
var transform = function(message) {
    // print (message) // may be handy
    if (skippedSource == message.sourceId && skippedSource == message.destinationId) {
        return null; // skip it
    }
    return message; // keep it
}

In this example, we compute the source ID used by CLIENT1 and skip all messages that use this ID either as the source (requests) or the destination (events).

Next, define the locations of your work and home directories, and then run the tool:

export EMBER_HOME=/deltix/emberhome\
export EMBER_WORK=/deltix/emberwork\
/deltix/ember/journal-transform transform.js

note

When ember is running under docker compose the following example can be used to run the tool using ember container (you may need to adjust version number and path to EMBER_HOME and EMBER_WORK directories):

docker run --entrypoint "/opt/deltix/ember/bin/journal-transform" -it -v "/home/r8fin/ember/config:/var/lib/emberhome" -v "/home/r8fin/ember/work:/var/lib/emberwork" "registry.deltixhub.com/deltix.docker/anvil/deltix-ember:1.14.149" transform.js

By default, the tool creates a new journal under $EMBER_WORK/journal/new. If you want to override this location, use the following command line argument:

/deltix/ember/journal-transform transform.js $EMBER_WORK/journal.new

note

Starting with Ember 1.14, the journal transformer does not use the output folder parameter anymore. Instead, the transformer always replaces the current journal with the new transformed version and creates a backup of the original journal in the $EMBER_WORK/journal-backup folder. On successful execution, the tool prints messages like the one below, indicating the exact backup location:

Successfully transformed journal using script in /deltix/emberwork/transform.js
Backup of Ember journal is stored in /deltix/emberwork/journal_backup_1

danger

WARNING: Journal transformer does not preserve journal FENCE files!!! If you have drop copy or data warehouse enabled you need to copy fence files from journal_backup.

note

The journal transformer rewrites journal files. New files are always named data1, data2, etc. and without snapshots. However, journal terms and sequence numbers are preserved (unless you modify sequence numbers as a part of your transformation function).

Transformer Samples

Here are some example transformations you can perform on the journal using the journal-transform tool.

Risk Projection Removal

This transformation removes messages related to the 'Trader/Destination/Currency' projection from the journal:

var transform = function(message) {
    if (message instanceof Packages.deltix.ember.message.risk.RiskUpdateRequest || 
        message instanceof Packages.deltix.ember.message.risk.RiskUpdateResponse) {
        if (message.projection == 'Trader/Destination/Currency') {
            print('Removing message: ' + message)
            return null;
        }
    }
    return message; // keep it
}

Multiple Projection Removal

This transformation removes messages related to 'TraderGroup/Exchange/Symbol' and 'Trader/Exchange' projections from the journal:

var transform = function(message) {
    if (message instanceof Packages.deltix.ember.message.risk.RiskUpdateRequest || 
        message instanceof Packages.deltix.ember.message.risk.RiskUpdateResponse ||
        message instanceof Packages.deltix.ember.message.trade.oms.PositionRequest) {
        if (message.projection == 'TraderGroup/Exchange/Symbol' || message.projection == 'Trader/Exchange') {
            print('Removing message: ' + message)
            return null;
        }
    }
    if (message instanceof Packages.deltix.ember.message.trade.oms.PositionSnapshot) {
        if (message.projectionPath == 'TraderGroup/Exchange/Symbol' || 
            message.projectionPath == 'Trader/Exchange') {
            print('Removing message: ' + message)
            return null;
        }
    }
    return message; // keep it
}

Risk limit patch

This transformer replaces limits defined as MaxPriceDifference to MaxAggressivePriceDifference:

function updateLimitNames(message) {
    message.commands.forEach(function(command) {
        if (command.limits) {
            command.limits.forEach(function(limit) {
                if (limit.name == "MaxPriceDifference") {
                    limit.name = "MaxAggressivePriceDifference";
                }
            });
        }
    });
}

var transform = function(message) {
    if (message instanceof Packages.deltix.ember.message.risk.RiskUpdateRequest) {
        if (message.projection == 'Source') {
            updateLimitNames(message)
            print('Updating message: ' + message)
        }
    }
    return message;
}

Order Modification

This transformation skips the order request for order ABC:1568945796826 and changes the limit price for order ABC:1568945796916:

var mySourceId = Number(Packages.deltix.anvil.util.codec.AlphanumericCodec.encode('ABC'))

var transform = function(message) {
    if (message instanceof Packages.deltix.ember.message.trade.OrderNewRequest) {
        if (mySourceId == message.sourceId && message.orderId == '1568945796826') {
            print('Skipping order: ' + message)
            return null; // skip it
        }
        if (mySourceId == message.sourceId && message.orderId == '1568945796916') {
            message.limitPrice = Packages.deltix.dfp.Decimal64Utils.parse('6001')
            print ("Modified limit price for " + message)
        }
    }
    return message;
}

Advanced Usage

Injecting Additional Messages

The JavaScript function has access to the 'writer' object, which acts as a consumer of journaled API Messages. This object allows injecting new messages into the journal, which will be inserted before the current message being processed.

For example, to inject a "Destination BINANCE is now connected" session status event (typically produced by a trading connector), you can use the following code snippet:

var SessionStatusEvent = Java.type('deltix.ember.message.trade.MutableSessionStatusEvent');
var sessionStatusEvent = new SessionStatusEvent();
sessionStatusEvent.timestamp = Date.parse("2019-01-01T00:00:00.000Z");
sessionStatusEvent.status = Packages.deltix.ember.message.trade.SessionStatus.CONNECTED;
sessionStatusEvent.sourceId = Number(Packages.deltix.anvil.util.codec.AlphanumericCodec.encode('BINANCE'));
writer.write(message);

danger

Injecting additional messages may lead to problems in the data warehouse. For example, if you insert 100 new messages in the middle of the journal, there is a possibility that upon restart, the data warehouse will append the last 100 messages as duplicates.

More Examples

Here are some more examples of transformations that can be performed using the journal-transform tool.

Removing a Bad Currency Update Message

This transformation removes a specific Currency Update message with the symbol "ETHUSD" and a base currency of Long.MIN_VALUE:

var transform = function(message) {
  if (message instanceof Packages.deltix.ember.message.smd.CurrencyUpdate) {
    if (message.symbol == "ETHUSD" && message.baseCurrency == Packages.java.lang.Long.MIN_VALUE) {
      return null; // skip it
    }
  }
  return message;
}

Renaming Account in Order Messages and Position Snapshots

This transformation replaces occurrences of the account name "Gold" with "Platinum" in all order messages and position snapshots:

originalAccount = "Gold"
replacementAccount = "PLATINUM"
originalProjection = "Account[" + originalAccount + "]"
replacementProjection = "Account[" + replacementAccount + "]"

var transform = function(message) {
    if (message instanceof Packages.deltix.ember.message.trade.OrderEvent ||
        message instanceof Packages.deltix.ember.message.trade.OrderRequest) {
      if (message.account == originalAccount) {
          message.account = replacementAccount;
          print ("Replaced account '" + message.account + "' in order message");
      }
    } else
    if (message instanceof Packages.deltix.ember.message.trade.oms.PositionSnapshot) {
        var projectionPath = message.projectionPath.toString();
        if (projectionPath.indexOf(originalProjection) >= 0) {
            message.projectionPath = projectionPath.replace(originalProjection, replacementProjection)
            print ("Replaced account '" + message.projectionPath + "' in position snapshot");
        }
    }
    
    return message;
}

Fixing Timestamp for Messages from a Specific Exchange

This transformation fixes the timestamp for all messages that came from the exchange with the ID "POWERTRADE" If a message has an original timestamp that is in the future (beyond tomorrow's date), the tool will print a warning. Additionally, it modifies the original timestamp to null for a specific order with the ID "1663060141859581" from the same exchange:

today = new Date()
tomorrow = new Date(today)
tomorrow.setDate(tomorrow.getDate() + 1)           
POWERTRADE = Packages.deltix.anvil.util.codec.AlphanumericCodec.encode("POWERTRADE")            
var transform = function(message) {
    if (message instanceof Packages.deltix.ember.message.trade.OrderTradeReportEvent) {
        if (POWERTRADE == message.exchangeId)) {

           if (message.originalTimestamp > tomorrow)
                print("Found trade in the future: " + message);
            
            if (message.orderId == "1663060141859581") {
                print("Fixing trade'originalTimestamp for " + message);
                message.originalTimestamp = -9223372036854775808; // null
            }            
        }
    }
    return message;
}