journal-transform
Overview
This document describes how to use the journal-transform tool. This tool can be used to edit the Ember journal on a customer site in the case of various special situations.
Disclaimer
The Ember system uses a replicated state machine approach, also known as "event sourcing". The trading State is stored in the journal as a sequence of trading requests and events that have occur within the system. When utilizing the journal-transform tool to modify the journal, special care must be taken to ensure consistency and avoid disrupting the trading state.
The tool operator bears full responsibility for any actions that may lead to an inconsistent trading state.
Examples of potentially harmful actions:
- Deletion of a
OrderNewRequestthat corresponds to a still active order. This could result in the system not recognizing any subsequent events for this order. - Deletion of any messages may cause a shift in message sequence numbers. As a result, trading connectors using message sequence to generate unique order IDs will be affected.
Prerequisites
Before using the journal-transform tool, ensure you meet the following requirements:
- Familiarize yourself with the Ember Trading Model. This document describes the format of messages the tool can transform.
- Starting from Ember version 1.14 this tool runs Rhino JavaScript Engine with most ES2015 and many ES6/ES2016 features supported. Sidenote: In earlier versions this tool used the Nashorn JavaScript engine.
- Make sure you have enough disk space to accommodate the transformed journal. Measure the current size of the journal directory, as you will likely need a similar amount of space for the transformed journal.
As the journal-transform tool rewrites the Ember Journal (which is normally owned by Ember), make sure to stop Ember's main and satellite processes, such as the journal compactor, before running this tool.
Quick Start
To get started with the journal-transform tool, create a JavaScript file that contains your transformation function. This tool reads the provided file and invokes the transform function for every message in the journal.
Let's create a file transform.js with the following content:
var skippedSource = Number(Packages.deltix.anvil.util.codec.AlphanumericCodec.encode('CLIENT1'))
var transform = function(message) {
if (skippedSource == message.sourceId && skippedSource == message.destinationId) {
return null; // skip it
}
return message; // keep it
}
In this example, we compute the source ID used by CLIENT1 and skip all messages that use this ID either as the source (requests) or the destination (events).
Next, define the locations of your work and home directories, and then run the tool:
export EMBER_HOME=/deltix/emberhome\
export EMBER_WORK=/deltix/emberwork\
/deltix/ember/journal-transform transform.js
When ember is running under docker compose the following example can be used to run the tool using ember container (you may need to adjust version number and path to EMBER_HOME and EMBER_WORK directories):
docker run --entrypoint "/opt/deltix/ember/bin/journal-transform" -it -v "/home/r8fin/ember/config:/var/lib/emberhome" -v "/home/r8fin/ember/work:/var/lib/emberwork" "registry.deltixhub.com/deltix.docker/anvil/deltix-ember:1.14.149" transform.js
By default, the tool creates a new journal under $EMBER_WORK/journal/new. If you want to override this location, use the following command line argument:
/deltix/ember/journal-transform transform.js $EMBER_WORK/journal.new
Starting with Ember 1.14, the journal transformer does not use the output folder parameter anymore. Instead, the transformer always replaces the current journal with the new transformed version and creates a backup of the original journal in the $EMBER_WORK/journal-backup folder. On successful execution, the tool prints messages like the one below, indicating the exact backup location:
Successfully transformed journal using script in /deltix/emberwork/transform.js
Backup of Ember journal is stored in /deltix/emberwork/journal_backup_1
WARNING: Journal transformer does not preserve journal FENCE files!!! If you have drop copy or data warehouse enabled you need to copy fence files from journal_backup.
The journal transformer rewrites journal files. New files are always named data1, data2, etc. and without snapshots. However, journal terms and sequence numbers are preserved (unless you modify sequence numbers as a part of your transformation function).
Transformer Samples
Here are some example transformations you can perform on the journal using the journal-transform tool.
Risk Projection Removal
This transformation removes messages related to the 'Trader/Destination/Currency' projection from the journal:
var transform = function(message) {
if (message instanceof Packages.deltix.ember.message.risk.RiskUpdateRequest ||
message instanceof Packages.deltix.ember.message.risk.RiskUpdateResponse) {
if (message.projection == 'Trader/Destination/Currency') {
java.lang.System.out.println('Removing message: ' + message)
return null;
}
}
return message; // keep it
}
Multiple Projection Removal
This transformation removes messages related to 'TraderGroup/Exchange/Symbol' and 'Trader/Exchange' projections from the journal:
var transform = function(message) {
if (message instanceof Packages.deltix.ember.message.risk.RiskUpdateRequest ||
message instanceof Packages.deltix.ember.message.risk.RiskUpdateResponse ||
message instanceof Packages.deltix.ember.message.trade.oms.PositionRequest) {
if (message.projection == 'TraderGroup/Exchange/Symbol' || message.projection == 'Trader/Exchange') {
java.lang.System.out.println('Removing message: ' + message)
return null;
}
}
if (message instanceof Packages.deltix.ember.message.trade.oms.PositionSnapshot) {
if (message.projectionPath == 'TraderGroup/Exchange/Symbol' ||
message.projectionPath == 'Trader/Exchange') {
java.lang.System.out.println('Removing message: ' + message)
return null;
}
}
return message; // keep it
}
Risk limit patch
This transformer replaces limits defined as MaxPriceDifference to MaxAggressivePriceDifference:
function updateLimitNames(message) {
message.commands.forEach(function(command) {
if (command.limits) {
command.limits.forEach(function(limit) {
if (limit.name == "MaxPriceDifference") {
limit.name = "MaxAggressivePriceDifference";
}
});
}
});
}
var transform = function(message) {
if (message instanceof Packages.deltix.ember.message.risk.RiskUpdateRequest) {
if (message.projection == 'Source') {
updateLimitNames(message)
java.lang.System.out.println('Updating message: ' + message)
}
}
return message;
}
Order Modification
This transformation skips the order request for order ABC:1568945796826 and changes the limit price for order ABC:1568945796916:
var mySourceId = Number(Packages.deltix.anvil.util.codec.AlphanumericCodec.encode('ABC'))
var transform = function(message) {
if (message instanceof Packages.deltix.ember.message.trade.OrderNewRequest) {
if (mySourceId == message.sourceId && message.orderId == '1568945796826') {
java.lang.System.out.println('Skipping order: ' + message)
return null; // skip it
}
if (mySourceId == message.sourceId && message.orderId == '1568945796916') {
message.limitPrice = Packages.deltix.dfp.Decimal64Utils.parse('6001')
java.lang.System.out.println ("Modified limit price for " + message)
}
}
return message;
}
Advanced Usage
Injecting Additional Messages
The JavaScript function has access to the 'writer' object, which acts as a consumer of journaled API Messages. This object allows injecting new messages into the journal, which will be inserted before the current message being processed.
For example, to inject a "Destination BINANCE is now connected" session status event (typically produced by a trading connector), you can use the following code snippet:
var SessionStatusEvent = Java.type('deltix.ember.message.trade.MutableSessionStatusEvent');
var sessionStatusEvent = new SessionStatusEvent();
sessionStatusEvent.timestamp = Date.parse("2019-01-01T00:00:00.000Z");
sessionStatusEvent.status = Packages.deltix.ember.message.trade.SessionStatus.CONNECTED;
sessionStatusEvent.sourceId = Number(Packages.deltix.anvil.util.codec.AlphanumericCodec.encode('BINANCE'));
writer.write(message);
Injecting additional messages may lead to problems in the data warehouse. For example, if you insert 100 new messages in the middle of the journal, there is a possibility that upon restart, the data warehouse will append the last 100 messages as duplicates.
More Examples
Here are some more examples of transformations that can be performed using the journal-transform tool.
Removing a Bad Currency Update Message
This transformation removes a specific Currency Update message with the symbol "ETHUSD" and a base currency of Long.MIN_VALUE:
var transform = function(message) {
if (message instanceof Packages.deltix.ember.message.smd.CurrencyUpdate) {
if (message.symbol == "ETHUSD" && message.baseCurrency == Packages.java.lang.Long.MIN_VALUE) {
return null; // skip it
}
}
return message;
}
Patching bad symbol names
// List of Future contracts that need U -> H transformation
var symbolsToTransform = [
'GNM FSV0025.U0026',
'GNM FSV0026.U0027',
'GNM FSV0027.U0028'
];
var transform = function (msg) {
// Check if the message symbol is in our list
if (symbolsToTransform.indexOf(msg.symbol) !== -1) {
msg.symbol = msg.symbol.replace('.U', '.H'); // Replace .U with .H in the symbol
}
return msg;
}
Renaming Account in Order Messages and Position Snapshots
This transformation replaces occurrences of the account name "Gold" with "Platinum" in all order messages and position snapshots:
originalAccount = "Gold"
replacementAccount = "PLATINUM"
originalProjection = "Account[" + originalAccount + "]"
replacementProjection = "Account[" + replacementAccount + "]"
var transform = function(message) {
if (message instanceof Packages.deltix.ember.message.trade.OrderEvent ||
message instanceof Packages.deltix.ember.message.trade.OrderRequest) {
if (message.account == originalAccount) {
message.account = replacementAccount;
java.lang.System.out.println ("Replaced account '" + message.account + "' in order message");
}
} else
if (message instanceof Packages.deltix.ember.message.trade.oms.PositionSnapshot) {
var projectionPath = message.projectionPath.toString();
if (projectionPath.indexOf(originalProjection) >= 0) {
message.projectionPath = projectionPath.replace(originalProjection, replacementProjection)
java.lang.System.out.println ("Replaced account '" + message.projectionPath + "' in position snapshot");
}
}
return message;
}
Fixing Timestamp for Messages from a Specific Exchange
This transformation fixes the timestamp for all messages that came from the exchange with the ID "POWERTRADE" If a message has an original timestamp that is in the future (beyond tomorrow's date), the tool will print a warning. Additionally, it modifies the original timestamp to null for a specific order with the ID "1663060141859581" from the same exchange:
today = new Date()
tomorrow = new Date(today)
tomorrow.setDate(tomorrow.getDate() + 1)
POWERTRADE = Packages.deltix.anvil.util.codec.AlphanumericCodec.encode("POWERTRADE")
var transform = function(message) {
if (message instanceof Packages.deltix.ember.message.trade.OrderTradeReportEvent) {
if (POWERTRADE == message.exchangeId)) {
if (message.originalTimestamp > tomorrow)
java.lang.System.out.println("Found trade in the future: " + message);
if (message.orderId == "1663060141859581") {
java.lang.System.out.println("Fixing trade'originalTimestamp for " + message);
message.originalTimestamp = -9223372036854775808; // null
}
}
}
return message;
}