journal-import

Overview

This document describes how to use the journal-import tool. This tool allows importing journal data from a backup created in a data warehouse.

Ember trading activity is usually archived from Ember Journal to a set of supported Data Warehouses:

This document describes the reverse procedure - restoring trading history from a data warehouse.

Some key points:

• After a successful import, the existing content of the journal directory is moved to a backup folder, while the newly imported data becomes the new journal.
• This tool restores the entire latest data warehouse term as the new copy of the journal. If the warehouse stores multiple journals terms, the current journal term is imported by default.
• Once the import is complete, you can restart Ember, Ember Journal Compaction, and other services. Ember Data warehouses will continue accumulating new trading activity.

Prerequisites

Before using the journal-import tool, ensure the following:

• The Ember process itself and all Ember satellite processes (Ember Monitor, Ember Data Warehouse, Ember Journal Compactor, and Deltix Trading Node) must be STOPPED during the journal import.
• Make sure you have enough disk space. All data warehouse messages related to the imported journal will be downloaded and used to create the journal files on disk. After the Ember Journal Compactor restarts, it compacts the restored journal and releases any exccess disk space.

Journal Sizing Estimate

The imported journal will require approximately N * 2500 bytes, where N is the number of orders stored in your data warehouse. This simple formula assumes the average order has 5 events (ack, fill, etc.). If you use very large orders with multiple fills, the formula is M * 500 bytes where M is the number of messages for each order.

Disclaimer

The system uses a replicated state machine approach (also known as "event sourcing"). The Trading State is stored in the journal as a sequence of trading requests and events that happened to the system. Special care must be taken when modifying warehouse data that could be imported back to the journal.

caution

The tool operator bears full responsibility for any action that may cause an inconsistent trading state.

Examples of potentially harmful actions:

• Deletion of a OrderNewRequest that corresponds to still active orders. The system likely won't recognize any subsequent events for this order.
• Deletion of any messages may cause a shift in message sequence numbers. As a result, trading connectors that use message sequence to generate unique order IDs will be affected.

Tool Arguments

The journal-import tool takes the following parameters:

Usage: journal-import [-force] [-term <arg>] -unit <arg>
 -force        Import and replace journal without prompting for confirmation
 -term <arg>   Journal term to import. By default will use current journal term
 -unit <arg>   Warehouse unit to import as specified in Ember config

The unit parameter is the only required one and must refer to the warehouse unit configured in ember.conf from which journal will be imported. Import is supported with these warehouse implementations: TimeBase, MS SQL, Redshift, ClickHouse, S3, and Kafka.

If the imported journal term is not specified via the -term argument, the tool will try to import the same term as the current journal (the one that originally resides in the EMBER_WORK/journal directory).

Once started, the tool will ask for confirmation:

~> journal-import -unit mssql
This tool will import journal from mssql warehouse.
Would you like to proceed? (y/n): 

The -force flag can be used to run journal-import without confirmation, which might be helpful when the import is executed from a script.

Journal Import Procedure

1. Make sure to define the EMBER_HOME and EMBER_WORK environment variables. This tool uses the Ember config and journal files located in the EMBER_HOME directory and temporarily stores the imported journal in the EMBER_WORK folder. It leaves a backup of the current journal there after it is finished. If you run this tool by attaching to the Ember Docker container, these environment variables should be already defined.

   export EMBER_HOME=/var/lib/emberhome
   export EMBER_WORK=/var/lib/emberwork

2. Stop all Ember services that may read or write to the journal.

3. Run the tool with the following command:

   journal-import -unit mssql -force
   This tool will import journal from sqlserver warehouse.

   During the import, the tool displays a progress indicator with the estimated percentage of processed data.

4. After the import is successfully finished, the tool makes a backup of the current journal, replaces it with the imported jornal, and then prints messages indicating the journal backup location, which will look like this:

   Successfully imported journal from mssql warehouse
   Backup of the Ember journal is stored in /var/lib/emberwork/journal/journal_backup_3

5. Restart Ember services to complete the journal import procedure.

Further Reading

• Ember Data Warehouses
• Ember Journal