Skip to main content

journal-clean

Overview​

This document describes how to use the journal-clean tool that allows removing messages from Ember journal linked to obsolete instruments.

Prerequisites​

Before using the journal-clean tool, ensure the following:

  • The Ember process itself and all Ember satellite processes (Ember Monitor, Ember Data Warehouse, Ember Journal Compactor, and Deltix Trading Node) must be STOPPED while the tool is used.
  • Make sure you have enough disk space. The tool will create a copy of the current journal with some messages removed which might require additional amount of disk space similar to the size to the original journal.

Disclaimer​

The system uses a replicated state machine approach (also known as "event sourcing"). The Trading State is stored in the journal as a sequence of trading requests and events that happened to the system.

caution

The tool operator bears full responsibility for any action that may cause an inconsistent trading state.

Examples of potentially harmful actions:

  • Deletion of a OrderNewRequest that corresponds to still active orders. The system likely won't recognize any subsequent events for this order.
  • Deletion of any messages may cause a shift in message sequence numbers. As a result, trading connectors that use message sequence to generate unique order IDs will be affected.

Tool Arguments​

The journal-clean tool takes the following parameters:

Usage: journal-clean [-symbols <arg>] [-symbolsFile <arg>] [-force]
 -symbols <arg>       Comma-separated list of instrument symbols to remove from the journal (use
                      this or -symbolsFile)
 -symbolsFile <arg>   Path to a CSV file of instrument symbols to remove (one or more per line; use
                      this or -symbols)
 -force               Rewrite and replace journal without prompting for confirmation

When started without the explicit -force option, the tool will ask for confirmation:

~> journal-clean -symbolsFile obsolete_instruments.csv
This tool will rewrite Ember journal and remove messages linked to the specified instruments.
Would you like to proceed? (y/n):

Removing references to obsolete instruments​

  1. Make sure to define the EMBER_HOME and EMBER_WORK environment variables. This tool uses the Ember config and journal files located in the EMBER_HOME directory and temporarily stores the imported journal in the EMBER_WORK folder. It leaves a backup of the current journal there after it is finished. If you run this tool by attaching to the Ember Docker container, these environment variables should be already defined.

    export EMBER_HOME=/var/lib/emberhome
    export EMBER_WORK=/var/lib/emberwork

  2. Stop all Ember services that may read or write to the journal.

  3. Run the tool with the following command:

    ~> journal-clean -symbolsFile obsolete_instruments.csv
    This tool will rewrite Ember journal and remove messages linked to the specified instruments.
    Would you like to proceed? (y/n):

    While processing the journal, the tool displays a progress indicator with the estimated percentage of processed data.

  4. After processing of the journal is successfully finished, the tool makes a backup of the current journal and replaces it with the new journal, and then prints messages indicating the journal backup location, which will look like this:

    Successfully cleaned the journal. Removed 1222 messages referencing removed symbols
    Backup of the Ember journal is stored in /var/lib/emberwork/journal/journal_backup_1

  5. Restart Ember services to complete the journal cleaning procedure.

Further Reading​