Skip to main content

Common Problems and Pitfalls

A Large Number of Errors Appears During Ember Startup​

During start up, Ember restores the system state by replaying historical orders and events.

Large numbers of WARN or ERROR level log messages during this phase may be caused by one of the following reasons:

Missing Instrument​

You may encounter large numbers of error messages when the system has orders for instruments that are no longer in your Security Metadata stream. Ember needs instrument metadata to correctly process each historical order and calculate its effect on positions.

Troubleshooting​

To troubleshoot this issue, try the following options:

  • Restore missing instruments.
  • Scrub the journal using the journal-transform tool by deleting all messages for expired instruments.

Missing Destination​

You may encounter large numbers of error messages when the system has orders for a destination that was removed from Ember configuration. For example, you previously traded on CME directly, but now use the Vela Trading gateway. Unless you keep trading, Ember will complain any time it sees an order with an invalid connector. The same applies to other execution algorithms.

Troubleshooting​

To fix this problem, in the engine settings, list the destinations you no longer need that have accumulated orders as nullDestination. Note that unless you maintain per destination positions, Ember will eventually expunge old orders from the journal and this problem can take care of itself.

Missing Risk Rule Projection​

You may encounter large numbers of error messages when your journal records limits for risk projections that are no longer configured.

Troubleshooting​

To clean up your journal, use the journal-transform tool.

FIX Messages Are Displayed Without Field Separators​

When troubleshooting order entry problems, one often has to look up a FIX message, sometimes via SSH.

To look up a FIX message via SSH, use this handy kung fu move in the command line:

tail -n 10 BOSONICMM-2021-04-14.99.messages | sed 's/\x01/\|/g'

How to Manually Backup or Remove a Journal​

In most situations, you should not need to remove the journal. In case you do, Ember warehouses can be used as backup. However, sometimes the direct removal or manual backup of a journal directory is required.

If Ember is running:

  • Use Ember's /opt/deltix/ember/bin/journal-compress tool to create a backup of a live system that will take care of unflushed in-memory buffers containing recent data.

If Ember is not running:

  • Backup the /var/deltix/emberwork/journal directory ($EMBER_WORK/journal).

If Ember is not running and the system is deployed under Kubernetes (and you have no easy access to Ember volume), temporarily redefine the entrypoint for Ember pod to /bin/sh using the following procedure:

  1. Go to the Kubernetes Dashboard GUI.

  2. Switch to Stateful Sets.

  3. On the ember row, click Edit.

  4. Locate the ember container and change command to the following code.
    Be careful with spaces:

              command:
                - /bin/sh
                - '-c'
                - tail -f /dev/null -n 10

    Entry point in Ember pod

  5. Click Save.

  6. Restart Ember by scaling Ember to 0, then back to 1.
    Now you can backup $EMBER_WORK/journal, which is mapped to /var/deltix/emberwork/journal under Docker.

  7. Go back, remove the command line entries, and repeat the steps.

If you use TimeBase data warehouse you may also want to clear/move existing data warehouse streams, to prepare for message sequence number reset. Other warehouses handle journal term.