Skip to main content

Tuning Ember for Market Maker - Part 1

This article describes set of steps to tune Ember and TimeBase for Market Maker.

December 2023

Trade-offs​

  • Market Maker Java algorithm allocates small amounts of memory on hot path. Future versions may reduce memory allocations, but currently GC tuning is required.
  • Some tuning steps like CPU isolation for critical threads and OS kernel parameters may be not practical for a typical Market Maker user.

Key tuning parameters​

Distilled set of tune ups:

Hardware​

  • Allocate more CPUs. Don't use overall load as indicator of spare CPU capacity.

OS​

  • OS: apply latency-performance tuned profile as described here.

Docker​

  • Run under host networking mode
  • Remove CPU quotas for ember and TimeBase or over-provision them.
  • Move all auxiliary services (Graylog, Grafana, TimeBase Admin, etc.) to another instance

JVM Memory​

For all key Java containers (TimeBase, Aggregator, Ember) configure JVM to use initial heap size equal to maximum heap size (for example, -Xms4G -Xmx4G). We also recommend pre-toallocating memory pages on JVM startup using -XX:+AlwaysPreTouch.

Use Java 17+ and ZeroGC Garbage Collection ( -XX:+UseZGC).

TimeBase​

  • Disable Prometheus telemetry for TimeBase (frequent metrics polling inflict GC). Configure file ./common/timebase-admin.properties:
TimeBase.enableMetrics=false

It may be a good idea to disable corresponding Prometheus exporter to avoid it polling TimeBase metrics every few seconds. This can be done by commenting out the following lines in ./monitoring-server/prometheus.yml

    #  - job_name: timebase
    #    static_configs:
    #      - targets: [ 'timebase:8011' ]
    #    metrics_path: /tb/metrics

Aggregator​

Set the following values in Advanced parameters section of Universe Configurator:

  • Starting from TimeBase 5.6.53 we recommend using newer 5.0 data storage format when recording market data. Older 4.3 stream format may still be recommended for environments that have small number of market data streams (less than ~10) and do not purge older market data history frequently.
  • Aggregator process settings: Set loader performance to MIN_LATENCY in Aggregator. MIN_LATENCY adds a thread doing "busy waiting" for new data to send downstream. This setting will require additional vCPU for Aggregator.

Market Maker Algorithm​

  • Set subscription performance to LOW_LATENCY (Available in Ember 1.12.72+)
  • Set idle strategy to NOOP (busy waiting mode). Ideally dedicate isolated CPU core to MM algo.
  • Switch to fast Order Book in MM algorithm settings. Fast Order book is relatively new version of Order Book processor (takes ~ 8x times less to process market data). We will make it default in the next major release.
algorithms {
  MM : ${template.algorithm.default} {
    ...
    subscription {
      performance = LOW_LATENCY                  // <--------------
    }
    idleStrategy = ${template.idleStrategy.noop} // <--------------
    settings {
      orderBookType = FAST                       // <--------------
    }
  }

Final recommendation​

For systems demanding the lowest latency, the golden rule is simplicity: avoid all non-essential tasks. When in doubt, delegate or offload.

Realistic expectations about internal market data latency for pragmatically (partially) tuned system:

  • 99% < 1 millisecond
  • 99.999% < 50 milliseconds

If you need better latency characteristics consider using TimeBase topics.

Please let Deltix support team know if you need help with system tuning.

Update: read Market Maker tuning - Part 2 continues with further optimization suggestions.