Memory Optimizations

[iyourshaw]

Optimize CIMMS memory usage.

Implements several best practices for managing memory usage.

Java Memory

Updates JVM options in the Dockerfile:

• Explicitly set the maximum Java heap space to 50% of the total container memory.
• Adds garbage collection and metaspace options similar to the recommendation for Kafka to minimize GC pauses:
  https://kafka.apache.org/documentation.html#java

RocksDB Native Memory

Implements Confluent recommendations for managing RocksDB state store memory.

• Use the recommended jemalloc allocator.
• Implements BoundedMemoryRocksDBConfig to be able to configure total RocksDB memory usage to a finite value. Attempted to choose reasonable default values. Configurable with environment variables.

[Michael7371]

Should the rocksdb variables be passed into the conflict monitor in the docker compose?

[iyourshaw]

Yes they should, good catch

[iyourshaw]

Updated

[John-Wiens]

The conflict visualizer has another docker-compose file that brings up the conflict monitor. Do we need to add this parameter to that file as well? This change would not be part of this PR, but would involve a separate PR to the conflict visualizer.

[iyourshaw]

The "privileged" setting was mainly for testing. I changed it to false by default. It doesn't need to be included anywhere else (and shouldn't be set in production). But the new environment variables below, starting with "ROCKSDB_" should be included anywhere there is a docker compose for cimms.

[Michael7371]

Might be best to combine these in one line and remove the downloaded apt lists, relevant docker best practices

[iyourshaw]

Ok, combined the commands, but amazon linux uses yum, not apt, so can't delete apt lists (and I can't find any info about if yum has some equivalent that should be removed)

[Michael7371]

looks good just minor comments

[John-Wiens]

Do we want to remove the deduplicator as part of this PR? Or should we remove it as a part of a separate PR? In either case, we should probably delete it entirely (including removing the source code) instead of just commenting it out.

[iyourshaw]

I had commented it out because, although it's not related to this PR, it was causing a conflict after pulling in the latest jpo-utils/develop branch. Now deleted entirely.