Memory Optimizations #131
Memory Optimizations #131
Conversation
docker-compose.yml
Outdated
| @@ -15,9 +15,11 @@ services: | |||
| MAVEN_GITHUB_TOKEN: ${MAVEN_GITHUB_TOKEN:?error} | |||
| MAVEN_GITHUB_ORG: ${MAVEN_GITHUB_ORG:?error} | |||
| image: jpo-conflictmonitor:latest | |||
| privileged: true # Set true to allow writing to /proc/sys/vm/drop_caches | |||
There was a problem hiding this comment.
Should the rocksdb variables be passed into the conflict monitor in the docker compose?
There was a problem hiding this comment.
Yes they should, good catch
There was a problem hiding this comment.
The conflict visualizer has another docker-compose file that brings up the conflict monitor. Do we need to add this parameter to that file as well? This change would not be part of this PR, but would involve a separate PR to the conflict visualizer.
There was a problem hiding this comment.
The "privileged" setting was mainly for testing. I changed it to false by default. It doesn't need to be included anywhere else (and shouldn't be set in production). But the new environment variables below, starting with "ROCKSDB_" should be included anywhere there is a docker compose for cimms.
There was a problem hiding this comment.
Awesome. Thanks for the clarification. I will open a PR in the Conflict Visualizer to match.
Dockerfile
Outdated
| # Use jemalloc for RocksDB per Confluent recommendation: | ||
| # https://docs.confluent.io/platform/current/streams/developer-guide/memory-mgmt.html#rocksdb | ||
| RUN amazon-linux-extras install -y epel | ||
| RUN yum install -y jemalloc-devel |
There was a problem hiding this comment.
Might be best to combine these in one line and remove the downloaded apt lists, relevant docker best practices
There was a problem hiding this comment.
Ok, combined the commands, but amazon linux uses yum, not apt, so can't delete apt lists (and I can't find any info about if yum has some equivalent that should be removed)
docker-compose.yml
Outdated
| kafka: | ||
| condition: service_healthy | ||
| required: false | ||
| # deduplicator: |
There was a problem hiding this comment.
Do we want to remove the deduplicator as part of this PR? Or should we remove it as a part of a separate PR? In either case, we should probably delete it entirely (including removing the source code) instead of just commenting it out.
There was a problem hiding this comment.
I had commented it out because, although it's not related to this PR, it was causing a conflict after pulling in the latest jpo-utils/develop branch. Now deleted entirely.
There was a problem hiding this comment.
I suspect the problem is that the new jpo-utils project contains its own version of the jpo-deduplicator. This would cause two versions of the deduplicator to start up and would break things since they will both try to allocate the same ports. So this fix seems reasonable to me.
|
Have these changes made it into the |
Optimize CIMMS memory usage.
Implements several best practices for managing memory usage.
Java Memory
Updates JVM options in the Dockerfile:
https://kafka.apache.org/documentation.html#java
RocksDB Native Memory
Implements Confluent recommendations for managing RocksDB state store memory.