add gearpump benchmark

[huafengw]

Gearpump is an Akka based real-time big data streaming engine initiated by Intel. Before opening this pull request, we did the benchmark tests in our own environment comparing with Storm and here is the brief result:

Cluster setup:

• 4 nodes in total, each with Intel Xeon processors running at 2.90 GHz, with total of 32 cores. Each node has 64G memory
• 10 Gb network
• 10 second windows
• 100 campaigns
• 10 ads per campaign
• 4 Kafka nodes with 4 partitions, not replication
• 1 Redis node
• 16 worker nodes (Storm)
• 3 zookeeper nodes

We launch multiple Kafka producers on each node to create heavy load and we mainly test the load at 170,000 events/s in total.

Storm setup:

• Storm 0.10.0
• No ack
• 4 supervisor nodes with default configuration

Gearpump setup:

• One Worker per node
• All default configuration

Comparison result at 170,000 events per second:

Generally our initial result shows that Gearpump's performance is fairly close to Storm's, sometimes Storm is better. Please note that Storm without acking can not have at-least-once guarantee while Gearpump still have that even if with high throughput.

When running sh stream-benchmarks.sh GEARPUMP_TEST , you can browser http:///localhost:8090 to see Gearpump's web UI, the default user name is admin ans the password is admin. You will be able to monitor the running streaming application something like:

Hope you guys can try our project and we are welcome to any question or advice:)

[knusbaum]

Nice! I'll take a closer look at this tomorrow.

[yahoocla]

Thank you for submitting this pull request, however I do not see a valid CLA on file for you. Before we can merge this request please visit https://yahoocla.herokuapp.com/ and agree to the terms. Thanks! 😄

[yahoocla]

CLA is valid!

[revans2]

Sorry this took so long. We are trying to fix things so that community participation is better taken care of.

The code looks great, but when we go to download the release, we get an HTML page, not the actual gearpump release, with a redirection link to AWS. Could you take a look at that please?

If I manually download the release everything works fine so I am +1 after this is fixed.

[huafengw]

Sorry about that, I'll let you know after I correct the code.

[revans2]

@huafengw it is not a big deal, spark changed things on us too and the build does not work out of the box for spark so I am working on a patch for that too.

[huafengw]

@revans2 Sorry for the late response. I just update the pull request and hope everything works fine.