README.md

Scribengin

#Contents#

1. Overview
2. The Problem
3. The Scribengin Solution
4. Read this to get Started
5. Performance Results
6. Quickstart Guides
7. Developer Info
8. Developer Guidelines
9. Releasing Code

####Overview Pronounced "Scribe Engine"

Scribengin is a highly reliable, highly available, and performant event/logging transport that registers data under defined schemas in a variety of end systems. Scribengin enables users to have multiple flows of data from a data source to a data sink. Scribengin will tolerate system failures of individual nodes and will do a complete recovery in the case of system failure.

####The Problem The core problem is how to reliably and scalably have a distributed application write data to multiple destination data systems. This requires the ability to do data mapping, partitioning, data enhancement, data transformation, and optional filtering to the destination system.

####The Scribengin Solution: A distributed, highly reliable ETL system that can handle multiple sources and sinks

####Read this to get started!

• High Level Scribengin Features
• Learn the Scribengin terminology, and get acquainted with Scribengin at a high level.

####Performance results

• AWS Performance Testing

####Quickstart Guides Get Scribengin launched quickly! This will walk users how to start a Scribengin instance in YARN.

• QuickStart Guide
• How to launch a cluster in any arbitrary environment i.e. in-house cluster, AWS, etc

####Developer Info

• Dev environment setup, learn to write a dataflow
• Operator Development Guide

####Developer Guidelines

• Code Conventions
• Code Organization
• Dataflow Performance and Validation
• Run Test

####Release

cd NeverwinterDP
./gradlew clean build release -x test

#Code and jars will be released to NeverwinterDP/release/build/release/