Merge pull request #659 from HubSpot/docs

Docs

Docs/containers.md

@@ -3,12 +3,12 @@
 Singularity supports containers in a few different contexts:
 
 ###Mesos Containerizer
-The default Mesos containerizer for processes which sets resource limits/etc. Enabled by adding `mesos` to the `--containerizers` argument when running `mesos-slave`. If this is enabled, commands with no `containerInfo` definition will run under this containerizer.
+The default [mesos containerizer](http://mesos.apache.org/documentation/latest/mesos-containerizer/) for processes which sets resource limits/etc. Enabled by adding `mesos` to the `--containerizers` argument when running `mesos-slave`. The mesos containerizer can isolate the task via cpu, memory or other parameters specified using the `--isolation` flag when starting the mesos slave. Deploys with no `containerInfo` definition will try to run under this containerizer by default.
 
 ###Mesos Docker Containerizer
-The [built-in docker containerizer](https://mesos.apache.org/documentation/latest/docker-containerizer/) which comes with Mesos. This will manage the starting and stopping of your docker container as well as mapping ports, adding environment variables, and mapping volumes in the container to the Mesos sandbox for that task. Enable this containerizer by adding `docker` to the arguments of `--containerizers` when running `mesos-slave`.
+The [docker containerizer](https://mesos.apache.org/documentation/latest/docker-containerizer/) that ships with Mesos, will manage the starting and stopping of your docker container as well as mapping ports, adding environment variables, and mapping volumes in the container to the Mesos sandbox for that task. You can enable this containerizer by adding `docker` to the arguments of `--containerizers` when running `mesos-slave`.
 
-To use Singularity with the Docker containerizer, add a `containerInfo` to the SingularityDeploy object when creating a deploy (without specifying a `customExecutorCmd`). The Singularity deploy object's `containerInfo` field mirrors the Mesos `containerInfo` definition:
+To use Singularity with the Docker containerizer, add a [`containerInfo` field](reference/api.md#model-SingularityContainerInfo) with a `type` of `DOCKER` to the [SingularityDeploy](reference/api.md#model-SingularityDeploy) object when creating a deploy (without specifying a `customExecutorCmd`). The Singularity deploy object's [`containerInfo` field](reference/api.md#model-SingularityContainerInfo) mirrors the Mesos `containerInfo` definition:
 
 ```
 {
@@ -20,13 +20,13 @@ To use Singularity with the Docker containerizer, add a `containerInfo` to the S
     "portMappings": [
       {
         "containerPortType": "FROM_OFFER", # FROM_OFFER or LITERAL
-        "containerPort": 0, # If type is FROM_OFFER this is the index of the port assigned by Mesos. (ie 0 -> $PORT0)
+        "containerPort": 0, # If type is FROM_OFFER this is the index of the port assigned by Mesos. (ie 0 -> first assigned port)
         "hostPortType": "FROM_OFFER",
         "hostPort": 0
       }
     ]
   },
-  "volumes": [
+  "volumes": [ # The sandbox for the task will always be added as a volume at /mnt/mesos/sandbox within the container
     {
       "containerPath": "/etc/example",
       "hostPath": "/etc/example"
@@ -49,17 +49,18 @@ When the SingularityExecutor is given a task with `containerInfo` of type `DOCKE
 - map all specified environment variables to the container
 - assign and map ports and specified volumes
 - map the Mesos sandbox task directory to `/mnt/mesos/sandbox` in the container
-- create and start the container, directing output to the configured `runContext.logFile`
+- create and start the container, directing output to the configured `logFile`
 - run a `docker stop` when receiving a `SIGTERM`, try to remove the stopped container, and exit with the container's exit code
 
-A few special notes and variables that are set:
-- `MESOS_SANDBOX`: The Mesos sandbox directory as seen inside the container (generally `/mnt/mesos/sandbox`)
-- `LOG_DIR`: The log directory that SingularityExecutor will use for logrotating/uploading/etc, generally mapped to `/mnt/mesos/sandbox/logs` in the container
+A few special notes and environment variables that are set:
+- Environment variables:
+  - `MESOS_SANDBOX`: The Mesos sandbox directory as seen inside the container (generally `/mnt/mesos/sandbox`)
+  - `LOG_DIR`: The log directory that SingularityExecutor will use for logrotating/uploading/etc, generally mapped to `/mnt/mesos/sandbox/logs` in the container
 - The Docker working directory will be set to the `taskAppDirectory` in the Mesos sandbox
-- The container name will be the task id
+- The container name will be a configured prefix (`se-` by default) and the the task id (`SingularityExcutorCleanup` uses this to optionally clean up old contaienrs that are managed by Singularity)
 - SingularityExecutor will explicitly try to pull the image (ie, must be from a repository reachable by the slave)
 
-Here is an example deploy to get you started:
+Here is an example deploy you can use with the [docker-compose](development/docker.md) setup to get you started:
 
 ```
 {

Docs/database.md

@@ -39,3 +39,16 @@ INFO  [2013-12-23 18:42:10,327] liquibase: ChangeSet migrations.sql::1::tpetr ra
 ```
 
 More information about `db` tasks can be found in the dropwizard-migrations [docs](http://dropwizard.io/manual/migrations), and more information about the migration file syntax can be found in the liquibase [docs](http://www.liquibase.org/documentation/yaml_format.html).
+
+### Purging Old Tasks
+
+You can optionally purge old task data from the db by specifying `historyPurging` configuration. In the configuration for SingularityService, you can have a section similar to the following (default values shown):
+
+```
+historyPurging:
+  deleteTaskHistoryAfterDays: 365                  # purge tasks older than this
+  deleteTaskHistoryAfterTasksPerRequest: 1000      # How many tasks per request before purge
+  deleteTaskHistoryBytesInsteadOfEntireRow: true   # Keep the row, just delete data to save space
+  checkTaskHistoryEveryHours: 24                   # how often to check for tasks to purge
+  enabled: false                                   # determines if we should run the purge
+```

Docs/details.md

@@ -6,7 +6,7 @@
 Singularity is an essential part of the HubSpot Platform and is ideal for deploying micro-services. It is optimized to manage thousands of concurrently running processes in hundreds of servers.
 
 ## How it Works
-Singularity is an [**Apache Mesos framework**](http://Mesos.apache.org/documentation/latest/Mesos-frameworks/). It runs as a *task scheduler* on top of **Mesos Clusters** taking advantage of Apache Mesos' scalability, fault-tolerance, and resource isolation. [Apache Mesos](http://Mesos.apache.org/documentation/latest/Mesos-architecture/) is a cluster manager that simplifies the complexity of running different types of applications on a shared pool of servers. In Mesos terminology, *Mesos applications* that use the Mesos APIs to schedule tasks in a cluster are called [*frameworks*](http://Mesos.apache.org/documentation/latest/app-framework-development-guide/).
+Singularity is an [**Apache Mesos framework**](http://mesos.apache.org/documentation/latest/mesos-frameworks/). It runs as a *task scheduler* on top of **Mesos Clusters** taking advantage of Apache Mesos' scalability, fault-tolerance, and resource isolation. [Apache Mesos](http://mesos.apache.org/documentation/latest/mesos-architecture/) is a cluster manager that simplifies the complexity of running different types of applications on a shared pool of servers. In Mesos terminology, *Mesos applications* that use the Mesos APIs to schedule tasks in a cluster are called [*frameworks*](http://mesos.apache.org/documentation/latest/app-framework-development-guide/).
 
 ![Mesos Frameworks](images/Mesos_Frameworks.png)
 
@@ -29,13 +29,28 @@ The *Mesos master* determines how many resources are offered to each framework a
 As depicted in the figure, Singularity implements the two basic framework components as well as a few more to solve common complex / tedious problems such as task cleanup and log tailing / archiving without requiring developers to implement it for each task they want to run:
 
 ### Singularity Scheduler
-The scheduler is the core of Singularity: a DropWizard API that implements the Mesos Scheduler Driver. The scheduler matches client deploy requests to Mesos resource offers and acts as a web service offering a JSON REST API for accepting deploy requests.
+The scheduler is the core of Singularity: a [DropWizard](http://www.dropwizard.io/) API that implements the Mesos Scheduler Driver. The scheduler matches client deploy requests to Mesos resource offers and acts as a web service offering a JSON REST API for accepting deploy requests.
 
 Clients use the Singularity API to register the type of deployable item that they want to run (web service, worker, cron job) and the corresponding runtime settings (cron schedule, # of instances, whether instances are load balanced, rack awareness, etc.). 
 
 After a deployable item (a **request**, in API terms) has been registered, clients can post *Deploy requests* for that item. Deploy requests contain information about the command to run, the executor to use, executor specific data, required cpu, memory and port resources, health check URLs and a variety of other runtime configuration options. The Singularity scheduler will then attempt to match Mesos offers (which in turn include resources as well as rack information and what else is running on slave hosts) with its list of *Deploy requests* that have yet to be fulfilled.
 
-Rollback of failed deploys, health checking and load balancing are also part of the advanced functionality the Singularity Scheduler offers. When a service or worker instance fails in a new deploy, the Singularity scheduler will rollback all instances to the version running before the deploy- keeping the deploys always consistent. After the scheduler makes sure that a Mesos task (corresponding to a service instance) has entered the TASK_RUNNING state it will use the provided health check URL and the specified health check timeout settings to perform health checks. If health checks go well, the next step is to perform load balancing of service instances. Load balancing is attempted only if the corresponding deployable item has been defined to be *loadBalanced*. To perform load balancing between service instances, Singularity supports a rich integration with a specific Load Balancer API. Singularity will post requests to the Load Balancer API to add the newly deployed service instances and to remove those that were previously running. Check [Integration with Load Balancers](development/lbs.md) to learn more. Singularity also provides generic webhooks which allow third party integrations, which can be registered to follow request, deploy, or task updates.
+<a name="deploys">
+Rollback of failed deploys, health checking and load balancing are also part of the advanced functionality the Singularity Scheduler offers. A new deploy for a long runing service will run as shown in the diagram below.
+
+![Singularity Deploy](images/deploy.png)
+
+When a service or worker instance fails in a new deploy, the Singularity scheduler will rollback all instances to the version running before the deploy, keeping the deploys always consistent. After the scheduler makes sure that a Mesos task (corresponding to a service instance) has entered the TASK_RUNNING state it will use the provided health check URL and the specified health check timeout settings to perform health checks. If health checks go well, the next step is to perform load balancing of service instances. Load balancing is attempted only if the corresponding deployable item has been defined to be *loadBalanced*. To perform load balancing between service instances, Singularity supports a rich integration with a specific Load Balancer API. Singularity will post requests to the Load Balancer API to add the newly deployed service instances and to remove those that were previously running. Check [Integration with Load Balancers](development/lbs.md) to learn more. Singularity also provides generic webhooks which allow third party integrations, which can be registered to follow request, deploy, or task updates.
+
+<a name="placement">
+#### Slave Placement
+
+When matching a Mesos resource offer to a deploy, Singularity can use one of several strategies to determine if the host in the offer is appropriate for the task in question, or `SlavePlacement` in Singularity terms. Available placement strategies are:
+
+- `GREEDY`: uses whatever slaves are available
+- `SEPARATE_BY_DEPLOY`/`SEPARATE`: ensures no 2 instances / tasks of the same request *and* deploy id are ever placed on the same slave
+- `SEPARATE_BY_REQUEST`: ensures no two tasks belonging to the same request (regardless if deploy id) are placed on the same host
+- `OPTIMISTIC`: attempts to spread out tasks but may schedule some on the same slave
 
 #### Singularity Scheduler Dependencies
 The Singularity scheduler uses ZooKeeper as a distributed replication log to maintain state and keep track of registered deployable items, the active deploys for these items and the running tasks that fulfill the deploys. As shown in the drawing, the same ZooKeeper quorum utilized by Mesos masters and slaves can be reused for Singularity.  
@@ -49,6 +64,7 @@ The [*Singularity UI*](ui.md) is a single page static web application served fro
 
 It is a fully-featured application which provides historical as well as active task information. It allows users to view task logs and interact directly with tasks and deploy requests.
 
+<a name="optional-components">
 ### Optional Slave Components
 
 #### Singularity Executor

Docs/development/docker.md

@@ -1,30 +1,32 @@
 ## Setup
 
-For developing or testing our Singularity with Docker, you will need to install [docker](https://docs.docker.com/installation/) and [docker-compose](https://docs.docker.com/compose/#installation-and-set-up).
+For developing or testing out Singularity with Docker, you will need to install [docker](https://docs.docker.com/installation/) and [docker-compose](https://docs.docker.com/compose/#installation-and-set-up).
 
 ## Example cluster with Docker Compose
 
 Run `docker-compose pull` first to get all of the needed images. *Note: This may take a few minutes*
 
-Then simply run `docker-compose up` to start containers for...
+Then simply run `docker-compose up` and it will start containers for...
 - mesos master
 - mesos slave (docker/mesos containerizers enabled)
-- zookeeper host
+- zookeeper
 - Singularity
 - [Baragon Service](https://github.com/HubSpot/Baragon) for load balancer management
 - [Baragon Agent](https://github.com/HubSpot/Baragon) + Nginx as a load balancer
 
 ...and the following UIs will be available:
-- Singularity UI => http://localhost:7099/singularity
-- Baragon UI => http://localhost:8080/baragon/v2/ui
+- Singularity UI => [http://localhost:7099/singularity](http://localhost:7099/singularity)
+- Baragon UI => [http://localhost:8080/baragon/v2/ui](http://localhost:8080/baragon/v2/ui)
 
 *if using [boot2docker](http://boot2docker.io/) or another vm, replace localhost with the ip of your vm*
 
+The docker-compose example clsuter will always run off of the most recent release tag.
+
 ## Developing With Docker
 
 ### `dev`
 
-In the root of this project is a `dev` wrapper script to make developing easier. You can do the following:
+In the root of this project is a `dev` wrapper script to make developing easier. It will run using images from the current snapshot version. You can do the following:
 
 ```
 ./dev pull           # Get the latest images from docker hub

Docs/install.md

@@ -6,20 +6,21 @@
 
 Singularity uses Zookeeper as its primary datastore -- it cannot run without it.
 
-Chef recipe: https://supermarket.chef.io/cookbooks/zookeeper
-Puppet module: https://forge.puppetlabs.com/deric/zookeeper
+Chef recipe: [https://supermarket.chef.io/cookbooks/zookeeper](https://supermarket.chef.io/cookbooks/zookeeper)
 
-More info on how to manually set up a Zookeeper cluster lives here: https://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup
+Puppet module: [https://forge.puppetlabs.com/deric/zookeeper](https://forge.puppetlabs.com/deric/zookeeper)
 
-For testing or local development purposes, a single-node cluster running on your local machine is fine.
+More info on how to manually set up a Zookeeper cluster lives [here](https://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup)
+
+For testing or local development purposes, a single-node cluster running on your local machine is fine. If using the [docker testing/development setup](development/docker.md), this will already be present.
 
 ### 2. Set up MySQL (optional)
 
 Singularity can be configured to move stale data from Zookeeper to MySQL after a configurable amount of time, which helps reduce strain on the cluster. If you're running Singularity in Production environment, MySQL is encouraged.
 
 ### 3. Set up a Mesos cluster
 
-Mesosphere provides a good tutorial for setting up a Mesos cluster: http://mesosphere.com/docs/getting-started/datacenter/install/. Don't bother setting up Marathon, it isn't necessary for Singularity.
+Mesosphere provides a good tutorial for setting up a Mesos cluster: http://mesosphere.com/docs/getting-started/datacenter/install/. You can skip the section on setting up Marathon since Singularity will be our framework instead.
 
 ### 4. Build or download the Singularity JAR
 
@@ -31,7 +32,7 @@ Run `mvn clean package` in the root of the Singularity repository. The Singulari
 
 #### Downloading a precompiled JAR
 
-Singularity JARs are published to Maven Central for each release. You can view the list of SingularityService (the executable piece of Singularity) here: http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22com.hubspot%22%20AND%20a%3A%22SingularityService%22
+Singularity JARs are published to Maven Central for each release. You can view the list of SingularityService (the executable piece of Singularity) JARs [here](http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22com.hubspot%22%20AND%20a%3A%22SingularityService%22)
 
 Be sure to only use the `shaded.jar` links -- the other JARs won't work.
 
@@ -40,6 +41,8 @@ Be sure to only use the `shaded.jar` links -- the other JARs won't work.
 Singularity requires a YAML file with some configuration values in order to start up. Here's an example:
 
 ```yaml
+
+# Run SingularityService on port 7099 and log to /var/log/singularity-access.log
 server:
   type: simple
   applicationContextPath: /singularity
@@ -49,8 +52,8 @@ server:
   requestLog:
     appenders:
       - type: file
-        currentLogFilename: ../logs/access.log
-        archivedLogFilenamePattern: ../logs/access-%d.log.gz
+        currentLogFilename: /var/log/singularity-access.log
+        archivedLogFilenamePattern: /var/log/singularity-access-%d.log.gz
 
 database:  # omit this entirely if not using MySQL
   driverClass: com.mysql.jdbc.Driver