Logging improvements to support field diagnosis

[quixotique]

From the NAF 2 contract:

In order for the Serval Project to provide ongoing support to deployments of OpenBTS, it will be necessary for the Serval DNA to collect diagnostic logs and for users to send those logs to Serval engineers upon request. Serval DNA already has its own internal logging system, but it only writes to a single output stream, typically a file on disk.
This will require development of a more general log output mechanism for Serval DNA that can automatically rotate log files and retain incidents of note. The extent of crash diagnosis will depend on the level of debugging support present on the OpenBTS platform. It will also require instructions for users to enable and disable logging and a mechanism for sending logs to the Serval Project.

The following improvements will achieve this goal:

• Provide another log level between INFO and DEBUG, called VERBOSE.
• Include the symbolic name of the debug flag in each DEBUG log message to aid filtering of large log files.
• Make all test scripts assert no ERROR or FATAL messages in server log files, and fix the failures that arise by converting WHY() messages to WARN() or INFO(), or by fixing the causes of the ERRORs directly.
• New code to rotate log files based on configurable size and time limits with sensible defaults.
• Expose a JNI command-line entry point to log to the Serval DNA logging system so that some Batphone activity, eg WiFi detection messages, can be logged to the Serval log files.
• New servald command to collect the current log files into a single Rhizome bundle addressed to Serval.
• Set up a servald daemon on serval1 to only receive pushed log bundles from Commotion OpenBTS sites.
• Configure OpenBTS units with the IP addresses (or host names, see Allow DNS names in Directory Service client config #18) of serval1.
• Documentation in servald-dna/doc with instructions for pushing log files and configuring debug flags and logging levels.

[quixotique]

Update on this issue:

The following logging features have been implemented so far:

• log files now contain the symbolic log flag in curly braces, eg, {rhizome}
• test scripts assert no ERROR messages in CLI invocations, but not in daemon logs
• configurable log file rotation
• CLI command to just log a message at a given level
• new log level HINT between INFO and WARN
• log outputter infrastructure makes it easy to add new logging back-ends