Enable buffered IO for log4j

Summary:
This patch enables bufferedIO for hadoop daemons, and sets the buffer
size to 4096, which should be slightly below 20 lines (the average size
in dfs1 is about 250bytes/line).

So obvoiusly we will not see synchronously those last 20 lines, but I
don't think it's a huge issue. On the other hand, my experiments show,
that with this settings we can speed up logging by a factor of 2.

I also added a hook which when the node is shutdown with hadoopctl
hadoop-daemon.sh stop - it will close the log manager which flushes the
contents of the buffer, so we won't miss those messages. Of course if we
kill a node with -9, we will loose up to those 20 lines.

-limited the classname to one from previous 2. The logs on dfs1 are
huge, we need
to make a pass I think over messages and limit these. In the last few
hours it's like 5GB per hour.

Test Plan: manually local setup, validated that log is flushed at normal shutdown

Reviewers: hkuang

Reviewed By: hkuang

conf/log4j.properties

@@ -34,7 +34,7 @@ log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
 log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
 
 # Pattern format: Date LogLevel LoggerName LogMessage
-log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
+log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c{1}: %m%n
 # Debugging Pattern format
 #log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
 

conf/log4j.properties.scribeappender

@@ -27,7 +27,7 @@ log4j.appender.DRFA.DatePattern=.yyyy-MM-dd-HH
 log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
 
 # Pattern format: Date LogLevel LoggerName LogMessage
-log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
+log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c{1}: %m%n
 # Debugging Pattern format
 #log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
 

src/core/org/apache/hadoop/util/StringUtils.java

@@ -36,6 +36,7 @@
 import java.util.Collection;
 
 import org.apache.hadoop.fs.*;
+import org.apache.log4j.LogManager;
 
 /**
  * General string utils
@@ -610,6 +611,7 @@ public static void startupShutdownMessage(Class<?> clazz, String[] args,
       public void run() {
         LOG.info(toStartupShutdownString("SHUTDOWN_MSG: ", new String[]{
           "Shutting down " + classname + " at " + hostname}));
+        LogManager.shutdown();
       }
     });
   }