Skip to content

Configuring istatd

Jump to bottom
ehohenstein edited this page Jun 18, 2014 · 4 revisions

Home - Configuring istatd

There are a few different ways to configure istatd, through commandline arguments, or config files. What method you use depends on how istatd will be run, and partially dependent on personal preference.

You can run istatd as a regular user, if you do not specify any directory or other file that the user is not allowed to access, and if you specify a very low number of maximum files, and you do not use the localstat option. Specifying any of those options requires that you start istatd as root, although istatd will drop privileges to a user you specify after start-up is done.

The number of open files is especially important: not only must you tell istatd what the maximum limit on open files should be (this should be larger than three times the maximum number of counter names you will collect) but you must also configure the limits in /etc/security/limits.conf (or similar for your distribution) to allow that many files to be open for the user in question. istatd cannot do this for you, and the default on most unix distributions is 1024, which is orders of magnitude too small for any real installation.

Also, you should figure out how metrics go into the system in the first place. On a single machine, sending metrics to the local istatd instance using TCP or UDP is straightforward. On a bigger server cluster, you want to choose between having each statistics client send the statistics directly to the central collecting instance ("master,") or running a small istatd copy on each machine, forwarding to the master, without persistently storing any data locally ("agent.") IMVU uses agent mode -- any system on any host can send data to istatd by sending a UDP or TCP line of text on the known port on the localhost. The istatd agent will buffer some amount of statistics in case of master connection disruption; when that buffer is exhausted, a drop-oldest-first replacement policy is used for dropping data to forward.

When running an istatd-agent on a host that is reporting a very large number of metrics, it may be necessary to increase the size of the UDP receive buffer. By default, istatd will attempt to set the UDP receive buffer to 256K. This is sufficient to avoid dropping UDP packets (and thus recorded metrics) for most agents under normal circumstances. However, if istatd receives a large burst of packets or a continuous stream of high volume packets, some packets may be lost due to the overflow of the UDP receive buffer. To make this less likely, the size UDP receive buffer can be increased using the --udp-buffer-size configuration parameter, which is specified in Kilobytes. Note that the OS may impose limits on the maximum size of a socket receive buffer which may even be smaller than the 256K default for this parameter. On Linux, you can set the system-wide maximum socket receive buffer size to 16MB using this command:

# sysctl -w net.core.rmem_max=16777216

As an Application

When running istatd directly as an executable, you can give it various command-line arguments (which you can also see by typing istatd --help):

  • --admin-port - When specified, port that exposes an admin telnet interface (Default: 8031)
  • --agent - Agent to forward to, rather than save data locally. (Default: none)
  • --agent-interval - Interval for batching values when forwarding. (Default: 10)
  • --config - Config file to read, if present. (Default: istatd.cfg)
  • --daemonize - Unattach stdin; detach to the background. (Default: 0)
  • --debug - When specified, turns on specific debugging options (as comma separated list). (Default: allKeys)
  • --fake-time - Fake current time to UNIX epoch (for testing). (Default: 0)
  • --files-dir - Where to serve built-in web files from. (Default: 60)
  • --flush - How often (in seconds) to flush stats files to disk. (Default: 60)
  • --http-port - Port to listen to for HTTP status service. 0 to disable. (Default: 8011)
  • --listen-address - IP address on which to start listeners. empty string to accept on all interfaces. (Default: none)
  • --local-stats - Gather local stats, counter names suffixed by the given string. (Default: localhost)
  • --log-file - When specified, errors and debugging information are logged to the specified file. Otherwise, this information is output on stdout/stderr. (Default: /var/db/istatd/istatd.log)
  • --log-level - How chatty to be on stdout/stderr. Higher is more. (0-4) (Default: 2)
  • --min-space - When specified, minimum free space required (in bytes). When below this amount, error messages will be generated periodically. (Default: -1)
  • --moving-average - When specified, calculates an exponential moving average at given granularity at given season length. (Default: none)
  • --num-files - Number of files to allocate (rlimit). (Default: 100000)
  • --pid-file - Write the PID of the daemon to the given file. (Default: /var/db/istatd/istatd.pid)
  • --replica-of - Source to pull-replicate from. Incompatible with stat-port. (Default: none)
  • --replica-port - Port that others can pull-replicate from. Requires store. (Default: 8021)
  • --retention - When specified, adjusts number and size of retention times. (Default: 10s:10d,5m:1y9d,1h:6y12d)
  • --rollup - How many levels to roll up counter values. (Default: 0)
  • --settings - Where to store settings (mostly used for HTTP app). (Default: /var/db/istatd/settings)
  • --stat-port - Port to listen to for incoming statistics. 0 to disable. (Default: 8001)
  • --store - Where to store statistics files. (Default: /var/db/istatd/store)
  • --test - When specified, run unit tests built into executable; don't start program. (Default: 0)
  • --thread-count - Number of threads to create. (Default: 3)
  • --user - User-id to set after adjusting rlimit. (Default: 1000)
  • --udp-buffer-size - Buffer size for UDP sockets, in Kilobytes. (Default: 256)

Here is an example invocation:

istatd --log-file /dev/null --flush 300 --agent myhost:8111

As a Service

When running istatd via /etc/init.d/istatd there are a few spots that can be modified.

First, there is /etc/default/istatd which looks something like this:

# 0: enable istatd when /etc/init.d/istatd script is invoked
# default: 0
ENABLE_ISTATD=1
EXTRA_ARGS=""

This is a shell script that gets sourced by the init.d script for istatd:

  • ENABLE_ISTATD - Whether or not istatd should be enabled to start as a service. A value of 1 will enable this.
  • CONFIG - The path containing a configuration file. (Default: /etc/istatd-server.cfg).
  • LOGFILE - The path where logs should be written for istatd. (Default: /var/log/istatd.log).
  • EXTRA_ARGS - Any extra arguments to istatd's command line (for most things, the config file is preferred).
  • There are other settings, too. Poke around in /etc/init.d/istatd for more.

To supply other options to istatd, a quick-and-dirty solution is to put them in the EXTRA_ARGS setting.

However, the preferred method is to put them in /etc/istatd (or wherever CONFIG is, in your /etc/default/istatd, if you've modified it).

Config files consist of # comment lines and lines containing options to supply istatd. The options here are the same as the ones you can specify on the command-line.

Here is an example file:

# This is an example!
--store /var/db/istatd/store
--settings /var/db/istatd/settings
--pid-file /var/run/istatd.pid
--log-file /var/log/istatd.log
--log-level 4
--num-files 100000
--flush 300
--thread-count 8
--stat-port 8111
--http-port 8011
--admin-port 8031
--files-dir /usr/share/istatd/files
--local-stats sometest
--debug http
Clone this wiki locally