-
Notifications
You must be signed in to change notification settings - Fork 34
Rrd2istatd
USAGE: python rrd2istatd.py [--counter] COUNTERMAPFILE RRADIR
COUNTERMAPFILE should contain lines of 'COUNTERID COUNTERNAME'
specify --counter if the values being converted should be stored as counters
otherwise the values will be assumed to be gauges
The "RRADIR" will be a directory of rra files dumped to xml format. The naming should be value_####.xml.
NOTE that it's important to know the difference between counters and gauges BEFORE transferring your data. This typing is FIXED when the stat is created in istatd and future values reported will all be treated the same as the first value seen, regardless of how future values are reported.
- Gauges are things which still "hold their value when not being reported" like %CPU and "bytes free on the disk"
- Counters are things which "happen a number of times" between samples, like "line reported to error log" or "byte sent over the wire to eth0" or "byte read from disk"
This is a simple text file mapping the cacti / rrd numeric id to the desired istatd counter name. The only whitespace allowed is the newlines to separate the lines, and the space between the counter id and the associated value.
counter_id counter.name.with.dots
counter_id counter.name.with.dots
counter_id counter.name.with.dots
...
e.g. 123 counter.one 234 counter.two 345 really.long.counter.name
Note that this requires that the rrd tool be installed. These files will take significantly more disk space than the rrd files being dumped.
cd [whereever the rra files currently live]
mkdir xml
for f in *.rrd ; do
g=`echo $f| sed -e "s/rrd/xml/"`
echo converting $f
rrdtool dump $f > xml/$g
done
- the counters must NOT already exist on istatd when the conversion is started. This is due to the limitation where istatd won't write "historical" counters back into past buckets
- the counter (gauge) names specified in the counter list file must only contain a-z . and _ ... no spaces or special characters!
If you have a beefy istatd machine (which of course you should), you're probably best off just copying the rra files over to the istatd machine for conversion and upload directly.
The conversion tool is single threaded and only works on a single counter at a time. Istatd will be able to accept MUCH more data than the python converter can throw at it. It's probably better to split your counter map file into chunks and invoke in parallel to speed up the import. I suggest using half your cores for python processes. On a 4 core box, use 2 python processes. 8 cores, 4 pythons, etc. Hyperthreaded cores count!!
Once you've staged your dumped rra fils into an xml subdirectory, go ahead and throw the data at istatd using netcat in parallel.
Example:
wc -l counters.txt
1000 counters.txt
grep processor /proc/cpuinfo | wc -l
16
# so, we'll say around 125 per file
split --lines=125 counters.txt chunk
# now we have a bunch of chunk files
for f in chunk* ; do
python rrd2istatd.py $f xml | netcat localhost 8111 &
done