A ruby webapp to collect and report mail server reputations, deliverability and rbl statistics. The UI is powered by Sinatra and the backend is provided by Redis. If you follow the installation steps outlined in this README, you should be up and running within 15 minutes.
The following data is collected and displayed for each postfix MTA you report on using mtarep:
- Return Path Sender Score
- Microsoft Smart Network Data Services
- Postfix SMTP Rejections
- Cloudmark CSI
- Brightmail Reputation Service
- RBL/DNSBL Listings
All the data is collected and inserted into a redis data store whenever the collector completes a run. The collector (collector.rb) can be scheduled via cron. The sinatra webapp (app.rb) handles generating the HTML and retrieving the data from redis.
Please note that the Microsoft SNDS data is only available for the previous day. Everyday at 3am EDT a process begins that aggregates data for the previous day from across various systems at Windows Live Hotmail. Due to the vast volume of data handled this process can take a few hours, so data may not be available for a few hours after 3am EDT.
When a provider block or rbl cell contains a listing, it becomes clickable. For provider blocks, when you click a block link a modal appears containing the most recent smtp rejection message from your server's maillog (location and access are configurable in the mtarep.yml file), as well details about the listing and a direct link to that provider's block removal form or instructions. The same scenario occurs for rbl listings but without the maillog info, as mtarep queries rbls directly via a dns lookup.
Each issue modal contains an acknowledgement button that can be clicked once you begin working an mtarep reported issue. When the acknowledgement button is clicked, a unique key is inserted back into redis that holds basic details about the issue, including a timestamp and the mtarep authenticated http username. Once acknowledged, the acknowledgement button is replaced with a timestamp and the mtarep authenticated http username that acknowledged the issue. This prevents multiple users working the same issue unknowingly.
Each column header in the web interface allows for rows sorting (eg: sort by rbl listings, microsoft SNDS filtering or trap hits, hostname, etc..)
Mtarep comes with a very basic REST api. I expect to add more functionality to the api as needed. At the moment, the api can only be queried for single host data.
For example, to get mtarep data for a public fqdn of 'mta-test1.domain.tld', you could do something like this:
require 'rest-client'
require 'json'
response = RestClient.get("http://username:[email protected]/api/#{public_fqdn}")
data_hash = JSON.parse(response)
=> {"hostname"=>"mta-test1.domain.tld", "senderscore"=>"98", "sndscolor"=>"green", "sndstraps"=>"2", "brightmail"=>"good", "provblocks"=>"no blocks", "listings"=>"unlisted"}A JSON blob of the requested public fqdn's mtarep data is returned.
To enable graphing you will need to configure a few things outside the scope of mtarep.
Sent, bounced, expired and feedback loop bar graphs can be configured for the domains of your choosing via the 'graph_domains' YAML array collection in the mtarep.yml configuration file. The graphing is provided by the HighCharts JS API. Individual bar graphs can be removed from view by clicking each type of graph in the graphing legend. This allows more granular detail on specific bar graphs, which can be particularly helpful if your sent total bar graph obfuscates the shit out of the bounce, fbl or expired graphs.
Each individual bar graph is calculated from midnight on the current day and continues to be calculated until 11:59pm on that same day. The data used to calculate the sent, bounced, fbl and expired bar graphs is not collected by mtarep. However, bar graphs will be calculated and rendered by mtarep if the appropriate data exists in the same redis db used by mtarep.
The mtarep graphing code will search your mtarep redis backend for HINCRBY based keys in the format of:
mycompany:20140208:expired
mycompany:20140208:fbl
mycompany:20140208:bounced
mycompany:20140208:sentWhere '20140208' is the current date and 'mycompany' is some type of identifier (perhaps your sending domain, company name or client name?)
These redis keys should contain an incrementing total for each unique domain that has been sent, bounced, expired or feedback-loop received. The exact redis operation is HINCRBY.
Using the redis-rb ruby client you could do something like this:
require 'redis'
@redis = Redis.new(:host => 'your redis server')
key = ['mycompany']
key << Time.now.strftime("%Y%m%d")
key << 'bounced'
@redis.hincrby(key.join(':'), 'gmail.com', 1)The above code would increment a counter for each bounced email to 'gmail.com' under the redis key 'mycompany:20140208:bounced'. This could be accomplished by creating a custom redis output plugin with Logstash to filter and ship this data directly from your postfix mail logs to either a pubsub broker like RabbitMQ or perhaps even directly to your redis datastore.
Mtarep comes with example configuration files for rackup (config.ru), thin (thin.yml) and mtarep itself (mtarep.yml). Please copy the example configs without the '.example' appendage and adjust each config according to your environment.
The individual mtarep.yml configuration file settings are as follows:
error_log:
The absolute path to your main mtarep error log file. All directories in the path must already exist. If the error log file does not exist, it will be created.
redis_server:
The DNS resolvable hostname of your redis server. Your redis server will be accessed over the default port of 6379 and the primary 'db0' database. Currently non standard ports and multiple redis databases are not supported.
snds_key:
Your organization's microsoft Smart Network Data Services data access key. If you do not use microsoft SNDS, you should signup immediately. There is a wealth of info regarding your inbox placement.
maillog_path:
The absolute path to your current postfix mail log file on each postfix MTA that you are using mtarep to report major provider rejections/blocks for. Currently only postfix log formats are supported.
ssh_key:
The absolute path to the ssh key file (on the server running mtarep) you want to use to access your remote postfix MTA mail logs with. The ssh key you provide here must have permissions to the remote 'maillog_path' specified above, and for the 'ssh_user' you specify below.
ssh_user:
The ssh username associated with the 'ssh_key' you specified above.
http_auth_file:
The absolute path to your mtarep app's http authentication file. This is used for authentcating login access credentials (username/password) to the mtarep web interface.
The format of this file must be:
username:{SHA}ME2JP/+546KPSPZQxQirw0qkUsQRyYWM=Currently only a SHA1 base64 digest is supported (Digest::SHA1.base64digest('password')).
mta_map:
This setting can contain either a YAML array collection or a YAML hash collection.
A YAML array collection is used when you want mtarep to use the same fqdn/hostname to lookup public fqdn/hostname/ip reputation data as well as the fqdn/hostname to ssh to for mail log parsing (ESP and ISP (provider) SMTP rejection log entries). Do not use this type of map if you ssh to your MTAs using a different fqdn/hostname or host alias from it's public mail fqdn/hostname.
The YAML hash collection support for this configuration setting should be used if you ssh to your MTAs using an internal hostname or alias that is different from the host's public fqdn hostname.
For example, if you have an MTA with a public HELO hostname of mta1.mydomain.com and you also access the MTA with ssh using that same hostname, then you want to use a YAML array collection here. If you have an MTA with a public HELO hostname of mta1.mydomain.com but you ssh to the MTA using the shortname 'mta1', then you want to use a YAML hash collection here (mta1: 'mta1.mydomain.com', etc..).
graph_domains: (optional)
The domains that you want mtarep to render sent, bounced, feedback-loop and expired bat graph totals for.
rbls:
A YAML array collection of RBL/DNSBL's you want mtarep to check your MTA public IP addresses against. The format of these must be the RBL hostnames that are queried for listings (eg: bl.spamcop.net)
provider_block_strings:
A YAML hash collection of external email provider names (the key) and a string (the value) that indicates a provider is blocking your MTA. These key/values are used by mtarep to search your remote MTA mail logs for particular rejection text patterns. The values for these are regular expressions. The rejection patterns for several major email providers are already included in the mtarep.yml.example file.
removal_links:
A YAML hash collection of external email provider names and rbl lists (the key) and a corresponding URL (the value) to that provider or rbl's listing info and/or removal form.
assistance_links: (optional)
A YAML hash collection of any custom documentation you maintain that outlines steps your organization has for resolving a specific mtarep reported issue. These links are used in the modal of a clicked issue in mtarep.
- git clone https://github.com/phantasm66/mtarep.git
- create config.ru, thin.yml and mtarep.yml configs from included example configs
- start the thin web server
- check the specified mtarep error_log location for any errors
- run mtarep/collector.rb manually to test your mtarep.yml (errors display on stderr)
- cron mtarep/collector.rb to run every 15 minutes (additional details below)
- go to http://hostname:port (according to your thin.yml config file)
You may need to adjust your cron scheduling interval for mtarep/collector.rb according to the amount of data you want mtarep to collect, the number and size of your MTA maillogs, etc. Before you schedule the cron, it might be wise to do a manual collection run with your production mtarep.yml and time it. However, if collector runs do overlap, the collector will safely terminate previously running collector processes. This prevents cascading collection runs, collisions, etc.