diff --git a/README.md b/README.md
index 07cdd4e..87b3f42 100644
--- a/README.md
+++ b/README.md
@@ -1,52 +1,10 @@
-# Overview
-
-
-
-**Performa** is a multi-server monitoring system with a web based front-end UI. It can monitor CPU, memory, disk, network, and of course your own custom metrics. Alerts can be configured to trigger on any expression, and send e-mails or fire web hooks. Timeline data can be stored on local disk or in Amazon S3.
-
-## Features at a Glance
-
-- Easy to install, configure and run
-- Monitor any number of servers
-- New servers are added to the system automatically
-- Assign servers to groups manually or automatically
-- Supports ephemeral servers (serverless, autoscale, etc.)
-- Metrics are collected every minute
-- Multiple graph scales: hourly, daily, monthly, yearly
-- Real-time views with auto-refreshing graphs
-- View graphs for individual servers or entire groups
-- Add custom commands for gathering your own metrics
-- Alerts with custom trigger expressions
-- Alert e-mails and web hooks for notification
-- Alert snooze feature to silence notifications
-- One click snapshot-to-URL-to-clipboard for graphs
-- Graph data can be kept indefinitely or auto-expired
-- Light and dark themes for the UI
-
-## Screenshots
-
-See Screenshots
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Table of Contents
+Table of Contents
-* [Glossary](#glossary)
+- [Overview](#overview)
+ * [Features at a Glance](#features-at-a-glance)
+ * [Screenshots](#screenshots)
+ * [Glossary](#glossary)
- [Installation](#installation)
- [Setup](#setup)
- [Configuration](#configuration)
@@ -86,6 +44,7 @@
+ [Built-in Alerts](#built-in-alerts)
+ [Alert Web Hooks](#alert-web-hooks)
* [Commands](#commands)
+ * [Snapshots](#snapshots)
- [Command Line](#command-line)
* [Starting and Stopping](#starting-and-stopping)
* [Storage Maintenance](#storage-maintenance)
@@ -99,6 +58,56 @@
* [Starting in Debug Mode](#starting-in-debug-mode)
- [License](#license)
+
+
+# Overview
+
+
+
+**Performa** is a multi-server monitoring system with a web based front-end UI. It can monitor CPU, memory, disk, network, and of course your own custom metrics. Alerts can be configured to trigger on any expression, and send e-mails or fire web hooks. Timeline data can be stored on local disk or in Amazon S3.
+
+## Features at a Glance
+
+- Easy to install, configure and run
+- Monitor any number of servers
+- New servers are added to the system automatically
+- Assign servers to groups manually or automatically
+- Supports ephemeral servers (serverless, autoscale, etc.)
+- Metrics are collected every minute
+- Multiple graph scales: hourly, daily, monthly, yearly
+- Real-time views with auto-refreshing graphs
+- View graphs for individual servers or entire groups
+- Add custom commands for graphing your own metrics
+- Alerts with custom trigger expressions
+- Alert e-mails and web hooks for notification
+- Alert snooze feature to silence notifications
+- Snapshot feature provides extra server details
+- One click snapshot-to-URL-to-clipboard for graphs
+- Graph data can be kept indefinitely or auto-expired
+- Light and dark themes for the UI
+
+## Screenshots
+
+See Screenshots
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
## Glossary
A quick introduction to some common terms used in Performa:
@@ -112,6 +121,7 @@ A quick introduction to some common terms used in Performa:
| **API Key** | A special key that can be used by external apps to send API requests into Performa. |
| **User** | A human user account, which has a username and a password. Passwords are salted and hashed with [bcrypt](https://en.wikipedia.org/wiki/Bcrypt). |
| **Satellite** | Our headless companion product, which silently collects metrics on your servers and sends them to the master server. See [Performa Satellite](#performa-satellite) below. |
+| **Snapshot** | A snapshot is a detailed record of everything happening on a server, including all processes and network sockets. Snapshots are taken when alerts trigger, and when being watched. See [Snapshots](#snapshots) below. |
# Installation
@@ -477,7 +487,7 @@ You can include any property from the main `conf/config.json` file by using the
# Performa Satellite
-Performa Satellite is our headless companion product, which silently collects metrics on your servers and sends them to the Performa master server. It has no dependencies and ships as a precompiled binary (for Linux and macOS), so it will be compatible with a wide range of systems. It does not run as a daemon, but instead launches via [cron](https://en.wikipedia.org/wiki/Cron) every minute, then exits. It uses about 25 MB of RAM while it is active.
+Performa Satellite is our headless companion product, which silently collects metrics on your servers and sends them to the Performa master server. It has no dependencies and ships as a precompiled binary (for Linux and macOS), so it will be compatible with a wide range of systems. It does not run as a daemon, but instead launches via [cron](https://en.wikipedia.org/wiki/Cron) every minute, then exits. It uses about 25 MB of RAM while it is active (usually only a few seconds per minute).
For more information about Performa Satellite, including installation and configuration instructions, please see the [Performa Satellite Github Repo](https://github.com/jhuckaby/performa-satellite).
@@ -714,14 +724,27 @@ Each command has the following properties:
| **Title** | A display title (label) for the alert, shown in the UI. |
| **Enabled** | A checkbox denoting whether the command is enabled (will be executed) or disabled (skipped). |
| **Groups** | A list of which server groups the command should be executed on (or you can select "all" groups). |
-| **Executable** | The executable command to run (e.g. `/bin/sh` or `/usr/bin/python`). |
+| **Executable** | The executable command to run (e.g. `/bin/sh`, `/usr/bin/python` or other). |
| **Script** | The script source code to pipe to the command (i.e. shell commands or other). |
| **Format** | If your command outputs JSON or XML, you can have this parsed for easier integration with monitors / alerts. |
-| **User ID** | Optionally run your command as a different user (i.e. for security). |
+| **User ID** | Optionally run your command as a different user on the server (i.e. for security purposes). |
| **Notes** | A notes text field is provided for your own internal use. |
If your command outputs raw text, you can use a regular expression to match the specific metric value (this is configured per each monitor that refers to the command). Or, if your command happens to output JSON or XML, then Performa can parse it, and provide more structured access in the [Data Source](#data-sources) system.
+## Snapshots
+
+A snapshot is a detailed report of everything happening on a server, including all the information we collect every minute (i.e. CPU, memory stats), but also:
+
+- A detailed list of all processes running on the server, including PIDs, commands, CPU and memory usage.
+- A detailed list of all network connections, their source and destination IP addresses and ports, and each connection state.
+- A list of all socket listeners, including which protocol, interface and port number.
+- A list of all filesystem mounts and their usage and free space.
+
+Performa automatically takes a snapshot of a server whenever any alert is triggered. This allows you to come back later to see exactly what was happening at the time of the alert (i.e. which processes and connections were open). You can access snapshots from the alert e-mail notification, as well as the "Snapshots" tab in the UI.
+
+In addition to automatic snapshots, you can "watch" any server or group for any amount of time. Setting a "watch" on a server or group means that it will generate a snapshot every minute for the duration of the watch timer. You can set watches by clicking the "Watch Server..." or "Watch Group..." buttons in the UI.
+
# Command Line
Here are all the Performa services available to you on the command line. Most of these are accessed via the following shell script:
diff --git a/htdocs/css/style.css b/htdocs/css/style.css
index 4db9684..0eb7455 100644
--- a/htdocs/css/style.css
+++ b/htdocs/css/style.css
@@ -19,6 +19,10 @@ div.container {
/* color: var(--header-text-color); */
}
+#d_footer {
+ margin: 12px 0px 17px 0px;
+}
+
/* Menus */
.tab_widget {
@@ -244,6 +248,10 @@ fieldset.dt_fs > legend {
/* Table Pagination Spacing Fix */
+fieldset div.pagination {
+ margin-top: 0;
+}
+
div.pagination > table tr td {
white-space: nowrap;
}
@@ -348,6 +356,10 @@ fieldset.inline_error {
font-size: 12px;
}
+#fe_ctrl_server, #fe_ctrl_group {
+ max-width: 200px;
+}
+
.chart_size_icon {
font-size: 18px;
cursor: pointer;
@@ -558,6 +570,7 @@ body.dark .apexcharts-xaxistooltip {
top: 0;
left: 0;
z-index: 2;
+ cursor: default;
}
.server_pie_container, .server_pie_graph, .server_pie_overlay {
@@ -591,6 +604,17 @@ body.dark .apexcharts-xaxistooltip {
text-overflow: ellipsis;
}
+.server_pie_widget {
+ position: absolute;
+ top: 0;
+ left: 0;
+ z-index: 2;
+ width: 180px;
+ height: 20px;
+ line-height: 20px;
+ text-align: right;
+}
+
/* Group Tab */
div.priv_group_admin, div.priv_group_other {
@@ -624,3 +648,37 @@ fieldset.overview_group > legend {
font-size: 12px;
color: var(--header-text-color);
}
+
+/* Snapshot Tab */
+
+.inline_table_scrollarea {
+ position: relative;
+ max-height: 75vh;
+ overflow-x: hidden;
+ overflow-y: scroll;
+}
+
+table.fieldset_table tr th.st_col_header {
+ white-space: nowrap;
+ font-weight: bold;
+ cursor: pointer;
+}
+table.fieldset_table tr th.st_col_header:hover {
+ color: var(--highlight-color);
+ text-decoration: underline;
+}
+table.fieldset_table tr th.st_col_header.active {
+ color: var(--highlight-color);
+}
+
+.percent_bar_container {
+ position: relative;
+ background-color: var(--border-color);
+ height: 15px;
+}
+
+.percent_bar_inner {
+ box-sizing: border-box;
+ border-right: 1px solid var(--header-text-color);
+ height: 15px;
+}
diff --git a/htdocs/index-dev.html b/htdocs/index-dev.html
index b7fe4c7..2525b22 100755
--- a/htdocs/index-dev.html
+++ b/htdocs/index-dev.html
@@ -32,14 +32,15 @@
';
+ return html;
+ },
+
+ getCPUTableHTML: function(cpus) {
+ // render HTML for CPU detail table
+ var self = this;
+ var html = '';
+ html += '';
+ html += '
';
+ html += '
';
+ html += '
CPU #
';
+ html += '
System %
';
+ html += '
User %
';
+ html += '
Nice %
';
+ html += '
I/O Wait %
';
+ html += '
Hard IRQ %
';
+ html += '
Soft IRQ %
';
+ html += '
Total %
';
+ html += '
';
+
+ var cpu_list = [];
+ for (var idx = 0, len = num_keys(cpus); idx < len; idx++) {
+ var key = 'cpu' + idx;
+ if (cpus[key]) cpu_list.push( cpus[key] );
+ }
+
+ cpu_list.forEach( function(cpu, idx) {
+ html += '
';
+ html += '
#' + Math.floor( idx + 1 ) + '
';
+ html += '
' + pct( cpu.system || 0, 100 ) + '
';
+ html += '
' + pct( cpu.user || 0, 100 ) + '
';
+ html += '
' + pct( cpu.nice || 0, 100 ) + '
';
+ html += '
' + pct( cpu.iowait || 0, 100 ) + '
';
+ html += '
' + pct( cpu.irq || 0, 100 ) + '
';
+ html += '
' + pct( cpu.softirq || 0, 100 ) + '
';
+
+ var total = 100 - (cpu.idle || 0);
+ html += '
' + self.getPercentBarHTML( total / 100, 200 ) + '
';
+ html += '
';
+ });
+
+ html += '
';
+ return html;
}
} );
diff --git a/htdocs/js/pages/Group.class.js b/htdocs/js/pages/Group.class.js
index c4c5f07..86a7ff8 100644
--- a/htdocs/js/pages/Group.class.js
+++ b/htdocs/js/pages/Group.class.js
@@ -115,6 +115,12 @@ Class.subclass( Page.Base, "Page.Group", {
var html = '';
// html += '
' + this.group.title + '
';
+ html += '
';
+ html += '' + this.group.title + "";
+ html += '
Watch Group...
';
+ html += '
Take Snapshot
';
+ html += '';
+ html += '
';
// insert alerts and server list table here
// (will be populated later)
@@ -676,6 +682,7 @@ Class.subclass( Page.Base, "Page.Group", {
html += '
Detail
';
html += '
Trigger
';
html += '
Date/Time
';
+ html += '
Actions
';
html += '';
all_alerts.forEach( function(alert) {
@@ -691,6 +698,10 @@ Class.subclass( Page.Base, "Page.Group", {
html += '
' + alert.message + '
';
html += '
' + alert_def.expression + '
';
html += '
' + get_nice_date_time( alert.date ) + '
';
+
+ var snap_id = alert.hostname + '/' + Math.floor( alert.date / 60 );
+ html += '
';
+
html += '';
});
@@ -711,7 +722,8 @@ Class.subclass( Page.Base, "Page.Group", {
// group info table: fs_group_info
var extra_server_info = config.extra_server_info;
var html = '';
- html += '';
+ // html += '';
+ html += '';
html += '
Use the menu below to optionally set watch timers on all current servers in the group. This will generate snapshots every minute until the timer expires.
';
+ watch_sel = 3600;
+
+ html += '
' +
+ // get_form_table_spacer() +
+ get_form_table_row('Watch For:', '') +
+ get_form_table_caption("Select the duration for the group watch.") +
+ '
';
+
+ app.confirm( 'Watch Group', html, "Set Watch", function(result) {
+ app.clearError();
+
+ if (result) {
+ var watch_time = parseInt( $('#fe_watch_time').val() );
+ var watch_date = time_now() + watch_time;
+ Dialog.hide();
+
+ app.api.post( 'app/watch', { hostnames: hostnames, date: watch_date }, function(resp) {
+ // update local state and show message
+ if (!state.watches) state.watches = {};
+
+ if (watch_time) {
+ app.showMessage('success', "Group will be watched for " + get_text_from_seconds(watch_time, false, true) + ".");
+ hostnames.forEach( function(hostname) {
+ state.watches[ hostname ] = watch_date;
+ });
+ }
+ else {
+ app.showMessage('success', "Group watch has been disabled.");
+ hostnames.forEach( function(hostname) {
+ delete state.watches[ hostname ];
+ });
+ }
+
+ } ); // api.post
+ } // user clicked set
+ } ); // app.confirm
+ },
+
+ takeSnapshot: function() {
+ // take a snapshot (i.e. 1 minute watch)
+ var args = this.args;
+ var state = config.state;
+ var watch_time = 60;
+ var watch_date = time_now() + watch_time;
+ var hostnames = this.hosts.map( function(host) { return host.hostname; } );
+
+ app.api.post( 'app/watch', { hostnames: hostnames, date: watch_date }, function(resp) {
+ // update local state and show message
+ if (!state.watches) state.watches = {};
+ app.showMessage('success', 'Your snapshot(s) will be taken within a minute, and appear on the Snapshots tab.');
+ hostnames.forEach( function(hostname) {
+ state.watches[ hostname ] = watch_date;
+ });
+ } ); // api.post
+ },
+
onThemeChange: function(theme) {
// user has changed theme, update graphs
if (this.graphs) {
diff --git a/htdocs/js/pages/Home.class.js b/htdocs/js/pages/Home.class.js
index dbe818e..2261da6 100644
--- a/htdocs/js/pages/Home.class.js
+++ b/htdocs/js/pages/Home.class.js
@@ -413,6 +413,7 @@ Class.subclass( Page.Base, "Page.Home", {
html += '
A watch is currently enabled on this server, and will be until ' + get_nice_date_time(state.watches[args.hostname], false, false) + ' (approximately ' + get_text_from_seconds(state.watches[args.hostname] - time_now(), false, true) + ' from now). Use the menu below to reset the watch, or disable it entirely.
';
+ watch_sel = 0;
+ }
+ else {
+ // watch is disabled
+ html += '
This server is not currently being watched. Use the menu below to optionally set a watch timer, which will generate snapshots every minute until the timer expires.
';
+ watch_sel = 3600;
+ }
+
+ html += '
' +
+ // get_form_table_spacer() +
+ get_form_table_row('Watch For:', '') +
+ get_form_table_caption("Select the duration for the server watch.") +
+ '
';
+
+ app.confirm( 'Watch Server', html, "Set Watch", function(result) {
+ app.clearError();
+
+ if (result) {
+ var watch_time = parseInt( $('#fe_watch_time').val() );
+ var watch_date = time_now() + watch_time;
+ Dialog.hide();
+
+ app.api.post( 'app/watch', { hostnames: [args.hostname], date: watch_date }, function(resp) {
+ // update local state and show message
+ if (!state.watches) state.watches = {};
+
+ if (watch_time) {
+ app.showMessage('success', "Server will be watched for " + get_text_from_seconds(watch_time, false, true) + ".");
+ state.watches[ args.hostname ] = watch_date;
+ }
+ else {
+ app.showMessage('success', "Server watch has been disabled.");
+ delete state.watches[ args.hostname ];
+ }
+
+ } ); // api.post
+ } // user clicked set
+ } ); // app.confirm
+ },
+
+ takeSnapshot: function() {
+ // take a snapshot (i.e. 1 minute watch)
+ var args = this.args;
+ var state = config.state;
+ var watch_time = 60;
+ var watch_date = time_now() + watch_time;
+
+ app.api.post( 'app/watch', { hostnames: [args.hostname], date: watch_date }, function(resp) {
+ // update local state and show message
+ if (!state.watches) state.watches = {};
+ app.showMessage('success', 'Your snapshot will be taken within a minute, and appear on the Snapshots tab.');
+ state.watches[ args.hostname ] = watch_date;
+ } ); // api.post
+ },
+
onThemeChange: function(theme) {
// user has changed theme, update graphs
if (this.graphs) {
diff --git a/htdocs/js/pages/Snapshot.class.js b/htdocs/js/pages/Snapshot.class.js
new file mode 100644
index 0000000..ba906b6
--- /dev/null
+++ b/htdocs/js/pages/Snapshot.class.js
@@ -0,0 +1,657 @@
+Class.subclass( Page.Base, "Page.Snapshot", {
+
+ default_sub: 'list',
+
+ onInit: function() {
+ // called once at page load
+ var html = '';
+ this.div.html( html );
+ },
+
+ onActivate: function(args) {
+ // page activation
+ if (!this.requireLogin(args)) return true;
+
+ if (!args) args = {};
+ if (args.id) args.sub = 'snapshot';
+ if (!args.sub) args.sub = this.default_sub;
+ this.args = args;
+
+ app.showTabBar(true);
+ this.showControls(false);
+
+ this.div.addClass('loading');
+ this['gosub_'+args.sub](args);
+
+ return true;
+ },
+
+ gosub_list: function(args) {
+ // show snapshot list
+ app.setWindowTitle( "Snapshot List" );
+
+ if (!args.offset) args.offset = 0;
+ if (!args.limit) args.limit = 25;
+ app.api.post( 'app/get_snapshots', copy_object(args), this.receive_snapshots.bind(this) );
+ },
+
+ receive_snapshots: function(resp) {
+ // receive page of snapshots from server, render it
+ var self = this;
+ var html = '';
+ this.div.removeClass('loading');
+
+ this.snapshots = [];
+ if (resp.rows) this.snapshots = resp.rows;
+
+ var cols = ['Hostname', 'Date/Time', 'Source', 'Alerts', 'Actions'];
+
+ html += '
';
+
+ html += '
';
+ html += 'Server Snapshot List';
+ // html += '';
+ html += '
'; // padding
+
+ this.div.html( html );
+ },
+
+ getNiceHostname: function(hostname, link, width) {
+ // get formatted hostname with icon, plus custom link
+ if (!width) width = 500;
+ if (!hostname) return '(None)';
+
+ var html = '