Version 1.0.2

- New feature added: Snapshots - New feature added: Server Watches (and group watches) - New feature added: CPU Core detail table - API Workflow change: submit API can now trigger a snapshot request - Locked down some APIs that weren't validating session cookies - Now chopping lists after 10K entries ('logs/activity', 'logs/snapshots') - Default config: Added `list_row_max`, removed `raw_file_paths` - Many UI tweaks - Server pie graphs now have tooltip details. - Added snapshot URLs to alert email templates. - Fixed network connection counter to only count ESTABLISHED conns - Updated dependencies. - Bumped pixl-boot to v2.0.0 for systemd / systemctl integration.
jhuckaby · Apr 22, 2019 · c8e8ebd · c8e8ebd
1 parent 7f1ae68
commit c8e8ebd
Show file tree

Hide file tree

Showing 20 changed files with 1,504 additions and 193 deletions.
diff --git a/README.md b/README.md
@@ -1,52 +1,10 @@
-# Overview
-
-![](https://pixlcore.com/software/performa/screenshots/light-dark.png)
-
-**Performa** is a multi-server monitoring system with a web based front-end UI.  It can monitor CPU, memory, disk, network, and of course your own custom metrics.  Alerts can be configured to trigger on any expression, and send e-mails or fire web hooks.  Timeline data can be stored on local disk or in Amazon S3.
-
-## Features at a Glance
-
-- Easy to install, configure and run
-- Monitor any number of servers
-- New servers are added to the system automatically
-- Assign servers to groups manually or automatically
-- Supports ephemeral servers (serverless, autoscale, etc.)
-- Metrics are collected every minute
-- Multiple graph scales: hourly, daily, monthly, yearly
-- Real-time views with auto-refreshing graphs
-- View graphs for individual servers or entire groups
-- Add custom commands for gathering your own metrics
-- Alerts with custom trigger expressions
-- Alert e-mails and web hooks for notification
-- Alert snooze feature to silence notifications
-- One click snapshot-to-URL-to-clipboard for graphs
-- Graph data can be kept indefinitely or auto-expired
-- Light and dark themes for the UI
-
-## Screenshots
-
-<details><summary>See Screenshots</summary>
-
-![](https://pixlcore.com/software/performa/screenshots/overview.png)
-
-![](https://pixlcore.com/software/performa/screenshots/group-detail.png)
-
-![](https://pixlcore.com/software/performa/screenshots/group-detail-2.png)
-
-![](https://pixlcore.com/software/performa/screenshots/server-detail-light.png)
-
-![](https://pixlcore.com/software/performa/screenshots/monitor-list.png)
-
-![](https://pixlcore.com/software/performa/screenshots/activity-log.png)
-
-![](https://pixlcore.com/software/performa/screenshots/edit-command.png)
-
-</details>
-
-## Table of Contents
+<details><summary>Table of Contents</summary>
 
 <!-- toc -->
-* [Glossary](#glossary)
+- [Overview](#overview)
+	* [Features at a Glance](#features-at-a-glance)
+	* [Screenshots](#screenshots)
+	* [Glossary](#glossary)
 - [Installation](#installation)
 - [Setup](#setup)
 - [Configuration](#configuration)
@@ -86,6 +44,7 @@
 		+ [Built-in Alerts](#built-in-alerts)
 		+ [Alert Web Hooks](#alert-web-hooks)
 	* [Commands](#commands)
+	* [Snapshots](#snapshots)
 - [Command Line](#command-line)
 	* [Starting and Stopping](#starting-and-stopping)
 	* [Storage Maintenance](#storage-maintenance)
@@ -99,6 +58,56 @@
 	* [Starting in Debug Mode](#starting-in-debug-mode)
 - [License](#license)
 
+</details>
+
+# Overview
+
+![](https://pixlcore.com/software/performa/screenshots/light-dark.png)
+
+**Performa** is a multi-server monitoring system with a web based front-end UI.  It can monitor CPU, memory, disk, network, and of course your own custom metrics.  Alerts can be configured to trigger on any expression, and send e-mails or fire web hooks.  Timeline data can be stored on local disk or in Amazon S3.
+
+## Features at a Glance
+
+- Easy to install, configure and run
+- Monitor any number of servers
+- New servers are added to the system automatically
+- Assign servers to groups manually or automatically
+- Supports ephemeral servers (serverless, autoscale, etc.)
+- Metrics are collected every minute
+- Multiple graph scales: hourly, daily, monthly, yearly
+- Real-time views with auto-refreshing graphs
+- View graphs for individual servers or entire groups
+- Add custom commands for graphing your own metrics
+- Alerts with custom trigger expressions
+- Alert e-mails and web hooks for notification
+- Alert snooze feature to silence notifications
+- Snapshot feature provides extra server details
+- One click snapshot-to-URL-to-clipboard for graphs
+- Graph data can be kept indefinitely or auto-expired
+- Light and dark themes for the UI
+
+## Screenshots
+
+<details><summary>See Screenshots</summary>
+
+![](https://pixlcore.com/software/performa/screenshots/overview.png)
+
+![](https://pixlcore.com/software/performa/screenshots/group-detail.png)
+
+![](https://pixlcore.com/software/performa/screenshots/group-detail-2.png)
+
+![](https://pixlcore.com/software/performa/screenshots/server-detail-light.png)
+
+![](https://pixlcore.com/software/performa/screenshots/snapshot-view.png)
+
+![](https://pixlcore.com/software/performa/screenshots/monitor-list.png)
+
+![](https://pixlcore.com/software/performa/screenshots/activity-log.png)
+
+![](https://pixlcore.com/software/performa/screenshots/edit-command.png)
+
+</details>
+
 ## Glossary
 
 A quick introduction to some common terms used in Performa:
@@ -112,6 +121,7 @@ A quick introduction to some common terms used in Performa:
 | **API Key** | A special key that can be used by external apps to send API requests into Performa. |
 | **User** | A human user account, which has a username and a password.  Passwords are salted and hashed with [bcrypt](https://en.wikipedia.org/wiki/Bcrypt). |
 | **Satellite** | Our headless companion product, which silently collects metrics on your servers and sends them to the master server.  See [Performa Satellite](#performa-satellite) below. |
+| **Snapshot** | A snapshot is a detailed record of everything happening on a server, including all processes and network sockets.  Snapshots are taken when alerts trigger, and when being watched.  See [Snapshots](#snapshots) below. |
 
 # Installation
 
@@ -477,7 +487,7 @@ You can include any property from the main `conf/config.json` file by using the
 
 # Performa Satellite
 
-Performa Satellite is our headless companion product, which silently collects metrics on your servers and sends them to the Performa master server.  It has no dependencies and ships as a precompiled binary (for Linux and macOS), so it will be compatible with a wide range of systems.  It does not run as a daemon, but instead launches via [cron](https://en.wikipedia.org/wiki/Cron) every minute, then exits.  It uses about 25 MB of RAM while it is active.
+Performa Satellite is our headless companion product, which silently collects metrics on your servers and sends them to the Performa master server.  It has no dependencies and ships as a precompiled binary (for Linux and macOS), so it will be compatible with a wide range of systems.  It does not run as a daemon, but instead launches via [cron](https://en.wikipedia.org/wiki/Cron) every minute, then exits.  It uses about 25 MB of RAM while it is active (usually only a few seconds per minute).
 
 For more information about Performa Satellite, including installation and configuration instructions, please see the [Performa Satellite Github Repo](https://github.com/jhuckaby/performa-satellite).
 
@@ -714,14 +724,27 @@ Each command has the following properties:
 | **Title** | A display title (label) for the alert, shown in the UI. |
 | **Enabled** | A checkbox denoting whether the command is enabled (will be executed) or disabled (skipped). |
 | **Groups** | A list of which server groups the command should be executed on (or you can select "all" groups). |
-| **Executable** | The executable command to run (e.g. `/bin/sh` or `/usr/bin/python`). |
+| **Executable** | The executable command to run (e.g. `/bin/sh`, `/usr/bin/python` or other). |
 | **Script** | The script source code to pipe to the command (i.e. shell commands or other). |
 | **Format** | If your command outputs JSON or XML, you can have this parsed for easier integration with monitors / alerts. |
-| **User ID** | Optionally run your command as a different user (i.e. for security). |
+| **User ID** | Optionally run your command as a different user on the server (i.e. for security purposes). |
 | **Notes** | A notes text field is provided for your own internal use. |
 
 If your command outputs raw text, you can use a regular expression to match the specific metric value (this is configured per each monitor that refers to the command).  Or, if your command happens to output JSON or XML, then Performa can parse it, and provide more structured access in the [Data Source](#data-sources) system.
 
+## Snapshots
+
+A snapshot is a detailed report of everything happening on a server, including all the information we collect every minute (i.e. CPU, memory stats), but also:
+
+- A detailed list of all processes running on the server, including PIDs, commands, CPU and memory usage.
+- A detailed list of all network connections, their source and destination IP addresses and ports, and each connection state.
+- A list of all socket listeners, including which protocol, interface and port number.
+- A list of all filesystem mounts and their usage and free space.
+
+Performa automatically takes a snapshot of a server whenever any alert is triggered.  This allows you to come back later to see exactly what was happening at the time of the alert (i.e. which processes and connections were open).  You can access snapshots from the alert e-mail notification, as well as the "Snapshots" tab in the UI.
+
+In addition to automatic snapshots, you can "watch" any server or group for any amount of time.  Setting a "watch" on a server or group means that it will generate a snapshot every minute for the duration of the watch timer.  You can set watches by clicking the "Watch Server..." or "Watch Group..." buttons in the UI.
+
 # Command Line
 
 Here are all the Performa services available to you on the command line.  Most of these are accessed via the following shell script:

diff --git a/htdocs/css/style.css b/htdocs/css/style.css
@@ -19,6 +19,10 @@ div.container {
 	/* color: var(--header-text-color); */
 }
 
+#d_footer {
+	margin: 12px 0px 17px 0px;
+}
+
 /* Menus */
 
 .tab_widget {
@@ -244,6 +248,10 @@ fieldset.dt_fs > legend {
 
 /* Table Pagination Spacing Fix */
 
+fieldset div.pagination {
+	margin-top: 0;
+}
+
 div.pagination > table tr td {
 	white-space: nowrap;
 }
@@ -348,6 +356,10 @@ fieldset.inline_error {
 	font-size: 12px;
 }
 
+#fe_ctrl_server, #fe_ctrl_group {
+	max-width: 200px;
+}
+
 .chart_size_icon {
 	font-size: 18px;
 	cursor: pointer;
@@ -558,6 +570,7 @@ body.dark .apexcharts-xaxistooltip {
 	top: 0;
 	left: 0;
 	z-index: 2;
+	cursor: default;
 }
 
 .server_pie_container, .server_pie_graph, .server_pie_overlay {
@@ -591,6 +604,17 @@ body.dark .apexcharts-xaxistooltip {
 	text-overflow: ellipsis;
 }
 
+.server_pie_widget {
+	position: absolute;
+	top: 0;
+	left: 0;
+	z-index: 2;
+	width: 180px;
+	height: 20px;
+	line-height: 20px;
+	text-align: right;
+}
+
 /* Group Tab */
 
 div.priv_group_admin, div.priv_group_other {
@@ -624,3 +648,37 @@ fieldset.overview_group > legend {
 	font-size: 12px; 
 	color: var(--header-text-color);
 }
+
+/* Snapshot Tab */
+
+.inline_table_scrollarea {
+	position: relative;
+	max-height: 75vh;
+	overflow-x: hidden;
+	overflow-y: scroll;
+}
+
+table.fieldset_table tr th.st_col_header {
+	white-space: nowrap;
+	font-weight: bold;
+	cursor: pointer;
+}
+table.fieldset_table tr th.st_col_header:hover {
+	color: var(--highlight-color);
+	text-decoration: underline;
+}
+table.fieldset_table tr th.st_col_header.active {
+	color: var(--highlight-color);
+}
+
+.percent_bar_container {
+	position: relative;
+	background-color: var(--border-color);
+	height: 15px;
+}
+
+.percent_bar_inner {
+	box-sizing: border-box;
+	border-right: 1px solid var(--header-text-color);
+	height: 15px;
+}
diff --git a/htdocs/index-dev.html b/htdocs/index-dev.html
@@ -32,14 +32,15 @@
 		</div>
 	</div>
 	<div class="container">
-				
+
 		<div class="master_content_container">
 			<!-- Main Content Area -->
 			<div class="tab_bar" style="display:none">
 				<div id="tab_Login" class="tab inactive" style="display:none"><span class="content"></span></div>
 				<div id="tab_Home" class="tab inactive"><span class="content"><i class="mdi mdi-chart-areaspline mdi-lg">&nbsp;&nbsp;</i>Overview</span></div>
-				<div id="tab_Group" class="tab inactive"><span class="content"><i class="mdi mdi-server-network mdi-lg">&nbsp;&nbsp;</i>Group Detail</span></div>
-				<div id="tab_Server" class="tab inactive"><span class="content"><i class="mdi mdi-desktop-tower mdi-lg">&nbsp;</i>Server Detail</span></div>
+				<div id="tab_Group" class="tab inactive"><span class="content"><i class="mdi mdi-server-network mdi-lg">&nbsp;&nbsp;</i>Groups</span></div>
+				<div id="tab_Server" class="tab inactive"><span class="content"><i class="mdi mdi-desktop-tower mdi-lg">&nbsp;</i>Servers</span></div>
+				<div id="tab_Snapshot" class="tab inactive"><span class="content"><i class="mdi mdi-history mdi-lg">&nbsp;&nbsp;</i>Snapshots</span></div>
 				<div id="tab_MyAccount" class="tab inactive"><span class="content"><i class="fa fa-user fa-lg2">&nbsp;&nbsp;</i>My Account</span></div>
 				<div id="tab_Admin" class="tab inactive" style="display:none"><span class="content"><i class="mdi mdi-lock mdi-lg">&nbsp;&nbsp;</i>Admin</span></div>
 
@@ -184,6 +185,7 @@
 				<div id="page_Home" style="display:none"></div>
 				<div id="page_Group" style="display:none"></div>
 				<div id="page_Server" style="display:none"></div>
+				<div id="page_Snapshot" style="display:none"></div>
 				<div id="page_MyAccount" style="display:none"></div>
 				<div id="page_Admin" style="display:none"></div>
 				<div id="page_Login" style="display:none"></div>
@@ -223,6 +225,7 @@
 	<script src="js/pages/Home.class.js"></script>
 	<script src="js/pages/Group.class.js"></script>
 	<script src="js/pages/Server.class.js"></script>
+	<script src="js/pages/Snapshot.class.js"></script>
 	<script src="js/pages/Login.class.js"></script>
 	<script src="js/pages/MyAccount.class.js"></script>
 	<script src="js/pages/Admin.class.js"></script>

diff --git a/htdocs/js/app.js b/htdocs/js/app.js
@@ -43,6 +43,7 @@ app.extend({
 			{ ID: 'Home' },
 			{ ID: 'Group' },
 			{ ID: 'Server' },
+			{ ID: 'Snapshot' },
 			{ ID: 'Login' },
 			{ ID: 'MyAccount' },
 			{ ID: 'Admin' }
@@ -200,8 +201,9 @@ app.extend({
 
 		var menu_html = '';
 		menu_html += '<option value="" disabled>Jump to Server</option>';
-		menu_html += '<option value="" disabled></option>';
-		menu_html += this.getRecentServerMenuOptionsHTML();
+		var temp_html = this.getRecentServerMenuOptionsHTML();
+		if (temp_html.match(/<optgroup/)) menu_html += '<option value="" disabled></option>';
+		menu_html += temp_html;
 		$('#fe_jump_to_server').empty().append( menu_html ).val('');
 
 		// jump to group menu
@@ -657,6 +659,13 @@ window.Debug = {
 	}
 };
 
+function short_float_str(num) {
+	// force a float (add suffix if int)
+	num = '' + short_float(num);
+	if (num.match(/^\-?\d+$/)) num += ".0";
+	return num;
+};
+
 // Debounce Function Generator
 // Fires once immediately, then never again until freq ms
 function debounce(func, freq) {