Merge pull request #61 from laimaretto/devel

Devel

README.md

@@ -13,7 +13,9 @@ The idea was born because of the need for a simple tool that could do pre-post c
 - [Usage](#usage)
   - [Configuration Options](#configuration-options)
   - [Templates](#templates)
+  - [Optional Features](#optional-features)
     - [Filtering Variables of Templates](#filtering-variables-of-templates)
+    - [Using plugins](#using-plugins)
 - [Results](#results)
   - [Tabulation of Logs - Assessment](#tabulation-of-logs---assessment)
   - [Comparision of Logs - Pre/Post Check](#comparision-of-logs---pre-and-post-check)
@@ -75,6 +77,7 @@ LogChecker can be configured through CLI as shown below. It is also possible to
 |`-ga` | Generate ATP document in `.docx` format, based on contents of json files from `taskAutom`. Default = no |
 |`-ic` | Adds new column (Idx Pre/Post) in changes detected table, when running comparision. Default = no  |
 |`-ug` | Using generic template. If `-ug=no`, logChecker only use the templates indicated in the `-tf` folder (and `-tf-post, if applicable). Default = yes |
+|`-up` | Additional plugins for manipulation of parsed information, creating new sheets. One plugin, use -up plugin1.py . For indicate a folder containing all the plugins: -up plugins/ . Default=''|
 |`-v` | Show version |
 
 ### Templates
@@ -89,7 +92,9 @@ There are situations in which a comparison of logs must be done when different v
 > When different TIMOS versions have been used, the variables of the several templates must be the same for the comparison to be succesful.
 
 <!---To find out a set of Templates that can be used, see [`here`](https://github.com/laimaretto/logTemplates)-->
-### Filtering Variables of Templates
+### Optional Features
+
+#### Filtering Variables of Templates
 
 Filtering of variables is possible. This can be done by simply adding the comments `#filterAction:` as `exclude` or `include-only`, and listing the variables in `#filterColumns:`, within the comment section of the templates at the very top.
 
@@ -98,13 +103,48 @@ When using `#filterAction: exclude`, all the variables listed under `#filterColu
 Similarly, if using `#filterAction: include-only`, the specified variables in `#filterColumns` are kept, while all the others are removed.
 
 
+[Go to Table of Contents](#table-of-contents)
+
+#### Using Plugins
+
+It is possible to use plugins to create new tabs in the Excel file. These plugins interact with the parsed data and should be customized as needed.
+
+If a plugin is used, specify it as `-up pluginName.py`.
+For multiple plugins, specify a folder, for example, using `-up pluginFolder/`. LogChecker will go through the folder's contents and use all `.py` files as plugins. 
+
+The plugin structure must follow:
+
+> pluginName.py
+```python
+def usePlugin(dict_parsed):
+    '''Plugin function that interacts with the dictionary containing the parsed data.
+
+    Args:
+        dict_parsed (dict): Dictionary with the parsed information.
+
+    Returns:
+        df_plg (dataFrame): Dataframe with the structure expected to be saved in new tab in Excel. In some cases, it is necessary to add the NAME or IP column, depending on the execution mode in -ri
+        valueKeys_plg (list): It is necessary to identify at least one column from the dataframe df_plg to be considered as valueKeys when performing the comparison task.
+        
+        Or use df_plg = None and valueKeys_plg = None to use a plugin without save a new sheet in Excel.
+
+    Notes:
+       - For example, to access the parsed data dataframe of a specific template, use: dict_parsed['sh_port.template']['dfResultDatos']
+    '''
+
+    return df_plg, valueKeys_plg
+```
+
+If `df_plg` (DataFrame) and `valueKeys_plg` (list) are returned, the plugin will save the information in a new sheet in Excel. If `return None, None` , the plugin will not save the information in the Excel.
+
+
 [Go to Table of Contents](#table-of-contents)
 
 ## Results
 
-Here are two examples of how to run logChecker.
+Here are two simpler examples of how to run logChecker.
 
-All the parameters are in [Configuration Options](#configuration-options).
+For more options, see the [available parameters](#configuration-options), as well as how to [filter template variable](#filtering-variables-of-templates) and how to [use plugins](#using-plugins). 
 
 
 ### Tabulation of logs - Assessment
@@ -126,6 +166,11 @@ Saving Excel
 
 Total running time: 0.45 seconds
 ```
+Below is the flowchart of the use case where [taskAutom](#https://github.com/laimaretto/taskAutom) is used to collect the data and logChecker is used to organize the collected information.
+
+|![Assessment with taskAutom and LogChecker](https://raw.githubusercontent.com/laimaretto/taskAutom/refs/heads/main/img/lc_assessment.png)|
+|:--:| 
+| *Use case: taskAutom and logChecker to make an assessment*|
 
 
 ### Comparision of logs - Pre and post check
@@ -150,4 +195,11 @@ Saving Excel
 Total running time: 0.46 seconds
 ```
 
+Below is the flowchart of the use case in which [taskAutom](#https://github.com/laimaretto/taskAutom) is used to collect the data and logChecker is used to compare the pre and post check.
+
+|![Pre-post check with taskAutom and LogChecker](https://raw.githubusercontent.com/laimaretto/taskAutom/refs/heads/main/img/lc_prepost.png)|
+|:--:| 
+| *Use case: taskAutom and logChecker to pre and post Check comparision*|
+
+
 [Go to Table of Contents](#table-of-contents)

changelog.md

@@ -1,3 +1,14 @@
+[4.5.9 - 2024-12-20]
+- Update `applyPlugin`: option to use `df_plg` and `valueKeys_plg` == None. In this case, plugin will not generate a new sheet in Excel.
+- Update `README.md`
+
+[4.5.8 - 2024-12-20]
+- Replacing `_platform` by `sys.platform`
+- New functionality: Use plugins to interact with parsed data.
+    - New paramater: `-up`/`--usePlugin`
+    - New functions: `verifyPlugin` and `applyPlugin`
+    - Update in `fncRun`
+
 [4.5.7 - 2024-12-10]
 - New parameter `-ug / --useGen` to enable/disable the use of generic template. By default=yes. Disable for the cases where we need to parse just the commands indicated in -tf folder.
 - Update `README.md`

setup.py

@@ -2,7 +2,7 @@
 
 setup(
     name='logChecker',
-    version='4.5.7',
+    version='4.5.9',
     description='A simple log analysis tool',
     long_description='A parsing tool to easily perform pre and post check comparisons after a maintenance window.',
     long_description_content_type='text/x-rst',

src/logChecker/__init__.py

@@ -1,2 +1,2 @@
-__version__ = "4.5.7"
+__version__ = "4.5.9"
 __author__ = 'Lucas Aimaretto, Beatriz Bonafe'

src/logChecker/logChecker.py

@@ -17,13 +17,15 @@
 import pandas as pd
 import glob
 import argparse
-from sys import platform as _platform
+import sys
 import json
 import re
 from ttp import ttp
 import os
 import io
 import time
+import importlib
+import traceback
 
 import docx
 from docx.enum.style import WD_STYLE_TYPE
@@ -367,17 +369,17 @@ def readLog(logFolder, formatJson):
 
 		ending = '*rx.txt'
 
-	if _platform == "linux" or _platform == "linux2" or _platform == "darwin":
+	if sys.platform == "linux" or sys.platform == "linux2" or sys.platform == "darwin":
 		# linux
 
 		listContent  = [f for f in glob.glob(logFolder  + ending)]
 
-	elif _platform == "win64" or _platform == "win32":
+	elif sys.platform == "win64" or sys.platform == "win32":
 		# Windows 64-bit
 
 		listContent  = [f.replace("\\", '/') for f in glob.glob(logFolder  + ending)]
 	else:
-		print(str(_platform) + ": not a valid platform. Quitting....")
+		print(sys.platform + ": not a valid platform. Quitting....")
 		quit()
 
 	d = {}
@@ -597,6 +599,77 @@ def mixAll(dTmpl, datosEquipo, routerId, cmdsLogs, datosCmdsLogs, tmpltName):
 	print(f'##### Logs from folder {logFolder} parsed #####')	
 	return datosEquipo
 
+def verifyPlugin(pluginFilename):
+	"""Verifies the plugin template
+
+	Args:
+		pluginFilename (str): Name of config template
+
+	Returns:
+		module: The module
+	"""
+
+	try:
+		if pluginFilename.split(".")[-1] == "py":
+			spec = importlib.util.spec_from_file_location("usePlugin",pluginFilename)
+			mod  = importlib.util.module_from_spec(spec)
+			sys.modules["usePlugin"] = mod
+			spec.loader.exec_module(mod)
+		else:
+			print("Verify extension of the file to be '.py'. Quitting...")
+			quit()
+	except Exception as e:
+		print(e)
+		print("----\nError importing plugin. Quitting ...")
+		quit()
+
+	return mod
+
+def applyPlugin(mod, plg, df_final, dTmplt):
+	'''Apply the plugin (from mod) to the data in the df_final. Updates the dTmplt and df_final dictionaries.
+	'''
+	try:
+		df_plg, valueKeys_plg = mod.usePlugin(df_final)
+		if df_plg is None and valueKeys_plg is None: #In this case, don't save new sheet.
+			pass
+		elif not (isinstance(df_plg, pd.DataFrame) and isinstance(valueKeys_plg, list)):
+			print("Plugin Error: df_plg must be a dataFrame and valueKeys_plg must be a list. Or both must be None, if there's no need to create a new sheet.")
+			quit()
+
+	except:
+		print(f'Plugin Error: Check the structure of your plugin {plg}')
+		for e in traceback.format_exc().splitlines()[-3:]:
+			print(e)
+		quit()
+
+	if (df_plg is not None) and (valueKeys_plg is not None):
+		#Keeping alphanumeric characters and '.', replacing other characters with '_'
+		plg_re = re.sub(r"[^\w.]", "_", plg)
+
+		df_final[plg_re] = {
+			'dfResultDatos' : df_plg,
+			'template'      : plg_re,
+			'command'       : plg,
+			'valueKeys'     : valueKeys_plg,
+			'parseStatus'   : 'ok',
+			'filterColumns' : df_plg.columns.tolist()
+		}
+
+		dTmplt[plg_re] = {
+			'templateColumns' : df_plg.columns.tolist(),
+			'commandKey'      : '',
+			'majorDown'       : ['down','dwn'],
+			'filterColumns'   : df_plg.columns.tolist(),
+			'valueKeys'       : valueKeys_plg,
+		}
+
+		print(f' {plg} - New sheet')
+
+	else:
+		print(f" {plg} - Does not create a new sheet")
+
+	return df_final, dTmplt
+
 def searchDiffAll(datosEquipoPre, datosEquipoPost, dTmplt, routerId, idxComp):
 	'''
 	Makes a new table, in which it brings the differences between two tables (post-pre)
@@ -1016,8 +1089,9 @@ def fncRun(dictParam):
 	genAtp             = dictParam['genAtp']
 	idxComp            = dictParam['idxComp']
 	useGen             = dictParam['useGen']
+	usePlugin          = dictParam['usePlugin']
 
-	if _platform == "win64" or _platform == "win32":
+	if sys.platform == "win64" or sys.platform == "win32":
 		templateFolder = templateFolder.replace('/', '\\')
 		if templateFolderPost != '':
 			templateFolderPost = templateFolderPost.replace('/','\\')
@@ -1028,6 +1102,12 @@ def fncRun(dictParam):
 		dLog   = readLog(preFolder, formatJson)
 
 		df_final    = parseResults(dTmplt, dLog, templateFolder, templateEngine, routerId, useGen, preFolder)
+		if len(usePlugin)>0:
+			print("##### Plugins: #####")
+			for plg in usePlugin:
+				mod = verifyPlugin(plg)
+				df_final, dTmplt = applyPlugin(mod, plg, df_final, dTmplt)
+
 		count_dif   = {}
 		searchMajor = {}
 
@@ -1063,6 +1143,13 @@ def fncRun(dictParam):
 		datosEquipoPre  = parseResults(dTmpltPre,  dLogPre,  templateFolder,     templateEngine, routerId, useGen, preFolder)
 		datosEquipoPost = parseResults(dTmpltPost, dLogPost, templateFolderPost, templateEngine, routerId, useGen, postFolder)
 
+		if len(usePlugin)>0:
+			print("##### Plugins: #####")
+			for plg in usePlugin:
+				mod = verifyPlugin(plg)
+				datosEquipoPre, dTmpltPre   = applyPlugin(mod, plg, datosEquipoPre, dTmpltPre)
+				datosEquipoPost, dTmpltPost = applyPlugin(mod, plg, datosEquipoPost, dTmpltPost)
+
 		count_dif       = searchDiffAll(datosEquipoPre, datosEquipoPost, dTmpltPre, routerId, idxComp)
 
 		searchMajor     = findMajor(count_dif, dTmpltPre, routerId, datosEquipoPre)
@@ -1093,10 +1180,18 @@ def main():
 	parser1.add_argument('-ga', '--genAtp',         type=str, default='no',  choices=['no','yes'], help='Generate ATP document in docx format, based on the contents of the json files from taskAutom. Default=no')
 	parser1.add_argument('-ic','--idxComp',         type=str, default='no',  choices=['yes','no'], help='Adds new column (Idx Pre/Post) in changes detected table with . Default=no')
 	parser1.add_argument('-ug','--useGen',          type=str, default='yes', choices=['yes','no'], help='Using generic template. If -ug=no, logChecker only use the templates indicated in the -tf and -tf-post folder. Default=yes')
-	parser1.add_argument('-v' ,'--version',         help='Version', action='version', version='(c) 2024 - Version: 4.5.7' )
+	parser1.add_argument('-up','--usePlugin',type=str, default='',help="Additional plugins for manipulation of parsed information, creating new sheets. One plugin, use -up plugin1.py . For indicate a folder containing all the plugins: -up plugins/ . Default='' ")
+	parser1.add_argument('-v' ,'--version',         help='Version', action='version', version='(c) 2024 - Version: 4.5.9' )
 
 	args = parser1.parse_args()
 
+	if args.usePlugin == '':
+		usePlugin = ''
+	elif args.usePlugin.endswith(('/', '\\')):
+		usePlugin = glob.glob(args.usePlugin + '*.py')
+	else:
+		usePlugin = args.usePlugin.split(',')
+
 	dictParam = dict(
 		preFolder          = args.preFolder,
 		postFolder         = args.postFolder,
@@ -1108,7 +1203,8 @@ def main():
 		routerId           = args.routerId,
 		genAtp             = True if args.genAtp == 'yes' else False,
 		idxComp            = True if args.idxComp == 'yes' else False,
-		useGen             = True if args.useGen == 'yes' else False, 
+		useGen             = True if args.useGen == 'yes' else False,
+		usePlugin          = usePlugin,
 	)
 
 	fncRun(dictParam)