LVM GEO Python Core (GISPython)

LVM GEO Python Core (GISPython) is an open source automation and scripting core developed by the LVM GEO team. Based on this core any developer can build simple and structured Python programming scripts with a rich functionality. The programming core allows management of all system's maintenance processes within a unified environment.

There are many automated maintenance operations necessary for every large geospatial information system (GIS). These operations are required for database and server maintenance, data validations and calculations, map preparing and caching, or data exchange with other systems. Within ERSI platform maintenance scripting is done by Python programming language and ArcPy library. The LVM GEO team has worked with ArcPy library for many years and has developed the LVM GEO Python Core complementing and enriching the Platform maintenance possibilities of the ESRI library that are not provided in the ArcPy standard:

• monitoring of automated scripts (for example Zabbix)
• script audit storage
• generation of automated e-mails (script progress status reports as well as automated data validation warning e-mails, etc.)
• data transfer using FTP and SFTP, data compressing and uncompressing
• SQL, PowerShell, Windows Shell file initiation and progress monitoring within an unified environment with ArcPy geoprocessing tools

The Core also includes tools that simplify usage of ArcPy and Python functions, significantly easing the development process:

• scripting of ArcGIS server administration
• scripting of file operations
• scripting of service caching
• unified script initiation from ArcGIS environment, Python Shell and from other tools
• etc.

The LVM GEO Python Core is already being used by several companies in Latvia, including JSC Latvia's State Forests for more than 200 automated processes every day. LVM GEO offers courses about LVM GEO Python to support development of an automation platform for companies and organizations.

GISPython package contains following modules:

• Main modules:
  • GISPythonModule
  • GISPythonTool
  • GISPythonToolBase
  • SysGISTools
  • SysTools_unittest
• Helper modules:
  • AGServerHelper
  • AGServerHelperNTLM
  • CachingHelper
  • FTPHelper
  • GDBHelper
  • GDPSynchroniserHelper
  • GDPSynchroniserHelper2
  • JsonParamsHelper
  • MailHelper
  • MyError
  • PublisherHelper
  • RarHelper
  • SFTPHelper
  • SimpleFileOps
  • SimpleFileOpsSafe
  • TimerHelper
  • xmlParamsHelper
  • ZipHelper

Dependencies

• ArcGIS 10.x /recommended with newest patches and service packs/ (GISPython is currently running on production systems based on ArcGIS 10.2.1, ArcGIS 10.3.1 and has been tested on ArcGIS 10.4)
• Python 2.7 (included in ArcGIS installation) (arcpy and numpy modules included)
• Additional python modules:
  • PyCrypto (manual installation)
  • NTLM: pip install python-ntlm (included in package setup process)
  • paramiko: pip install paramiko (included in package setup process)
  • patool: pip install patool (included in package setup process)
  • simpleJson: pip install simplejson (included in package setup process)

If pip isn’t installed, you can get it here!

Installation

GISPython is available on the Python Package Index, so you can get it via pip: pip install GISPython. Before installing it is recomended to upgrade pip to latest version using pip install --upgrade pip. To upgrade GISPython use pip install gispython --upgrade.

Configuration & basic usage

Before using GISPython modules in custom geoprocessing scripts, you need to set up your scripting environment with SetupDefaultEnvironment module which also includes template for user scripts.

SetupDefaultEnvironment module also includes basic parameters (variable paramsFileSource) for parameter file (e.g. SysGISParams.py) which is important, because GISPython relies of several parameters to be present to function successfully:

• OutDir - directory for storing script output log files OutDir = r'C:\GIS\Log\Outlog\'
• OutDirArh - directory for storing script output log file archive (all non active files) OutDirArh = r'C:\GIS\Log\Outlog\Archive\'
• ErrorLogDir - directory for storing script error log files ErrorLogDir = r'C:\GIS\Log\ErrorLog\' (Important! This directory can be monitored for non empty files. If this directory has a file that is non empty - this indicates that a script has failed)
• ErrorLogDirArh - directory for storing script error log files ErrorLogDirArh = r'C:\GIS\Log\ErrorLog\Archive'
• TmpFolder - Temp folder TmpFolder = r'C:\GIS\tmp'
• encodingPrimary - encoding of Windows shell encodingPrimary = 'cp775'
• encodingSecondary - encoding of Windows unicode language used encodingSecondary = 'cp1257'
• SetLogHistory - enable or disable Geoprocessing history logging SetLogHistory = False

It is recommended to define additional script parameters in SysGISParams.py file, to keep the main code clean. Our approach is to define all the parameters that define current system environment be kept in this one file. In case of moving environment (e.g. test system and production system) this one file has the specific connections and can be easily modified without changing the scripts.

Recommendations

Set up the variables at the beggining of the main function, to shorten the main code:

Tool = self.Tool
gp = Tool.gp
callGP = Tool.callGP
pj = os.path.join

Basic operations:

ArcPy function call:

gpCaller = self.Tool.callGP
slay = 'some_layer'
callGP('AddField_management', slay, 'Day_TXT', 'TEXT', '#', '#', 10)
callGP('AddField_management', slay, 'CAR', 'TEXT', '#', '#', 128)
callGP('AddField_management', slay, 'WorkID', 'DOUBLE', 12, 0)
callGP('AddField_management', slay, 'REC_DATE_FROM', 'DATE')

Tool message output:

Tool = self.Tool
self.Tool.AddMessage(u'This is a message')
self.Tool.AddWarning(u'This is a warning')
self.Tool.AddError(u'This is an error')

Documentation

Documentation includes information about all GISPython modules & examples of use. Documentation can be found on: http://gispython.readthedocs.io.

Examples

Main modules

GISPythonModule

Main module, which contains frame for all GISPython package modules. Module allows the code unification, and ensures the code execution from both the ArcGIS Desktop Python console and the Command Prompt.

Example:

Executes the tool if it's to be executed within an another Python tool

from  GISPython import TimerHelper
import OnlineCompress
with TimerHelper.TimedSubprocess(Tool, u'Compress DB'): # Adds a message to the tool output
    with TimerHelper.TimedSubprocess(Tool, u'disconnect users from DB', 2): # Adds a message to the tool output
        self.Tool.RunSQL('KillUsers', Pr.u_sde, Pr.p_sde) # Runs custom SQL script
    OnlineCompress.MainModule(None, False).runInsideJob(Tool) # Runs SDE Compression procedure from OnlineCompress module (custom geoprocessing module)

GISPythonTool

Module defines abstract classes for the ESRI Toolbox tool definition and contains functions which helps to create an ArcGIS Toolbox, validates the tool's parameter values and controls a behavior of the tool's dialog.

Example:

Define parameters in ESRI Python toolbox

from  GISPython import GISPythonTool

class ToolAtributeValidator(GISPythonTool.GISPythonTool):
    def __init__(self):
        """Define tool (tool name is the class name)"""
        self.label = u"Tool for attribute data validation (test mode)"
        self.description = u"Tool for attribute data validation (test mode)"
        self.category = 'GEO Maintenance'

    def getParameterInfo(self):
        """Define the parameters"""
        param_0 = arcpy.Parameter(
            displayName=r'Key:',
            name=u"key",
            datatype=u"String",
            parameterType=u"Required",
            direction=u"Input")

        ret_val = [param_0]
        return ret_val

    def execute(self, parameters, messages):
        """Tool execution"""
        AtributeValidator.MainModule(parameters[0].valueAsText).DoJob()
        return

SysGISTools

Base module which contains GISPython scripting framework, logging, error processing and automation mechanisms for different operations, and module and interface base classes.

Examples:

Run a script from Shell with parameters

Tool = self.Tool
# Executes your custom process script (mainly maintenance scripts) within runShell function,
# which implements _runProcess function (message output in terminal window).
# Function executes script seperately from main process (Detached=True) and indicates, 
# that the errors doesn't have to be logged (noErr=True).
Tool.runShell('SomeProcess.py', Detached = True, noErr = True)
time.sleep(10) # after 10 seconds launch another runShell process

Executes a custom SQL script file (only Oracle sqlplus supported)

from  GISPython import TimerHelper
Tool = self.Tool
# Executes process from SQL file
with TimerHelper.TimedSubprocess(self.Tool, u'datu atlasi no nogabaliem'): # Adds a message to the tool output
    Tool.RunSQL('LoadSomeDataFromTable') # Runs SQL file within RunSQL function, which implements GetSQL function (gets the SQL file location)

Duplicates path seperator symbol '' for external execution compatibility

from  GISPython import SimpleFileOps
from  GISPython import TimerHelper
Tool = self.Tool
#...
with TimerHelper.TimedSubprocess(Tool, u'prepare environment'): # Adds a message to the tool output
    DirHelper = SimpleFileOps.SimpleFileOps(Tool) # Set variable for SimpleFileOps module functions
    tmpFolder = os.path.join(Pr.TmpFolder, "Soil") # Set tmp folder path
    #...
    Tool.AddMessage(u'\n        ...delete previous tmp data') # Add tool output message
    DirHelper.CheckCreateDir(Tool.CorrectStr(tmpFolder)) # Check/create tmp directory (with modified path seperators)
    DirHelper.ClearDir(Tool.CorrectStr(tmpFolder)) # Clear tmp directory (with modified path seperators)

Helper modules

MailHelper

Module for e-mail operations. Module contains functions for typical SMTP operations, and parameter processing from user parameter file.

Examples:

Send e-mail using parameters from parameter file

from  GISPython import MailHelper
MailHelper.GISPythonMailHelper(self.Pr, ['***@mail.com'], 'Subject', 'e-mail content')

This script depends on following parameters which needs to be configured in SysGISParams.py file:

• Mailserver - mail server name Mailserver = 'mail.server.com'
• MailserverPort - mail server port number MailserverPort = 587
• MailserverUseTLS - use TLS in mail server? MailserverUseTLS = True
• MailserverUseSSL - use SSL in mail server? MailserverUseSSL = False
• MailserverUser - user name MailserverUser = 'UserName
• MailserverPWD - user password MailserverPWD = r'userPassword'
• MailFromAdress - e-mail adress from which to send the e-mail MailFromAdress = '[email protected]'

Send e-mail

from  GISPython import MailHelper
mailSender = MailHelper.MailHelper('[email protected]', ['***@mail.com'], 'Subject', u'e-mail content') # Set up the e-mail for sending
mailSender.SendMessage('smtp.server.com', 587, 'username', 'password', useTLS=True) # Send e-mail

SimpleFileOps

File and filesystem operations module. Module contains functions for typical file and filesystem operations, and locking control and processing. This module uses windows PowerShell to address file locking situations. For module with the same functionality without PowerShell script usage use SimpleFileOpsSafe.

Examples:

Check for the directory, and clear its contents

from  GISPython import SimpleFileOps
FO = SimpleFileOps.SimpleFileOps(self.Tool)
workDir = pj(Pr.TmpFolder, 'WorkingDirectory') # Set the working directory
FO.CheckCreateDir(workDir) # Check if directory exists, if not, create the directory
FO.ClearDir(workDir) # Clear directory contents

Find the newest file in the directory and create a backup

from  GISPython import SimpleFileOps
FO = SimpleFileOps.SimpleFileOps(self.Tool)
workDir = pj(Pr.TmpFolder, 'WorkingDirectory') # Set the working directory
backupDir = pj(Pr.TmpFolder, 'BackupDirectory') # Set the working directory
newFile = FO.FindNewestFile(workDir, '*') # Find newest file with any extension
FO.BackupOneFile(newFile, backupDir)

TimerHelper

Timing module. Module contains functions for countdown procedures in a code block.

Example:

Add a message to the tool output

from  GISPython import TimerHelper
with TimerHelper.TimedSubprocess(self.Tool, u'some process'):
    self.Tool.AddMessage(u'some action')

ZipHelper

Zip file operations module. Module contains functions for common Zip file operations.

Examples:

Archiving procedure

from  GISPython import ZipHelper
ZH = ZipHelper.ZipHelper()
workDir = 'c:\\tmp\someDirectory' # Directory to archive
zipFile = 'c:\\tmp\fileName' + self.Tool.MyNowFileSafe() + '.zip' # New zip file with formatted date (simple example)
zipFile = 'c:\\tmp\fileName{0}.zip'.format(self.Tool.MyNowFileSafe()) # New zip file with formatted date (good example)
ZH.CompressDir(workDir, zipFile)

Extraction procedure

from  GISPython import ZipHelper
ZH = ZipHelper.ZipHelper()
workDir = 'c:\\tmp\someDirectory' # Directory in which to extract the Zip file
zipFile = 'c:\\tmp\fileName{0}.zip'.format(self.Tool.MyNowFileSafe()) # Zip file with formatted date
ZH.ExtractZipFile(zipFile, workDir)

Contributing

We encourage developers to use LVM GEO Python Core code and also contribute to the project. LVM GEO team is open for cooperation in developing new solutions, provides LVM GEO Python courses and offers to implement a fully functional automation platform for companies and organizations for their business needs.

Licensing

GPL-3.0+

Copyright