Releases: google/testrun
Releases · google/testrun
v2.1.1
This release improves the user experience when running Testrun for pilot assessments whilst also introducing a number of smaller improvements and bug fixes.
New features
- Starting status is now shown whilst Testrun is starting
- When an unexpected device is connected this is logged in the command line
Improvements
- Language used for pilot assessments is now more appropriate
- TLS server tests scans for TLS services on every TCP port
- NTPv3 is now allowed provided that NTPv4 is also used
- The device profile is updated on every ZIP export (in case device information has changed)
- Documentation now states which tests are affected by single interface mode
Bug fixes
- Risk profile name is now validated correctly
- Updated Cryptogaphy and Jinja2 libraries
v2.1
This release provides a number of bug fixes and feature improvements aiming to increase the stability of Testrun and improving the user experience.
New features
- Provided documentation in additional configuration options
- Allow firmware and target MAC address to be passed in the command line
- New communication.network.type test implemented
- Extended TCP port scan to cover all ports
- New module report for the TLS module
- Outbound connections are listed in the TLS module report
Improvements
- Upgraded the docker base image to Ubuntu 22
- Most modules use Python 3.10 slim docker image, decreasing installation times
- Greater coverage in unit testing
- Updated multiple Python dependencies
- Added automated actions to verify that Testrun starts on different operating systems (Ubuntu 20, 22, 24)
- Remove excess logging from MQTT client
- Don't start the UI when running in net only mode
- Only run tests if they are included in the chosen test pack
- Added the BACnet port to the services open ports table
Bug fixes
- Locked some sub dependencies
- Corrected permissions of the resources folder
- Check for certificate properties when uploading a CA certificate
- Added missing tests to the documentation
- Better exception handling if the USB adapter does not support port stats
- Added exception handling around missing leases during tests
- Better exception handling in the connection test module
- More accurate result if device does not respond to BACnet discovery
- Fixed results for TLS server if public certificate could not be resolved
- Allow reports to be downloaded from external hosts
- Adapter changes between test attempts are now handled correctly
v2.0.1
v2.0
Version 2.0 of Testrun provides new functionality to support new users hoping to submit a device for a pilot assessment. In addition, new features support a number of use cases for both Google and non-Google users.
New features
- Device create and edit modal has been updated and requires further device information
- Device testing type (Device qualification or pilot assessment) has been added to device object
- DHCP disconnect and DHCP disconnect IP change tests have been implemented
- Test module reports now provide further information
- Device testing type has been added to the reports list page
- Optional recommendations have been added to the pilot assessment device report
- TLS v1.0 test has been implemented
Improvements
- Documentation has been reviewed and updated
- UI is now built during package, not installation
- Informational test result color has changed
- PDF report design has been improved
- Unit testing has been enabled in GitHub actions
- Installation time is reduced by using slim Python images for test modules
Bug fixes
- DHCP NAK messages no longer trigger a Non Compliant result in the DHCP snooping test
- Steps to resolve are no longer present on tests with Error or Informational test result
- Single IP test will no longer detect the testing adapters
Known issues
- We currently have an accessibility issue in the create/edit device modal where some text does not display correctly when using accessibility tools.
- If you are using a python version lower than 3.9, you will experience the following error:
Traceback (most recent call last):
File "framework/python/src/core/test_runner.py", line 25, in <module>
from testrun import Testrun
File "/usr/local/testrun/framework/python/src/core/testrun.py", line 30, in <module>
from session import TestrunSession
File "/usr/local/testrun/framework/python/src/core/session.py", line 80, in <module>
class TestrunSession():
File "/usr/local/testrun/framework/python/src/core/session.py", line 638, in TestrunSession
def _remove_invalid_questions(self, questions: list[dict]) -> list[dict]:
TypeError: 'type' object is not subscriptable
To resolve, you will need to upgrade to 3.9 by doing the following:
-
Upgrade python:
apt install python3.9 -
Set python3.9 as default python installation by running the following:
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 1
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 2
sudo update-alternatives --config python3
- Confirm python3.9 is now the default:
python3 --version
Python 3.9.5
- Install 3.9 venv:
sudo apt install python3.9-venv
v1.4
During Q3 of 2024, Testrun will see a number of minor releases containing just a handful of new features, improvements and bug fixes. Whilst it is not required that you update your installation of Testrun, it is recommended so that you have a better experience with the application.
New features
- Addition of a web-sockets server for real-time communication between front and backend
- Receive a notification when an adapter is connected or disconnected
- Implemented automated unit testing to improve stability
- Addition of the informational dns.mdns test
- Added GRPC server to allow host network control
- TLS 1.3 server and client informational tests have been added
Improvements
- Duplicate DNS queries are now combined in the PDF report
- Improved security of the package workflow
- Disabled the UI during network only mode
- Disabled the monitoring period when in network only mode
- Tests always report Informational if they are informational tests
- All pylint issues have been resolved
- Google CLA, pylint and ESLint checks are now enforced
- Added a no profile option in the Testrun zip download
- Backend not responding error message has been rephrased
Bug fixes
- Individual test module configuration changes are no longer ignored
- DNS queries from the DNS server to 8.8.8.8 are now hidden
- Draft risk profiles can now expire after 1 year
v1.4-alpha
NOTE: This is a pre-release. It is not ready for official use. Feel free to give it a go and let us know if you find any bugs ahead of the release. For official use, please always download the latest version.
New features
- Addition of a web-sockets server for real-time communication between front and backend
- Implemented automated unit testing to improve stability
- Addition of the informational dns.mdns test
- Addition of the TLS client and server v1.3 tests
- TLS 1.3 server and client informational tests have been added
- Added risk level to each individual question in the PDF report
- Check for internet connectivity during a Testrun (wifi icon in top right)
Improvements
- Duplicate DNS queries are now combined in the PDF report
- Improved security of the package workflow
- Disabled the UI during network only mode
- Disabled the monitoring period when in network only mode
- Tests always report Informational if they are informational tests
- UI is now pre-built to speed up installation times
- All pylint issues have been resolved
- Google CLA, pylint and ESLint checks are now enforced
- Added a no profile option in the Testrun zip download
- Backend not responding error message has been rephrased
- Added automated unit testing
- More complete documentation
Bug fixes
- Individual test module configuration changes are no longer ignored
- DNS queries from the DNS server to 8.8.8.8 are now hidden
- Draft risk profiles can now expire after 1 year
v1.3.1
v1.3
NOTE: Please ensure ethool is available when using Testrun. Install using sudo apt-get install ethtool
Testrun v1.3 is one of our biggest updates yet! New additions to Testrun include an automated device risk assessment, certificate management and 3 new connection tests have been automated. See below for a detailed list of all changes:
New features
- Ability to upload local CA certificates into the user interface
- Answer questions to form a risk assessment for your device and attach to a ZIP export
- 3x new connection tests (port_link, port_duplex, port_speed)
- Mockoon and Postman files for the API have been shared
- Test modules are now loaded into the user interface dynamically
- The number of successful tests out of the total number executed is now in the PDF report
- Feature Not Detected test result has been added
- Informational test result has been added
Improvements
- Version numbers are no longer repeated in multiple locations in the code
- Test modules can be changed for one Testrun when quick launching a device test
- Added more exception handling in API endpoints
- Testrun version is now reported in the JSON report
- Updated documentation
- Move ZIP files to a temp folder
Bug fixes
- The device under test now has a working internet connection
- Testrun now stops if you disconnect the device during the monitor period
- ARP responses from link local addresses are now accepted
- QUIC UDP protocol is now allowed in the TLS 1.2 client test
- Fixed an issue where a second Testrun would be cancelled automatically
- Prevent creating a device where a folder for that device already exists
v1.2.2
Testrun v1.2.2 is a small release which adds some minor functionality and resolves some bugs:
Improvements
- Updated documentation on test statuses
- Ability to shutdown Testrun from the user interface
- Modify quick start icon on device tile
- Package script improved
- Shortened maximum device manufacturer and model name to 28 characters
- Sets default report limit to infinite
- Added steps to resolve to the PDF report
- PDF test report edge cases considered
Bug fixes
- Better error handling of test module results
- Testrun now stops if device is disconnected during monitor period
- Fix issue with TLS client test
- Increase API timeout when stopping Testrun
- Fixed issue when failed to fetch the latest version number
- Fixed issue with module reports duplicating on second test attempt
- Rendering issue with error test result resolved