Merge pull request #188 from nautobot/prep-4.0.0

Prep 4.0.0

.github/workflows/ci.yml

@@ -106,7 +106,7 @@ jobs:
       fail-fast: true
       matrix:
         python-version: ["3.11"]
-        nautobot-version: ["2.2.2"]
+        nautobot-version: ["2.2.3"]
     env:
       INVOKE_NAUTOBOT_DEVICE_ONBOARDING_PYTHON_VER: "${{ matrix.python-version }}"
       INVOKE_NAUTOBOT_DEVICE_ONBOARDING_NAUTOBOT_VER: "${{ matrix.nautobot-version }}"
@@ -152,7 +152,7 @@ jobs:
         include:
           - python-version: "3.11"
             db-backend: "postgresql"
-            nautobot-version: "2.2.2"
+            nautobot-version: "2.2.3"
           # - python-version: "3.11"
           #   db-backend: "mysql"
           #   nautobot-version: "stable"

README.md

@@ -3,7 +3,7 @@
 <p align="center">
   <img src="https://raw.githubusercontent.com/nautobot/nautobot-app-device-onboarding/develop/docs/images/icon-DeviceOnboarding.png" class="logo" height="200px">
   <br>
-  <a href="https://github.com/nautobot/nautobot-app-device-onboarding/actions"><img src="https://github.com/nautobot/nautobot-app-device-onboarding/actions/workflows/ci.yml/badge.svg?branch=main"></a>
+  <a href="https://github.com/nautobot/nautobot-app-device-onboarding/actions"><img src="https://github.com/nautobot/nautobot-app-device-onboarding/actions/workflows/ci.yml/badge.svg?branch=develop"></a>
   <a href="https://docs.nautobot.com/projects/device-onboarding/en/latest/"><img src="https://readthedocs.org/projects/nautobot-plugin-device-onboarding/badge/"></a>
   <a href="https://pypi.org/project/nautobot-device-onboarding/"><img src="https://img.shields.io/pypi/v/nautobot-device-onboarding"></a>
   <a href="https://pypi.org/project/nautobot-device-onboarding/"><img src="https://img.shields.io/pypi/dm/nautobot-device-onboarding"></a>
@@ -17,11 +17,54 @@ The `nautobot-device-onboarding` plugin is using the [netmiko](https://github.co
 
 Regardless, the Onboarding App greatly simplifies the onboarding process by allowing the user to specify a small amount of info and having the app populate a much larger amount of device data in Nautobot.
 
+### Support Matrix (Sync Devices From Network)
+
+|     Data Attribute      | Cisco IOS          | Cisco XE           | Cisco NXOS         | Cisco WLC          | Juniper Junos      | Arista EOS         | F5  |
+| ----------------------- | :----------------: |  :--------------:  |  :--------------:  |  :--------------:  |  :--------------:  |  :--------------:  | :-: |
+| Hostname                | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: |
+| Platform                | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: |
+| Manufacturer            | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: |
+| Serial Number           | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: |
+| Device Type             | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: |
+| Mgmt Interface          | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: |
+| Mgmt IP Address         | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: |
+
+### Support Matrix (Sync Data From Network)
+
+|     Interfaces          | Cisco IOS          | Cisco XE           | Cisco NXOS         | Cisco WLC          | Juniper Junos      | Arista EOS         | F5  |
+| ----------------------- | :----------------: |  :--------------:  |  :--------------:  |  :--------------:  |  :--------------:  |  :--------------:  | :-: |
+| Name           | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: |
+| IP Address     | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: |
+| Type           | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: |
+| MTU            | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: |
+| Description    | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: |
+| Mac Address    | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: |
+| Link Status    | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: |
+| 802.1Q mode    | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: |
+| Lag Member     | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: |
+| Vrf Membership | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: |
+
+|     VLANS          | Cisco IOS          | Cisco XE           | Cisco NXOS         | Cisco WLC          | Juniper Junos      | Arista EOS         | F5  |
+| ----------------------- | :----------------: |  :--------------:  |  :--------------:  |  :--------------:  |  :--------------:  |  :--------------:  | :-: |
+| Untagged VLANs       | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: |
+| Tagged VLANs        | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: |
+
+|     Cabling          | Cisco IOS          | Cisco XE           | Cisco NXOS         | Cisco WLC          | Juniper Junos      | Arista EOS         | F5  |
+| ----------------------- | :----------------: |  :--------------:  |  :--------------:  |  :--------------:  |  :--------------:  |  :--------------:  | :-: |
+|  Terminations A      | :test_tube: | :test_tube: | :test_tube: | :x: | :test_tube: | :x: | :x: |
+|  Terminations B      | :test_tube: | :test_tube: | :test_tube: | :x: | :test_tube: | :x: | :x: |
+
+| Legend |
+| :---- |
+| :white_check_mark: - Supported and stable. |
+| :x: - No current support. |
+| :test_tube: - Supported, but has limited testing. |
+
 ### Screenshots
 
 Device Onboarding is a Job that allows you to provide a few required pieces of information and onboard the device.
 
-![job input](https://raw.githubusercontent.com/nautobot/nautobot-app-device-onboarding/develop/docs/images/do_job_inputs.png)
+![job input](https://raw.githubusercontent.com/nautobot/nautobot-app-device-onboarding/develop/docs/images/sync_devices_inputs.png)
 
 ## Try it out!
 

development/Dockerfile

@@ -6,8 +6,8 @@
 # -------------------------------------------------------------------------------------
 # !!! USE CAUTION WHEN MODIFYING LINES BELOW
 
-# Accepts a desired Nautobot version as build argument, default to 2.0.3
-ARG NAUTOBOT_VER="2.0.3"
+# Accepts a desired Nautobot version as build argument, default to 2.2.3
+ARG NAUTOBOT_VER="2.2.3"
 
 # Accepts a desired Python version as build argument, default to 3.11
 ARG PYTHON_VER="3.11"
@@ -28,7 +28,7 @@ ENV INVOKE_NAUTOBOT_DEVICE_ONBOARDING_LOCAL=true
 # Since this is only used for development and we don't ship this container, pinning Poetry back is not expressly necessary
 # We also don't need virtual environments in container
 RUN which poetry || curl -sSL https://install.python-poetry.org | python3 - && \
-    poetry config virtualenvs.create false
+  poetry config virtualenvs.create false
 
 # !!! USE CAUTION WHEN MODIFYING LINES ABOVE
 # -------------------------------------------------------------------------------------

development/development.env

@@ -36,3 +36,5 @@ POSTGRES_DB=${NAUTOBOT_DB_NAME}
 MYSQL_USER=${NAUTOBOT_DB_USER}
 MYSQL_DATABASE=${NAUTOBOT_DB_NAME}
 MYSQL_ROOT_HOST=%
+
+NAUTOBOT_SSOT_HIDE_EXAMPLE_JOBS="True"

development/docker-compose.base.yml

@@ -13,7 +13,6 @@ x-nautobot-base: &nautobot-base
     - "creds.env"
   tty: true
 
-version: "3.8"
 services:
   nautobot:
     depends_on:

development/docker-compose.dev.yml

@@ -3,7 +3,6 @@
 # any override will need to include these volumes to use them.
 # see:  https://github.com/docker/compose/issues/3729
 ---
-version: "3.8"
 services:
   nautobot:
     command: "nautobot-server runserver 0.0.0.0:8080"

development/docker-compose.mysql.yml

@@ -1,6 +1,4 @@
 ---
-version: "3.8"
-
 services:
   nautobot:
     environment:

development/docker-compose.postgres.yml

@@ -1,6 +1,4 @@
 ---
-version: "3.8"
-
 services:
   nautobot:
     environment:

development/docker-compose.redis.yml

@@ -1,5 +1,4 @@
 ---
-version: "3.8"
 services:
   redis:
     image: "redis:6-alpine"

development/nautobot_config.py

@@ -93,11 +93,11 @@
         "disable_existing_loggers": False,
         "formatters": {
             "normal": {
-                "format": "%(asctime)s.%(msecs)03d %(levelname)-7s %(name)s : %(message)s",
+                "format": "%(asctime)s.%(msecs)03d %(levelname)-7s %(name)s :\n  %(message)s",
                 "datefmt": "%H:%M:%S",
             },
             "verbose": {
-                "format": "%(asctime)s.%(msecs)03d %(levelname)-7s %(name)-20s %(filename)-15s %(funcName)30s() : %(message)s",
+                "format": "%(asctime)s.%(msecs)03d %(levelname)-7s %(name)-20s %(filename)-15s %(funcName)30s() :\n  %(message)s",
                 "datefmt": "%H:%M:%S",
             },
         },
@@ -127,13 +127,24 @@
 #
 
 # Enable installed Apps. Add the name of each App to the list.
-PLUGINS = ["nautobot_device_onboarding"]
+PLUGINS = ["nautobot_device_onboarding", "nautobot_ssot", "nautobot_plugin_nornir"]
 
 # Apps configuration settings. These settings are used by various Apps that the user may have installed.
 # Each key in the dictionary is the name of an installed App and its value is a dictionary of settings.
-# PLUGINS_CONFIG = {
-#     'nautobot_device_onboarding': {
-#         'foo': 'bar',
-#         'buzz': 'bazz'
-#     }
-# }
+PLUGINS_CONFIG = {
+    "nautobot_device_onboarding": {},
+    "nautobot_ssot": {
+        "hide_example_jobs": is_truthy(os.getenv("NAUTOBOT_SSOT_HIDE_EXAMPLE_JOBS")),
+    },
+    "nautobot_plugin_nornir": {
+        "nornir_settings": {
+            "credentials": "nautobot_plugin_nornir.plugins.credentials.nautobot_secrets.CredentialsNautobotSecrets",
+            "runner": {
+                "plugin": "threaded",
+                "options": {
+                    "num_workers": 20,
+                },
+            },
+        },
+    },
+}

docs/admin/compatibility_matrix.md

@@ -7,3 +7,4 @@
 | 1.2.0                     | 1.4.0                          | 1.99                          |
 | 2.0.0                     | 2.0.0                          | 2.99                          |
 | 3.0.0                     | 2.0.3                          | 2.99                          |
+| 4.0.0                     | 2.2.3                          | 2.99                          |

docs/admin/install.md

@@ -10,25 +10,22 @@ Here you will find detailed instructions on how to **install** and **configure**
 !!! note
     Please check the [dedicated page](compatibility_matrix.md) for a full compatibility matrix and the deprecation policy.
 
-### Access Requirements
+### Application Dependencies
 
-#### NAPALM Credentials
+Device onboarding 4.0 added two additional application dependencies.
 
-The Onboarding App uses NAPALM. You can configure a default NAPALM username and password in `nautobot_config.py`.
+- [nautobot_ssot](https://docs.nautobot.com/projects/ssot/en/latest/)
+- [nautobot_plugin_nornir](https://docs.nautobot.com/projects/plugin-nornir/en/latest/)
 
-When `NAPALM_USERNAME`, `NAPALM_PASSWORD` and `NAPALM_ARGS` are configured in `nautobot_config.py`, the user does not have to use the Credentials/SecretGroup fields in the Device Onboarding job, unless they wish to override the values in `nautobot_config.py`:
+### Access Requirements
 
-```python
-# Credentials that Nautobot will use to authenticate to devices when connecting via NAPALM.
-NAPALM_USERNAME = "<napalm username>"
-NAPALM_PASSWORD = "<napalm pwd>"
-NAPALM_ARGS = {"secret": "<enable secret pwd>"}
-```
+- The original OnboardingTask Job used NAPALM Credentials.
+- The new SSoT based jobs use Nautobot Secrets in a similar method to other apps. This is a more flexible and supportable way for the future.
 
 ## Install Guide
 
 !!! note
-    Apps can be installed from the [Python Package Index](https://pypi.org/) or locally. See the [Nautobot documentation](https://docs.nautobot.com/projects/core/en/stable/user-guide/administration/installation/app-install/) for more details. The pip package name for this app is [`nautobot-device-onboarding`](https://pypi.org/project/nautobot-device-onboarding/).
+    Apps can be installed manually or using Python's `pip`. See the [nautobot documentation](https://nautobot.readthedocs.io/en/latest/plugins/#install-the-package) for more details. The pip package name for this app is [`nautobot-device-onboarding`](https://pypi.org/project/nautobot-device-onboarding/).
 
 The app is available as a Python package via PyPI and can be installed with `pip`:
 
@@ -49,7 +46,7 @@ Once installed, the app needs to be enabled in your Nautobot configuration. The
 
 ```python
 # In your nautobot_config.py
-PLUGINS = ["nautobot_device_onboarding"]
+PLUGINS = ["nautobot_plugin_nornir", "nautobot_ssot", "nautobot_device_onboarding"]
 
 # PLUGINS_CONFIG = {
 #   "nautobot_device_onboarding": {
@@ -58,6 +55,22 @@ PLUGINS = ["nautobot_device_onboarding"]
 # }
 ```
 
+For the `Sync Data from Network` job to work the below must be added to the `PLUGINS_CONFIG`.
+
+```python
+    "nautobot_plugin_nornir": {
+        "nornir_settings": {
+            "credentials": "nautobot_plugin_nornir.plugins.credentials.nautobot_secrets.CredentialsNautobotSecrets",
+            "runner": {
+                "plugin": "threaded",
+                "options": {
+                    "num_workers": 20,
+                },
+            },
+        },
+    },
+```
+
 Once the Nautobot configuration is updated, run the Post Upgrade command (`nautobot-server post_upgrade`) to run migrations and clear any cache:
 
 ```shell
@@ -74,14 +87,14 @@ Then restart (if necessary) the Nautobot services which may include:
 sudo systemctl restart nautobot nautobot-worker nautobot-scheduler
 ```
 
-## App Configuration
+## App Configuration (Original)
 
 Although the app can run without providing any settings, the app behavior can be controlled with the following list of settings defined in `nautobot_config.py`:
 
 - `create_platform_if_missing` boolean (default True). If True, a new platform object will be created if the platform discovered by netmiko do not already exist and is in the list of supported platforms (`cisco_ios`, `cisco_nxos`, `arista_eos`, `juniper_junos`, `cisco_xr`)
 - `create_device_type_if_missing` boolean (default True), If True, a new device type object will be created if the model discovered by Napalm do not match an existing device type.
 - `create_manufacturer_if_missing` boolean (default True), If True, a new manufacturer object will be created if the manufacturer discovered by Napalm is do not match an existing manufacturer, this option is only valid if `create_device_type_if_missing` is True as well.
-- `create_device_role_if_missing` boolean (default True), If True, a new device role object will be created if the device role provided was not provided as part of the onboarding and if the `default_device_role` do not already exist.
+- `create_device_role_if_missing` boolean (default True), If True, a new device role object will be created if the device role provided was not provided as part of the onboarding and if the `default_device_role` does not already exist.
 - `create_management_interface_if_missing` boolean (default True), If True, add management interface and IP address to the device. If False no management interfaces will be created, nor will the IP address be added to Nautobot, while the device will still get added.
 - `default_device_status` string (default "Active"), status assigned to a new device by default.
 - `default_ip_status` string (default "Active"), status assigned to a new device management IP.

docs/admin/release_notes/version_4.0.md

@@ -0,0 +1,27 @@
+# v4.0 Release Notes
+
+!!! warning
+    Nautobot Device Onboarding v4.0.0 completely revamps the applications design and framework. The original `OnboardingTask` job is still packaged with the app to provide a backwards compatible way for users that have used its extensions framework in the past to solve complex problems. However, that job is now hidden by default to avoid confusion with the two new SSoT based onboarding jobs that v4.0.0 exposes.
+
+## [v4.0.0 (2024-08-05)](https://github.com/nautobot/nautobot-app-device-onboarding/releases/tag/v4.0.0)
+
+### Added
+
+- [#181](https://github.com/nautobot/nautobot-app-device-onboarding/issues/181) - Sync Devices from Network job was added which utilizes the SSoT framework to onboard devices.
+- [#181](https://github.com/nautobot/nautobot-app-device-onboarding/issues/181) - Sync Data from Network job was added which utilizes the SSoT framework to onboard devices.
+- [#181](https://github.com/nautobot/nautobot-app-device-onboarding/issues/181) - Git Datasource object to be able to use a Git Repo to overload new SSoT job YAML file definitions.
+- [#181](https://github.com/nautobot/nautobot-app-device-onboarding/issues/181) - Create a Nornir inventory `EmptyInventory` to support ondemand inventory population for `Sync Devices` job.
+- [#181](https://github.com/nautobot/nautobot-app-device-onboarding/issues/181) - Add `nautobot-app-nornir` dependency to reuse `NautobotORMInventory` to support inventory creation for `Sync Data` job.
+- [#201](https://github.com/nautobot/nautobot-app-device-onboarding/issues/201) - - Add ability to sync in cables for cisco ios,nxos and Juniper via neighbor discovery protocol commands.
+
+### Changed
+
+- [#151](https://github.com/nautobot/nautobot-app-device-onboarding/issues/151) - Replaced pydocstyle with ruff.
+- [#181](https://github.com/nautobot/nautobot-app-device-onboarding/issues/181) - The `OnboardingTask` job is changed to `hidden` by default.
+
+### Housekeeping
+
+- [#167](https://github.com/nautobot/nautobot-app-device-onboarding/issues/167) - Re-baked from the template `nautobot-app-v2.2.1`.
+- [#194](https://github.com/nautobot/nautobot-app-device-onboarding/issues/194) - - Add @housebpass to Codeowners
+- [#194](https://github.com/nautobot/nautobot-app-device-onboarding/issues/194) - - Add ntc-templates to new issue template
+- [#203](https://github.com/nautobot/nautobot-app-device-onboarding/issues/203) - - Add compatibility matrix for what platforms support which data fields to sync.

docs/dev/arch_decision.md

@@ -0,0 +1,9 @@
+# Architecture Decision Records
+
+The intention is to document deviations from a standard Model View Controller (MVC) design. And other general Application decisions.
+
+## Handling the Nornir Inventory
+
+In order for Nornir to function an inventory is created. There are multiple supported inventory sources that fit many needs; however there is a unique requirement that this plugin is trying to solve. The problem is specifically around the first SSoT job (Sync Devices from Network); how can we create an inventory when there is no source "yet"? Our solution to this problem is to generate a empty inventory, and then process the ip addresses from the job form to create a inventory in an on demand fashion and inject the credentials into the inventory based on the secrets group selected.
+
+For the general application constraint for this ADR see the [Credentials Section](../user/app_getting_started.md#device-credentials-functionality).