Skip to content

Commit

Permalink
feat: fully implementing paloalto
Browse files Browse the repository at this point in the history
  • Loading branch information
@bl4ko
bl4ko committed Mar 21, 2024
1 parent 17f1269 commit c527493
Show file tree
Hide file tree
Showing 16 changed files with 498 additions and 211 deletions.
56 changes: 28 additions & 28 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ Currently, the supported external data sources types are:
- [`vmware`](https://www.vmware.com/products/vcenter.html)
- [`dnac`](https://www.cisco.com/site/us/en/products/networking/catalyst-center/index.html)
- [`proxmox`](https://www.proxmox.com/en/)
- [`paloalto`](https://www.paloaltonetworks.com/network-security/next-generation-firewall)
- firewall

> [!WARNING]
> **This project is still under heavy development, use with caution.**
Expand Down Expand Up @@ -59,27 +61,27 @@ Example configuration can be found [here](#example-config).

### Source

| Parameter | Description | Source Type | Type | Possible values | Default | Required |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------ | --------------- | -------- | --------------------- | ---------- | -------- |
| `source.name` | Name of the data source. | all | str | any | "" | Yes |
| `source.type` | Type of the data source. | all | str | [ovirt, vmware, dnac] | "" | Yes |
| `source.httpScheme` | Http scheme for the source | all | str | [ http,https] | https | no |
| `source.hostname` | Hostname of the data source. | all | str | any | "" | Yes |
| `source.port` | Port of the data source. | all | int | 0-65536 | 443 | No |
| `source.username` | Username of the data source account. | all | str | any | "" | Yes |
| `source.password` | Password of the data source account. | all | str | any | "" | Yes |
| `source.validateCert` | Enforce TLS certificate validation. | all | bool | [true, false] | false | No |
| `source.tagColor` | TagColor for the source tag. | all | string | any | Predefined | No |
| `source.ignoredSubnets` | List of subnets, which will be ignored (e.g. IPs won't be synced). | all | []string | any | [] | No |
| `source.interfaceFilter` | Regex representation of interface names to be ignored (e.g. `(cali\|vxlan\|flannel\|[a-f0-9]{15})`) | all | string | any | [] | No |
| `source.hostSiteRelations` | Regex relations in format `regex = siteName`, that map each host that satisfies regex to site. | [vmware, ovirt] | []string | any | [] | No |
| `source.clusterSiteRelations` | Regex relations in format `regex = siteName`, that map each cluster that satisfies regex to site. | [vmware, ovirt] | []string | any | [] | No |
| `source.clusterTenantRelations` | Regex relations in format `regex = tenantName`, that map each cluster that satisfies regex to tenant. | [vmware, ovirt] | []string | any | [] | No |
| `source.hostTenantRelations` | Regex relations in format `regex = tenantName`, that map each host that satisfies regex to tenant. | all | []string | any | [] | No |
| `source.vmTenantRelations` | Regex relations in format `regex = tenantName`, that map each vm that satisfies regex to tenant. | [vmware, ovirt] | []string | any | [] | No |
| `source.vlanGroupRelations` | Regex relations in format `regex = vlanGroup`, that map each vlan that satisfies regex to vlanGroup. | all | []string | any | [] | No |
| `source.vlanTenantRelations` | Regex relations in format `regex = tenantName`, that map each vlan that satisfies regex to tenant. | all | []string | any | [] | No |
| `source.customFieldMappings` | Mappings of format `customFieldName = option`. Currently, supported options are `contact`, `owner`, `description`. | [ vmware ] | []string | any | [] | No |
| Parameter | Description | Source Type | Type | Possible values | Default | Required |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ----------- | -------- | ---------------------------------------- | ---------- | -------- |
| `source.name` | Name of the data source. | all | str | any | "" | Yes |
| `source.type` | Type of the data source. | all | str | [ovirt, vmware, dnac, proxmox, paloalto] | "" | Yes |
| `source.httpScheme` | Http scheme for the source | all | str | [ http,https] | https | no |
| `source.hostname` | Hostname of the data source. | all | str | any | "" | Yes |
| `source.port` | Port of the data source. | all | int | 0-65536 | 443 | No |
| `source.username` | Username of the data source account. | all | str | any | "" | Yes |
| `source.password` | Password of the data source account. | all | str | any | "" | Yes |
| `source.validateCert` | Enforce TLS certificate validation. | all | bool | [true, false] | false | No |
| `source.tagColor` | TagColor for the source tag. | all | string | any | Predefined | No |
| `source.ignoredSubnets` | List of subnets, which will be ignored (e.g. IPs won't be synced). | all | []string | any | [] | No |
| `source.interfaceFilter` | Regex representation of interface names to be ignored (e.g. `(cali\|vxlan\|flannel\|[a-f0-9]{15})`) | all | string | any | [] | No |
| `source.hostSiteRelations` | Regex relations in format `regex = siteName`, that map each host that satisfies regex to site. | all | []string | any | [] | No |
| `source.clusterSiteRelations` | Regex relations in format `regex = siteName`, that map each cluster that satisfies regex to site. | all | []string | any | [] | No |
| `source.clusterTenantRelations` | Regex relations in format `regex = tenantName`, that map each cluster that satisfies regex to tenant. | all | []string | any | [] | No |
| `source.hostTenantRelations` | Regex relations in format `regex = tenantName`, that map each host that satisfies regex to tenant. | all | []string | any | [] | No |
| `source.vmTenantRelations` | Regex relations in format `regex = tenantName`, that map each vm that satisfies regex to tenant. | all | []string | any | [] | No |
| `source.vlanGroupRelations` | Regex relations in format `regex = vlanGroup`, that map each vlan that satisfies regex to vlanGroup. | all | []string | any | [] | No |
| `source.vlanTenantRelations` | Regex relations in format `regex = tenantName`, that map each vlan that satisfies regex to tenant. | all | []string | any | [] | No |
| `source.customFieldMappings` | Mappings of format `customFieldName = option`. Currently, supported options are `contact`, `owner`, `description`. | [ vmware ] | []string | any | [] | No |

### Example config

Expand Down Expand Up @@ -130,15 +132,13 @@ source:
- Creator = owner
- Description = description

- name: testvmare
type: vmware
hostname: vcenter-test.example.com
- name: pa-uk
type: paloalto
hostname: 192.168.1.52
username: user
password: passw0rd
customFieldMappings: # Here we define map of our custom field names, to 3 option [email, owner, description]
- Email = email
- Maintainer = owner
- Notes = description
hostTenantRelations:
- .* = SRC


- name: dnacenter
Expand Down
31 changes: 23 additions & 8 deletions internal/constants/constants.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,15 @@ const (
CustomFieldHostMemoryDescription = "Amount of memory on the host"
)

// Device Role constants.
const (
DeviceRoleFirewall = "Firewall"
DeviceRoleFirewallColor = "f57842"

DeviceRoleServer = "Server"
DeviceRoleServerColor = "00add8"
)

// Constants used for variables in our contexts.
type CtxKey int

Expand All @@ -137,31 +146,37 @@ const (
// Here all mappings are defined so we don't hardcode api paths of objects
// in our code.
const (
// Tenancy paths.
ContactGroupsAPIPath = "/api/tenancy/contact-groups/"
ContactRolesAPIPath = "/api/tenancy/contact-roles/"
ContactsAPIPath = "/api/tenancy/contacts/"
TenantsAPIPath = "/api/tenancy/tenants/"
ContactAssignmentsAPIPath = "/api/tenancy/contact-assignments/"

// IPAM paths.
PrefixesAPIPath = "/api/ipam/prefixes/"
VlanGroupsAPIPath = "/api/ipam/vlan-groups/"
VlansAPIPath = "/api/ipam/vlans/"
IPAddressesAPIPath = "/api/ipam/ip-addresses/"

// Virtualization paths.
ClusterTypesAPIPath = "/api/virtualization/cluster-types/"
ClusterGroupsAPIPath = "/api/virtualization/cluster-groups/"
ClustersAPIPath = "/api/virtualization/clusters/"
VirtualMachinesAPIPath = "/api/virtualization/virtual-machines/"
VMInterfacesAPIPath = "/api/virtualization/interfaces/"

DevicesAPIPath = "/api/dcim/devices/"
DeviceRolesAPIPath = "/api/dcim/device-roles/"
DeviceTypesAPIPath = "/api/dcim/device-types/"
InterfacesAPIPath = "/api/dcim/interfaces/"
SitesAPIPath = "/api/dcim/sites/"
ManufacturersAPIPath = "/api/dcim/manufacturers/"
PlatformsAPIPath = "/api/dcim/platforms/"

// DCIM paths.
DevicesAPIPath = "/api/dcim/devices/"
DeviceRolesAPIPath = "/api/dcim/device-roles/"
DeviceTypesAPIPath = "/api/dcim/device-types/"
InterfacesAPIPath = "/api/dcim/interfaces/"
SitesAPIPath = "/api/dcim/sites/"
ManufacturersAPIPath = "/api/dcim/manufacturers/"
PlatformsAPIPath = "/api/dcim/platforms/"
VirtualDeviceContextsAPIPath = "/api/dcim/virtual-device-contexts/"

// Extras paths.
CustomFieldsAPIPath = "/api/extras/custom-fields/"
TagsAPIPath = "/api/extras/tags/"
)
55 changes: 54 additions & 1 deletion internal/netbox/inventory/add_items.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -479,8 +479,8 @@ func (nbi *NetboxInventory) AddDeviceType(ctx context.Context, newDeviceType *ob
}

func (nbi *NetboxInventory) AddPlatform(ctx context.Context, newPlatform *objects.Platform) (*objects.Platform, error) {
newPlatform.Tags = append(newPlatform.Tags, nbi.SsotTag)
nbi.PlatformsLock.Lock()
newPlatform.Tags = append(newPlatform.Tags, nbi.SsotTag)

Check warning on line 483 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L483 

Added line #L483 was not covered by tests
defer nbi.PlatformsLock.Unlock()
if _, ok := nbi.PlatformsIndexByName[newPlatform.Name]; ok {
// Remove id from orphan manager, because it still exists in the sources
Expand Down Expand Up @@ -515,6 +515,7 @@ func (nbi *NetboxInventory) AddDevice(ctx context.Context, newDevice *objects.De
nbi.DevicesLock.Lock()
defer nbi.DevicesLock.Unlock()
newDevice.Tags = append(newDevice.Tags, nbi.SsotTag)
addSourceNameCustomField(ctx, &newDevice.NetboxObject)
if newDevice.Site == nil {
return nil, fmt.Errorf("device %s is not assigned to a site, but it should be", newDevice)

Check warning on line 520 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L518-L520 

Added lines #L518 - L520 were not covered by tests
}
Expand Down Expand Up @@ -549,6 +550,46 @@ func (nbi *NetboxInventory) AddDevice(ctx context.Context, newDevice *objects.De
return nbi.DevicesIndexByNameAndSiteID[newDevice.Name][newDevice.Site.ID], nil
}

// AddVirtualDeviceContext adds new virtual device context to the local inventory.
func (nbi *NetboxInventory) AddVirtualDeviceContext(ctx context.Context, newVDC *objects.VirtualDeviceContext) (*objects.VirtualDeviceContext, error) {
nbi.DevicesLock.Lock()
defer nbi.DevicesLock.Unlock()
newVDC.Tags = append(newVDC.Tags, nbi.SsotTag)
if newVDC.Device == nil {
return nil, fmt.Errorf("VirtualDeviceContext %s is not assigned to a device, but it should be", newVDC)

Check warning on line 559 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L554-L559 

Added lines #L554 - L559 were not covered by tests
}
addSourceNameCustomField(ctx, &newVDC.NetboxObject)
if _, ok := nbi.VirtualDeviceContextsIndexByNameAndDeviceID[newVDC.Name][newVDC.Device.ID]; ok {
oldVDC := nbi.VirtualDeviceContextsIndexByNameAndDeviceID[newVDC.Name][newVDC.Device.ID]
delete(nbi.OrphanManager[constants.VirtualDeviceContextsAPIPath], oldVDC.ID)
diffMap, err := utils.JSONDiffMapExceptID(newVDC, oldVDC, false, nbi.SourcePriority)
if err != nil {
return nil, err

Check warning on line 567 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L561-L567 

Added lines #L561 - L567 were not covered by tests
}
if len(diffMap) > 0 {
nbi.Logger.Debug(ctx, "VirtualDeviceContext ", newVDC.Name, " already exists in Netbox but is out of date. Patching it...")
patchedVDC, err := service.Patch[objects.VirtualDeviceContext](ctx, nbi.NetboxAPI, oldVDC.ID, diffMap)
if err != nil {
return nil, err

Check warning on line 573 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L569-L573 

Added lines #L569 - L573 were not covered by tests
}
nbi.VirtualDeviceContextsIndexByNameAndDeviceID[newVDC.Name][newVDC.Device.ID] = patchedVDC
} else {
nbi.Logger.Debug(ctx, "VirtualDeviceContext ", newVDC.Name, " already exists in Netbox and is up to date...")

Check warning on line 577 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L575-L577 

Added lines #L575 - L577 were not covered by tests
}
} else {
nbi.Logger.Debug(ctx, "VirtualDeviceContext ", newVDC.Name, " does not exist in Netbox. Creating it...")
newDevice, err := service.Create[objects.VirtualDeviceContext](ctx, nbi.NetboxAPI, newVDC)
if err != nil {
return nil, err

Check warning on line 583 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L579-L583 

Added lines #L579 - L583 were not covered by tests
}
if nbi.VirtualDeviceContextsIndexByNameAndDeviceID[newDevice.Name] == nil {
nbi.VirtualDeviceContextsIndexByNameAndDeviceID[newDevice.Name] = make(map[int]*objects.VirtualDeviceContext)

Check warning on line 586 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L585-L586 

Added lines #L585 - L586 were not covered by tests
}
nbi.VirtualDeviceContextsIndexByNameAndDeviceID[newDevice.Name][newDevice.Device.ID] = newDevice

Check warning on line 588 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L588 

Added line #L588 was not covered by tests
}
return nbi.VirtualDeviceContextsIndexByNameAndDeviceID[newVDC.Name][newVDC.Device.ID], nil

Check warning on line 590 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L590 

Added line #L590 was not covered by tests
}

func (nbi *NetboxInventory) AddVlanGroup(ctx context.Context, newVlanGroup *objects.VlanGroup) (*objects.VlanGroup, error) {
nbi.VlanGroupsLock.Lock()
defer nbi.VlanGroupsLock.Unlock()
Expand Down Expand Up @@ -586,6 +627,7 @@ func (nbi *NetboxInventory) AddVlan(ctx context.Context, newVlan *objects.Vlan)
nbi.VlansLock.Lock()
defer nbi.VlansLock.Unlock()
newVlan.Tags = append(newVlan.Tags, nbi.SsotTag)
addSourceNameCustomField(ctx, &newVlan.NetboxObject)

Check warning on line 630 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L630 

Added line #L630 was not covered by tests
if _, ok := nbi.VlansIndexByVlanGroupIDAndVID[newVlan.Group.ID][newVlan.Vid]; ok {
// Remove id from orphan manager, because it still exists in the sources
oldVlan := nbi.VlansIndexByVlanGroupIDAndVID[newVlan.Group.ID][newVlan.Vid]
Expand Down Expand Up @@ -622,6 +664,7 @@ func (nbi *NetboxInventory) AddInterface(ctx context.Context, newInterface *obje
nbi.InterfacesLock.Lock()
defer nbi.InterfacesLock.Unlock()
newInterface.Tags = append(newInterface.Tags, nbi.SsotTag)
addSourceNameCustomField(ctx, &newInterface.NetboxObject)

Check warning on line 667 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L667 

Added line #L667 was not covered by tests
if _, ok := nbi.InterfacesIndexByDeviceIDAndName[newInterface.Device.ID][newInterface.Name]; ok {
// Remove id from orphan manager, because it still exists in the sources
delete(nbi.OrphanManager[constants.InterfacesAPIPath], nbi.InterfacesIndexByDeviceIDAndName[newInterface.Device.ID][newInterface.Name].ID)
Expand Down Expand Up @@ -658,6 +701,7 @@ func (nbi *NetboxInventory) AddVM(ctx context.Context, newVM *objects.VM) (*obje
nbi.VMsLock.Lock()
defer nbi.VMsLock.Unlock()
newVM.Tags = append(newVM.Tags, nbi.SsotTag)
addSourceNameCustomField(ctx, &newVM.NetboxObject)

Check warning on line 704 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L704 

Added line #L704 was not covered by tests
if _, ok := nbi.VMsIndexByName[newVM.Name]; ok {
// Remove id from orphan manager, because it still exists in the sources
delete(nbi.OrphanManager[constants.VirtualMachinesAPIPath], nbi.VMsIndexByName[newVM.Name].ID)
Expand Down Expand Up @@ -728,6 +772,7 @@ func (nbi *NetboxInventory) AddIPAddress(ctx context.Context, newIPAddress *obje
newIPAddress.Tags = append(newIPAddress.Tags, nbi.SsotTag)
nbi.IPAddressesLock.Lock()
defer nbi.IPAddressesLock.Unlock()
addSourceNameCustomField(ctx, &newIPAddress.NetboxObject)

Check warning on line 775 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L775 

Added line #L775 was not covered by tests
if _, ok := nbi.IPAdressesIndexByAddress[newIPAddress.Address]; ok {
// Delete id from orphan manager, because it still exists in the sources
delete(nbi.OrphanManager[constants.IPAddressesAPIPath], nbi.IPAdressesIndexByAddress[newIPAddress.Address].ID)
Expand Down Expand Up @@ -795,3 +840,11 @@ func (nbi *NetboxInventory) AddPrefix(ctx context.Context, newPrefix *objects.Pr
}
return nbi.PrefixesIndexByPrefix[newPrefix.Prefix], nil
}

// Helper function that adds source name to custom field of the netbox object.
func addSourceNameCustomField(ctx context.Context, netboxObject *objects.NetboxObject) {
if netboxObject.CustomFields == nil {
netboxObject.CustomFields = make(map[string]string)

Check warning on line 847 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L845-L847 

Added lines #L845 - L847 were not covered by tests
}
netboxObject.CustomFields[constants.CustomFieldSourceName] = ctx.Value(constants.CtxSourceKey).(string) //nolint

Check warning on line 849 in internal/netbox/inventory/add_items.go

View check run for this annotation

Codecov / codecov/patch

internal/netbox/inventory/add_items.go#L849 

Added line #L849 was not covered by tests
}
Loading

0 comments on commit c527493

Please sign in to comment.