New appliance format

[grossmj]

With GNS3 v3.0 around the corner, it would be a great opportunity to change the appliance file format to add flexibility for custom settings / hardware configuration for specific versions and also add some other improvements:

Here is what I have in mind for now:

• Support for more than one emulator settings / hardware configurations. One config can be assigned to a specific version using its name. The default config is used (default field must be configured if there are multiple configs) or the only config present if none is explicitly assigned to a version. Should we refer to this as "configs" or "settings"?
• Support for different checksum algorithms by adding a checksum_type field (md5, sha256 etc.) and replacing md5sum by a checksum field in the images section. What algorithms should we support in addition of md5 and sha256?
• Should we add default_username and default_password fields?

Example:

{
    "appliance_id": "709c2a9b-5dc3-4362-b147-fb848a0df963",
    "name": "My appliance",
    "category": "router",
    "description": "This is my new appliance",
    "vendor_name": "Cisco",
    "vendor_url": "http://www.cisco.com/",
    "documentation_url": "https://www.cisco.com/c/en/us/support/routing/xxx",
    "product_name": "Appliance product xxx",
    "product_url": "https://www.cisco.com/c/en/us/products/xxx/index.html",
    "registry_version":8,
    "status": "experimental",
    "maintainer": "GNS3 Team",
    "maintainer_email": "[email protected]",
    "usage": "This is how to use my appliance",
    "default_username": "cisco",
    "default_password": "admin",
    "symbol": "router.svg",
    "first_port_name": "eth0",
    "port_name_format": "eth{port1}",
    "settings": [
        {
            "name": "Default settings",
            "type": "qemu",
            "default": true,
            "adapter_type": "e1000",
            "adapters": 2,
            "ram": 4096,
            "cpus": 1,
            "hda_disk_interface": "scsi",
            "arch": "x86_64",
            "console_type": "vnc",
            "boot_priority": "cd",
            "kvm": "require",
            "options": ""
        },
        {
            "name": "Custom settings for version 7.10.2",
            "type": "qemu",
            "adapters": 4,
            "ram": 8192,
            "cpus": 1
        }
    ],
    "images": [
        {
            "filename": "file.iso",
            "version": "7.10.2",
            "checksum": "ef8712e655fcbc92dc1a1551ee2e4a80",
            "checksum_type": "md5",
            "filesize": 1287245824,
            "download_url": "https://software.cisco.com/download/home/286307342/type/286307754/release/7.10.2"
        },
        {
            "filename": "file2.iso",
            "version": "6.10.4",
            "checksum": "e5b844cc57f57094ea4585e235f36c78c1cd222262bb89d53c94dcb4d6b3e55d",
            "checksum_type": "sha256",
            "filesize": 3949459594,
            "download_url": "https://software.cisco.com/download/home/286307342/type/286307754/release/6.10.4"
        },
        {
            "filename": "empty100G.qcow2",
            "version": "1.0",
            "checksum": "1e6409a4523ada212dea2ebc50e50a65",
            "checksum_type": "md5",
            "filesize": 198656,
            "download_url": "https://sourceforge.net/projects/gns-3/files/Empty%20Qemu%20disk/",
            "direct_download_url": "https://sourceforge.net/projects/gns-3/files/Empty%20Qemu%20disk/empty100G.qcow2/download"
        }
    ],
    "versions": [
        {
            "name": "7.10.2",
            "settings": "Custom settings for version 7.10.2",
            "images": {
                "hda_disk_image": "empty100G.qcow2",
                "cdrom_image": "file.iso"
            }
        },
        {
            "name": "6.10.4",
            "images": {
                "hda_disk_image": "empty100G.qcow2",
                "cdrom_image": "file2.iso"
            }
        }
    ]
}

Please let me know if you have comments / suggestions.

[grossmj]

We could move away from JSON and use Yaml for instance?

[eantowne]

• What would be the benefit of YAML vs JSON, beyond human readability?

• I think that more needs to be moved from the base of the config into the "settings". Vendors do occasionally change things. I think the base portion should simply be metadata about the .gns3a (maintainer, vendor, vendor url, status as examples).

Fields that definitely should be moved into "settings"
"usage", "default_username", "default_password", "symbol", "first_port_name","port_name_format"

Examples of why this is necessary:

"default_username", "default_password" - Nvidia (Cumulus) changed their defaults in 4.0 or 5.0 (cannot remember which) of CumulusVX
"first_port_name","port_name_format" - Linux used to use friendly names (ethX), now interface names start with en or wl

• Hash algorithms: MD5, SHA-I, SHA-256, and SHA-512, these are all of the widely used, and fairly standardized algo's. There are more out there, but these are the predominant variants. I think that GNS3 should specify one as the standard for a registry version. If the client or compute has to run multiple hashes on 50 files in a folder to determine which files may already be available, there will be a significant delay for the user experience, even more than there already is. I would select whichever is the most precise with least chance of collision.

[grossmj]

What would be the benefit of YAML vs JSON, beyond human readability?

Well, you are right, it would only add human readability and would give us just more work for not many benefits.

I think that more needs to be moved from the base of the config into the "settings". Vendors do occasionally change things. I think the base portion should simply be metadata about the .gns3a (maintainer, vendor, vendor url, status as examples).

Good point. Just one thing to note is that the fields in settings could just be taken and used to run the given emulator (Qemu, Dynamips etc.) Now we need to know which ones are metadata and which ones are to be used with an emulator. Maybe we could further divide settings to show that separation, something like this:

    "settings": [
        {
            "name": "Default settings",
            "type": "qemu",
            "default": true,
            "default_username": "root",
            "default_password": "admin",
            "properties": {
                "adapter_type": "e1000",
                "adapters": 2,
                "ram": 4096,
                "cpus": 1,
                "hda_disk_interface": "scsi",
                "arch": "x86_64",
                "console_type": "vnc",
                "boot_priority": "cd",
                "kvm": "require",
                "options": ""
           }
        }

Hash algorithms: MD5, SHA-I, SHA-256, and SHA-512, these are all of the widely used, and fairly standardized algo's. There are more out there, but these are the predominant variants. I think that GNS3 should specify one as the standard for a registry version. If the client or compute has to run multiple hashes on 50 files in a folder to determine which files may already be available, there will be a significant delay for the user experience, even more than there already is. I would select whichever is the most precise with least chance of collision.

MD5, SHA-I, SHA-256, and SHA-512 would be a good start however you raised a good point, we originally have chosen MD5 because it is fast and hasn't a big chance of collision. We will have to carefully consider this.

[eantowne]

From my perspective, the "metadata" fields would be those that describe the .gns3a file, or that are not necessary to the mechanics of creating/running the appliance. Everything else becomes a setting.

I assume that if a given "settings" set does not include a field, then the value for that field would be pulled from "Default Settings"?

I will try take some time this week and take a look through the v7 schema and see if I can come up with some framework for metadata vs mechanics data and how that can be normalized across emulators and such. (work dependent)

[grossmj]

I assume that if a given "settings" set does not include a field, then the value for that field would be pulled from "Default Settings"?

Correct 👍

I will try take some time this week and take a look through the v7 schema and see if I can come up with some framework for metadata vs mechanics data and how that can be normalized across emulators and such. (work dependent)

Sounds good, thanks.

[eantowne]

This is my first stab at dividing up the fields within the schema based on versioning the "configuration" relevant portions of the schema. I also took the opportunity to shameless try and rename and consolidate some of the fields. I additionally and shamelessly used the opportunity to try and remove the restriction of only 4 drives on QEMU appliances.

One of the key concepts here would be that a "default" values section would exist, and entries in a "version" would override the defaults. Conversely, this would also mean that if the field is not specified in the "version", then the "version" uses the "default" value. Conceivably, this could even apply to disk images (empty30g.qcow2 as an example) that are generic, and reused throughout versions.

[i] indicates a field/section that is repeatable, with "i" being an index

Metadata Fields:

CRITERIA: Fields that have no bearing on the functionality/installation/configuration of an appliance. Fields that are used to describe the appliance in a general nature or provide metadata reference the .gns3a file itself.

appliance_id
name
category
description
vendor_name
vendor_url
registry_version
product_name
maintainer Could conceivably be a repeatable field instead?
maintainer_email
symbol
linked_clone

Versioned Administrative Fields:

CRITERIA: Fields that do not affect the functionality of the appliance, but may need to be versioned, if field not specified in a version, the "default" section value is used.

version
documentation_url
product_url
status
availability
usage

Versioned Configuration Fields:

first_port_name
port_name_format
port_segment_size
emulator
emulator_version Could be used to specify specific QEMU versions?
interface_count Equivalent of "ethernet_adapters", "adapters",
console_type Docker has different console types than the other emulators
image
ram
cores Replaces "cpus" field, and allows future expansion for sockets vs. cores more easily/syntax accuracy ;)
startup_config Could also be used for Docker "start_command"?
nvram

Emulator Specfic Configuration Fields (versioned)

Docker:

start_command Replace with "startup_config"?
environment
console_http_port
console_http_path

IOU:

serial_adapters

Dynamips:

chassis
platform
wic[i]
slot[i]
midplane
npe
idlepc

QEMU:

adapter_type Should this just be removed and every interface is custom?
custom_adapters Includes all of the sub-fields specified in appliance_v7.json
disk[i] Make this section repeatable to allow for more than 4 drives and allow for the options listed below
disk_type Sub-field of disk[i], CD-ROM, OS, bios, HDD, initrd, kernel
disk_bus_type Sub-field of disk[i], SCSI,ide,sata,etc.
disk_image_name Sub-field of disk[i], to specify/map to image entry
disk_size Sub-field of disk[i], same as appliance_v7.json
disk_hash_type Sub-field of disk[i], hashing algorithm used
disk_hash Sub-field of disk[i], hashed value of file
disk_download_url Sub-field of disk[i], same as appliance_v7.json
disk_direct_download_url Sub-field of disk[i], same as appliance_v7.json
disk_compression Sub-field of disk[i], same as appliance_v7.json
arch
boot_priority
kernel_command_line
kvm
qemu_options Was just "options", provides more clarity when viewing a .gns3a file without knowledge of the schema
cpu_throttling
tpm
on_close
process_priority

[grossmj]

@b-ehlers do you have any ideas or comments?

[ghost]

Do you have any ideas or comments?

Honestly no. I'm quite happy with GNS3 v2.2 and will take time before upgrading to v3.0. While I see the benefit to make the registry schema more flexible, I currently have no need for any changes.

[eantowne]

Do you have any ideas or comments?

Honestly no. I'm quite happy with GNS3 v2.2 and will take time before upgrading to v3.0. While I see the benefit to make the registry schema more flexible, I currently have no need for any changes.

I think that might be the most ringing endorsement of GNS3 2.2 I have ever seen. :)

[grossmj]

I am getting close to have defined the new appliance format, please have a look at #734 (comment)