A simple file parser for nmap XML data.
- Build a REST API to import a single nmap scan result.
- The API should accept a single nmap scan file.
- Detail which you chose and why.
- Ingest the nmap result into a sqlite database.
- Build a UI that allows a user to view the results of nmap scans by IP address.
In the above challenge, feel free to use any language. For your submission, package source code and files into a zip file. Include a README.md outlining:
- Precise instructions to launch and use your submission including runtimes and versions.
- Any assumptions you made.
- Additional thoughts about the project.
One of the things I love the most about software creation is the great fact that no matter how far you progress into the disciplines, you've merely scratched the surface. There is always another layer that can be added to make a program better.
I love building things and I really want to build them well. Perfection is probably not possible, but striving to become better at our craft and seeking better ways of doing things can be a part of every day and every task. Kaizen, improvement.
The biggest assumption I made was that the purpose of the exercise is to demonstrate principles, values, patterns and my ideas about how software should be built. I tried to do that faithfully. I have interviewed many software engineering candidates and know what I look for, I have tried to deliver indications of my values.
Some guiding principles that I tried to convey here:
- Separation of concerns
- Security should be considered
- Documentation should be adequate
- Unit/Integration test should back the code
- CI protects the codebase
- Issues should be tracked and changes to code should reference the issue that required the change
- Source control should be used effectively and should tell a story about the development of a project
I also made some compromises for the sake of expediency:
-
There's some duplication in test setup between
/cmdand/pkg. I normally would use DB mocks in my http handler tests but am experimenting with this idea that better tests use an actual DB instead of a mock. I'm not sure how I feel about this but wanted to try it. -
I didn't have time to write Swagger/Openapi docs for the REST endpoints, so I just loosely documented them here in the README.
-
I would probably revise the data model if I were to do this over again. Once I got into implementing the UI (last!) I found some characteristics of the data models that I didn't like.
The most reliable way to distribute the application is as a docker image. You will need docker installed on your machine to run the Dockerfile. If you do not have Docker installed on your machine, you may be able to build and run the Go binary, but it will depend on your having SQLite installed. "It works on my machine", but to make sure you can run it too, I've packaged the app in a docker image.
-
Clone the repo
git clone [email protected]:jcorry/nmap-scan-api.git -
Build the image
docker build -t nmap-api . -
Run the image in a container
docker run -it -e "PORT=8080" -p 8080:8080 nmap-api -
Make requests! My Postman Collection
Upload an nmap XML file to be parsed and saved to the DB.
Content-Type=multipart/form-data
{
"file": "nmap.results.xml"
}
201
created
400
ERROR_MESSAGE string
Get a paginated list of nmap hosts
-
start
- In: query
- Valid: int
- Matches: The starting index of records to retrieve
-
length
- In: query
- Valid: int (max: 1000)
- Matches: The number of records to retrieve
Example: http://localhost:8080/api/v1/nmap?start=0&length=400
200
{
"meta": {
"start": 0,
"length": 100,
"total": 492
},
"items": [
{
"fileid": string,
"starttime": timestamp,
"endtime": timestamp,
"comment": string
"status": string,
"hostnames": [
"name": string,
"type": string,
],
"addresses": [
"addr": string,
"addrtype": string
],
"ports": [
"protocol": string,
"portid": int,
"owner": string,
"service": string,
]
},
...
]
}
400
bad request error
-
start
- In: query
- Valid: int
- Matches: The starting index of records to retrieve
-
length
- In: query
- Valid: int (max: 1000)
- Matches: The number of records to retrieve
Example: http://localhost:8080/nmap/list?start=0&length=400
200
HTML list view of nmap host scans
nmap data must be mapped to internal structs and SQL tables.
I added a file import table to log the importation of files with a unique hash per file. This way, I can prevent duplicate data imports. I'm filing an issue to come back and handle the case where the file was accepted on upload and its hash saved to the imports table but the batch insert of hosts fails, rollsback and leaves the hash in the imports table. This case would prevent the user from ever successfully uploading that file.
I've never seen nmap data before today but found a Go package used for parsing nmap XML files. The author clearly knows way more than me about nmap data structures so I am going to use their parser and model my app around their structs. I used the XML data format from the available files because it was the first I found suitable tools for parsing it.
There's a ton of data that is available in nmap results, for our demo project we will focus on only a subset of that data. My API response as designed above will inform the SQL structure and the internal structs that query data is scanned to. Much to my dismay, the structure of the XML does not really map well to a relational DB. Ports and Hostnames are (I think) related to Addresses, but there's no relationship between these depicted in the XML structure or its parsed equivalent.
One mistake I made was in the sqlite HostRepo.list() function. The way I handled scanning row results into structs and
then storing those structs in maps so they can be keyed to the host prevents DB based sorting. This should be refactored
and can be remedied by using a []*models.Host as the parent structure and eliminating the hostMap.
The UI is a simple HTML grid supplied by Bootstrap 4. The HTML is generated from templates using Go's html/template package
which is sparse, but adequate for very basic display.
The list view also makes a JSON representation available. An API URL is provided. If the Content-Type request header
is application/json the handler will respond with JSON.