diff --git a/README.adoc b/README.adoc deleted file mode 100644 index 032cd8eb5da..00000000000 --- a/README.adoc +++ /dev/null @@ -1,194 +0,0 @@ - -= Wat is OpenKAT? - -*NOTE: You can find the most recent English documentation at https://minvws.github.io/nl-kat-coordination/* - -OpenKAT heeft als doel het monitoren, registreren en analyseren van de status van informatiesystemen. Het uitgangspunt is dat veel van de grote beveiligingsincidenten worden veroorzaakt door kleine fouten en bekende kwetsbaarheden en dat als je die tijdig kunt vinden je systemen en infrastructuur een stuk veiliger worden. - -OpenKAT scant, verzamelt, analyseert en rapporteert in een permanent proces: - -image::https://user-images.githubusercontent.com/76487016/172068892-d8bb4552-5d4e-42d5-bd94-d1fb1b6d18b7.png[Flow van OpenKAT,640,] - -OpenKAT scant netwerken, analyseert kwetsbaarheden en maakt toegankelijke rapporten. Het integreert de meest gebruikte netwerktools en scansoftware in een modulair framework, heeft toegang tot externe databases zoals shodan en combineert de informatie uit al deze bronnen in overzichtelijke rapportages. - -Het pakket is nuttig als je een complex of omvangrijk systeem wilt monitoren en wilt weten of hierin bekende kwetsbaarheden of configuratiefouten te vinden zijn. Door de modulaire opbouw en uitbreidbaarheid is OpenKAT in een veelheid aan situaties toepasbaar, dat wil zeggen dat je het aan kunt passen en naar je eigen hand kunt zetten. - -= Documentatie - -Wil je meer weten over de werking van OpenKAT, lees dan de uitgebreide documentatie. Een toegankelijk startpunt is de link:https://github.com/minvws/nl-kat-coordination/wiki/Algemene-uitleg-OpenKAT-en-bijbehorende-figuren[informatiebrochure], waarin ook bovenstaand model wordt uitgelegd. Alle informatie over OpenKAT en de functionaliteit van het pakket is te vinden link:https://github.com/minvws/nl-kat-coordination/wiki[op de wiki]. - -== Actuele release - -De actuele release van OpenKAT zie je bij de releasetags op github. Op de link:https://github.com/minvws/nl-kat-coordination/wiki[wiki] tref je ook de releasenotes aan. - -== Hoe is OpenKAT ontstaan? - -Het Ministerie van VWS heeft de 'Kwetsbaarheden Analyse Tool' gebouwd om de eigen infrastructuur en het stelsel rondom CoronaCheck te monitoren. OpenKAT is door eigen programmeurs van het ministerie gebouwd. - -De dynamiek van de vaccinatiecampagne leverde voor de informatiebeveiliging een aantal kwesties op: - -* Schaal: - -Bij de vaccinatiecampagne zijn enorm veel organisaties betrokken waarmee informatie wordt uitgewisseld. - -* Dynamiek: - -COVID houdt zich niet aan een net en planbaar schema maar vroeg zeker in de eerste fase om directe actie. 's Avonds deed de minister een belofte in het parlement, na een nacht doorwerken werden de volgende ochtend weer nieuwe systemen of functies uitgerold. - -* Focus: - -De grootste veiligheidsincidenten in dit soort systemen worden veroorzaakt door bekende kwetsbaarheden, configuratiefouten en slordigheid. Ook foutjes bij een update of een nieuwe uitrol na de initiele pentest willen we kunnen vinden. - -=== Hoe lost OpenKAT deze kwesties op? - -Toen OpenKAT werd bedacht was vooral duidelijk dat er monitoring moest komen en wel geautomatiseerd, flexibel en herleidbaar. De opbouw van het systeem geeft een indicatie van de mogelijkheden: - -* Framework: - -OpenKAT is een framework dat ingezet kan worden voor informatieverzameling, opslag en verwerking. Het is zo flexibel dat 'de stukjes er bijna uitvallen': zo goed als alles wat gescheiden kan worden is dat ook. Zo kan er op nieuwe ontwikkelingen worden gereageerd en konden nieuwe functies worden opgenomen. - -* Plugins voor informatieverzameling: - -De 'Boefjes' halen informatie binnen: het zijn plugins varierend van een klein scriptje of scraper tot een externe tool die in een eigen container draait. Is er een nieuw issue dat nog niet wordt gedekt, dan maak je er een boefje voor dat de informatie ophaalt. - -* Forensische borging: - -De ruwe data wordt opgeslagen met een hash en een externe timestamp. Hierdoor is terug te halen welke informatie op welk moment is opgeslagen. Komen er nieuwe kwetsbaarheden voor een bepaalde software versie? Dan is die al in het systeem bekend en hoeft er niet apart gescand te worden. - -* Datamodel: - -Om alle inputs te verwerken wordt data omgezet in objecten, die passen in een vooraf uitgewerkt datamodel. Zo is een IP adres een object, dat via verschillende routes kan worden gevonden en logische relaties heeft met andere objecten. Het datamodel is uitbreidbaar. - -* Automatisch scannen: - -Het pakket zoekt zelf naar informatie, aan de hand van de logische relaties in het datamodel. De resultaten van de scans leiden weer tot nieuwe acties, net als het verstrijken van tijd leidt tot herhaling van eerdere scans. - -* Vrijwaringen per gebruiker en organisatie: - -De intensiteit van een scan wordt bepaald aan de hand van de beschikbare vrijwaring. OpenKAT kan genoeg tools aanspreken om een systeem zwaar te belasten en daarvoor is toestemming nodig. Is die er niet, dan kan altijd via 'derden' informatie worden verzameld zoals met shodan en vergelijkbare databases. - -* Bevindingen en rapportages: - -De resultaten van de analyse zijn eenvoudig te bekijken, per gebruiker, organisatie, object etc. Rapportages zijn beschikbaar voor veelvoorkomende vragen en eenvoudig uitbreidbaar. - -== Welke code bevat OpenKAT? - -OpenKAT omvat de volgende repositories: - -=== link:https://github.com/minvws/nl-kat-coordination[NL-KAT-Coordination] - -De centrale repo van OpenKAT bevat alle documentatie en informatie om OpenKAT zelf te installeren. - -=== link:https://github.com/minvws/nl-kat-mula[NL-KAT-mula] - -Mula is de scheduler, die in OpenKAT de boefjes aanstuurt. - -=== link:https://github.com/minvws/nl-kat-octopoes[NL-KAT-octopoes] - -Octopoes is het datamodel met alle objecten. Octopoes omvat ook de XTDB, waarin alle objecten zijn opgeslagen. - -=== link:https://github.com/minvws/nl-kat-rocky[NL-KAT-rocky] - -Rocky is de frontend van OpenKAT. Rocky maakt gebruik van Manon Open voor de scheiding van stijl en inhoud. - -=== link:https://github.com/minvws/nl-kat-bytes[NL-KAT-bytes] - -Bytes bevat de database met ruwe informatie en metadata, met externe signing voor de forensische borging. - -=== link:https://github.com/minvws/nl-kat-boefjes[NL-KAT-boefjes] - -Boefjes bevat twee onderdelen van OpenKAT: boefjes, de plugins die scans uitvoeren en whiskers, de normalizers die de data normaliseren en er objecten van maken. - -== Hoe kan ik OpenKAT installeren en gebruiken? - -OpenKAT kan direct worden geinstalleerd met behulp van link:https://github.com/minvws/nl-kat-coordination/wiki/Installatiehandleiding-KAT[de installatiehandleiding op de wiki]. De standaard installatie werkt in elk geval met Ubuntu en met MacOS X. Er zijn ook debian packages beschikbaar. - -Bij het bouwen van een productieomgeving bepalen de beschikbare bronnen en de toepassing hoe ver je de systemen splitst en schaalt. link:https://github.com/minvws/nl-kat-coordination/wiki/Infrastructuur-en-voorbeeldinstallatie[Voorbeelden van de installatiemogelijkheden] zijn beschikbaar. - -== Welke ondersteuning krijgt het project? - -OpenKAT is gebouwd door het Ministerie van VWS, Directie Informatiebeleid, programma Realisatie Digitale Ondersteuning. Dit is een tijdelijk programma in verband met de pandemie. De komende periode is er ondersteuning voor de doorontwikkeling van OpenKAT. Onder andere Z-Cert heeft ontwikkeltijd ter beschikking gesteld. Het team staat open voor samenwerking met gebruikers en andere partijen. - -== Wat levert dit op voor andere open source projecten? - -OpenKAT en onderdelen ervan kunnen onder de voorwaarden van de EU PL 1.2 licentie worden toegepast in andere projecten. Zo maakt de frontend gebruik van Manon-Open, een framework waarin content en styling zijn gescheiden en dat goed bruikbaar is voor andere projecten. Daarnaast is het mogelijk om OpenKAT te integreren in andere systemen. Het uitgangspunt is dat het als framework functioneert en aanpasbaar is aan verschillende situaties. - -= Licenties - -== Onder welke licentie is OpenKAT vrijgegeven? - -OpenKAT is beschikbaar onder link:https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12[de EU PL 1.2 licentie]. Deze licentie is gekozen omdat het een redelijke mate van vrijheid biedt, maar wel het publieke karakter waarborgt. De EU PL 1.2 licentie blijft behouden bij verdere verspreiding van de software. Wijzigingen en toevoegingen kunnen plaatsvinden onder de EU PL 1.2 licentie of onder verenigbare licenties, die een vergelijkbaar karakter hebben. - -De tools die door OpenKAT worden aangesproken kunnen hun eigen licentie hebben, uit het OS/S domein of vanuit commerciele toepassing. De eigenaar van het systeem dat deze tools aanspreekt is hier zelf verantwoordelijk voor. De opname van nieuwe boefjes in de KAT-alogus wordt geregeld in een aparte overeenkomst. - -== Plugins bouwen - -Het gebruik van plugins zoals boefjes (scraper), whiskers (normalizer) of bits (businessrule) die informatie uit andere tools analyseren maakt het mogelijk om systemen met een ander type licentie met OpenKAT te laten samenwerken. Plugins link:https://github.com/minvws/nl-kat-coordination/wiki/Plugins-maken:-Boefjes,-Whiskers-en-Bits[zijn eenvoudig te bouwen] en vallen onder de EU PL 1.2 licentie, voor zover je ze in de KATalogus wilt laten opnemen voor verdere verspreiding. OpenKAT als systeem kan hierdoor prima in een corporate omgeving functioneren. - -= Meedoen! - -== Hoe kan ik meedoen en meehelpen? - -Je kunt direct meedoen en betrokken zijn bij de ontwikkeling van OpenKAT: - -* Installeer het systeem en gebruik het, geef ons feedback -* Boef je eigen boefjes, whiskers en bits -* Help mee om het datamodel uit te breiden -* Stel nieuwe features voor -* Stuur link:https://github.com/minvws/nl-kat-coordination/issues[bugreports in als issue] -* Help mee met het beschikbaar maken van OpenKAT voor andere operating systems - -== Kunnen externe ontwikkelaars ook code toevoegen aan het project? - -Ja, dat is zeker de bedoeling van het openbaar maken van de broncode. We zijn op zoek naar mensen die willen meehelpen. In eerste instantie ligt de coordinatie van het project bij het ontwikkelteam bij het Ministerie van VWS, maar we staan open voor alle bijdragen. De opzet is om rond OpenKAT een community op te bouwen die de software gebruikt en helpt ontwikkelen, om er zo voor te zorgen dat het een goede bijdrage kan leveren aan de informatiebeveiliging. - -== Hoe kan ik wijzigingen zoals bugfixes, patches en nieuwe features toevoegen? - -Je kunt direct PR's insturen via Github, of contact opnemen met de community manager via meedoen@openkat.nl. - -OpenKAT hanteert de volgende uitgangspunten voor het schrijven van code: - -* python 3.8 -* Alle code via pullrequests met reviews -* link:https://peps.python.org/pep-0008/[Python met PEP8: ] -* Pylint -* link:https://pypi.org/project/black/[Black], 120 tekens regellengte: -* Type hinting -* Tests - -Op Github tref je een development branch aan. Hiervoor kunnen pull requests voor review worden aangeleverd. Op basis van de development branch wordt de main branch gevoed, ten behoeve van productiereleases. De reviews worden gedaan door VWS developers. - -Als je wilt dat je boefje wordt opgenomen in de KAT-alogus geldt er een aparte regeling, waar we je graag over vertellen. Stuur een mailtje naar meedoen@openkat.nl. - -== Ik run Arch/NetBSD/OpenVMS of iets anders leuks, hoe kan ik zorgen dat OpenKAT het ook op mijn systeem doet? - -OpenKAT gaat er vanuit dat je ubuntu of debian gebruikt, maar de community manager kreeg het onder Mac OS X zo aan de gang. Probeer het dus gerust, en help ons vooral met fixes en documentatie voor de installatie op je favoriete systeem! - -== Testen of ontwikkelen via GitPod == - -Via gitpod kan iedereen (met een github, gitlab account) snel een OpenKAT omgeving opstarten en testen. Tijdens deze installatie kun je zelf een gebruikersnaam en wachtwoord invoeren. - -https://gitpod.io/#github.com/minvws/nl-kat-coordination - -Eenmaal opgestart zal de Rocky interface beschikbaar zijn op de dienst die op poort 8000 draait. - -= Internationalisatie - -== In welke talen is OpenKAT beschikbaar? - -OpenKAT ondersteunt op dit moment de volgende talen: - -- Engels -- Nederlands -- Papiamento - -De meeste documentatie in de software zelf is in het Engels geschreven. De handleidingen en de wiki zijn in het Nederlands, maar willen we graag ook in andere talen beschikbaar maken. - -= Contact met het team - -Er een aantal opties om contact te leggen het met team van OpenKAT: - -* Direct contact: meedoen@openkat.nl -* Forum: link:https://github.com/minvws/nl-kat-coordination/discussions[Github Discussions] of de OpenKAT groep op link:https://www.linkedin.com/[Linkedin] -* IRC: #openkat op irc.libera.chat -* Signal: https://signal.group/#CjQKIIS4T1mDK1RcTqelkv-vDvnzrsU4b2qGj3xIPPrqWO8HEhDISi92dF_m4g7tXEB_QwN_ diff --git a/README.rst b/README.rst new file mode 100644 index 00000000000..74faa2d9261 --- /dev/null +++ b/README.rst @@ -0,0 +1,177 @@ +================ +What is OpenKAT? +================ + +OpenKAT aims to monitor, record and analyze the status of information systems. The basic premise is that many of the major security incidents are caused by small errors and known vulnerabilities, and that if you can find them in time your systems and infrastructure become a lot more secure. + +OpenKAT scans, collects, analyzes and reports in an ongoing process: + +.. image:: docs/source/introduction/img/flowopenkat.png + :alt: flow of OpenKAT + +OpenKAT scans networks, finds vulnerabilities and creates accessible reports. It integrates the most widely used network tools and scanning software into a modular framework, accesses external databases such as shodan, and combines the information from all these sources into clear reports. It also includes lots of cat hair. + +OpenKAT is useful if you want to monitor a complex system and want to know whether it contains known vulnerabilities or configuration errors. Due to its modular structure and extensibility, OpenKAT can be applied in a multitude of situations. You can customize it and put it to your own use. + +Documentation +============= + +Brochures +********* + +The high level documentation on OpenKAT explain the purpose and operation of OpenKAT at the management level: + +- `the 'TL;DR' of 2 pages (English) `_ +- `the extensive brochure on OpenKAT (Dutch) `_ + +Technical documentation +*********************** + +`The full documentation of OpenKAT can be found here `_. It includes information such as: + +- Introduction to the system +- Modules +- Guidelines +- Templates +- Technical documentation + +Principles behind OpenKAT +========================= + +The Ministry of Health, Welfare and Sport built the "Vulnerability Analysis Tool" to monitor systems during the pandemic. OpenKAT was built by the ministry's own programmers. Because of the scale and dynamics of the campaign, monitoring had to be automated, flexible and traceable. The structure of the system gives an indication of the possibilities: + +Framework +********* + +OpenKAT is a framework that can be used for information collection, storage and processing. It is so flexible that "the pieces almost fall out": just about everything that can be separated is. Thus, it can respond to new developments and new functions can be included. + +Plugins for information collection +********************************** + +The 'Boefjes' retrieve information: they are plugins ranging from a small script or scraper to an external tool running in its own container. If there is a new issue that is not yet covered, create a boefje for it that retrieves the information. + +Forensic assurance +****************** + +The raw data is stored with a hash and an external timestamp. This allows retrieval of what information was stored at what time. Are there new vulnerabilities coming out for a particular software version? Then it is already known in the system and no separate scanning is required. + +Data model +********** + +In order to process all inputs, data is converted into objects, which fit into a predetermined data model. For example, an IP address is an object, which can be found through various routes and has logical relationships with other objects. The data model can be extended to include all sorts of objects with logical relations. + +Automatic scanning +****************** + +The package itself searches for information, based on the logical relationships in the data model. The results of the scans in turn lead to new actions, just as the passage of time leads to repetition of previous scans. + +Clearances per user and organization +************************************ + +The intensity of a scan is determined by the indemnity available. OpenKAT can invoke enough tools to put a heavy load on a system and permission is required to do so. If there is none, information can always be gathered through "third parties" such as with shodan and similar databases. + +Findings and reports +******************** + +The results of the analysis are easy to view, by user, organization, object, etc. Reports are available for common questions and easily expandable. + +Current release +=============== + +The current release of OpenKAT can be found via the `release page on this repository `_. + +What code does OpenKAT contain? +=============================== + +OpenKAT includes the following repositories: + +:Overview: `NL-KAT-Coordination `_ + +:Mula: `NL-KAT-mula `_ + +:Octopoes: `NL-KAT-octopoes `_ + +:Rocky: `NL-KAT-rocky `_ + +:Bytes: `NL-KAT-bytes `_ + +:Boefjes and normalizers: `NL-KAT-boefjes `_ + +Which licence applies to OpenKAT? +================================= + +OpenKAT is available under the `EU PL 1.2 license `_. This license was chosen because it provides a reasonable degree of freedom while ensuring public character. The EU PL 1.2 license is retained upon further distribution of the software. Modifications and additions can be made under the EU PL 1.2 license or under compatible licenses, which are similar in nature. + +The tools addressed by OpenKAT may have their own license, from the OS/S domain or from commercial application. This is the responsibility of the owner of the system addressing these tools. The inclusion of new boefjes in the KAT catalog is governed by a separate agreement. + +Participate +=========== + +You can directly participate and be involved in the development of OpenKAT. There is a community around OpenKAT with active developers and organizations working on implementing their own OpenKAT setup. If you want to start slowly, there are nice options: + +- Install the system and use it, give us feedback +- Build your own boefjes, whiskers and bits +- Help extend the data model +- Suggest new features +- Submit `bugreports `_ as an issue +- Help make OpenKAT available for other operating systems + +Test or develop via GitPod +************************** + +Through gitpod, anyone (with a github, gitlab account) can quickly start up and test an OpenKAT environment. During this installation, you can enter your own username and password. + +`Gitpod test environment `_ + +Once started, the Rocky interface will be available on the service running on port 8000. + +Can I also add code? +******************** + +That is most welcome! The coordination of the project lies with the development team at the Ministry of Health, Welfare and Sport, which is open to all contributions. Please get in touch, there are many people working on OpenKAT and combined efforts make the whole system stronger. + +How can I add changes such as bug fixes, patches and new features? +****************************************************************** + +You can submit PRs directly via Github, or contact the community manager at meedoen@openkat.nl. Check out the templates and coding guidelines. + +OpenKAT uses the following principles for writing code: + +* python 3.8 +* All code via pullrequests with reviews +* `Python with PEP8 `_. +* Pylint +* `[Black], 120 characters line length: `_ +* Type hinting +* Tests + +On Github you will find a development branch. Pull requests can be submitted for review. Based on the development branch, the main branch is fed for production releases. The reviews are done by VWS developers. + +If you want your boefje to be included in the KAT catalog, a separate arrangement applies, which we would be happy to tell you about. Send an email to meedoen@openkat.nl. + +I run Arch/NetBSD/OpenVMS or something else +******************************************* + +How can I make sure OpenKAT works on my system? OpenKAT assumes you're running ubuntu or debian, but the community manager got it working under Mac OS X in no time. So feel free to try it and help us with fixes and documentation for installation on your favorite system! + +Internationalization +==================== + +OpenKAT currently supports the following languages: + +- English +- Dutch +- Papiamento + +Most of the documentation in the software itself is written in English. Some of the general documentation is in Dutch, but we would like to make it available in other languages as well. + +Contact +======= + +There several options to contact the OpenKAT team: + +- Direct contact: meedoen@openkat.nl +- `Github Discussions `_ +- `OpenKAT group on Linkedin `_ (search for OpenKAT) +- IRC: #openkat on irc.libera.chat +- `Signal group `_ diff --git a/docs/source/_static/openkat.css b/docs/source/_static/openkat.css index fdc2d565e28..9211486c8e2 100644 --- a/docs/source/_static/openkat.css +++ b/docs/source/_static/openkat.css @@ -1,3 +1,7 @@ -.wy-nav-content:has(.wy-table-responsive){ - max-width: fit-content; +.wy-nav-content:has(.wy-table-responsive) { + max-width: fit-content; +} + +.wy-side-nav-search > div.version { + color: hsla(0, 0%, 100%, .6); } \ No newline at end of file diff --git a/docs/source/conf.py b/docs/source/conf.py index de643641d4c..c4210356943 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -42,7 +42,12 @@ html_theme = "sphinx_rtd_theme" -html_theme_options = {"collapse_navigation": False} +html_logo = "keiko-hero.jpg" + +html_theme_options = { + "collapse_navigation": False, + "style_nav_header_background": "#ca005d", +} html_context = { "display_github": True, diff --git a/docs/source/guidelines/contributions.rst b/docs/source/guidelines/contributions.rst index 457de1fb1eb..863f93cbaaf 100644 --- a/docs/source/guidelines/contributions.rst +++ b/docs/source/guidelines/contributions.rst @@ -41,7 +41,7 @@ Contributing to the documentation benefits everyone who uses OpenKAT. We encourage you to help us improve the documentation, and you don't have to be an expert using OpenKAT to do so. There are many sections that are better off written by non-experts. If something in the docs doesn't make sense to you, updating the relevant section might be a great way to ensure it will help the next person. -You're welcome to propose edits to almost every text, including comments and docstrings in the code, the wiki, and other files. +You're welcome to propose edits to almost every text, including comments and docstrings in the code, this documentation, and other files. You could help us out with the following sections: diff --git a/docs/source/guidelines/development.rst b/docs/source/guidelines/development.rst index 6fafcbdae68..3e78ca4aab8 100644 --- a/docs/source/guidelines/development.rst +++ b/docs/source/guidelines/development.rst @@ -99,7 +99,7 @@ It has a simple syntax and many plugins available that should improve our test c Development Environment ======================= -See the `KAT wiki `_ for the overall installation instructions. +See :ref:`Developer or local install` for the overall installation instructions. In a development context, we strongly recommend to use the Docker setup to test and make changes in the codebase (and not production packages). When it comes to development there is no specific IDE that must be used, although many of us would choose PyCharm as the preferred IDE. @@ -153,4 +153,4 @@ These are automatically rendered by GitHub and the online Sphinx docs. Mermaid has support for things like PlantUML and ERD's. -### require time estimate and timeframe for an issue \ No newline at end of file +.. require time estimate and timeframe for an issue \ No newline at end of file diff --git a/docs/source/guidelines/project_statuses.md b/docs/source/guidelines/project_statuses.md index 915eda7c0c3..fcc35a236aa 100644 --- a/docs/source/guidelines/project_statuses.md +++ b/docs/source/guidelines/project_statuses.md @@ -1,40 +1,41 @@ # Project statuses -**Updated on 18-11-2022** +**Updated on 13-01-2022** | | | [rocky](https://github.com/minvws/nl-kat-rocky/) | [boefjes](https://github.com/minvws/nl-kat-boefjes/) | [bytes](https://github.com/minvws/nl-kat-bytes/) | [octopoes](https://github.com/minvws/nl-kat-octopoes/) | [mula](https://github.com/minvws/nl-kat-mula/) | [keiko](https://github.com/minvws/nl-kat-keiko/) | -|--------------------------------- |------------------------------------------------------------------------------------- |----------------------------------------------------------- |--------------------------------------------------------------- |----------------------------------------------------------- |----------------------------------------------------------------- |--------------------------------------------------------- |----------------------------------------------------------- | -| **GitHub Templates** | | | | | | | | -| | Feature Requests | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | -| | Bug Reports | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | -| | Pull Requests | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | -| **(Python) Code Quality Tools** | | | | | | | | -| | [General-purpose pre-commit hooks](https://github.com/pre-commit/pre-commit-hooks) | ❌ (awaiting PR) | ✅ | ✅ | ✅ | ❌ | ✅ | -| | [black](https://github.com/psf/black) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -| | [flake8](https://github.com/PyCQA/flake8) | 🟡 (Many checks are skipped) | ✅ | ❌ | ✅ | ❌ | ✅ | -| | [pylint](https://github.com/PyCQA/pylint) | ❌ (awaiting PR) | ❌ | ✅ | ✅ | ✅ | ✅ | -| | [pydocstyle](https://github.com/PyCQA/pydocstyle) | ❌ (awaiting PR) | ❌ | ❌ | 🟡 (Not yet automated in CI) | ❌ | ✅ | -| | [mypy](https://github.com/pre-commit/mirrors-mypy) strict, ignoring missing imports | ❌ (awaiting PR) | ❌ | ✅ | ✅ | ✅ | ✅ | -| | [vulture](https://github.com/jendrikseipp/vulture) with 90% min confidence | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | -| | [eradicate](https://github.com/myint/eradicate) | ❌ (awaiting PR) | ❌ | ✅ | ✅ | ❌ | ✅ | -| | [robotidy](https://github.com/MarketSquare/robotframework-tidy) | ✅ | ⚪️ | ⚪️ | ✅ | ⚪️ | ✅ | -| | [pyupgrade](https://github.com/asottile/pyupgrade) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | -| **Automated Testing** | | | | | | | | -| | Unit tests (for all supported Python versions) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -| | Integration tests | 🟡 (Not yet automated in CI) | ❌ | ✅ | ✅ | ✅ | ✅ | -| | Code coverage | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | -| | Dependabot | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -| **Automated Builds** | | | | | | | | -| | Debian packages | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -| | Docker container images | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -| **GitHub Workflow** | | | | | | | | -| | Linked to the central project board | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -| | README and/or Wiki are sufficient for a standalone setup | ✅ | | | | | | -| | Documentation is centralized in the README and/or Wiki | ✅ | | | | | | -| | Merge is not possible if automated checks are failing | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | -| | Merge requires approval from reviewer(s) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -| | Merge must be squashed or rebased | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | -| | Configured security policy and advisories | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -| | `.env` files are well-documented | ✅ (Mostly self-explanatory) | | | | | | -| | `Makefile` is used for helper/convenience functionalities | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -| | LICENSE - European Union Public License 1.2 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | \ No newline at end of file +|--------------------------------- |------------------------------------------------------------------------------------- |-------------------------------------------------- |------------------------------------------------------ |-------------------------------------------------- |-------------------------------------------------------- |------------------------------------------------ |-------------------------------------------------- | +| **GitHub Templates** | | | | | | | | +| | Feature Requests | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | +| | Bug Reports | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | +| | Pull Requests | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | +| **(Python) Code Quality Tools** | | | | | | | | +| | [General-purpose pre-commit hooks](https://github.com/pre-commit/pre-commit-hooks) | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | +| | [black](https://github.com/psf/black) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| | [flake8](https://github.com/PyCQA/flake8) | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | +| | [pylint](https://github.com/PyCQA/pylint) | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | +| | [pydocstyle](https://github.com/PyCQA/pydocstyle) | ❌ | ❌ | ❌ | 🟡 (Not yet automated in CI) | ❌ | ✅ | +| | [mypy](https://github.com/pre-commit/mirrors-mypy) strict, ignoring missing imports | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | +| | [vulture](https://github.com/jendrikseipp/vulture) with 90% min confidence | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | +| | [eradicate](https://github.com/myint/eradicate) | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | +| | [robotidy](https://github.com/MarketSquare/robotframework-tidy) | ✅ | ⚪️ | ⚪️ | ✅ | ⚪️ | ✅ | +| | [pyupgrade](https://github.com/asottile/pyupgrade) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| | [refurb](https://github.com/dosisod/refurb) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| **Automated Testing** | | | | | | | | +| | Unit tests (for all supported Python versions) | 🟡 (Not all Python versions are tested) | ✅ | ✅ | ✅ | ✅ | ✅ | +| | Integration tests | 🟡 (Not yet automated in CI) | ❌ | ✅ | ✅ | ✅ | ✅ | +| | Code coverage | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | +| | Dependabot | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| **Automated Builds** | | | | | | | | +| | Debian packages | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| | Docker container images | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| **GitHub Workflow** | | | | | | | | +| | Linked to the central project board | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| | README and/or Wiki are sufficient for a standalone setup | ✅ | | | | | | +| | Documentation is centralized in the README and/or Wiki | ✅ | | | | | | +| | Merge is not possible if automated checks are failing | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| | Merge requires approval from reviewer(s) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| | Merge must be squashed or rebased | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | +| | Configured security policy and advisories | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| | `.env` files are well-documented | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| | `Makefile` is used for helper/convenience functionalities | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| | LICENSE - European Union Public License 1.2 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | \ No newline at end of file diff --git a/docs/source/index.rst b/docs/source/index.rst index b3e1452cb62..3818fdc5e17 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -7,6 +7,8 @@ Welcome to the OpenKAT documentation! :maxdepth: 4 :caption: Contents + introduction/index guidelines/index templates/index - technical_design/index \ No newline at end of file + technical_design/index + modules/index diff --git a/docs/source/introduction/howdoesitwork.rst b/docs/source/introduction/howdoesitwork.rst new file mode 100644 index 00000000000..daeb2b1eff1 --- /dev/null +++ b/docs/source/introduction/howdoesitwork.rst @@ -0,0 +1,111 @@ +====================== +How does OpenKAT work? +====================== + +General notes +============= + +OpenKAT aims to monitor, record and analyze the status of information systems. OpenKAT scans networks, analyzes vulnerabilities and creates accessible reports. It integrates the most widely used network tools and scanning software into a modular framework, accesses external databases such as and combines the information from all these sources into clear reports. + +What OpenKAT adds to the available security and monitoring tools is the ability to combine the output from different sources for the purpose of analysis. Thanks to its object-oriented data model and forensically secured database, OpenKAT contains a complete overview and timeline of monitored systems. This makes the development through time insightful for analysis and provable for audits and controls. + +.. image:: img/stappenopenkat.png + :alt: steps in OpenKAT + +OpenKAT uses a configurable data model. All information is translated into objects, which are stored in the database. This contains both the original information and the objects. Analysis is done on the database based on business rules, noting changes. OpenKAT delivers findings in a dashboard or via reporting software. + + +Basic concepts +============== + +Central to OpenKAT are objects and the data model. Objects are created by collecting and analyzing information. The objects found are analyzed using business rules, leading to findings that are included as objects in the data model. + +The data model helps in the search for more information, through the logical coherence of objects. When an object is found, OpenKAT automatically checks whether related objects are also present. Based on this, it searches for information again, completing the circle. + +Objects, the data model and recursivity +*************************************** + +The information collected by OpenKAT is stored as objects. For example, an object is "an IP address" or "a hostname". If there is a hostname, based on the data model, OpenKAT also expects an IP address and possible open ports. + +Depending on the clearance or indemnification given this is then scanned for, which in turn yields more information, which in turn can trigger new scans. This process continues until OpenKAT has searched the entire data model for this hostname. How far OpenKAT goes in searching depends on the clearances. + +.. image:: img/flowopenkat.png + :alt: flow of OpenKAT + +The basic installation of OpenKAT includes a data model based on networks and information systems. The system has scanning software to map, analyze and report on networks. The strength of OpenKAT is that it is modular and easy to expand. This can be done based on the existing data model. It can also be extended to other domains, for example in the area of compliance, as long as there is a logical coherence of information. + +OpenKAT checks key aspects of configuration, accessibility and presence of vulnerabilities. In addition, the premise is that changes should be noticed: after all, a change is a peculiarity, which can signal a security risk. The temporal database used by OpenKAT also makes it possible to look back, that is, "from when to when" a particular situation occurred. + +Indemnification +*************** + +OpenKAT works with a system of indemnities for scanning, linked to intrusion levels. An organization gives an indemnification for a certain intensity of scan. For each intrusion level, the level is specified, in order to prevent unexpected problems for a production system. + +Intrusion levels or indemnities: +:L1: 'do not touch' +:L2: 'touch at normal user level'. +:L3: 'detectable scanning'. +:L4: 'intensive scanning'. + +If scanning with OpenKAT poses a risk then it applies to all actors who have access to this particular system and is already worthy of a finding. + +Users and organizations +*********************** + +Scanning and reporting in OpenKAT are different systems with separate users. The red team user issues the command and determines the safeguard, the reporting user has read access and views the results. + +:Red team user: Gives the system a certain command and safeguard ('scan this network, with this intrusion level'). Based on this, OpenKAT collects information. + +:Reporting user: Has read only access to the objects, can look through time at the scans done and see what findings the system has created. + +System build +============ + +The system has four parts: information collection, storage, analysis and reporting. + +.. image:: img/modulesopenkat.png + :alt: modules in OpenKAT + +Collection: Boefjes and Whiskers +******************************** + +Boefjes collect the information for OpenKAT. They are scripts that can call a tool or collect information themselves. Whiskers are the normalizers that try to filter out objects. These objects fit the data model being used. The scheduler "Mula" deploys boefjes depending on the query, the available clearance (the "intrusion level") and the information found. Thus, a scan leads to new data, which produces findings based on the business rules. This in turn can lead to new deployment of boefjes, who scan additional parts of a system. + +Storage: Bytes and Octopoes +*************************** + +Objects are stored in Octopoes, the database of objects accessible for analysis. Here objects can be viewed in logical context and over time. All original information, along with metadata, is stored separately in Bytes. This can be used to verify the creation of objects and findings. + +Bytes stores all original information including the full metadata in timestamped and signed records. This makes the observation by OpenKAT and the process that led to an object and any conclusions reproducible and traceable, which is also called forensic assurance. The metadata also includes, for example, the software version of OpenKAT, of the boefjes and of any external scanning tools, so that it is possible to trace why certain observations were or were not made. + +The standard installation of Octopoes includes a data model suitable for information security. This can be supplemented or adapted to the specific situation in which OpenKAT is used. There are already parties involved in the development phase that want to use OpenKAT for checking certain administrative aspects of certification, which fits well with the current application. + +Analysis: Bits +************** + +The objects in the database can be analyzed using business rules, which are included in Bits. For example, a list of open ports associated with an IP address is fine in one situation, but leads to a finding in another. A finding associated with a particular object is also stored as an object in Octopoes, and can lead to more scans or other actions. Bits, like Boefjes and Whiskers, are modular, customizable and easy to add. A finding based on a business rule can lead to additional scans or actions from OpenKAT. + +Reports +******* + +Reports can be created in a couple of ways. In the standard installation of OpenKAT, there are a number of options for creating reports: by object and thematically. For example, there are findings reports with all findings based on the business rules such as: + +- configurations +- old software +- ports +- missing headers +- SSL problems and certificates +- SPF and mail configuration + +Specific reports: + +- DNS reporting +- Internet.com (partial) +- SSL report with certificates + +GraphQL: + +- Simple input for queries +- Build complex queries yourself + +OpenKAT can generate reports in a number of formats, including LaTeX and PDF. An API is also available that can interface with other systems. diff --git a/docs/source/introduction/img/boefjeinfopage.png b/docs/source/introduction/img/boefjeinfopage.png new file mode 100644 index 00000000000..a703a9e3065 Binary files /dev/null and b/docs/source/introduction/img/boefjeinfopage.png differ diff --git a/docs/source/introduction/img/boefjes.png b/docs/source/introduction/img/boefjes.png new file mode 100644 index 00000000000..08f2d58b316 Binary files /dev/null and b/docs/source/introduction/img/boefjes.png differ diff --git a/docs/source/introduction/img/crisisroom.png b/docs/source/introduction/img/crisisroom.png new file mode 100644 index 00000000000..076068988b8 Binary files /dev/null and b/docs/source/introduction/img/crisisroom.png differ diff --git a/docs/source/introduction/img/findingdetail.png b/docs/source/introduction/img/findingdetail.png new file mode 100644 index 00000000000..8c60af0d1ca Binary files /dev/null and b/docs/source/introduction/img/findingdetail.png differ diff --git a/docs/source/introduction/img/findings.png b/docs/source/introduction/img/findings.png new file mode 100644 index 00000000000..475fca365e9 Binary files /dev/null and b/docs/source/introduction/img/findings.png differ diff --git a/docs/source/introduction/img/findingsreportperobject.png b/docs/source/introduction/img/findingsreportperobject.png new file mode 100644 index 00000000000..93b49583507 Binary files /dev/null and b/docs/source/introduction/img/findingsreportperobject.png differ diff --git a/docs/source/introduction/img/flowopenkat.png b/docs/source/introduction/img/flowopenkat.png new file mode 100644 index 00000000000..a9c6a022814 Binary files /dev/null and b/docs/source/introduction/img/flowopenkat.png differ diff --git a/docs/source/introduction/img/katalogus.png b/docs/source/introduction/img/katalogus.png new file mode 100644 index 00000000000..0fcd22cae99 Binary files /dev/null and b/docs/source/introduction/img/katalogus.png differ diff --git a/docs/source/introduction/img/landingpage.png b/docs/source/introduction/img/landingpage.png new file mode 100644 index 00000000000..a368c881ea2 Binary files /dev/null and b/docs/source/introduction/img/landingpage.png differ diff --git a/docs/source/introduction/img/modulesopenkat.png b/docs/source/introduction/img/modulesopenkat.png new file mode 100644 index 00000000000..d996e2af743 Binary files /dev/null and b/docs/source/introduction/img/modulesopenkat.png differ diff --git a/docs/source/introduction/img/normalizers.png b/docs/source/introduction/img/normalizers.png new file mode 100644 index 00000000000..745bb39d28a Binary files /dev/null and b/docs/source/introduction/img/normalizers.png differ diff --git a/docs/source/introduction/img/objectlist.png b/docs/source/introduction/img/objectlist.png new file mode 100644 index 00000000000..9b8bbbff128 Binary files /dev/null and b/docs/source/introduction/img/objectlist.png differ diff --git a/docs/source/introduction/img/report.png b/docs/source/introduction/img/report.png new file mode 100644 index 00000000000..71622c46911 Binary files /dev/null and b/docs/source/introduction/img/report.png differ diff --git a/docs/source/introduction/img/stappenopenkat.png b/docs/source/introduction/img/stappenopenkat.png new file mode 100644 index 00000000000..f434484b976 Binary files /dev/null and b/docs/source/introduction/img/stappenopenkat.png differ diff --git a/docs/source/introduction/index.rst b/docs/source/introduction/index.rst new file mode 100644 index 00000000000..e052762034b --- /dev/null +++ b/docs/source/introduction/index.rst @@ -0,0 +1,13 @@ +Introduction +############ + +Contains introduction into OpenKAT + +.. toctree:: + :maxdepth: 4 + :caption: Contents + + intro + howdoesitwork + usermanual + makeyourown diff --git a/docs/source/introduction/intro.rst b/docs/source/introduction/intro.rst new file mode 100644 index 00000000000..9875bbdbbf3 --- /dev/null +++ b/docs/source/introduction/intro.rst @@ -0,0 +1,76 @@ +============ +Introduction +============ + +What is OpenKAT +=============== + +OpenKAT aims to monitor, record and analyze the status of information systems. OpenKAT scans networks, analyzes vulnerabilities and creates accessible reports. It integrates the most commonly used network tools and scanning software into a modular framework, accesses external databases and combines the information from all these sources into clear reports. + +.. image:: img/flowopenkat.png + :alt: flow of OpenKAT + +Two brochures about OpenKAT have been created with a general explanation, both are in Dutch: + +- :download:`the 'TL;DR' of 2 pages (English) ` +- :download:`the extensive brochure on OpenKAT (Dutch) ` + +These, like the information in the high on documentation you are reading now, are aimed at a general understanding of the system. The documentation for each module goes into more detail about how the software itself works. + +If, when using OpenKAT, your computer gives a purring sound, it is just a sign of a well functioning instance of this software. Just stay away from the nails while playing. + +Securityconcept +=============== + +The premise behind OpenKAT is that most security incidents are caused by known vulnerabilities and configuration errors. Making mistakes is human, so you can't prevent it. Therefore, the goal is to find known vulnerabilities and configuration errors and fix them as quickly as possible. OpenKAT provides the tools for this. The Ministry of Health in the Netherlands made it publicly available under the EU PL 1.2 licence, to be applied as widely as possible. + +Who is OpenKAT for? +=================== + +OpenKAT is built to monitor a larger number of systems, such as the IT systems during the pandemic. Typical users are (network) organizations that want to monitor their systems, like Z-Cert does in healthcare or the Dutch Ministry of Health with the test providers that want to connect to CoronaCheck. + +The nicest playground for OpenKAT is a situation where many systems are active. In the user group around OpenKAT we see larger organizations from the non-profit sector, their service providers, hosting providers, auditors and others involved in information security. + +Where do I start with OpenKAT? +============================== + +The documentation explains how the system works and what the main principles are. This gives an impression, but after reading the documentation, trying it yourself is the best way to find out how OpenKAT works. In addition to the documentation, read `the readme from the general repository `_ (dutch). + +The easiest way to get to know the system is a local installation. If you don't have a debian or ubuntu machine (yet), try the Gitpod test environment. The installation chapter has a comprehensive roadmap for creating a local installation. + +Where is the software located? +============================== + +OpenKAT consists of separate modules that each perform a specific task. These are included in a number of repositories. + +OpenKAT includes the following repositories: + +:Overview: `NL-KAT-Coordination `_ + +:Mula: `NL-KAT-mula `_ + +:Octopoes: `NL-KAT-octopoes `_ + +:Rocky: `NL-KAT-rocky `_ + +:Bytes: `NL-KAT-bytes `_ + +:Boefjes: `NL-KAT-boefjes `_ + +The 'Modules of OpenKAT' section of the documentation goes into detail on each of these modules. + +Responsible disclosure +====================== + +OpenKAT scans for vulnerabilities. If you find any, it is valid that you deal with them properly. If you come across a vulnerability in a central government system you can report it to the `NCSC `_. + +Many organizations have their contact information in ``security.txt`` in the root of their domain, so you get straight to the right people. Not every organization handles it equally professionally, but that's no reason not to want to use that standard yourself. + +What are the plans for the future? +================================== + +OpenKAT was created during the pandemic. Publishing the source code is one way to give the software built during this period a longer life. With OpenKAT, the Ministry of Health is contributing to the `National Cybersecurity Strategy `_ (dutch) and supports the continued development of the system. + +Since the source code was published, 'OpenKAT days' have been organized regularly, the community around OpenKAT has grown, and developers from various other organizations are working on modules for the system. It is the first government project to be developed in this way. If you also want to help, contact the team at meedoen@openkat.nl. + +The long-term goal is for OpenKAT to play a permanent role in information security in healthcare and in the Netherlands as a whole. The system itself provides a good basis for this and its modular structure makes it easily adaptable to a specific context. Thanks to the EU PL 1.2 license, such contributions will be made available to all users as much as possible. diff --git a/docs/source/introduction/makeyourown.rst b/docs/source/introduction/makeyourown.rst new file mode 100644 index 00000000000..7a26b8b393e --- /dev/null +++ b/docs/source/introduction/makeyourown.rst @@ -0,0 +1,412 @@ +=============================================== +Plugins for OpenKAT: boefjes, whiskers and bits +=============================================== + +OpenKAT is modular and can be easily extended. This guide provides a first step for the development of new plugins: boefjes that scan, whiskers that collect objects, and bits that contain businessrules. + +OpenKAT comes with a KATalog of boefjes, which can be viewed through the front end of the system. The premise is that all information is processed and stored in the smallest unit, ensuring the modular nature of OpenKAT. + +OpenKAT can access multiple KATalogs, so it is possible to create your own overview of boefjes in addition to the official KATalog. This is of interest to organizations that use boefjes for specific purposes or with software that has a different licensing model. + +This guide explains how the plugins work and how they are created, and gives an overview of which plugins already exist. + +The community is working on sample boefjes with a ``enter your code here`` option and a repository with a prebuilt CI that provides boefjes as artifacts. Please send an email to meedoen@openkat.nl if you would like to participate in this. + +What types of plugins are there? +================================ + +There are three types of plugins, deployed by OpenKAT to collect information, translate it into objects for the data model and then analyze it. Boefjes gather facts, Whiskers structure the information for the data model and Bits determine what you want to think about it; they are the business rules. Each action is cut into the smallest possible pieces. + +- Boefjes gather factual information, such as by calling an external scanning tool like nmap or using a database like shodan. + +- Whiskers analyze the information and turn it into objects for the data model in Octopoes. + +- Bits contain the business rules that do the analysis on the objects. + +Boefjes and Whiskers are linked together through the mime-type that the boefje passes along to the information. For each mime-type, multiple Boefjes and Whiskers are possible, each with its own specialization. Thus, the data from a crook can be delivered to multiple whiskers to extract a different object each time. Bits are linked to objects and assess the objects in the data model. + +How does it work? +================= + +A hostname given as an object to OpenKAT, for example, is used as input to a search by the matching boefjes. Based on the data model, logically related objects are searched to get a complete picture. + +Thus, OpenKAT is like a snowball rolling through the network based on the data model. The logical connections between objects point the way, and OpenKAT keeps looking for new boefjes until the model is complete. + +The new objects in the data model are evaluated by Bits, the business rules. This produces findings, which are added as objects. For example, the hostname includes a dns configuration, which must meet certain requirements. If it goes outside the established parameters it leads to a finding. + +Where to start? +=============== + +The first question is what information you need. If you know this, there are a number of options, which determine what is best to do: + +- the information is already present in the data model -> create a businessrule (bit) +- the information is present in the output of an existing boefje -> create a normalizer (whiskers) +- the information is not yet available -> create a boefje, modify the data model and create a normalizer + +If you want to add factual information, use a boefje. Want to add an opinion or analysis, use a bit. + +OpenKAT assumes that you collect and process all information in the smallest possible units so that they can contribute back to other combinations and results. This is how you maintain the modular nature of the package. + +To make a finding about a CVE to a software version, you have a string of objects: the finding of the software, the version, the CVE. That combination then leads to the object of the finding. + +Existing boefjes +================ + +The existing boefjes can be viewed via the KATalog in OpenKAT and are on `GitHUB in the boefjes repository. `_ + +Example: the boefje for shodan +------------------------------ + +The boefje calling shodan gives a good first impression of its capabilities. The boefje includes the following files. + +- __init.py__, which remains empty +- boefje.json, containing the normalizers and objects in the data model +- cover.jpg, with a matching cat picture for the KATalog +- description.md, simple documentation of the boefje +- main.py, the actual boefje +- normalize.py, the normalizer (whiskers) +- normalizer.json, which accepts and supplies the normalizer +- requirements.txt, with the requirements for this boefje +- schema.json, settings for the web interface + +boefje.json +*********** + +boefje.json is the definition of the boefje, with its position in the data model, the associated normalizer, the objects and the findings that the combination of boefje and normalizer can deliver. + +The objects associated with this boefje are IPAddressV4, IPAddressV6, Finding, CVEFindingType. This boefje consumes IP addresses and produces findings about the open ports, supplemented by the information about these ports. + +Shodan comes with an API key, which you can add in the web interface. + +.. code-block:: + + { + "id": "shodan", + "name": "Shodan", + "description": "Use Shodan to find open ports with vulnerabilities that are found on that port", + "consumes": [ + "IPAddressV4", + "IPAddressV6" + ], + "produces": [ + "Finding", + "IPPort", + "CVEFindingType" + ], + "environment_keys": ["SHODAN_API"], + "scan_level": 1 + } + +Using the template as a base, you can create a boefje.json for your own boefje. The template starts with the name of your new boefje: + + +.. code-block:: + + { + "id": "boefje", + "name": "Boefje", + "description": "Beschrijving", + +Your boefje collects information to turn it into objects. Specify the objects your boefje needs. Those objects come from the data model. Should the information you want to retrieve not yet be incorporated into the data model, you need to modify it separately. How this works is described in general terms later in this document. + +.. code-block:: + + "consumes": [ + "object uit het datamodel", + "nog een object uit het datamodel" + ], + "produces": [ + "informatie", + "informatie" + ], + +The boefje can also bring variables from the web interface, like in Shodan the API key. There are more possibilities, you can be creative with this and let the end user bring settings from the web interface. + +.. code-block:: + + "environment_keys": ["SHODAN_API"], + "scan_level": 1 + + +schema.json +*********** + +To allow the user to add information through the web interface, add the schema.json file to the folder where your boefje is located. This json is used as the basis for a form for the user. In this case, it can contain an API key, but it can also be something else that your boefje responds to. This Schema must conform to the https://json-schema.org/ standard. + +Currently, however, OpenKAT only understands fairly shallow structures. For example, not all field types are supported, nor does OpenKAT understand references. You can test whether your Schema is neatly understood by checking the settings form in Rocky's KAT catalog for your boefje. + +.. code-block:: + + { + "title": "Arguments", + "type": "object", + "properties": { + "SHODAN_API": { + "title": "SHODAN_API", + "maxLength": 128, + "type": "string", + "description": "A Shodan API key (see https://developer.shodan.io/api/requirements)." + } + }, + "required": [ + "SHODAN_API" + ] + } + +main.py +******* + +The boefje itself imports the shodan api module, assigns an IP address to it and accepts the output. This output goes to Bytes and is analyzed by one (or more) normalizers. The link between the normalizer and the byte is made via the mime-type, which you can give in the ``set`` function in the byte. The code block below also contains a check, to prevent you from asking for non-public IP addresses. + +.. code-block:: + + import json + import logging + from typing import Tuple, Union, List + + import shodan + + from os import getenv + from ipaddress import ip_address + + from boefjes.job_models import BoefjeMeta + + + def run(boefje_meta: BoefjeMeta) -> List[Tuple[set, Union[bytes, str]]]: + api = shodan.Shodan(getenv("SHODAN_API")) + input_ = boefje_meta.arguments["input"] + ip = input_["address"] + results = {} + + if ip_address(ip).is_private: + logging.info("Private IP requested, I will not forward this to Shodan.") + else: + try: + results = api.host(ip) + except shodan.APIError as e: + if e.args[0] != "No information available for that IP.": + raise + logging.info(e) + + return [(set(), json.dumps(results))] + +Normalizers +----------- + +The normalizer imports the raw information, extracts the objects from it and gives them to Octopoes. Since OpenKAT 1.3.0, the normalizers are fully self-contained. They consist of the following files: + +- __init__.py +- normalize.py +- normalizer.json + +normalizer.json +*************** + +The normalizers translate the output of a boefje into objects that fit the data model. Each normalizer defines what input it accepts and what it provides. In the case of the shodan normalizer, it involves the entire output of the shodan boefje (created based on IP address), where findings and ports come out. The normalizer.json defines these: + +.. code-block:: + + { + "id": "kat_shodan_normalize", + "consumes": [ + "shodan" + ], + "produces": [ + "Finding", + "IPPort", + "CVEFindingType" + ] + } + +normalize.py +************ + +The file normalize.py contains the actual normalizer. From octopoes, the normalizer retrieves the objects and their references: from the findings list the CVEFindingType for the CVEs and the Finding for the findings, from the network objects list the IPPort, the Protocol and the PortState. Then the information about those objects is extracted from the imported data and stored as objects. + +.. code-block:: + + import json + from typing import Iterator, Union + + from octopoes.models import OOI, Reference + from octopoes.models.ooi.findings import CVEFindingType, Finding + from octopoes.models.ooi.network import IPPort, Protocol, PortState + + from boefjes.job_models import NormalizerMeta + + def run(normalizer_meta: NormalizerMeta, raw: Union[bytes, str]) -> Iterator[OOI]: + results = json.loads(raw) + ooi = Reference.from_str(normalizer_meta.boefje_meta.input_ooi) + + for scan in results["data"]: + port_nr = scan["port"] + transport = scan["transport"] + + ip_port = IPPort( + address=ooi, + protocol=Protocol(transport), + port=int(port_nr), + state=PortState("open"), + ) + yield ip_port + + if "vulns" in scan: + for cve, _ in scan["vulns"].items(): + ft = CVEFindingType(id=cve) + f = Finding(finding_type=ft.reference, ooi=ip_port.reference) + yield ft + yield f + +Adding objects +============== + +If you want to add an object, you need to know with which other objects there is a logical relationship. An object is as simple as possible. As a result, a seemingly simple query sometimes explodes into a whole tree of parts. + +Adding objects to the data model requires an addition in octopus. Here, an object can be added if it is connected to other objects. Visually this is well understood using the `Graph explorer `_. The actual code is `in the Octopoes repo `_. + +As with the boefje for shodan, here we again use the example from the functional documentation. A description of an object in the data model, in this case an IPPort, looks like this: + + +.. code-block:: + + class IPPort(OOI): + object_type: Literal["IPPort"] = "IPPort" + + address: Reference = ReferenceField(IPAddress, max_issue_scan_level=0, max_inherit_scan_level=4) + protocol: Protocol + port: conint(gt=0, lt=2 ** 16) + state: Optional[PortState] + + _natural_key_attrs = ["address", "protocol", "port"] + _reverse_relation_names = {"address": "ports"} + _information_value = ["protocol", "port"] + + +Here it is defined that to an IPPort belongs an IPadress, a Protocol and a PortState. It also specifies how scan levels flow through this object and specifies the attributes that format the primary/natural key: "_natural_key_attrs = ["address", "protocol", "port"]". More explanation about scan levels / indemnities follows later in this document. + +The PortState is defined separately. This can be done for information that has a very specific nature so you can describe it. + +.. code-block:: + + class PortState(Enum): + OPEN = "open" + CLOSED = "closed" + FILTERED = "filtered" + UNFILTERED = "unfiltered" + OPEN_FILTERED = "open|filtered" + CLOSED_FILTERED = "closed|filtered" + +Bits: businessrules +=================== + +Bits are businessrules that assess objects. Which ports are allowed to be open, which are not, which software version is acceptable, which is not. Does a system as a whole meet a set of requirements associated with a particular certification or not? + +In the hostname example, that provides an IP address, and based on the IP address, we look at which ports are open. These include some ports that should be open because certain software is running and ports that should be closed because they are not used from a security or configuration standpoint. + +The example below comes from the functional documentation and discusses the Bit for the IPPort object. The bit used for the analysis of open ports consists of three files: + +- __init.py__, an empty file +- bit.py, which defines the structure +- port_classification.py, which contains the business rules + +Bit.py gives the structure of the bit, containing the input and the businessrules against which it is tested. An example is included below. The bit accepts input belonging to the objects IPPort and IPAddress. It then calls the module port_classification, which contains the businessrules. + + +.. code-block:: + + from bits.definitions import BitParameterDefinition, BitDefinition + from octopoes.models.ooi.network import IPPort, IPAddress + + BIT = BitDefinition( + id="port-classification", + consumes=IPPort, + parameters=[], + module="bits.port_classification.port_classification", + ) + +The businessrules are contained in the module port_classification, in the file port_classification.py. This bit grabs the IPPort object and supplies the KATFindingType and Finding objects. The businessrules in this case distinguish three types of ports: the COMMON_TCP_PORTS that may be open, SA_PORTS that are for management purposes and should be closed, and DB_PORTS that indicate the presence of certain databases and should be closed. + +The specification for a bit is broad, but limited by the data model. Boefjes retrieve information externally, bits only look at the objects in Octopus. Analysis of the information can then be used to create new objects, such as the KATFindingTypes which in turn correspond to a set of specific reports in OpenKAT. + +.. code-block:: + + from typing import List, Iterator + + from octopoes.models import OOI + from octopoes.models.ooi.findings import KATFindingType, Finding + from octopoes.models.ooi.network import IPPort + + COMMON_TCP_PORTS = [25, 53, 110, 143, 993, 995, 80, 443] + SA_PORTS = [21, 22, 23, 3389, 5900] + DB_PORTS = [1433, 1434, 3050, 3306, 5432] + + + def run( + input_ooi: IPPort, + additional_oois: List, + ) -> Iterator[OOI]: + + port = input_ooi.port + if port in SA_PORTS: + open_sa_port = KATFindingType(id="KAT-560") + yield open_sa_port + yield Finding( + finding_type=open_sa_port.reference, + ooi=input_ooi.reference, + description=f"Port {port} is a system administrator port and should not be open.", + ) + + if port in DB_PORTS: + ft = KATFindingType(id="KAT-561") + yield ft + yield Finding( + finding_type=ft.reference, + ooi=input_ooi.reference, + description=f"Port {port} is a database port and should not be open.", + ) + + if port not in COMMON_TCP_PORTS and port not in SA_PORTS and port not in DB_PORTS: + kat = KATFindingType(id="KAT-562") + yield kat + yield Finding( + finding_type=kat.reference, + ooi=input_ooi.reference, + description=f"Port {port} is not a common port and should possibly not be open.", + ) + +Bits can recognize patterns and derive objects from them. The Bit for internet.nl can thus deduce from a series of objects whether a particular site meets the requirements of internet.nl or not. This bit retrieves findings from a series of items and draws conclusions based on them. The analysis underlying this is built up from small steps, which go around OpenKAT several times before enough information is available to draw the right conclusions. + +.. code-block:: + + from bits.definitions import BitParameterDefinition, BitDefinition + from octopoes.models.ooi.dns.zone import Hostname + from octopoes.models.ooi.findings import Finding + from octopoes.models.ooi.web import Website + + BIT = BitDefinition( + id="internet-nl", + consumes=Hostname, + parameters=[ + BitParameterDefinition(ooi_type=Finding, relation_path="ooi"), # findings on hostnames + BitParameterDefinition(ooi_type=Finding, relation_path="ooi.website.hostname"), # findings on resources + BitParameterDefinition(ooi_type=Finding, relation_path="ooi.resource.website.hostname"), # findings on headers + BitParameterDefinition(ooi_type=Finding, relation_path="ooi.hostname"), # findings on websites + BitParameterDefinition(ooi_type=Finding, relation_path="ooi.netloc"), # findings on weburls + BitParameterDefinition(ooi_type=Website, relation_path="hostname"), # only websites have to comply + ], + module="bits.internetnl.internetnl", + ) + +Add Boefjes +=========== + +There are a number of ways to add your new boefje to OpenKAT. + +- Put your boefje in the local folder with the other boefjes +- Do a commit of your code, after review it can be included +- Add an image server in the KAT catalog config file ``*`` + +``*`` If you want to add an image server, join the ongoing project to standardize and describe it. The idea is to add an image server in the KAT catalog config file that has artifacts from your boefjes and normalizers as outputted by the Github CI. + +The goal is to set up a separate Github repo with a complete CI to create artifacts based on a template boefje. You can clone this repo. Your OpenKAT installation points you to the artifacts so they are usable from your system. This is now being worked on by the OpenKAT community. Send an email to meedoen@openkat.nl if you want to help. (status: Dec. 2022) + diff --git a/docs/source/introduction/pdf/OpenKAT handout_ENG.pdf b/docs/source/introduction/pdf/OpenKAT handout_ENG.pdf new file mode 100644 index 00000000000..54094789292 Binary files /dev/null and b/docs/source/introduction/pdf/OpenKAT handout_ENG.pdf differ diff --git a/docs/source/introduction/pdf/introductie OpenKAT V20220621.pdf b/docs/source/introduction/pdf/introductie OpenKAT V20220621.pdf new file mode 100644 index 00000000000..e5930ccf117 Binary files /dev/null and b/docs/source/introduction/pdf/introductie OpenKAT V20220621.pdf differ diff --git a/docs/source/introduction/usermanual.rst b/docs/source/introduction/usermanual.rst new file mode 100644 index 00000000000..bd6e2f88a24 --- /dev/null +++ b/docs/source/introduction/usermanual.rst @@ -0,0 +1,259 @@ +========== +User Guide +========== + +This manual covers the day-to-day use of OpenKAT via the web interface. The concepts behind OpenKAT are explained in the "How does OpenKAT work" section. When using OpenKAT for the first time, the on-boarding flow is available, see the section in this chapter. + +.. image:: img/landingpage.png + :alt: landingpage + +Web interface +============= + +The user interface of OpenKAT consists of five screens, which provide access to the information and main functions of the system: + +- Crisis Room (main) +- KAT catalog +- Findings +- Objects +- Tasks + +Crisis Room +----------- + +The Crisis Room provides the overview of all findings, which can be viewed for different times. The time of day can be selected with the option button after which the findings that were applicable at that time become visible. + +.. image:: img/crisisroom.png + :alt: crisisroom + +KAT catalog +----------- + +The KAT catalog contains all the tools that this instance of KAT has access to, all the boefjes and normalizers. Click on a boefje for more information, such as the objects it can search for. + +Boefjes can be deployed automatically or manually. New boefjes can be added by the administrator, either locally or by adding an external KAT catalog in Rocky's config file. + +Automatic deployment of boefjes depends on the safeguard level, which can be set for each object. If no safeguard is set, it can be derived from a logically connected object for which it is. + +.. image:: img/katalogus.png + :alt: KAT catalog + +Each boefje has an info page with information about the tools used, the associated objects and the safeguard level required to use the boefje. + +.. image:: img/boefjeinfopage.png + :alt: Findings + +Findings +-------- + +The findings made by KAT can be seen on the Findings page. Use the filters to select the findings. Click on the finding for more information or to generate a report on this finding. + +.. image:: img/findings.png + :alt: Findings + +A finding is also an object in the data model, and can also be found on the objects page. + + +Objects +------- + +The Objects page lists all the objects in Octopus. For each object there is information about: + +- properties +- relationship with other objects +- findings +- level of safeguarding + +The objects page is a practical page to find information about a host or system. The objects can be filtered and a report per object can be easily created and exported. + +.. image:: img/findingsreportperobject.png + :alt: findings per object + +The object detail page provides more information about a specific object, such as the number of findings for this object. More information can be requested per finding: + +.. image:: img/findingdetail.png + :alt: finding detail + +Tasks +----- + +The scans of KAT can be found on the Tasks page as tasks. A task is created per boefje and per normalizer, with a status. Per task, the object is displayed and the json with metadata can be downloaded. This includes the hash of the scan performed. + +.. image:: img/boefjes.png + :alt: tasks + + +Users and organizations +======================= + +OpenKAT has administrators, users and organizations. + +Organizations +------------- + +Organizations own the systems for which KAT is deployed. From KAT, multiple organizations can be monitored simultaneously, each with its own settings. The 1.4rc2 includes additional options for creating new organizations via an API. Please contact meedoen@openkat.nl if you would like to help test and develop this. + +Users +----- + +The administrator is responsible for the system. Besides the administrator two usertypes have been defined: the red team user who can launch new scans and the regular user who has read access and can request reports. + +User management +--------------- + +Users and organizations can be created in the on boarding flow, in the Web interface or automated. The administrator of the system can create organizations and do user management. The administrator of an organization in turn can create users within the organization. The django interface provides additional capabilities for user management via the command line, for use in an automated deployment and linkage to external user management. + +OpenKAT Objects +=============== + +Adding an initial object with an appropriate safeguard puts OpenKAT to work. This can be done in on-boarding, but objects can also be added individually or as CSV files. + +Properties +---------- + +Objects can be viewed via the 'Objects' page in OpenKAT's main menu. Here are the already created objects with the type and safeguard level for each object. Objects can be added, scanned, filtered and there is an export option. + +New objects can be created via the 'add' option. This can be done individually or per CSV. The specification of the CSV is included on the page where it can be provided. + +Start scan +---------- + +Based on the object and the clearance, OpenKAT provides an overview of available boefjes. All users can perform a manual scan appropriate to the given safeguard level. The manual scan is accelerated by the scheduler. The results appear as findings with the object. + +View Findings +------------- + +Findings appear on the general findings page, but can also be viewed by object. + + +Scan levels and indemnities +=========================== + +Boefjes can collect information with varying intensity. OpenKAT has a system of safeguards to control permission to perform scans and prevent damage to the systems under test. + +For each object, the 'indemnification level' menu indicates how deeply scanning is allowed. Here the user gives an agreement on the risks of the scans and permission to store the information gathered on these systems. + +The levels used range from level 0 to level 4, from 'do not scan' to 'very intrusive'. Scanning levels are distributed in the data model, either by inheritance or by user statements. The different levels are qualitative in nature. L1 'do not touch' is obvious, but the difference between L2 'normal user' and L3 'detectable scanning' is at the discretion of the developer and administrator. The use of NMAP, for example, falls in between and depends heavily on the arguments the tool brings. + +.. list-table:: Scan levels + :widths: 25 50 + :header-rows: 1 + + * - Level + - Description + * - L0 + - do not scan + * - L1 + - do not touch + * - L2 + - normal user + * - L3 + - detectable scan + * - L4 + - intensive scan + + +Indemification by user +---------------------- + +The user's statement counts as an indemnification for scanning a particular object. This obtains permission to scan and store the information. The statement is given at the start of a new scan or specifically for certain objects. + +Inheritance +----------- + +Objects are linked to other objects in the data model. Underlying objects receive the same safeguard level, parent objects a lower level. For example, a hostname has an ip address for which the same safeguard level applies, but it also has a DNS server that may be outside the organization's domain and receives a lower level. + +Extended profiles +----------------- + +L0: Do not scan +*************** + +The user can explicitly indicate that certain systems should not be scanned. For example, because he is not the owner of these. + +L1: Do not touch +**************** + +OpenSource and passive data collection. For this profile, objects are viewed through various freely available data and sources via the Internet. These can be sources that do not have explicit permission (e.g. LinkedIn, DNS, leaked password databases). The goal here is to detect public information that could be a risk to the client: information that could be misused by an attacker in a targeted attack. + +Examples of sources/tools used: + +- Shodan (via API) +- HaveIbeenPnwed +- DNS + +L2: Touching at the normal user level +************************************* + +Targeted scans, limited intrusive. Scan will be dosed and skip known sensitive scans. The scanned target usually continues to function without problems. + +Example of scanning tools useful for this purpose: + +- Nmap +- Nikto +- Burp passive scanner + +L3: Detectable scan +******************* + +This scan will be more intrusive: connect to services to find out versions, try to log in with commonly used (default) login credentials, automated testing of found vulnerabilities whether they are vulnerable, more intensive guessing of urls and more intensive crawling of web pages. + +A greater number of scans will be performed, resulting in a spike in data traffic. The infrastructure may not be designed for this. + +Example of useful scanning tools and methods: + +- Nessus, Nexpose, Acunetix +- Burp Intruder, active scanner + +L4: Intensive scan +****************** + +The premise of the test profile is to verify whether an attacker can exploit vulnerabilities to give himself more extensive access to the tested environment. Thus, known exploit code is applied in this level. + +Reports +======= + +OpenKAT displays all findings in the crisis room, the entry point for all current information from the system. In addition, OpenKAT can create thematic reports and display findings per object. The reports are available in the front end and as PDF, based on a LaTeX parser. The organization's house style can also be incorporated. It is also possible to link to other reporting and alerting systems. + +.. image:: img/report.png + :alt: Report + +My first scan +============= + +If you are using OpenKAT for the first time you can use the on-boarding flow. The on-boarding flow helps you through the full cycle of OpenKAT. After following this flow, you will have a functioning OpenKAT installation running a first set of scans. By adding more objects, releasing and selecting boefjes, you can find out more information and perform analysis. + +The on-boarding flow uses the following steps to get you going: + +- Create admin account with 2FA + +The administrator account in the front end uses a login, password and two-factor authentication with one-time passwords. The code for creating the one time passwords is available as a string and as a QR code. + +- Organization creation + +The organization is the entity that "owns" the systems to be scanned and on whose behalf the user can provide an indemnification. From an OpenKAT installation, multiple organizations can be scanned, each with its own settings and its own objects. Organizations can be created automatically from release 1.5 on the basis of an API, which is relevant for larger systems. + +- User creation + +Users in OpenKAT are the red team and the read-only user. + +- Choosing a report ("what question do you ask OpenKAT?") + +OpenKAT starts with a question, for example about the situation around the DNS configuration of a particular domain. For this, choose the relevant report. + +- Creating an object ('what should OpenKAT look at first?') + +Add the objects that OpenKAT can take as a starting point for the scan, for example a hostname. + +- Specify clearance level ('how intensive should OpenKAT search?') + +Specify the intensity of the scan: how intensely may OpenKAT scan? The heavier, the greater the impact on the system being scanned. + +- Select boefjes and have OpenKAT scan them + +Based on the report, object and safeguard, select the relevant boefjes for your first scan and run the scan. + +- View results: in the web interface or as a PDF report + +The scan is an ongoing process, looking for information based on derivation and logical connections in the data model. The results of the scan appear over time, any findings can be viewed by object, at Findings and in the Crisis Room. In each context, reports can also be generated. + diff --git a/docs/source/keiko-hero.jpg b/docs/source/keiko-hero.jpg new file mode 100644 index 00000000000..25e2b7c722f Binary files /dev/null and b/docs/source/keiko-hero.jpg differ diff --git a/docs/source/modules/index.rst b/docs/source/modules/index.rst new file mode 100644 index 00000000000..3d6e8ce799d --- /dev/null +++ b/docs/source/modules/index.rst @@ -0,0 +1,10 @@ +Modules +####### + +Contains references to the documentations of the modules of OpenKAT + +.. toctree:: + :maxdepth: 4 + :caption: Contents + + modules diff --git a/docs/source/modules/modules.rst b/docs/source/modules/modules.rst new file mode 100644 index 00000000000..1b2b2165e59 --- /dev/null +++ b/docs/source/modules/modules.rst @@ -0,0 +1,58 @@ +======= +Modules +======= + +OpenKAT consists of individual modules, each of which performs a subtask of the system. The modules are specific to OpenKAT. Rocky is the frontend, Mula is the scheduler, Bytes the storage of raw data, Octopoes contains the data model, Keiko is the PDF engine for reports and Boefjes and Whiskers and are separate components stored in the KATalogus. OpenKAT uses Manon for the design in order to easily comply with accessibility and style requirements. + +The source code and technical documentation is included for each module in its own github repository. This document refers to these repos and their documentation. If you have any questions, please contact the team, see 'Contact' in the readme of the NL-KAT-Coordination repo. + +Rocky +===== + +Rocky is the front end for OpenKAT and based on a Django framework. Through Rocky, OpenKAT can be accessed. It provides a user interface for all information present in the system. It is also possible to access OpenKAT only through the APIs. + +`NL-KAT-rocky `_ is the repository containing Rocky's source code and documentation. + +Mula +==== + +Mula is the scheduler, which controls OpenKAT. Mula has extensive technical documentation and design schematics. + +`NL-KAT-mula `_ is the repository containing Mula's source code and documentation. + +Octopoes +======== + +Octopoes contains the data model and the XTDB with all objects. The documentation accompanying Octopoes covers the technical side of the logic behind OpenKAT and includes extensive information for people who want to work with the data model themselves. + +`NL-KAT-octopoes `_ is the repostory containing the source code and documentation of Octopoes. + +Bytes +===== + +Bytes is a relatively simple datastore for the raw data provided by OpenKAT. In addition to storage, Bytes also handles assurance by having a timestamp created from the stored data. + +`NL-KAT-bytes `_ is the repository containing Bytes' source code and documentation. + +Boefjes and whiskers +==================== + +The KATalog contains all boefjes and whiskers. The KATalog is easy to add new boefjes and normalizers. The general documentation contains some pointers for writing boefjes. The team is happy to help you if you want to get started with these. The readme of `NL-KAT-Coordination repo `_. contains all the ways to contact the team. + +`NL-KAT Boefjes `_ is the repository containing the source code and documentation of Boefjes, Whiskers and Bits. + +Keiko +===== + +Keiko is OpenKAT's PDF generator, used for generating reports. The design of the reports is done based on a LaTeX template, making the style of the reports customizable. + +`NL-KAT-Keiko `_ is the repository containing Keiko's source code. + +Manon +===== + +Content and form are separated in Rocky. The form is contained in Manon, which as far as OpenKAT is applied in the context of the central government helps to comply with the central government house style. For Web sites outside this context, Manon is also applicable, but with a different style. + +`NL-RDO-Manon `_ is the repository containing Manon's source code. + + diff --git a/docs/source/technical_design/hardening.rst b/docs/source/technical_design/hardening.rst new file mode 100644 index 00000000000..7a98601fcb4 --- /dev/null +++ b/docs/source/technical_design/hardening.rst @@ -0,0 +1,117 @@ +================= +Hardening OpenKAT +================= + +Hardening is making your environment secure. The default installation of OpenKAT is suitable for local use. If you are installing the software in a production environment, make sure you are running a secure configuration. The following modifications are a first step: + +Password Django installation +============================ + +In the django config (/admin on rocky), change the default password of the admin:admin user. + +SESSION_COOKIE_AGE +================== + +Rocky is the web interface. It sets cookies for sessions of 1209600 seconds, or 2 weeks. This is quite long and can be adjusted with a new entry in rocky/settings.py. +Sessions are limited to 2 hours based on the default configuration. + +Security headers +================ + +Rocky expects a reverse proxy that can handle TLS. This is a good place to set the security headers: + ++-------------------------------------------+------------------------------------------+ +| Header | Settings | ++-------------------------------------------+------------------------------------------+ +| X-Frame-Options | Deny | ++-------------------------------------------+------------------------------------------+ +| X-Content-Type-Options | nosniff | ++-------------------------------------------+------------------------------------------+ +| Content-Security-Policy | default-src 'self'; object-src 'none'; | +| | child-src 'self'; frame-ancestors 'none';| +| | upgrade-insecure-requests; | +| | block-all-mixed-content | ++-------------------------------------------+------------------------------------------+ +| X-Permitted-Cross-Domain-Policies | none | ++-------------------------------------------+------------------------------------------+ +| Referrer-Policy | no-referrer | ++-------------------------------------------+------------------------------------------+ +| Clear-Site-Data | "cache","cookies","storage" (Opt. bij | +| | reverse NGINX proxies, "cookies" weglaten| ++-------------------------------------------+------------------------------------------+ +| Cross-Origin-Embedder-Policy (COEP) | require-corp | ++-------------------------------------------+------------------------------------------+ +| Cross-Origin-Opener-Policy (COOP) | same-origin | ++-------------------------------------------+------------------------------------------+ +| Cross-Origin-Resource-Policy (CORP) | same-origin | ++-------------------------------------------+------------------------------------------+ +| Permissions-Policy | accelerometer=(),autoplay=(),camera=(), | +| | display-capture=(),document-domain=(), | +| | encrypted-media=(),fullscreen=(), | +| | geolocation=(), gyroscope=(), | +| | magnetometer=(), microphone=(), midi=(), | +| | payment=(), picture-in-picture=(), | +| | publickey-credentials-get=(), | +| | screen-wake-lock=(),sync-xhr=(self), | +| | usb=(),web-share=(), | +| | xr-spatial-tracking=() | ++-------------------------------------------+------------------------------------------+ +| Cache-Control | no-store, max-age=0 | ++-------------------------------------------+------------------------------------------+ +| Pragma | no-cache | ++-------------------------------------------+------------------------------------------+ +| X-DNS-Prefetch-Control | off | ++-------------------------------------------+------------------------------------------+ +| Expect-CT | max-age=86400, enforce | ++-------------------------------------------+------------------------------------------+ + +SSL/TLS on nginx +================ + +Use the following versions and settings for SSL/TLS on nginx: + +- ssl_protocols TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE +- ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384; +- ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0 +- ssl_session_tickets off; # Requires nginx >= 1.5.9 +- ssl_stapling on; # Requires nginx >= 1.3.7 +- ssl_stapling_verify on; # Requires nginx => 1.3.7 +- ssl_prefer_server_ciphers on; +- ssl_session_timeout 10m; +- ssl_session_cache shared:SSL:10m; + +Optional use of HSTS, including for subdomains. + +``add_header Strict-Transport-Security "max-age=31104000;" always;`` + +Obscuring errors to the clients +=============================== + +By default, errors in OpenKAT are visible to the user. The reverse proxy can restrict this and return a generic error message. + +``proxy_intercept_errors on; +error_page 500 502 503 504 /error.html;`` + +In addition, it makes sense to prevent the proxy itself from sending its version along with each response: + +``server_tokens off;`` + +Web Application Firewall +======================== + +Installing KAT (rocky) behind a WAF provides an additional layer of security. Modsecurity, which is part of the reverse proxy, can be used for this purpose. More information can be found on the project's Github page: + +- https://github.com/SpiderLabs/ModSecurity +- https://github.com/SpiderLabs/ModSecurity-nginx + +Continue reading +================ + +Much more information is available on this topic. When applying OpenKAT in a production environment, the following links offer a first step: + +- https://developer.mozilla.org/en-US/docs/Learn/Server-side/Django/web_application_security +- https://owasp.org/www-project-secure-headers/ +- https://docs.djangoproject.com/en/4.0/topics/security/ + + + diff --git a/docs/source/technical_design/img/infraopenkat.png b/docs/source/technical_design/img/infraopenkat.png new file mode 100644 index 00000000000..f23c82a864c Binary files /dev/null and b/docs/source/technical_design/img/infraopenkat.png differ diff --git a/docs/source/technical_design/index.rst b/docs/source/technical_design/index.rst index d7e248e0351..e76ba5d60c5 100644 --- a/docs/source/technical_design/index.rst +++ b/docs/source/technical_design/index.rst @@ -7,4 +7,7 @@ Contains documentation for developers and contributors. :maxdepth: 4 :caption: Contents - containers \ No newline at end of file + install + containers + localinstall + hardening diff --git a/docs/source/technical_design/install.rst b/docs/source/technical_design/install.rst new file mode 100644 index 00000000000..3028844f199 --- /dev/null +++ b/docs/source/technical_design/install.rst @@ -0,0 +1,37 @@ +================== +Installation Guide +================== + +OpenKAT can be installed in a number of ways. You can use OpenKAT in a way that suits your situation. For developers and for introductory purposes, there is a local installation, there are debian packages that are automatically built from each release, there are pre-built docker containers, and there is community work on kubernetes and nomad scripts. At VWS, OpenKAT is installed based on ansible script. Use the hardening guide for a production install. + +Pre-built Docker images +======================= + +The pre-built docker images are on the Github Container Registry. The deployment manual based on the pre-built docker images is included in this manual. A kubernetes script is being worked on for automatic deploy. + +make kat +======== + +The 'developer option' or local install of OpenKAT, which builds the system from the source using docker containers. The manual for this install explains how to set up your computer for an install of OpenKAT. + +Debian packages +=============== + +The Debian packages are now available as artifacts from the Github actions in the repository of each module. There is an unofficial installation script to merge them into a working installation, and a repository to enable ``apt install kat`` is under construction. + +There is a beta community for the debian packages. If you want to get involved in this, send an email to meedoen@openkat.nl. + +Example infrastructure +====================== + +A larger installation of KAT can scale both horizontally and vertically. The setup depends on your own preferences. We do not have an estimate for your hardware planning, but most work is done by Mula and Octopoes. + +An example is shown in the diagram below. OpenKAT runs behind a proxy with firewalls, with Rocky accessible as the front end. From rocky, a second proxy connects to OpenKAT's components. Most components can be duplicated to distribute load and ensure availability. The arrows in the drawing indicate the direction in which the connections are initiated. + +.. image:: img/infraopenkat.png + :alt: Infra example of OpenKAT + +At the backend, a management interface can be added, adapted to the situation where OpenKAT is used. Each module has a healthpoint for monitoring. + +Backups are particularly relevant for the raw data in Bytes and the userdata, possibly for speed the objects in Octopoes. Based on Bytes and the userdata, the system can in principle be redeployed and restored. + diff --git a/docs/source/technical_design/localinstall.rst b/docs/source/technical_design/localinstall.rst new file mode 100644 index 00000000000..64d4d196e42 --- /dev/null +++ b/docs/source/technical_design/localinstall.rst @@ -0,0 +1,175 @@ +========================== +Developer or local install +========================== + +make kat +======== + +Install OpenKAT on your own machine with ``make kat`` or ``make kat-stable``. KAT-stable, as the name implies, is the last major release. If you want to deploy OpenKAT in a production environment use the hardening settings as well. + +Requirements +------------ + +You need the following things to install OpenKAT: + +- A computer with a Linux installation. In this document we use Ubuntu, but on many other distributions it works in a similar way. Later we will also add instructions for macOS. +- Docker. If you don't already have this, install it first in Chapter 2. + +- OpenKAT's `GitHub repository: `_ + +Before installing +----------------- + +OpenKAT is installed in Docker, and therefore Docker must be installed first. Do this according to the instructions of your (Unix) operating system. You can read the instructions for Ubuntu below. On the `website of Docker `_ you will see installation instructions for other distributions. + +Docker install +************** + +Open a terminal of your choice, such as gnome-terminal on Ubuntu. + +- We won't assume you have older versions of Docker running, but if you do, you need to uninstall them with the following command: + +.. code-block:: + + $ sudo apt-get remove docker docker-engine docker.io containerd runc yarnpkg + +If apt-get does not run through because of missing packages, try the command again without the name that apt-get stumbled over. + +- Then install some required packages that allow *apt* to use packages over HTTPS: + +.. code-block:: + + $ sudo apt-get install apt-transport-https ca-certificates curl gnupg lsb-release + +The packages are checked and updated as needed. If an installation is required, apt asks if you want to continue with the installation ("Do you want to continue?"). Type 'Y' and press enter. + +-Next add Docker's official GPG key: + +.. code-block:: + + $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg + $ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null + + +- Update your packages and install the latest Docker version: + +.. code-block:: + + $ sudo apt-get update + $ sudo apt-get install docker-ce docker-ce-cli containerd.io + $ sudo usermod -aG docker ${USER} + + +When asked if you want to continue the installation, type 'Y' again and press enter. + +Install docker-compose +********************** + +Install the latest version of docker-compose and give the tool appropriate permissions with the following two commands: + +.. code-block:: + + $ sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose + $ sudo chmod +x /usr/local/bin/docker-compose + + +Install dependencies +******************** + +Dependencies are packages required for OpenKAT to work. Run the following commands to install them: + + +.. code-block:: + + $ curl -fsSL https://deb.nodesource.com/setup_16.x | sudo -E bash - + $ sudo apt-get install -y nodejs gcc g++ make python3-pip docker-compose + $ curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | gpg --dearmor | sudo tee /usr/share/keyrings/yarnkey.gpg >/dev/null + $ echo "deb [signed-by=/usr/share/keyrings/yarnkey.gpg] https://dl.yarnpkg.com/debian stable main" | sudo tee /etc/apt/sources.list.d/yarn.list + $ sudo apt-get update && sudo apt-get install yarn + +Getting Started +--------------- + +Now the installation of OpenKAT can begin. We do this via git. + +Default installation +********************* + +- Clone the repository: + +.. code-block:: + + $ git clone https://github.com/minvws/nl-kat-coordination.git + +- Go to the folder: + +.. code-block:: + + $ cd nl-kat-coordination + +- Make KAT: + +.. code-block:: + + $ make kat + +Other options are "make clone" and "make pull" to either clone only or update the repositories. the above command performs this itself. + +Currently, the make cat instruction only works for the first user on a ``*nix`` system. This is a known problem which will be solved soon. The current user must be user 1000. You can check this by executing `id`. + +In some cases this may not work because Docker does not yet know your user name. You solve this with the following commands, entering your user name instead of $USER: + +.. code-block:: + + $ sudo gpasswd -a $USER docker + $ newgrp docker + +Then OpenKAT is built, including all the parts such as Octopoes and Rocky. + +Specific builds +*************** + +If you want to create a specific build, you have a number of options. You can also look in the `Makefile `_. Below are some examples. + +- Clone only relevant repositories + +.. code-block:: + + $ make clone + +- Start a separate container + +.. code-block:: + + $ docker-compose up --build -d {container_name} + + Set up a superuser with custom credentials (fill in the parameters as preferred for your installation) + + +By default a user named 'admin', with the password 'admin' should be available. + +- Optional seed of the database with OOI information + +.. code-block:: + + $ docker exec -it nl-kat-coordination_rocky_1 python3 /app/rocky/manage.py loaddata OOI_database_seed.json + +- install octopus-core in your local python environment with a symlink (after cloning) + +.. code-block:: + + $ pip install -e nl-kat-coordination-octopoes-core + +Updates +------- + +Updating an existing installation can be done with the new make update. + +Go to the directory containing openkat: + +.. code-block:: + + $ cd nl-kat-coordination + $ make update + +Create a new superuser for the new version. You can delete the old superuser after the update. This is not pretty, but has the advantage that your databases remain intact. Check that you are on the most recent version everywhere, especially Rocky sometimes hangs because of yarn.lock.