Towards better inference: bits → nibbles

docs/source/developer-documentation/octopoes.md

@@ -650,3 +650,41 @@ The unit tests `octopoes/tests` are run using
 ```shell
 python -m unittest discover octopoes/tests
 ```
+
+## Nibbles
+
+**What are nibbles?**
+Nibbles are the flexible replacement for bits. We learned a lot on what does and doesn't work around bits and ran into some limitations. The nibbles are solving these limitations. We hope that the nibbles are 4 times as efficient as bits, but this isn't officially bench marked (and is more of a bits-nibbles joke ;)).
+
+Nibbles process normalized latent data in OpenKAT, they correlate this data and try to extract as much information as possible, based on the specified rules. Nibbles ensure for a mapping of one or more objects to another bunch of objects. Where bits would trigger on objects, nibbles can trigger on sets of objects. This makes it possible to map a single object to N objects (1 to N objects), but also map multiple objects to multiple objects (N to M objects).
+
+**Limitation example of bits**
+One example of the limitations from bits was that if one of the bits (business rules) would identify an open port in network A, and for network B this port would identify as closed, how would you classify the port? With bits it was not possible to correctly identify data mismatches and decide upon this. Nibbles allow you to detect these differences in data observations. If there are any inconsistencies, this could other nibbles to be triggered to resolve the inconsistency.
+
+**How are nibbles better than bits?**
+Nibbles are a successor for bits, all the mistakes and issues we learned from bits should be solved with nibbles.
+
+- Dynamic loading from the user interface. This allows a user to enable, run or add nibbles from the web interface, instead of having to edit the code.
+- Objects do not require a relation: it is now possible to search for specific objects (e.g. give me all URLs with port 80 open) and return all data. With bits it was not possible to query this kind of information, as the parameters (such as URL and port) were only accessible in the bit and not on the overall dataset.
+- Automatic updates: if a nibble gets updated, the affects are automatically recalculated.
+- Visual representation: it is now possible to get a better visual representation in a graph on which inputs are available for nibbles and how they relate to each other.
+- Efficient: using nibbles it is much more efficient to query for the right data and even make queries that were not previously possible. Nibbles also reduce the number of read and writes to XTDB, which reduces the database load.
+
+**How can I use nibbles?**
+Nibbles are enabled by default. Users can choose to enable or disable the nibbles themselves and determine where objects originated from and which nibbles are also applicable on an object or set of objects.
+
+Nibbles are accessible from the web interface, to allow for enabling, disabling, editing and adding of nibbles.
+
+Using the XTDB EDN query language you can perform queries for nibbles. You can query a set of objects and process this in your nibble, including all possible combinations on the queried set (of objects). One of the downsides is that this could result in a large number of sub-queries that are structurally the same, but are slightly different (e.g. flags that are different). It is therefore important to properly document your queries to still comprehend them in the future (and make it easier to catch bugs).
+
+The `xtdb-cli` tool can be used to query on XTDB for nibbles. The query below searches all variables where the object type is an URL:
+
+```shell
+octopoes/tools/xtdb-cli.py query '{:query {:find [(pull ?var [*])] :where [[?var :object_type "URL"]]}}'
+```
+
+Make it visually pretty with jq:
+
+```shell
+octopoes/tools/xtdb-cli.py query '{:query {:find [(pull ?var [*])] :where [[?var :object_type "URL"]]}}' | jq
+```

docs/source/manual/user-manual.rst

@@ -452,6 +452,15 @@ Almost all bits are enabled by default and be disabled by adding the bit to `BIT
 
 Note that if you enable a bit that was previously enabled the bit won't be automatically run for every object it should have run on, but only when it is triggered again after a new scan or other bit that has run. When a bit that was previously enabled is disabled the resulting objects from that bit will also not be automatically removed. Only when the bit triggers instead of running the bit the resulting OOIs of the previous run will be deleted. This also means that if the bit isn't triggered the old objects will not be removed.
 
+Nibbles
+=======
+Nibbles are the flexible replacement for bits. We learned a lot on what does and doesn't work around bits and ran into some limitations. The nibbles are solving these limitations. We hope that the nibbles are 4 times as efficient as bits, but this isn't officially bench marked (and is more of a bits-nibbles joke ;)).
+
+Nibbles process the data in OpenKAT, they correlate data and try to extract as much information as possible based on the specified rules. Nibbles ensure for a mapping of one or more objects to another bunch of objects.
+
+A more technical explanation on Nibbles can be found at: `https://docs.openkat.nl/developer-documentation/octopoes.html#nibbles <https://docs.openkat.nl/developer-documentation/octopoes.html#nibbles>`_.
+
+
 Reports
 =======