Audit Logging of Data Flow and Data Usage

[janpmeyer]

A topic that has come up with us is the special tracking of a data history and the use of this data in particular, so separate from a logging which deals with the state of the data ecosystem / the connector and its component state.

For this I/we would develop an extension which offers an interface internally, like the Monitor already offers, that within the Connector as soon as data is manipulated or moved, this must be logged actively, or an observation on it can take place (passively). So that a data transfer from Connector A to Connector B arrive also logged.

Besides the internal logging there should also be an external interface where external certified software can write logs how they use the data or where they move the data to for example.

For this the first idea would be to write a simple and small extension which can store in-memory logs. A simple API interface which could log logs from external with a POST method. In the first step an active logging mechanism is easier to implement, but if possible in a next step it should become an observing logging, so that the error-proneness within the system would be reduced.

Accordingly, the following information should first be logged over the data:
• DatenID
• SourceId
• Timestamp
• Action (USAGE, DATATRANSFER)
• LogMessage (More detailed description but free text)

What should be logged first?

• Access/use to data
• Data manipulation
• Move data

Critical is the concrete implementation of the identification of data (data ID), so that the logs can be assigned to them.

[jimmarino]

@ndr-brt is already working on a general eventing system that auditing can be built on top of as opposed to introducing a specific interface into the runtime to deal with auditing. I would sync with him.

[lholthof]

Great, thanks for instant sharing! 👍

And you (@ndr-brt ) now adopted this, by publishing events for lifecycle changes of the TransferProcess (and ContractNegotation is following soon)?! How about data mgt APIs, is the same concept intended to be used there as well?

Usually an audit log should be written before the action is taken, to ensure every action is logged even if app crashes. As far as I see the events have been put in post-action methods, which is appropriate for other event based mechanisms (e.g. notification). Did you elaborate adding it to pre-action methods already?

[ndr-brt]

@lholthof currently the event system is not usable for audit because events are not stored and they can be lost. For the moment they can be used for notifications, but in the future they could be eligible for auditing implementing some sort of persistence in a transactional context.

pre-action events are not events :)

(contract negotiation events will be implemented in the next days)

[lholthof]

@ndr-brt, is there already an alternative approach for audit logging being discussed somewhere? Or how would I trigger it?

To me the events look quite close, just the implementation of EventSubscriber has to connect to some persistence and errors have to be thrown to prevent action being taken without log.

[janpmeyer]

@lholthof Im going to try something like an audit logging as an extension like you describe it, but im just focused on data flows (like described in the initial discussion entry). So i probably dont gonna use every Event existing.

[ndr-brt]

@ndr-brt, is there already an alternative approach for audit logging being discussed somewhere? Or how would I trigger it?

To me the events look quite close, just the implementation of EventSubscriber has to connect to some persistence and errors have to be thrown to prevent action being taken without log.

no discussion has started yet, currently we've been focusing on notifications.
The pre-action subscription can be achieved on TransferProcess and ContractNegotiation registering a Listener on the Observable service (take a look at the extensions/azure/events module for an example).
Just pay attention as, if with the events you could currently miss an event, also with pre-action you could count an action that at the end won't happen.