Refact: Separation of anomaly detection from drift detection #1133

fhswf · Feb 13, 2025 · e1730f8 · e1730f8
1 parent 151a9bd
commit e1730f8
Show file tree

Hide file tree

Showing 82 changed files with 1,114 additions and 702 deletions.
diff --git a/doc/rtd/content/03_machine_learning/mlpro_oa/main.rst b/doc/rtd/content/03_machine_learning/mlpro_oa/main.rst
@@ -2,9 +2,12 @@
 MLPro-OA - Online Adaptivity
 ============================
 
-This framework addresses topics of online machine learning and is already implemented and ready to be used by early adopters. The 
-documentation is still in preparation but the API description is already done. Just browse through the menu structure and follow 
-the links provided...
+This framework addresses the challenge of continuously adapting to changing conditions by processing new information in real time. Unlike traditional offline approaches, 
+which rely on predefined models trained on historical data, online-adaptive methods dynamically refine their behavior as new data becomes available. This enables 
+continuous learning, rapid adaptation to non-stationary environments, and increased robustness in uncertain or evolving scenarios. Such methods are essential in domains 
+requiring real-time decision-making and continuous model updates, including autonomous systems, predictive analytics, and self-optimizing processes.
+
+The following subfields are available in MLPro-OA:
 
 .. toctree::
    :maxdepth: 2

diff --git a/doc/rtd/content/03_machine_learning/mlpro_oa/sub/01_oa_stream_processing.rst b/doc/rtd/content/03_machine_learning/mlpro_oa/sub/01_oa_stream_processing.rst
@@ -0,0 +1,32 @@
+.. _target_oa_streams:
+Online-adaptive data stream processing (OADSP)
+==============================================
+
+This sub-framework of MLPro-OA is directly related to the research topic of online machine learning (OML). It deals with
+online-adaptive stream tasks embedded in a stream workflow as part of an extended process model. This process model extends
+the non-adaptive DSP process model of sub-framework MLPro-BF-Streams by advanced adaptation mechanisms like 
+
+   - Event-oriented adaptation
+   - Cascaded adaptation
+   - Reverse adaptation
+
+which are explained in more detail below.
+
+The description is still under contruction but parts of the sub-menu and first howtos and API specifications are already available.
+Browse the menu and see Section 'Cross reference' for further details.
+
+
+**Learn more**
+
+.. toctree::
+   :maxdepth: 3
+   :glob:
+
+   streams/*
+
+
+**Cross reference**
+  - :ref:`Howtos MPro-OA <target_appendix1_OA>`
+  - :ref:`API reference: MLPro-OA-Streams <target_api_oa_streams>`
+  - :ref:`API reference: MLPro-OA-Streams - Pool of objects <target_api_pool_oa_streams>`
+  - :ref:`Basics of data stream processing in MLPro <target_bf_streams>`
diff --git a/doc/rtd/content/03_machine_learning/mlpro_oa/sub/01_overview.rst b/doc/rtd/content/03_machine_learning/mlpro_oa/sub/01_overview.rst
diff --git a/doc/rtd/content/03_machine_learning/mlpro_oa/sub/02_getstarted.rst b/doc/rtd/content/03_machine_learning/mlpro_oa/sub/02_getstarted.rst
diff --git a/doc/rtd/content/03_machine_learning/mlpro_oa/sub/03_oa_stream_processing.rst b/doc/rtd/content/03_machine_learning/mlpro_oa/sub/03_oa_stream_processing.rst
diff --git a/...arning/mlpro_oa/sub/04_oa_systems.rst.off → ...arning/mlpro_oa/sub/03_oa_systems.rst.off b/...arning/mlpro_oa/sub/04_oa_systems.rst.off → ...arning/mlpro_oa/sub/03_oa_systems.rst.off
@@ -1,11 +1,11 @@
 .. _target_oa_systems:
-Online Adaptive Systems
-=======================
+Online-adaptive state-based systems
+===================================
 
 Further descriptions coming soon...
 
 .. toctree::
    :maxdepth: 2
    :glob:
 
-   layer1_oa_systems/*
+   systems/*
diff --git a/...yer0_oa_stream_processing/01_overview.rst → ...ning/mlpro_oa/sub/streams/01_overview.rst b/...yer0_oa_stream_processing/01_overview.rst → ...ning/mlpro_oa/sub/streams/01_overview.rst
diff --git a/...cessing/01_overview/01_oa_stream_task.rst → ...streams/01_overview/01_oa_stream_task.rst b/...cessing/01_overview/01_oa_stream_task.rst → ...streams/01_overview/01_oa_stream_task.rst
@@ -1,5 +1,5 @@
 .. _target_oa_stream_tasks:
-Online Adaptive Stream Tasks
+Online-adaptive stream tasks
 ============================
 
 ...

diff --git a/...ing/01_overview/02_oa_stream_workflow.rst → ...ams/01_overview/02_oa_stream_workflow.rst b/...ing/01_overview/02_oa_stream_workflow.rst → ...ams/01_overview/02_oa_stream_workflow.rst
@@ -1,5 +1,5 @@
 .. _target_oa_stream_workflows:
-Online Adaptive Stream Workflows
+Online-adaptive stream workflows
 ================================
 
 

diff --git a/...am_processing/02_oa_boundary_detector.rst → ...a/sub/streams/02_oa_boundary_detector.rst b/...am_processing/02_oa_boundary_detector.rst → ...a/sub/streams/02_oa_boundary_detector.rst
@@ -1,5 +1,5 @@
 .. _target_oa_boundary_detector:
-Boundary Detection
+Boundary detection
 ==================
 
 Further descriptions coming soon...

diff --git a/...stream_processing/03_oa_normalization.rst → ...ro_oa/sub/streams/03_oa_normalization.rst b/...stream_processing/03_oa_normalization.rst → ...ro_oa/sub/streams/03_oa_normalization.rst
diff --git a/...sing/03_oa_normalization/01_oa_minmax.rst → ...eams/03_oa_normalization/01_oa_minmax.rst b/...sing/03_oa_normalization/01_oa_minmax.rst → ...eams/03_oa_normalization/01_oa_minmax.rst
diff --git a/...ng/03_oa_normalization/02_oa_zscaling.rst → ...ms/03_oa_normalization/02_oa_zscaling.rst b/...ng/03_oa_normalization/02_oa_zscaling.rst → ...ms/03_oa_normalization/02_oa_zscaling.rst
diff --git a/...stream_processing/10_cluster_analysis.rst → ...ro_oa/sub/streams/10_cluster_analysis.rst b/...stream_processing/10_cluster_analysis.rst → ...ro_oa/sub/streams/10_cluster_analysis.rst
@@ -1,5 +1,5 @@
 .. _target_oa_cluster_analyzer:
-Cluster Analysis
+Cluster analysis
 ================
 
 Further descriptions coming soon...

diff --git a/...m_processing/10_cluster_analysis/.gitkeep → .../sub/streams/10_cluster_analysis/.gitkeep b/...m_processing/10_cluster_analysis/.gitkeep → .../sub/streams/10_cluster_analysis/.gitkeep
diff --git a/...tream_processing/20_anomaly_detection.rst → ...o_oa/sub/streams/20_anomaly_detection.rst b/...tream_processing/20_anomaly_detection.rst → ...o_oa/sub/streams/20_anomaly_detection.rst
@@ -1,19 +1,19 @@
 .. _target_oa_anomaly_detection:
-Anomaly Detection
+Anomaly detection
 =================
 
 Anomaly detection involves identifying instances that are structurally or dimensionally similar to non-anomalous data but deviate significantly from the normal data distribution or pattern. In real-world problems, anomaly detection helps uncover unusual activities in banking and finance, abnormalities in medical test results, uncommon behavior or sensor readings of machines, defective products in manufacturing lines, or malicious activities in network traffic monitoring. Detecting and analyzing these instances or behaviors is crucial for taking immediate action, preventing future occurrences of undesirable events, and ensuring data quality. Moreover, anomaly detection plays a pivotal role in making unbiased and accurate decisions across various domains.
 
 Anomaly detection techniques can be broadly classified into two categories based on the under- lying principles and methodologies. The two categories are Statistical anomaly detectors and Machine Learning anomaly detectors.
 
-**Types of Anomalies**
+**Types of anomalies**
 There are three main types of anomalies- Point anomalies, Contextual anomalies and Collective anomalies.
 
   (a) Point Anomalies : Type I anomalies or point anomalies are individual data instances that are significantly different from the rest of the dataset. Also known as global anomalies, these do not fit the normal distribution or pattern of the dataset.
   (b) Contextual Anomalies : Type II anomalies or contextual anomalies are data instances that are anomalies only in a particular context or subset of the dataset. Also known as conditional anomalies, these are not necessarily anomalies in the context of the whole dataset but anomalous within a specific context or condition.
   (c) Group Anomalies : Type III anomalies or group anomalies or collective anomalies are anomalous data instances when taken as a group or subset of the dataset. They may or may not be anomalies when considered individually. Also known as group anomalies, these occur when there is a deviation or unexpected relationship or behaviour among a group of data instances from the normal distribution of data.
 
-**Classification of Anomaly Detectors**
+**Classification of anomaly detectors**
 Anomaly detection techniques can be broadly classified into two categories based on the under- lying principles and methodologies. The two categories are Statistical anomaly detectors and Machine Learning anomaly detectors.
 
   (a) Statistical Anomaly Detectors : Statistical anomaly detectors use statistical methods to find data point deviations from the normal distribution. Common algorithms for this category of anomaly detectors are Z-score, Kernel Density Estimate, and Gaussian Mixture Models (GMM).
@@ -29,7 +29,7 @@ Anomaly detection techniques can be broadly classified into two categories based
    20_anomaly_detection/*
 
 
-**Cross Reference**
+**Cross reference**
 
 - Selected open access papers
 - Howtos

diff --git a/...ocessing/20_anomaly_detection/01_cbad.rst → .../streams/20_anomaly_detection/01_cbad.rst b/...ocessing/20_anomaly_detection/01_cbad.rst → .../streams/20_anomaly_detection/01_cbad.rst
@@ -1,5 +1,5 @@
 .. _target_oa_cbad:
-Cluster-based Anomaly Detection
+Cluster-based anomaly detection
 ===============================
 
 Cluster-based anomaly detection uses clustering algorithms to form clustered data and identify anomolous behaviours of these clusters, which are flagged as anomalies.
@@ -18,7 +18,8 @@ Can uncover contextual anomalies (data points that are anomalous in one cluster
 Can identify global anomalies (points far from all clusters) and local anomalies (outliers within a cluster).
 Scalability with various clustering algorithms, such as k-means for simpler scenarios or DBSCAN for non-spherical and dense data distributions.
 
-**New Types of Anomalies**
+
+**New types of anomalies**
 
   Cluster-based methods introduce nuanced anomaly categorizations:
 
@@ -28,9 +29,10 @@ Scalability with various clustering algorithms, such as k-means for simpler scen
 
   (c) Cluster Structural Anomalies: Unusual clusters themselves, such as unexpected densities, shapes, or sizes, signaling broader irregularities.
 
-**Special Dependencies on Cluster Algorithms**
 
-  Cluster-based anomaly detection heavily depends on the choice of clustering algorithm, as it directly impacts the detection process:
+**Special dependencies on cluster algorithms**
+
+Cluster-based anomaly detection heavily depends on the choice of clustering algorithm, as it directly impacts the detection process:
 
 k-means: Effective for spherical clusters but may miss anomalies in datasets with non-convex shapes or varying densities.
 DBSCAN: Ideal for discovering density-based anomalies but sensitive to hyperparameter tuning (e.g., minPts, ε).
@@ -39,7 +41,8 @@ Gaussian Mixture Models (GMM): Handles soft clustering and detects probabilistic
 Spectral Clustering: Good for identifying anomalies in non-linear manifolds but computationally intensive for large datasets.
 Effective anomaly detection requires understanding the clustering algorithm’s limitations and ensuring it aligns with the data characteristics and problem context.
 
-**Cross Reference**
+
+**Cross reference**
 
 - Howtos
 - API
diff --git a/...ream_processing/30_anomaly_prediction.rst → ...sub/streams/21_anomaly_prediction.rst.off b/...ream_processing/30_anomaly_prediction.rst → ...sub/streams/21_anomaly_prediction.rst.off
@@ -1,5 +1,5 @@
 .. _target_oa_anomaly_prediction:
-Anomaly Prediction
+Anomaly prediction
 ==================
 
 Further descriptions coming soon...
@@ -11,10 +11,10 @@ Further descriptions coming soon...
    :maxdepth: 2
    :glob:
 
-   20_anomaly_prediction/*
+   21_anomaly_prediction/*
 
 
-**Cross Reference**
+**Cross reference**
 
 - Selected open access papers
 - Howtos

diff --git a/...processing/30_anomaly_prediction/.gitkeep → ...ub/streams/21_anomaly_prediction/.gitkeep b/...processing/30_anomaly_prediction/.gitkeep → ...ub/streams/21_anomaly_prediction/.gitkeep
diff --git a/doc/rtd/content/03_machine_learning/mlpro_oa/sub/streams/30_drift_detection.rst b/doc/rtd/content/03_machine_learning/mlpro_oa/sub/streams/30_drift_detection.rst
@@ -0,0 +1,21 @@
+.. _target_oa_drift_detection:
+Drift detection
+===============
+
+Further descriptions coming soon...
+
+
+**Learn more**
+
+.. toctree::
+   :maxdepth: 2
+   :glob:
+
+   30_drift_detection/*
+
+
+**Cross reference**
+
+- Selected open access papers
+- Howtos
+- API
diff --git a/...content/03_machine_learning/mlpro_oa/sub/streams/30_drift_detection/01_cbdd.rst b/...content/03_machine_learning/mlpro_oa/sub/streams/30_drift_detection/01_cbdd.rst
@@ -0,0 +1,21 @@
+.. _target_oa_cbdd:
+Cluster-based drift detection
+=============================
+
+Further descriptions coming soon...
+
+
+**New types of drift**
+
+...
+
+
+**Special dependencies on cluster algorithms**
+
+...
+
+
+**Cross reference**
+
+- Howtos
+- API
diff --git a/...g/mlpro_oa/sub/layer1_oa_systems/.gitkeep → .../sub/streams/31_drift_prediction/.gitkeep b/...g/mlpro_oa/sub/layer1_oa_systems/.gitkeep → .../sub/streams/31_drift_prediction/.gitkeep
diff --git a/doc/rtd/content/03_machine_learning/mlpro_oa/sub/systems/.gitkeep b/doc/rtd/content/03_machine_learning/mlpro_oa/sub/systems/.gitkeep
diff --git a/doc/rtd/content/99_appendices/appendix1/sub/mlpro_oa/01_streams.rst b/doc/rtd/content/99_appendices/appendix1/sub/mlpro_oa/01_streams.rst
@@ -1,8 +1,8 @@
-Online Adaptive Stream Processing
-=================================
+Online-adaptive data stream processing
+======================================
 
 .. toctree::
    :maxdepth: 1
    :glob:
 
-   oa_dsp/* 
+   streams/* 
diff --git a/doc/rtd/content/99_appendices/appendix1/sub/mlpro_oa/streams/howto_oa_pp_009.rst b/doc/rtd/content/99_appendices/appendix1/sub/mlpro_oa/streams/howto_oa_pp_009.rst
diff --git a/doc/rtd/content/99_appendices/appendix1/sub/mlpro_oa/streams/howto_oa_pp_121.rst b/doc/rtd/content/99_appendices/appendix1/sub/mlpro_oa/streams/howto_oa_pp_121.rst
@@ -0,0 +1,26 @@
+.. _Howto_OA_PP_121:
+Howto OA-PP-121: Complex preprocessing with parallel tasks
+==========================================================
+
+
+**Executable code**
+
+.. literalinclude:: ../../../../../../../../test/howtos/oa/streams/hybrid/howto_oa_streams_pp_121_complex_preprocessing.py
+	:language: python
+
+
+
+**Results**
+
+After starting the Howto, the Workflow and Tasks windows appear. These can now be arranged on 
+the screen before the actual processing is started with a keystroke... 
+
+.. image::
+    images/howto_oa_pp_121.gif
+    :width: 700px
+
+
+
+**Cross reference**
+
+- :ref:`API Reference - Online-adaptive data stream processing <target_api_oa_streams>`
diff --git a/...pro_oa/streams/images/howto_oa_pp_009.gif → ...pro_oa/streams/images/howto_oa_pp_121.gif b/...pro_oa/streams/images/howto_oa_pp_009.gif → ...pro_oa/streams/images/howto_oa_pp_121.gif
diff --git a/doc/rtd/content/99_appendices/appendix2/sub/core/mlpro_oa/01_streams.rst b/doc/rtd/content/99_appendices/appendix2/sub/core/mlpro_oa/01_streams.rst
@@ -1,6 +1,6 @@
 .. _target_api_oa_streams:
-OA Stream Processing
-====================
+Online-adaptive data stream processing
+======================================
 
 .. image:: streams/images/MLPro-OA-Stream-Processing_class_diagram.drawio.png
    :scale: 50%

diff --git a/doc/rtd/content/99_appendices/appendix2/sub/pool/mlpro_oa/10_streams.rst b/doc/rtd/content/99_appendices/appendix2/sub/pool/mlpro_oa/10_streams.rst
@@ -1,6 +1,6 @@
 .. _target_api_pool_oa_streams:
-OA Stream Processing
-====================
+Online-adaptive data stream processing
+======================================
 
 .. toctree::
    :maxdepth: 2

diff --git a/...ppendix2/sub/pool/mlpro_oa/20_systems.rst → ...dix2/sub/pool/mlpro_oa/20_systems.rst.off b/...ppendix2/sub/pool/mlpro_oa/20_systems.rst → ...dix2/sub/pool/mlpro_oa/20_systems.rst.off
diff --git a/...d/content/99_appendices/appendix2/sub/pool/mlpro_oa/streams/01_stream_tasks.rst b/...d/content/99_appendices/appendix2/sub/pool/mlpro_oa/streams/01_stream_tasks.rst
@@ -1,6 +1,6 @@
 .. _target_api_oa_streams_tasks:
-OA Stream Tasks
-===============
+Online-adaptive stream tasks
+============================
 
 .. toctree::
    :maxdepth: 2

diff --git a/.../streams/tasks/01_preprocessing_tasks.rst → .../streams/tasks/10_preprocessing_tasks.rst b/.../streams/tasks/01_preprocessing_tasks.rst → .../streams/tasks/10_preprocessing_tasks.rst
@@ -1,12 +1,12 @@
 .. _target_api_oa_stream_tasks_prepro:
-OA Data Preprocessing
-=====================
+Preprocessing
+=============
 
 .. image:: images/MLPro-OA-Preprocessing-Tasks_class_diagram.drawio.png
    :scale: 50%
 
 
-Boundary Detector
+Boundary detector
 -----------------
 
 .. automodule:: mlpro.oa.streams.tasks.boundarydetectors

diff --git a/...oa/streams/tasks/02_cluster_analyzers.rst → ...oa/streams/tasks/20_cluster_analyzers.rst b/...oa/streams/tasks/02_cluster_analyzers.rst → ...oa/streams/tasks/20_cluster_analyzers.rst
@@ -1,12 +1,12 @@
 .. _target_api_oa_stream_tasks_clu:
-Cluster Analysis
+Cluster analyzer
 ================
 
 .. image:: images/MLPro-OA-Cluster_Analyzers_class_diagram.drawio.png
    :scale: 50%
 
 
-Template for Cluster Algorithms
+Template for cluster algorithms
 -------------------------------
 
 .. automodule:: mlpro.oa.streams.tasks.clusteranalyzers.basics

diff --git a/...oa/streams/tasks/03_anomaly_detectors.rst → ...oa/streams/tasks/30_anomaly_detectors.rst b/...oa/streams/tasks/03_anomaly_detectors.rst → ...oa/streams/tasks/30_anomaly_detectors.rst
@@ -1,5 +1,5 @@
 .. _target_api_oa_stream_tasks_ad:
-Anomaly Detectors
+Anomaly detectors
 =================
 
 .. image:: anomaly_detectors/images/MLPro-OA-Anomaly-Detectors_class_diagram.drawio.png