Implement Windows Logcollector Module

[cborla]

Description

This issue is a section of #201, focuses on implementing the Windows Logcollector module in the Wazuh Agent 5.0.0. The Windows collector will utilize the Event Channel (eventchannel) API to gather system logs, ensuring seamless integration and log management on Windows platforms.

Functional Requirements

• Windows Collector Implementation
  • Develop a collector that leverages the Windows Event Channel API to collect system and application logs.
• Persistent State Management
  • Store and manage the persistent state specific to Windows logs, including:
    • Reading Position: Track the last read event to resume log collection accurately after restarts.
• State Reload on Startup
  • On agent startup, reload the stored state and resume log collection from the last known position to maintain consistency.
• Log Transmission
  • Queue collected Windows logs as stateless messages in a json wrapper.

Non-functional Requirements

• Default Configuration
  • Provide a default configuration for the Windows collector within the codebase, enabling out-of-the-box functionality.
• Module Integration
  • Ensure that the Windows collector is implemented as part of the Logcollector module rather than as a standalone executable, maintaining modularity and ease of integration.

Deliverables

1. Wazuh Agent Executable with Windows Collector
   • An updated Wazuh Agent build that includes the implemented Windows collector within the Logcollector module.
2. Design Documentation for Windows Collector
   • Comprehensive documentation detailing the architecture, components, and workflow of the Windows collector.

Acceptance Criteria

• The Windows collector successfully collects system and application logs using the Event Channel API.
• Persistent state (reading position and last modification date) is correctly stored and reloaded on agent startup.
• Logs are queued as stateless messages without data loss.
• Default and customizable configurations are properly implemented and documented.
• The module integrates seamlessly into the Wazuh Agent executable without introducing regressions.

[LucioDonda]

Update 7/10/2024 and 8/10/2024

Analysis and comparisson of libraries:

1. Windows Event Log API* (EvtQuery, EvtNext, etc.) :

   • Older and more traditional C-style approach
   • Part of the Windows SDK
   • Provides direct access to event logs
   • Can be used with C++, but may require more wrapper code for modern C++ integration
   • Pros: Well-documented, stable, and widely supported
   • Cons: More verbose, may require more boilerplate code in C++

2. Event Tracing for Windows (ETW):

   • More modern approach to event tracing and logging
   • Provides real-time and historical event collection
   • Offers high-performance event tracing
   • Can be used with C++ and integrates well with modern C++ features
   • Pros: High performance, flexible, supports real-time monitoring
   • Cons: Steeper learning curve, may be overkill for simple event log queries

3. krabsetw:

   • Open-source C++ library for ETW
   • Provides a modern C++ wrapper around ETW functionality
   • Designed to work well with C++11 and later
   • Pros: Easier to use than raw ETW, good C++ integration
   • Cons: Third-party library, may have less documentation compared to official Microsoft solutions

4. Integrating PowerShell:

   • Allows using PowerShell cmdlets from C++ code
   • Can leverage PowerShell's Get-WinEvent cmdlet for querying event logs
   • Requires setting up PowerShell hosting in C++
   • Pros: Flexible, can use PowerShell's rich event querying capabilities
   • Cons: Additional overhead of PowerShell runtime, may be slower for large-scale querying

First Approach

Some basic design and tentative approach on how can this be achieved, this should be double checked based on the different libraries limitations and needs:

Class Diagram:

classDiagram
    class EventChannelQuerySystem {
        - Session session
        - Provider provider
        - Consumer consumer
        -BookmarkManager bookmarkMgr
        -QueryBuilder queryBuilder
        +initialize()
        +startQuery(channelName, queryString)
        +stopQuery()
        +getResults()
        +setReconnectTime(time)
        +rotateLog()
    }
    class Session {
        -sessionName
        -sessionProperties
        +open()
        +close()
        +enableProvider(provider)
    }
    class Provider {
    }
    class Consumer {
        -eventCallback
        +processEvent(event)
        +setEventCallback(callback)
    }
    class BookmarkManager {
        -bookmarks
        +createBookmark(event)
        +getBookmark(id)
        +saveBookmarks()
        +loadBookmarks()
    }
    class QueryBuilder {
        +buildQuery(channelName, conditions)
        +addWildcardMatch(field, pattern)
    }

    EventChannelQuerySystem -- Session
    EventChannelQuerySystem -- Provider
    EventChannelQuerySystem -- Consumer
    EventChannelQuerySystem -- BookmarkManager
    EventChannelQuerySystem -- QueryBuilder

Loading

1. EventChannelQuerySystem: The main class that orchestrates the entire querying process.
2. Session: Manages the ETW session, including opening, closing, and enabling providers.
3. Provider: Represents an ETW provider, which can be enabled or disabled.
4. Consumer: Processes the events received from the ETW session.
5. BookmarkManager: Handles the creation, saving, and loading of bookmarks.
6. QueryBuilder: Constructs queries, including support for wildcard matches.

Flow chart:

1. The system starts by initializing the ETWQuerySystem.
2. It opens an ETW session and enables the required provider.
3. The ETW consumer is set up to handle incoming events.
4. Bookmarks are loaded from persistent storage.
5. A query is built using the QueryBuilder, incorporating channel name and specific query conditions.
6. The query is started, and the system enters an event processing loop.
7. For each received event:
   • The event is processed.
   • Bookmarks are updated.
   • Log rotation is performed if needed.
8. The system checks if reconnection is needed and performs it if necessary.
9. When the query is stopped, bookmarks are saved, and the session is closed.

graph TD
    A[init] --> B[Initialize - ETWQuerySystem]
    B --> C[Open - ETWSession]
    C --> D[Enable - ETWProvider]
    D --> E[Set up - ETWConsumer]
    E --> F[Load Bookmarks -> store]
    E --> W[(Store)]
    F --> G[Build N Queries]
    G --> H[run]
    H --> I{While Event Received?}
    I -->|Yes| J[Process Event]
    J --> K[Update Bookmark]
    K --> I
    I -->|No| N{Query Stopped or shutdown signal ?}
    N -->|Yes| O[Save Bookmarks]
    O --> P[Close Session]
    P --> Q[End]
    N -->|No| R{Reconnect Needed?}
    R -->|Yes| S[Reconnect]
    S --> I
    R -->|No| I

Loading

[LucioDonda]

Update 9/10/2024

• Working on a base example for getting events in real-time on windows using ETW.
• Poor and hard to follow documentation.
• hard to reuse examples.
• Getting some base events. but looking for an alternative.

[LucioDonda]

Update 10/10/2024

• Working on krabsetw aproach:

  • Better documented and easy to find example on git.
  • Easy to use and to integrate with vcpkg.
  • Still working on getting events from any provider.

  Code using krabsetw can be as simple as:

    krabs::user_trace trace;
    krabs::provider<> provider(krabs::guid(L"{A0C1853B-5C40-4B15-8766-3CF1C58F985A}"));
    provider.any(0xf0010000000003ff);
    provider.add_on_event_callback([](const EVENT_RECORD &record, const krabs::trace_context &trace_context) {
        krabs::schema schema(record, trace_context.schema_locator);
        std::wcout << L"Event " << schema.event_id();
        std::wcout << L"(" << schema.event_name() << L") received." << std::endl;
        if (schema.event_id() == 7937) {
            krabs::parser parser(schema);
            std::wstring context = parser.parse<std::wstring>(L"ContextInfo");
            std::wcout << L"\tContext: " << context << std::endl;
        }
    });
  
    trace.enable(provider);
    trace.start();

[LucioDonda]

Update 14/10/2024

Seccond Approach

Because we're using krabsetw and it needs to create a thread for each provider, this is the rework of the design:

Class Diagram:

classDiagram
    class EventChannelQuerySystem {
        - Provider provider
        - Consumer consumer
        -BookmarkManager bookmarkMgr
        -QueryBuilder queryBuilder
        +initialize()
        +setEventCallback(callback)
        +start()
        +qttyOfResults()
        +setReconnectTime(time)
        +rotateLog()
    }
    class Provider {
        -channel
        -providerName
    }
    class Consumer {
        -eventCallback
        -queryString
        +processEvent(event)
    }
    class BookmarkManager {
        -bookmarks
        +createBookmark(event)
        +getBookmark(id)
        +saveBookmarks()
        +loadBookmarks()
    }
    class QueryBuilder {
        +buildQuery(channelName, conditions)
        +addWildcardMatch(field, pattern)
    }

    EventChannelQuerySystem -- Session
    EventChannelQuerySystem -- Provider
    EventChannelQuerySystem -- Consumer
    EventChannelQuerySystem -- BookmarkManager
    EventChannelQuerySystem -- QueryBuilder

Loading

Flow chart Changes:

1. For loop for every channel neeeded
2. In each case a thread is created for that purpose.
3. Process events should fire an async task to the logcollector module.

Details

flowchart TD
    A["init"] --> C["Initialize Channel / Provider"]
    C --> E["Initialice Trace Session"]
    E --> F["Load Bookmarks"] & W[("Store")]
    F --> G["Build / Check Queries"]
    G --> H["Start Provider Thread"]
    H --> I{"Receiveing events"} & B["Next Provider"]
    I -- Yes --> J["Process Event"]
    J --> K["Update Bookmark"]
    K --> I
    I -- No --> N{"Stopped ?"}
    N -- Yes --> O["Save Bookmarks"]
    O --> P["Close Session"]
    P --> Q["End"]
    N -- No --> R{"Reconnection?"}
    R -- Yes --> S["Reconnect"]
    S --> I
    R -- No --> I
    B --> A
    I --> n1["Untitled Node"]

Loading

Example of reading Application events -> ⚠️ User should be SYSTEM:

void EventLogApplication::start()
{
    // While Adminstrator is sufficent to view the Security EventLog,
    // SYSTEM is required for the Microsoft-Windows-Security-Auditing provider.
    char user_name[128] = { 0 };
    DWORD user_name_length = 128;
    if (!GetUserNameA(user_name, &user_name_length) || !strcmp(user_name, "SYSTEM") == 0)
    {
        std::wcout << L"Microsoft-Windows-Security-Auditing can only be traced by SYSTEM" << std::endl;
        return;
    }

    krabs::user_trace trace(L"EventLog-Application");
    krabs::provider<> provider(L"Microsoft-Windows-Security-SPP");
    provider.any((ULONGLONG)-1);
    provider.add_on_event_callback([](const EVENT_RECORD &record, const krabs::trace_context &trace_context) {
        krabs::schema schema(record, trace_context.schema_locator);
        std::wcout << L"Event " << schema.event_id();
        std::wcout << L"(" << schema.event_name() << L") received." << std::endl;

        if (schema.event_id() == 4703) {
            krabs::parser parser(schema);
            std::wstring enabled_privilege_list = parser.parse<std::wstring>(L"EnabledPrivilegeList");
            std::wstring disabled_privilege_list = parser.parse<std::wstring>(L"DisabledPrivilegeList");
            std::wcout << L"\tEnabledPrivilegeList=" << enabled_privilege_list << std::endl;
            std::wcout << L"\tDisabledPrivilegeList=" << disabled_privilege_list << std::endl;
        }
    });

    trace.enable(provider);
    trace.start();
}

[LucioDonda]

Update 15/10/2024

• Integrating base case development with main issue branch.
• solving some issues with vcpkg
• Adding UT for base execution.

[LucioDonda]

Update 16/10/2024

• New limitation for krabsetw I'm not able to receive old events, waiting for dev's answer here. ⚠️
• Integrating base case development with main issue branch:
  • able to run tests calling harcoded methods.
  • able to receive fixed header fields (pending search for limitations in this ground). Also non header ones.
  • Pending branch on updated base. and rest of the integration with logcollector.

[LucioDonda]

Update 17/10/2024

• Integrating base case development with main issue branch:
  • Rebase and analysis of changes on the base branch.
  • modifying all necessary code to work along the base branch PR.
  • stuck on unicode issue and transforming infinite loop into a awaitable.

[LucioDonda]

Update 18/10/2024

• Final rebase with master.
• Creating some unit test (in progress)
• Due to lack of time to finish before the MVP deadline and based on the limitations of the krabsetw library it was decided to rework on the solution to reach a more suitable product for the future.

[cborla]

Moved to on hold until the release testing stage is completed.