x-pack/filebeat/input/salesforce: Implement a new Salesforce input

[kush-elastic]

Description

Introducing a new filebeat input -- Salesforce. The existing input used for the Salesforce module was not a good choice and it didn't give us the flexibility to address some issues we were facing with HTTPJSON.

New Salesforce input is introduced to have greater control over authentication, how data is fetched and processed, and more. There are several clients available for Salesforce written in Go that execute SOQL queries against the Salesforce server and do the other needful steps (e.g., fetching EventLogFile) which were previously done in HTTPJSON. Please note that this will be completely independent of HTTPJSON input and hence there would be no relation.

Please read the technical doc to read more about the decisions taken here.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to Run this Input

Configuration

Following is the sample configuration for salesforce input:

- type: salesforce
  enabled: true
  version: 56
  auth.oauth2:
    user_password_flow:
      enabled: true
      client.id: client-id
      client.secret: client-secret
      token_url: https://instance-id.develop.my.salesforce.com
      user: [email protected]
      password: salesforce-instance-password
    jwt_bearer_flow:
      enabled: true
      client.id: client-id
      client.username: [email protected]
      client.key_path: server_client.key
      url: https://login.salesforce.com
  url: https://instance-id.develop.my.salesforce.com
  event_monitoring_method:
    event_log_file:
      enabled: true
      interval: 1h
      query:
        default: "SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE EventType = 'Login' ORDER BY CreatedDate ASC NULLS FIRST"
        value: "SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE EventType = 'Login' AND CreatedDate > [[ .cursor.event_log_file.last_event_time ]] ORDER BY CreatedDate ASC NULLS FIRST"
      cursor:
        field: "CreatedDate"
    object:
      enabled: true
      interval: 5m
      query:
        default: "SELECT FIELDS(STANDARD) FROM LoginEvent"
        value: "SELECT FIELDS(STANDARD) FROM LoginEvent WHERE EventDate > [[ .cursor.object.first_event_time ]]"
      cursor:
        field: "EventDate"

• Type: Identifies the input type as salesforce.

• Enabled: Indicates whether the Salesforce input is active. Example: true

• Version: Specifies the version of the Salesforce API to be used for the input. Example: 56

• Auth/OAuth2 Configuration:

  • User Password Flow:

    • Enabled: Enables OAuth2 User Password Flow for authentication. Example: true
    • client.id: The OAuth2 client ID used for authentication. Example: 3MMG9pAzvMkjMb6nzW6iSPJU2rxCryzVeeo7KAuRlI1uqDsJhxow76lQ5Eks2KSfr1o8D1PoMoGQCPPILldkB
    • client.secret: The OAuth2 client secret used for authentication. Example: C497A710C21497FB0A0A68B8DAF78ALFBE010C65BFBE076E85B9B37553D6CBCA
    • token_url: The Base URL for obtaining OAuth2 tokens. Example: https://devabcdin-org-ed.develop.my.salesforce.com
    • username: The Salesforce instance username or user ID for authentication. Example: [email protected]
    • password: The Salesforce user's password for authentication. Example: salesforce-instance-password

  • JWT Bearer Flow:

    • Enabled: Enables OAuth2 JWT Bearer Flow for authentication. Example: true
    • client.id: The OAuth2 client ID used for authentication. Example: 3MMG9pAzvMkjMb6nzW6iSPJU2rxCryzVeeo7KAuRlI1uqDsJhxow76lQ5Eks2KSfr1o8D1PoMoGQCPPILldkB
    • client.username: The Salesforce username used for authentication. Example: [email protected]
    • client.key_path: The path to the private key file used for JWT authentication. Example: server_client.key
    • url: The Salesforce authentication endpoint URL. Example: https://login.salesforce.com

• URL: Specifies the base URL of the Salesforce instance. Example: https://devabcdin-org-ed.develop.my.salesforce.com

• Event Monitoring Method:

  • Event Log File:

    • Enabled: Enables monitoring of Salesforce Event Log Files. Example: true
    • Interval: Checks for new events at regular intervals. Example: 1h
    • Query:
      • default: Default SOQL query to retrieve Event Log Files. Example: "SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE EventType = 'Login' ORDER BY CreatedDate ASC NULLS FIRST"
      • value: Dynamic SOQL query with a cursor condition based on the last event timestamp. Example: "SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE EventType = 'Login' AND CreatedDate > [[ .cursor.event_log_file.last_event_time ]] ORDER BY CreatedDate ASC NULLS FIRST"
    • Cursor:
      • field: Specifies the field used as a cursor for tracking changes. Example: "CreatedDate"

  • Object:

    • Enabled: Enables monitoring of Salesforce objects. Example: true
    • Interval: Checks for new events at regular intervals. Example: 5m
    • Query:
      • default: Default SOQL query to retrieve Salesforce objects. Example: "SELECT FIELDS(STANDARD) FROM LoginEvent"
      • value: Dynamic SOQL query with a cursor condition based on the first event timestamp. Example: "SELECT FIELDS(STANDARD) FROM LoginEvent WHERE EventDate > [[ .cursor.object.first_event_time ]]"
    • Cursor:
      • field: Specifies the field used as a cursor for tracking changes. Example: "EventDate"

Related issues

• Relates https://github.com/elastic/obs-infraobs-team/issues/1248
• Relates https://github.com/elastic/obs-infraobs-team/issues/1274

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b new_salesforce_input upstream/new_salesforce_input
git merge upstream/main
git push upstream new_salesforce_input

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kush-elastic? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

❕ Build Aborted

Either there was a build timeout or someone aborted the build.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Duration: 135 min 39 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[shmsr]

@tommyers-elastic / @lalit-satapathy We have addressed all your review comments, we plan to merge this so we can implement the module changes. Please review and let us know if we can merge.

It's okay to review the full code later; we can address any missed items (if any) in module PR. For now, we can merge the code for input. Let us know what we should do?

[lalit-satapathy]

LGTM for early release and future updates as needed.

[shmsr]

Pinging @elastic/elastic-agent-data-plane / @elastic/ingest-eng-prod for their approval. We have made changes to the filebeat's docs and some common files and that's why your approval is required.

[alexsapran]

CODEOWNERS changes looks good

[kush-elastic]

/test

[belimawr]

It seems that because the codeowners change was made after the PR was created, it is still requiring an approval from the data plane. As everybody else already approved this PR, I'm approving it without a review to unblock it.