Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[devops] Improved diagnostics and implement deadlocked process termination. #21317

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

[devops] Improved diagnostics and implement deadlocked process termination. #21317

wants to merge 17 commits into from

Conversation

rolfbjarne
Copy link
Member

  • Unify the code to collect diagnostic information about a bot.
  • Unify some of the cleanup code to prepare a bot as well.
  • Implement code to terminate processes on a bot that's used more than a day
    of CPU (presumably these processes are stuck for some reason).

@rolfbjarne
[devops] Add a bit more info when showing the enviroment.
6294877 
Also show:

* The network configuration
* The load on the system / cpu usage (top)
@rolfbjarne
[devops] Improved diagnostics and implement deadlocked process termin…
fc710fc 
…ation.

* Unify the code to collect diagnostic information about a bot.
* Unify some of the cleanup code to prepare a bot as well.
* Implement code to terminate processes on a bot that's used more than a day
  of CPU (presumably these processes are stuck for some reason).
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
dalexsoto
dalexsoto reviewed Sep 27, 2024
View reviewed changes
tools/devops/automation/scripts/bash/clean-bot.sh Outdated
@@ -12,6 +12,14 @@ df -h
# We don't care about errors in this section, we just want to clean as much as possible
set +e

# Clean workspace
(
if test -d "$SYSTEM_DEFAULTWORKINGDIRECTORY/xamarin-macios"; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rolfbjarne Can you make the name of the repo a variable somewhere central? Maybe first line in the top level Make.config? Some sed'ing may be involved heh but since we are migrating the repo over we can start adding to the machinery to no depend on the magic string xamarin-macios

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dalexsoto fixed!

@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2
Copy link
Collaborator

📚 [CI Build] Artifacts 📚

Artifacts were not provided.

Pipeline on Agent
Hash: [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

✅ API diff for current PR / commit

NET (empty diffs)
  • iOS: (empty diff detected)
  • tvOS: (empty diff detected)
  • MacCatalyst: (empty diff detected)
  • macOS: (empty diff detected)

✅ API diff vs stable

.NET (No breaking changes)

ℹ️ Generator diff

Generator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes)

Pipeline on Agent
Hash: f2858ee783b5cdd92fa3c21499a71c36bf8e7a73 [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

💻 [CI Build] Tests on macOS M1 - Mac Big Sur (11) passed 💻

✅ All tests on macOS M1 - Mac Big Sur (11) passed.

Pipeline on Agent
Hash: f2858ee783b5cdd92fa3c21499a71c36bf8e7a73 [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

💻 [CI Build] Tests on macOS M1 - Mac Monterey (12) passed 💻

✅ All tests on macOS M1 - Mac Monterey (12) passed.

Pipeline on Agent
Hash: f2858ee783b5cdd92fa3c21499a71c36bf8e7a73 [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

💻 [CI Build] Tests on macOS M1 - Mac Ventura (13) passed 💻

✅ All tests on macOS M1 - Mac Ventura (13) passed.

Pipeline on Agent
Hash: f2858ee783b5cdd92fa3c21499a71c36bf8e7a73 [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

💻 [CI Build] Windows Integration Tests passed 💻

✅ All Windows Integration Tests passed.

Pipeline on Agent
Hash: f2858ee783b5cdd92fa3c21499a71c36bf8e7a73 [PR build]

@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@rolfbjarne rolfbjarne marked this pull request as ready for review October 1, 2024 09:49
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2
Copy link
Collaborator

🚀 [CI Build] Test results 🚀

Test results

✅ All tests passed on VSTS: test results.

🎉 All 97 tests passed 🎉

Tests counts

✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (iOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (tvOS): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ generator: All 1 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. Html Report (VSDrops) Download
✅ introspection: All 4 tests passed. Html Report (VSDrops) Download
✅ linker: All 40 tests passed. Html Report (VSDrops) Download
✅ monotouch (iOS): All 7 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 7 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ monotouch (macOS): All 8 tests passed. [attempt 3] Html Report (VSDrops) Download
✅ monotouch (tvOS): All 7 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. Html Report (VSDrops) Download

Pipeline on Agent
Hash: f2858ee783b5cdd92fa3c21499a71c36bf8e7a73 [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

✅ API diff for current PR / commit

NET (empty diffs)
  • iOS: (empty diff detected)
  • tvOS: (empty diff detected)
  • MacCatalyst: (empty diff detected)
  • macOS: (empty diff detected)

✅ API diff vs stable

.NET (No breaking changes)

ℹ️ Generator diff

Generator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes)

Pipeline on Agent
Hash: d66ed0abb923ce0b9b2888733ca1e9e6a04315eb [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

💻 [CI Build] Tests on macOS X64 - Mac Sonoma (14) passed 💻

✅ All tests on macOS X64 - Mac Sonoma (14) passed.

Pipeline on Agent
Hash: d66ed0abb923ce0b9b2888733ca1e9e6a04315eb [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

💻 [CI Build] Tests on macOS M1 - Mac Big Sur (11) passed 💻

✅ All tests on macOS M1 - Mac Big Sur (11) passed.

Pipeline on Agent
Hash: d66ed0abb923ce0b9b2888733ca1e9e6a04315eb [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

💻 [CI Build] Tests on macOS M1 - Mac Monterey (12) passed 💻

✅ All tests on macOS M1 - Mac Monterey (12) passed.

Pipeline on Agent
Hash: d66ed0abb923ce0b9b2888733ca1e9e6a04315eb [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

💻 [CI Build] Tests on macOS M1 - Mac Ventura (13) passed 💻

✅ All tests on macOS M1 - Mac Ventura (13) passed.

Pipeline on Agent
Hash: d66ed0abb923ce0b9b2888733ca1e9e6a04315eb [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

📚 [CI Build] Artifacts 📚

Artifacts were not provided.

Pipeline on Agent
Hash: [PR build]

@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2
Copy link
Collaborator

🔥 [CI Build] Test results 🔥

Test results

❌ Tests failed on VSTS: test results

0 tests crashed, 2 tests failed, 97 tests passed.

Failures

❌ monotouch tests (iOS)

2 tests failed, 7 tests passed.
  • monotouch-test/iOS Unified 64-bits - simulator/Debug (managed static registrar) [dotnet]: TimedOut
  • monotouch-test/iOS Unified 64-bits - simulator/Release (managed static registrar, all optimizations) [dotnet]: Failed

Html Report (VSDrops) Download

Successes

✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (iOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (tvOS): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ generator: All 1 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. Html Report (VSDrops) Download
✅ introspection: All 4 tests passed. Html Report (VSDrops) Download
✅ linker: All 40 tests passed. Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 7 tests passed. Html Report (VSDrops) Download
✅ monotouch (macOS): All 8 tests passed. Html Report (VSDrops) Download
✅ monotouch (tvOS): All 7 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. Html Report (VSDrops) Download

Pipeline on Agent
Hash: d66ed0abb923ce0b9b2888733ca1e9e6a04315eb [PR build]

@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
1 similar comment
@vs-mobiletools-engineering-service2

This comment has been minimized.

Sign in to view
@vs-mobiletools-engineering-service2
Copy link
Collaborator

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: d66ed0abb923ce0b9b2888733ca1e9e6a04315eb [PR build]

mcumming
mcumming approved these changes Oct 2, 2024
View reviewed changes
tools/devops/automation/scripts/run-generator-tests-on-windows.ps1 Outdated
@@ -1,5 +1,5 @@
# Dump the environment to see what we're working with.
& "$Env:SYSTEM_DEFAULTWORKINGDIRECTORY\xamarin-macios\tools\devops\automation\scripts\show_env.ps1"
& "$Env:SYSTEM_DEFAULTWORKINGDIRECTORY\xamarin-macios\tools\devops\automation\scripts\show_bot_info.ps1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
& "$Env:SYSTEM_DEFAULTWORKINGDIRECTORY\xamarin-macios\tools\devops\automation\scripts\show_bot_info.ps1"
& "$Env:SYSTEM_DEFAULTWORKINGDIRECTORY\$Env:BUILD_REPOSITORY_NAME\tools\devops\automation\scripts\show_bot_info.ps1"

Copy link
Member Author

@rolfbjarne rolfbjarne Oct 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$Env:BUILD_REPOSITORY_NAME is xamarin/xamarin-macios, so this replacement doesn't work as-is.

Copy link
Member Author

@rolfbjarne rolfbjarne Oct 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated all of these to not have xamarin-macios hardcoded anymore.

tools/devops/automation/scripts/run-generator-tests-on-windows.ps1
@@ -1,5 +1,5 @@
# Dump the environment to see what we're working with.
& "$Env:SYSTEM_DEFAULTWORKINGDIRECTORY\xamarin-macios\tools\devops\automation\scripts\show_env.ps1"
& "$Env:SYSTEM_DEFAULTWORKINGDIRECTORY\xamarin-macios\tools\devops\automation\scripts\show_bot_info.ps1"

# Set a few variables
$Env:DOTNET = "$Env:BUILD_SOURCESDIRECTORY\xamarin-macios\tests\dotnet\Windows\bin\dotnet\dotnet.exe"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
$Env:DOTNET = "$Env:BUILD_SOURCESDIRECTORY\xamarin-macios\tests\dotnet\Windows\bin\dotnet\dotnet.exe"
$Env:DOTNET = "$Env:BUILD_SOURCESDIRECTORY\$Env:BUILD_REPOSITORY_NAME\\tests\dotnet\Windows\bin\dotnet\dotnet.exe"

tools/devops/automation/templates/build/api-diff-process-results.yml Outdated
@@ -27,8 +27,8 @@ steps:
repositoryAlias: ${{ parameters.repositoryAlias }}
commit: ${{ parameters.commit }}

- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1
displayName: 'Show Environment'
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
- pwsh: $(System.DefaultWorkingDirectory)/$(Build.Repository.Name)/tools/devops/automation/scripts/show_bot_info.ps1

tools/devops/automation/templates/common/configure.yml Outdated
@@ -65,8 +65,8 @@ steps:
name: decisions
displayName: 'Make decisions'

- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1
displayName: 'Show Environment'
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
- pwsh: $(System.DefaultWorkingDirectory)/$(Build.Repository.Name)/tools/devops/automation/scripts/show_bot_info.ps1

tools/devops/automation/templates/common/load_configuration.yml Outdated
@@ -75,8 +75,8 @@ steps:
name: decisions
displayName: 'Make decisions'

- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1
displayName: 'Show Environment'
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
- pwsh: $(System.DefaultWorkingDirectory)/$(Build.Repository.Name)/tools/devops/automation/scripts/show_bot_info.ps1

tools/devops/automation/templates/sign-and-notarized/upload-azure.yml Outdated
@@ -96,8 +96,8 @@ steps:
Azure.DropPrefix: ${{ parameters.uploadPrefix }}package
GitHub.Context: 'vsts-devdiv artifacts'

- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1
displayName: 'Show Environment'
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
- pwsh: $(System.DefaultWorkingDirectory)/$(Build.Repository.Name)/tools/devops/automation/scripts/show_bot_info.ps1

tools/devops/automation/templates/tests/publish-html.yml Outdated
@@ -49,8 +49,8 @@ steps:

- template: download-artifacts.yml

- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1
displayName: 'Show Environment'
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
- pwsh: $(System.DefaultWorkingDirectory)/$(Build.Repository.Name)/tools/devops/automation/scripts/show_bot_info.ps1

tools/devops/automation/templates/windows/build.yml Outdated
@@ -37,8 +37,8 @@ steps:
repositoryAlias: ${{ parameters.repositoryAlias }}
commit: ${{ parameters.commit }}

- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1
displayName: 'Dump Environment'
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
- pwsh: $(System.DefaultWorkingDirectory)/$(Build.Repository.Name)/tools/devops/automation/scripts/show_bot_info.ps1

tools/devops/automation/templates/windows/build.yml Outdated
@@ -127,8 +127,8 @@ steps:
displayName: "Write and verify id_rsa"
continueOnError: true

- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1
displayName: 'Show Environment'
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
- pwsh: $(System.DefaultWorkingDirectory)/$(Build.Repository.Name)/tools/devops/automation/scripts/show_bot_info.ps1

tools/devops/automation/templates/windows/reenable-mac.yml Outdated
@@ -19,8 +19,8 @@ steps:
repositoryAlias: ${{ parameters.repositoryAlias }}
commit: ${{ parameters.commit }}

- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1
displayName: 'Dump Environment'
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1
- pwsh: $(System.DefaultWorkingDirectory)/$(Build.Repository.Name)/tools/devops/automation/scripts/show_bot_info.ps1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dalexsoto dalexsoto dalexsoto left review comments

@mcumming mcumming mcumming approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@rolfbjarne @vs-mobiletools-engineering-service2 @dalexsoto @mcumming