-
Notifications
You must be signed in to change notification settings - Fork 511
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[devops] Improved diagnostics and implement deadlocked process termination. #21317
base: main
Are you sure you want to change the base?
Changes from 12 commits
6294877
fc710fc
0dffb7b
4cdb1d1
9a547e9
97f8669
3781225
bda0b67
4476e8a
eea7714
f2858ee
d66ed0a
cc68818
7c6c893
96c75e8
7edd442
db3c0a6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
#!/bin/bash -e | ||
|
||
echo "Looking for processes that have been stuck for more than a day, and will try to kill them." | ||
|
||
# Collect the list of processes for the current user, including the CPU time. | ||
# We then split the CPU time into separate fields, so that it's easier to figure out the total number of minutes later on. | ||
IFS=$'\n' | ||
PROCESSES=() | ||
while IFS='' read -r line; do PROCESSES+=("$line"); done < <(ps -o cputime=,pid=,user=,lstart=,args= -U "$USER" -w -w -w | sed -e 's/\([0-9]*\):\([0-9][0-9]\)\.\([0-9][0-9]\)/\1 m \2 s \3 ms/' | sort -nr) | ||
|
||
IFS=' ' | ||
for process in "${PROCESSES[@]}"; do | ||
IFS=" " read -r -a FIELDS <<< "$process" | ||
minutes=${FIELDS[0]} | ||
pid=${FIELDS[6]} | ||
|
||
echo "$process" | ||
|
||
# looking for processes that have spent more than a day using the CPU (24h * 60min = 1440min) | ||
if (( minutes > 1440 )); then | ||
echo " This process has been stuck for more than $minutes minutes, so assuming it's deadlocked and we'll try to kill it:" | ||
echo " kill -9 $pid" | ||
kill -9 "$pid" | sed 's/^/ /' || true | ||
fi | ||
done | ||
|
||
echo "No (more) processes stuck for more than a day." |
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -1,5 +1,5 @@ | ||||||
# Dump the environment to see what we're working with. | ||||||
& "$Env:SYSTEM_DEFAULTWORKINGDIRECTORY\xamarin-macios\tools\devops\automation\scripts\show_env.ps1" | ||||||
& "$Env:SYSTEM_DEFAULTWORKINGDIRECTORY\xamarin-macios\tools\devops\automation\scripts\show_bot_info.ps1" | ||||||
|
||||||
# Set a few variables | ||||||
$Env:DOTNET = "$Env:BUILD_SOURCESDIRECTORY\xamarin-macios\tests\dotnet\Windows\bin\dotnet\dotnet.exe" | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
This file was deleted.
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -27,8 +27,8 @@ steps: | |||||
repositoryAlias: ${{ parameters.repositoryAlias }} | ||||||
commit: ${{ parameters.commit }} | ||||||
|
||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1 | ||||||
displayName: 'Show Environment' | ||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1 | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Show Bot Info' | ||||||
|
||||||
- pwsh: | | ||||||
if (Test-Path "$Env:SYSTEM_DEFAULTWORKINGDIRECTORY/Artifacts" -PathType Container) { | ||||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -65,8 +65,8 @@ steps: | |||||
name: decisions | ||||||
displayName: 'Make decisions' | ||||||
|
||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1 | ||||||
displayName: 'Show Environment' | ||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1 | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Show Bot Info' | ||||||
|
||||||
- pwsh: | | ||||||
Import-Module $Env:SYSTEM_DEFAULTWORKINGDIRECTORY/xamarin-macios/tools/devops/automation/scripts/MaciosCI.psd1 | ||||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -75,8 +75,8 @@ steps: | |||||
name: decisions | ||||||
displayName: 'Make decisions' | ||||||
|
||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1 | ||||||
displayName: 'Show Environment' | ||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1 | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Show Bot Info' | ||||||
|
||||||
- pwsh: | | ||||||
Import-Module $Env:SYSTEM_DEFAULTWORKINGDIRECTORY/xamarin-macios/tools/devops/automation/scripts/MaciosCI.psd1 | ||||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -17,30 +17,14 @@ steps: | |||||
- bash: $(Build.SourcesDirectory)/xamarin-macios/tools/devops/automation/scripts/bash/fix-github-ssh-key.sh | ||||||
displayName: 'Fix GitHub SSH host key' | ||||||
|
||||||
- bash: cd $(System.DefaultWorkingDirectory)/xamarin-macios/ && git clean -xdf | ||||||
displayName: 'Clean workspace' | ||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1 | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Show Bot Info' | ||||||
|
||||||
- bash: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/bash/clean-bot.sh | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Clean bot' | ||||||
continueOnError: true | ||||||
timeoutInMinutes: 60 | ||||||
|
||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1 | ||||||
displayName: 'Show Environment' | ||||||
|
||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_python_env.ps1 | ||||||
displayName: 'Show Python information' | ||||||
|
||||||
- bash: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/bash/delete-library-dirs.sh | ||||||
displayName: 'Delete library folders' | ||||||
timeoutInMinutes: 5 | ||||||
|
||||||
- bash: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/bash/clean-results-dir.sh | ||||||
workingDirectory: $(System.DefaultWorkingDirectory) | ||||||
displayName: 'Clear results directory' | ||||||
timeoutInMinutes: 5 | ||||||
continueOnError: true | ||||||
|
||||||
- bash: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/bash/remove-ui-prompt.sh | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
env: | ||||||
OSX_KEYCHAIN_PASS: ${{ parameters.keyringPass }} | ||||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -64,18 +64,16 @@ steps: | |||||
condition: succeededOrFailed() # we do not care about the previous process cleanup | ||||||
continueOnError: true | ||||||
|
||||||
- bash: cd $(System.DefaultWorkingDirectory)/xamarin-macios/ && git clean -xdf | ||||||
displayName: 'Clean workspace' | ||||||
|
||||||
# download the packages that have been created, install them, later download the zip files that contain the already built | ||||||
# tests and execute them. | ||||||
|
||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1 | ||||||
displayName: 'Show Environment' | ||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1 | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Show Bot Info' | ||||||
|
||||||
- bash: | | ||||||
ioreg -l | grep -e Manufacturer -e 'Vendor Name' | ||||||
displayName: 'Dump Hardware' | ||||||
- bash: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/bash/clean-bot.sh | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Clean bot' | ||||||
continueOnError: true | ||||||
timeoutInMinutes: 60 | ||||||
|
||||||
- bash: | | ||||||
if [[ $(ioreg -l | grep -e 'VMware' | wc -l) -ne 0 ]]; then | ||||||
|
@@ -95,11 +93,6 @@ steps: | |||||
|
||||||
displayName: 'Set VM Vendor' | ||||||
|
||||||
- bash: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/bash/clean-bot.sh | ||||||
displayName: 'Clean bot' | ||||||
continueOnError: true | ||||||
timeoutInMinutes: 60 | ||||||
|
||||||
# Use a cmdlet to check if the space available in the devices root system is larger than 50 gb. If there is not | ||||||
# enough space available it: | ||||||
# 1. Set the status of the build to error. It is not a failure since no tests have been ran. | ||||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -53,8 +53,8 @@ jobs: | |||||
- checkout: release-scripts | ||||||
clean: true | ||||||
|
||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1 | ||||||
displayName: 'Show Environment' | ||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1 | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Show Bot Info' | ||||||
|
||||||
- bash: | | ||||||
sudo rm -Rf $(Build.SourcesDirectory)/package | ||||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -53,8 +53,8 @@ steps: | |||||
version: 3.x | ||||||
displayName: 'Install .NET Core SDK 3.x needed for ESRP' | ||||||
|
||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1 | ||||||
displayName: 'Show Environment' | ||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1 | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Show Bot Info' | ||||||
|
||||||
- task: MicroBuildSigningPlugin@4 | ||||||
displayName: 'Install Signing Plugin' | ||||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -96,8 +96,8 @@ steps: | |||||
Azure.DropPrefix: ${{ parameters.uploadPrefix }}package | ||||||
GitHub.Context: 'vsts-devdiv artifacts' | ||||||
|
||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1 | ||||||
displayName: 'Show Environment' | ||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1 | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Show Bot Info' | ||||||
|
||||||
- pwsh: | | ||||||
Import-Module $Env:SYSTEM_DEFAULTWORKINGDIRECTORY\\xamarin-macios\\tools\\devops\\automation\\scripts\\MaciosCI.psd1 | ||||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -49,8 +49,8 @@ steps: | |||||
|
||||||
- template: download-artifacts.yml | ||||||
|
||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1 | ||||||
displayName: 'Show Environment' | ||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1 | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Show Bot Info' | ||||||
|
||||||
# build a message with all the content of all tests, to do so, we get the labels and to pass them to pwsh we do a join with ; | ||||||
# as the separator | ||||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -37,8 +37,8 @@ steps: | |||||
repositoryAlias: ${{ parameters.repositoryAlias }} | ||||||
commit: ${{ parameters.commit }} | ||||||
|
||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1 | ||||||
displayName: 'Dump Environment' | ||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1 | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Show Bot Info' | ||||||
|
||||||
- ${{ if or(contains(variables['Build.Reason'], 'ResourceTrigger'), contains(variables['Build.Reason'], 'BuildCompletion'), contains(variables['Build.DefinitionName'], 'xamarin-macios-ci-tests'), contains(variables['Build.DefinitionName'], 'xamarin-macios-pr-tests')) }}: | ||||||
- download: macios | ||||||
|
@@ -127,8 +127,8 @@ steps: | |||||
displayName: "Write and verify id_rsa" | ||||||
continueOnError: true | ||||||
|
||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1 | ||||||
displayName: 'Show Environment' | ||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1 | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Show Bot Info' | ||||||
|
||||||
- pwsh: | | ||||||
Import-Module $Env:SYSTEM_DEFAULTWORKINGDIRECTORY\\xamarin-macios\\tools\\devops\\automation\\scripts\\MaciosCI.psd1 | ||||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -19,8 +19,8 @@ steps: | |||||
repositoryAlias: ${{ parameters.repositoryAlias }} | ||||||
commit: ${{ parameters.commit }} | ||||||
|
||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_env.ps1 | ||||||
displayName: 'Dump Environment' | ||||||
- pwsh: $(System.DefaultWorkingDirectory)/xamarin-macios/tools/devops/automation/scripts/show_bot_info.ps1 | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
displayName: 'Show Bot Info' | ||||||
|
||||||
- task: AzureKeyVault@2 | ||||||
inputs: | ||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
$Env:BUILD_REPOSITORY_NAME
isxamarin/xamarin-macios
, so this replacement doesn't work as-is.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've updated all of these to not have
xamarin-macios
hardcoded anymore.