Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

♻️ CI/CD | Fix Generate Commit Data Workflow #1521

Open
1 of 2 tasks
tkapa opened this issue Oct 1, 2024 · 4 comments
Open
1 of 2 tasks

♻️ CI/CD | Fix Generate Commit Data Workflow #1521

tkapa opened this issue Oct 1, 2024 · 4 comments
Assignees
Labels
Type: Refactor A code quality improvement e.g. Tech debt

Comments

@tkapa
Copy link
Member

tkapa commented Oct 1, 2024

Cc:

Hi

Pain

Build + Deployment workflows are being choked up by a failing piece of the GitHub Workflow.

A resource request Times out, causing the workflow to fail more often than not. Either the resource should be improved to cache the required responses, or we should be more lenient with the timeout.

Image

What code could be improved?

.github/workflows

Tasks

  • Investigate
  • Fix

More Information

Screenshots

Thanks!

@tkapa tkapa added the Type: Refactor A code quality improvement e.g. Tech debt label Oct 1, 2024
@babakamyljanovssw
Copy link
Member

Update:

I did investigation on timeout failures of Azure function GetCommitHistorySyncHash in workflow runs:

  • This function logic is pretty simple get string type value which is the last commit hash from Cosmos DB and return it to client, so there is no need to implement caching in the code as this function call usually takes a few seconds when it is successful.
  • These timeouts happened before executing any function operations, so the code is not causing the timeouts.
  • I think timeouts happened due to hitting monthly service limitations for Azure App Service Plan - https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale#service-limits
  • And Production workflow hasn't failed due to Azure function timeout since October 1st, 2024. The last time this function GetCommitHistorySyncHash timed out is on September 30th, 2024. So timeout failures stopped since October.
  • The App Service Plan service limits are not per function individually, it is per Azure Function App. So I did checked which function is being used mostly and it is GetReactionsFunction, 30000 logs was returned since August 1st, 2024, due to most usage of this function, it has failed more with the timeout. I tested the reaction feature of rules page, and seems it is not working properly. All requests to GetReactionsFunction, ReactFunction, RemoveReactionFunction are giving CORS error from rules page. We might need new PBI for reaction feature to investigate it and handle this feature more properly

Image
Figure: Liking and removing like are giving CORS error

@babakamyljanovssw
Copy link
Member

babakamyljanovssw commented Oct 12, 2024

Drafted issues raised after investigations of Azure Functions

#1556
#1557

@babakamyljanovssw
Copy link
Member

Update:

The timeout failures of Azure Functions are not happening anymore in workflow runs

@babakamyljanovssw
Copy link
Member

Since the timeout are not happening in recent workflows, I've sent email to myself to check this issue in one month if it is occurring again.

Email subject: Azure Functions - Timeout failures on workflows

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Refactor A code quality improvement e.g. Tech debt
Projects
None yet
Development

No branches or pull requests

3 participants