Proposal: Add a develop Branch for Improved Workflow

[bdemann]

Summary:
We are considering introducing a develop branch that feature branches would merge into. The develop branch would then merge into main less frequently, potentially only during releases. This workflow aims to address several challenges we’re currently facing with the existing main-centric approach.

---

Challenges Addressed:

1. Documentation Deployment:

   • Currently, documentation is deployed as soon as changes are merged into main.
   • This means users may see documentation for unreleased features, causing confusion when the docs don’t match their current version.
   • A develop branch would allow us to update documentation incrementally without exposing unfinished or unreleased features.

2. Maintaining a Pristine main Branch:

   • Merging directly into main makes it harder to ensure every commit is fully vetted.
   • By merging into develop first, we could better verify the stability of changes before promoting them to main, improving confidence in the main branch.

3. Test Visibility and Reliability:

   • Tests for main run in the background with low visibility. If they fail, the code is already on main, reducing the effectiveness of the testing process.
   • A develop branch would allow us to catch and address issues before they reach main.

4. CI/CD Bottlenecks on High Merge Days:

   • On days with frequent PR merges, our CI/CD pipeline becomes overwhelmed, as new test jobs are launched for every merge into main.
   • Limited machines for running tests lead to delays and reduced efficiency.
   • A develop branch would consolidate changes and reduce the number of test jobs triggered for main, alleviating bottlenecks.

---

Proposed Solution:

• Introduce a develop branch for feature branch merges.
• Merge develop into main only during releases or after a comprehensive verification process.
• Update the documentation workflow to align with the new branching strategy, ensuring users only see docs for released features.
• Increase visibility of develop branch tests and prioritize resolving failures before merging into main.

Potential Benefits:

• More accurate and user-friendly documentation.
• Improved confidence in the stability of main.
• Reduced CI/CD bottlenecks, especially on busy merge days.
• Better test visibility and control over the release process.

[bdemann]

For the time being we are limiting the tests that run on a merge from a feature branch to main to the same tests that run during the pr, this will help point 4 for right now #2374