A CI system for Edk2-Platforms

[spbrogan]

The Tools and CI meeting (held every Monday at 4:30 Pacific time) is going to discuss changes to edk2-platforms to enable CI. To make this more practical there are going to be changes necessary to each platform and the repository. Most likely platforms without an active maintainer will need to be "archived".

If you have ideas and are willing to help, please use this discussion topic to communicate them. Next Monday (12/12) at the tools and ci meeting we will recap the thread and try to finalize a plan.

Thanks

[spbrogan]

A few of the topics we could discuss. Please share your own ideas. If you are an active maintainer, what would you like to see? What would help improve quality and lower effort to maintain?

1. Removing unsupported or inactive platforms
2. Splitting the repo up so that each repo is smaller and can target repo level assets at the platform(s) that it supports:
   1. readme can apply to the platform
   2. contribution and support can be targeted at model desire by platform maintainers
   3. tools and dependencies can be targeted (ideally ext_deps)
   4. submodules can be used for co-development repository development
   5. Common feature code vs platform code is differentiated, and the different life cycles and requirements are acknowledged and adhered to.
3. automations that can be enabled to lower overhead (dependabot, labels, etc)
4. Releases and release management
5. Aligning build process using stuart tools for easy CI integration and future enhancements

[spbrogan]

Thanks for the context and I share many of your thoughts and frustrations but where I differ is that Stuart doesn't help "simple platforms" and/or makes it worse. :)

Stuart makes CI and local builds more consistent. In the context of platform CI, it significantly lowers the cost/effort to maintain the server-based build steps while still allowing enough platform flexibility. Your QemuOpenBoardPkg still needs a NASM compiler and ASL compiler. It still only has compatibility with some C compilers and tool chains. It still either assumes everything it needs is on the "path" or has some sort of environment variable set which is then used to resolve things. It still has the dependency on the tools required from Edk2 which it must include somewhere and are everchanging. When it is built on Windows vs Linux vs Cloud hosted VM vs container vs WSL these are all different and users have problems setting them up consistently and/or knowing what the suggested/supported configuration and environment is. The configuration/environment is not in code and not managed by git and thus building something from 6 month ago often fails and requires a different set of dependencies and this information is not captured consistently.

Stuart also provides a framework for adding useful functionality. For Platform CI in QemuOpenBoardPkg, i would expect we would want to first add booting to the shell and making sure that was successful. That means launching QEMU with a whole bunch of parameters. That means mapping a drive. That means building a startup.nsh file. That means determining if it was successful. Next, we would want to run some shell-based tests and confirm that they pass. From there who knows what else. Maybe we would add QemuSbsaPkg and want the same things. Well, Stuart would enable a lot of that with minimal code. See the PlatformCI in edk2 (OvmfPkg, ArmVirtPkg, and EmulatorPkg). Or better yet, see the PlatformCI in https://github.com/microsoft/mu_tiano_platforms as the enhancements there enable a lot of automatic validation on a qemu based platform.

For more philosophical arguments - I don't see how adding more tooling on top of Tianocore's already very proprietary build tooling is a good idea. If you need custom prebuild and postbuild steps, I would encourage you to add support to BaseTools for that.

I view it as a necessary evil and not that much more evil. Stuart tries to use the same files and follow the same rules, so it tries not to add too much more custom stuff. It actually tries to use the same information so that you don't need another set of defines/configuration switches that have to be kept in-sync. Ideally, it would use the same exact parsers from basetools but as you noticed no one is willing to make fundamental changes/improvements to make that work (and if someone was willing to invest the energy in that we probably should start over and align with an existing build tools). This conversation has been ongoing for years with little traction other than most agree it would be nice.

Finally, there is a lot of things I don't like about Stuart and a lot of limitations and problems that I don't plan to invest in fixing (like the usage of PCDs as build configuration flags which breaks QemuOpenBoardPkg). As we have mentioned, it is built on top of a very shaky foundation of edk2 basetools. But it works for the scenarios I use it and it makes managing a very broad set of platforms and CI practical. I don't see a sustainable way to enable CI in edk2-platforms or edk2 without using Stuart and without asking each of the platform maintainers to convert and maintain their build using Stuart.

Again, I look forward to other feedback in this discussion and more importantly the contributions others are willing to make. Thanks

[nate-desimone]

@spbrogan - You could have chosen to make the new features of stuart part of BaseTools from the very beginning. Instead you choose to do your own thing in the context of the MU project with zero collaboration with the wider TianoCore community during the early formative phases of mu_tools. Now, you want everyone else in the TianoCore community to do build/CI your way. And moreover, you threaten to withhold CI on edk2-platforms if we don't do it your way. If that is going to be your attitude, then we will just do CI for edk2-platforms without you.

[spbrogan]

Wow @nate-desimone. I'll try to avoid reading too much into your remarks and language choices as the goal here is to have a productive discussion and find solutions to shared community challenges. Stuart, also known as edk2-pytools, has been maintained within the Tianocore organization since 2019. Lots of opportunities for community members to collaborate. If you were not aware check them out here: https://github.com/tianocore/edk2-pytool-extensions and https://github.com/tianocore/edk2-pytool-library

Many of the investment over the last 4 years have been to support Tianocore projects for the Tianocore community. To do Edk2 CI, the community not only needs the edk2-pytools projects but also all the plugins in Edk2, build file maintenance, CI server resources, troubleshooting, etc. Anyway, my point is in those 4 years, community collaboration would have been great, but it has been very limited. Going forward collaboration would be awesome, but if the platform owners in edk2-platforms decide they don't want to use a Stuart based system then I probably am not the person to invest/lead that effort. This isn't me threatening anything, but more a reality of the time and investment I can give.

I look forward to any constructive feedback you may have.

[mhaeuser]

@spbrogan I'm not sure you can really sell the building side of things here. I know the rough idea of what Stuart is trying to achieve and it's noble, but I don't feel like within the community, there is any demand for this. Everyone has their own workflows already. It may make edk2 more tangible for newcomers, but my experience with it was I followed some GitHub doc page trying to build mainline edk2 and it threw some weird error with no details on the update command. This is not a bug report, thus I will omit checking what the issue was, but as you can imagine, that was as far as I went with a tool that doesn't really provide any benefit to me, at least locally.

However, for automated testing, that's a whole different story. I don't have the time to read the documentation tonight, but if I understood you correctly that you are automating OVMF-based tests with Stuart? Please. A good example for why something like this should be mandatory is this BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4234. Something like booting a minimal Linux and printing success to serial would already be a great milestone. I had a brief discussion with @heatd and for Ext4Pkg (and actually also QemuOpenBoardPkg, though I'm not involved there), we would like to enable whatever CI system ends up being used, so long as down the road it will allow to enable automated testing with QEMU.

@nate-desimone Is that really the discussion climate we want here? We are where we are and we have what we have. As a private contributor with a very limited amount of contributions attempted, I have a shocking amount failures on record already. Just looking at the last year: An uncontroversial patch to introduce advanced alignment macros and static assertions to validate related spec conformance (with a precedence of this actually detecting a critical bug)? One review, rest ignored, never merged. A patch to fix said terrible bug in the CLANG38 toolchain where it mis-aligns structs compared to the spec requirements? Ignored, merged only a year later by the efforts of @heatd. Various discussion sub-topics for the GSoC secure PE loader project? Minimal feedback, some of which "we cannot accept this, but also cannot tell you why", rest ignored. Dozens of filed BZs, some of which under embargo for security concerns and with patches attached? Ignored. I had about 50 branches with edk2 changes in the pipeline, some of which prerequisite for the new loader, some of which bugs I fixed along the way, and I could neither push them forward, as the fundamental patches were ignored, nor could I afford to wait as they were about to all be messed up by the Uncrustify commit series. But they were. They sit there, to this day, on my GitHub fork, never rebased against Uncrustify, as I'm not a one-man-army. The last thing on Earth I'd dream shaming @spbrogan for is not trying too hard to collaborate on literally any design with this community. I think the Mu team has a very efficient strategy (which we also tried to employ in the past), i.e., to completely design solutions that satisfy one's own needs in-house, submit them, and then see what the complaints are. The latter is where we are at right now. So, if he is asking for feedback and offers support with integrating the stack, it should be evaluated and receive feedback accordingly. I can only speak for a very limited number of folks out there publicly, but with the "if you don't want to collaborate, we'll do it without you", you may be surprised how not-alone TianoCore is, just in the opposing direction. So can we leave destructive nonsense to the side and focus on what must be improved now?

[heatd]

Going forward collaboration would be awesome, but if the platform owners in edk2-platforms decide they don't want to use a Stuart based system then I probably am not the person to invest/lead that effort. This isn't me threatening anything, but more a reality of the time and investment I can give.

To be clear, I don't oppose stuart but rather I oppose stuart as the standard invocation of the build system itself - I see no problem in using it for CI tasks as they are different in nature to normal builds.

[spbrogan]

As discussed on Dec 12th meeting, we will start enabling some of the feature packages first. If you own a feature package, then please enable your package and work to make it pass the CI. If you make changes to the CI files you can submit a PR to my branch. If you make changes to existing edk2-platforms files you can do that in edk2-platforms repo.

Branch can be seen here: https://github.com/spbrogan/edk2-platforms/tree/enable_ci_v1
Readme has directions: https://github.com/spbrogan/edk2-platforms/tree/enable_ci_v1/.pytool#readme

If you own a platform and want to enable CI please respond here and we can start to get that enabled.

[nate-desimone]

Hi all sorry for the delay in response.

@spbrogan Thank you for taking the time to get this started! Per your suggestions:

1. Removing unsupported or inactive platforms

Agreed.

2. GitHub PRs/code reviews.

Yes please! That would be a wonderful.

3. Splitting the repo up so that each repo is smaller and can target repo level assets at the platform(s) that it supports.

I think there is too many opinions on what an immaculate repository layout looks like to really make a good call here. Some think that having 1 repo per SoC is too many. Some think that having a monorepo for the entirety of edk2 is too few. I think the right answer is probably some number in between those two. I could see edk2-platforms splitting into maybe 2-4 different repos... but 10+ repos would be a needless amount of abstraction.

I don't think having a per-board repo with recursive submodules is the right idea. I'm in a position where I get to talk to many companies across the PC industry and the vast majority of them prefer a repository layout more akin to the Android Open Source Project (multiple top level independent repos). And I think that makes sense because in a lot of ways the ecosystem level considerations that we try to solve with edk2 and the TianoCore project are very similar to the ones seen by ASOP. Just like ASOP, we have a "core", then multiple 3rd party SoC/silicon vendors, and then multiple OEMs with multiple end-user designs. I think that style of enabling model naturally lends itself to an ASOP style repo layout and that is what we use here at Intel.

Personally I'd recommend we table the discussion of splitting up edk2-platforms for now. I just don't see a clear path to consensus. Besides, just getting CI working on the existing edk2-platforms would be a huge improvement so lets not make perfect the enemy of good.

4. Automations that can be enabled to lower overhead (dependabot, labels, etc)

What kind of automations did you have in mind?

5. Releases and release management

This is really something that we should have as higher priority for edk2-platforms. Right now it would be very hard to use edk2-platforms out-of-the-box and a big part of that is a complete lack of release engineering.

6. Aligning build process using stuart tools for easy CI integration and future enhancements

Agreed with @heatd here. Stuart is not something that I or most of my customers would use. The high level goals of stuart are all very admirable... the implementation however is absolutely terrible. And stuart is highly opinionated on doing platform development in a very specific way that does not match most users of edk2. Everyone that I have talked to outside of Microsoft (including several large PC manufacturers) has echoed a high level of distaste for stuart.

@mhaeuser Per your discussion on package dependencies:

There are some packages in edk2-platforms that I think are more "core" than "platform". MinPlatformPkg and AdvancedFeaturePkg for example.

The biggest thing MinPlatformPkg does is provide an abstraction from the edk2 package hierarchy to prevent the constant moving of LibraryClass/Driver .infs from impacting code that depends on those modules. AdvancedFeaturePkg does a similar job of collecting all the modules required to enable any given advanced feature together so that *BoardPkg code doesn't need to change as advanced features have modules added/removed over time.

[spbrogan]

@nate-desimone Thanks for the feedback. Let's dig a little deeper.

For 3. before tabling the discussion, I want to understand more. My position has generally been "align the repos with groups that are aligned." Here, alignment means they are committed to the same goals, they communicate and follow similar patterns and thus they can release at the same time, and they carry the same common dependencies (like edk2 version). To me that could mean between 1 to N number of platforms can be in the same repository. Finally, due to out of tree platforms wanting to consume some of the same "common features" it would be great if the common feature packages were in a repo independent of platforms. Much like how Edk2 gets consumed into many out of tree platforms, if successful, many advanced features could be consumed without major modifications that would be ideal for all.

To continue, the problem with just trying to get CI working is the mixing of dependencies, versions, and methods of doing things. If they are all different CI becomes a huge mess.

For 6. I also want to understand more. What exactly is the feedback for Stuart? What do you and/or others have distaste for? What opinion does Stuart have that you disagree with? How are you using Edk2 that doesn't fit with Stuart?

Looking forward to discussion.

Thanks
Sean

[mhaeuser]

@nate-desimone Just a comment regarding shelving. I’m not going to try to drive yet another discussion that’s nothing but pointless in the end. However, I don’t see how shelving makes sense. This is not some irrational fight to get our monorepo back, or for @spbrogan to get as many repos as he personally prefers. We are in a situation where everyone suffers exclusively from drawbacks. This could be a book example for why for “the truth lies somewhere in the middle” is a scam. No, it really doesn’t. We don’t get the simplicity and the nice inherent versioning of the monorepo, and the Mu team, as to my understanding, does not benefit from the separation as-is either. At Acidanthera, we maintain a duplicate of Ext4Pkg in AUDK (our edk2 monorepo), because edk2-platforms versioning is terrible and we don’t want to depend on it. And Mu splits up the repositories manually anyway. This is not “perfect is the enemy of good”, this is miserable. My stance is not “give us the monorepo”, but “give us the monorepo or at least fully leverage the advantages of a multi-repo design”. The current design is pure vanity and edk2-platforms functions more like a junkyard rather than a separate management unit.

Speaking of core / platform-agnostic things, Pedro’s RFC to move Ext4Pkg to edk2 has, of course, been ignored: https://edk2.groups.io/g/devel/message/96440
This change would not fix the design at all, but for @heatd, me as a reviewer, and the entire Acidanthera team, this would make things a lot less unbearable.

@spbrogan I’m a bit disappointed that you did not address most of the points here. This includes @bcran’s comments regarding vague and confusing stuart errors (I’ve run into such myself), @heatd’s comment regarding stuart adding complexity to the stack (the core issue is not really stuart, but BaseTools already has so many hidden issues that one rather wants less layers, not more), and the entirety of my posts (including overlapping points made by @heatd). I’m not saying this happens out of ill will at all, maybe you literally don’t have the time to even read all those things. But saying so would really help.

From here on, I rather want to address the entire community. For a bit of context, the sole reason I applied as a reviewer for Ext4Pkg was that I saw how terribly @heatd’s work was treated and how the previous reviewing model was slowing him down. Naturally, I needed to get up to speed on edk2 things again and I happened to engage again, here and there. And same as with GSoC, or the patches from Acidanthera before, I end up regretting every single action I take that exceeds the bare minimum of effort. @spbrogan obviously put a lot of effort in this series of proposals and so did we with our comments. Can we maybe get a demonstration of what kind of response we can expect from this community but shooting anything and anyone down or outright being ignored?

[mhaeuser]

@nate-desimone Sorry, @heatd just told me he understood that last paragraph as hostile against you and I want to clarify: NO, not at all! :) You kind of took the “shooting this down” role, but that actually is a very important role in discussion. In fact, I often take this stance myself (sometimes incorrectly so), @heatd can probably confirm. :) The issue is when that is the only kind of response ever (and that makes me look at the vague direction of those that stay silent). This remark was probably heavily influenced by a patch series titled “Disabling safe string constraint assertions”, which aimed to remove an obviously incorrect ASSERT. It was a 9-revision series that evolved from removing a single line, to changing all DebugLib instances, to abstracting a sort of DebugCommonLib, right back to removing the single line (but now with bells and whistles). It even spawned a separate discussion thread out of desperation. There was a lot of interesting discussion with non-maintainers, but each patch that evolved from their comments was shot down by the maintainers with no alternative provided. It was only saved by Laszlo and Mike concluding that trying to invent solutions to preserve bad behaviour was worse than just changing the behaviour and fixing whatever assumed it previously (thanks again!).

[spbrogan]

@mhaeuser - Sorry for not responding above in the first thread. I have been trying to keep to the "threads" in this discussion and trying not to respond to myself or make it harder to follow. Hopefully @heatd can add some context or provide some feedback on the first thread. Nate started a "new" thread here so it gave me a chance to reply. Anyway, I'll leave this as I think most of our questions here are for @nate-desimone.

[ghost]

I've just spent some time looking through edk2-pytool-extensions to see if I could work out how to add macOS and FreeBSD (and hopefully others in future, like Illumos, NetBSD, OpenBSD etc.) support, but it looks like it's not straightforward.

Once the mono invocation of NuGet.exe is fixed, then it falls over because it doesn't expect the OS to be Darwin or FreeBSD when looking for directories. I'm guessing then there will be problems fetching binaries via NuGet, so I'm giving up for now. I'll do things the old way and install dependencies using the OS' package manager.

[nate-desimone]

@mhaeuser - I don't think it is fair to say the BaseTools DSC/DEC/FDF file format is proprietary. All the tooling, documentation, and specifications for those file formats is open source. I think the words you are looking for are "unique" or perhaps "esoteric". This might be a case of English not being a first language 😄.

@spbrogan, @mhaeuser, @heatd - So it seems like there is a lot of interest in point 3 so let's start there. First of all, it wasn't my intention to "shoot it down"; my comments are more based on my previous experience working with TianoCore. Whenever I personally try to drive changes like this it tends to be 3x more effort than it should be and I tend to get burned in the process. My comment was more along the lines of "It is really worth the effort to litigate?"

Given that it seems all of you have a great level of interest in driving changes here, let's see what we can do to come with with a compromise. So it sounds like we are all agreed that monorepo isn't really great for TianoCore due to the high level of out-of-tree use, so at least for now I'm not going to think a whole lot about what that would look like. If someone wants to come up with a proposal there I would of coarse be happy to look at it. So what are some alternate repo structures that might make sense? Well I can think of 2 options that would probably work well:

Option 1 - Add an edk2-boards repo (My preference)

Change definition of edk2-platforms from "a dumpster for everything not core" to "common platform code". Here is how I think things would end up looking in this case (note that I'm not well versed with the architecture of the ARM platforms, so there might be some stuff that belongs in edk2-platforms that I'm missing here):

edk2-platforms
├───Drivers/*
├───Features/*
├───Platform
│   ├───Intel
│   │   ├───BoardModulePkg
│   │   ├───MinPlatformPkg
│   │   └───Tools
│   └───StandaloneMm
│       └───PlatformStandaloneMmPkg
└───Silicon
    └───Intel
        ├───IntelSiliconPkg
        └───Tools

edk2-boards
├───Platform
│   ├───96Boards/*
│   ├───AMD/*
│   ├───Ampere/*
│   ├───ARM/*
│   ├───BeagleBoard/*
│   ├───Hisilicon/*
│   ├───Intel
│   │   ├───CometlakeOpenBoardPkg/*
│   │   ├───KabylakeOpenBoardPkg/*
│   │   ├───PurleyOpenBoardPkg/*
│   │   ├───QuarkPlatformPkg/*
│   │   ├───SimicsOpenBoardPkg/*
│   │   ├───TigerlakeOpenBoardPkg/*
│   │   ├───Vlv2TbltDevicePkg/*
│   │   ├───WhiskeylakeOpenBoardPkg/*
│   │   └───WhitleyOpenBoardPkg/*
│   ├───LeMaker/*
│   ├───Loongson/*
│   ├───Marvell/*
│   ├───NXP/*
│   ├───Phytium/*
│   ├───Qemu/*
│   ├───RaspberryPi/*
│   ├───RISC-V/*
│   ├───SiFive/*
│   ├───Socionext/*
│   ├───SoftIron/*
│   └───SolidRun/*
└───Silicon
    ├───AMD/*
    ├───Ampere/*
    ├───ARM/*
    ├───Atmel/*
    ├───Broadcom/*
    ├───Hisilicon/*
    ├───Intel
    │   ├───CoffeelakeSiliconPkg/*
    │   ├───KabylakeSiliconPkg/*
    │   ├───PurleyRefreshSiliconPkg/*
    │   ├───QuarkSocPkg/*
    │   ├───SimicsIch10Pkg/*
    │   ├───SimicsX58SktPkg/*
    │   ├───TigerlakeSiliconPkg/*
    │   ├───Vlv2DeviceRefCodePkg/*
    │   └───WhitleySiliconPkg/*
    ├───Marvell/*
    ├───Maxim/*
    ├───NXP/*
    ├───Openmoko/*
    ├───Phytium/*
    ├───Qemu/*
    ├───RISC-V/*
    ├───SiFive/*
    ├───Socionext/*
    ├───Synopsys/*
    └───TexasInstruments*

Option 2 - Add multiple ISA specific edk2-boards repos

Similar to Option 1, but with platforms broken out by ISA. This might be useful for CI because the compilers/toolchains tend to vary by ISA. The downside to this is added barriers to collaboration between different silicon companies on edk2 development. I see cross collaboration as a big positive for the TianoCore community so for that reason I don't like Option 2 as much as Option 1.

edk2-platforms
├───Drivers/*
├───Features/*
├───Platform
│   ├───Intel
│   │   ├───BoardModulePkg
│   │   ├───MinPlatformPkg
│   │   └───Tools
│   └───StandaloneMm
│       └───PlatformStandaloneMmPkg
└───Silicon
    └───Intel
        ├───IntelSiliconPkg
        └───Tools

edk2-boards-x86
├───Platform
│   ├───Intel
│   │   ├───CometlakeOpenBoardPkg/*
│   │   ├───KabylakeOpenBoardPkg/*
│   │   ├───PurleyOpenBoardPkg/*
│   │   ├───QuarkPlatformPkg/*
│   │   ├───SimicsOpenBoardPkg/*
│   │   ├───TigerlakeOpenBoardPkg/*
│   │   ├───Vlv2TbltDevicePkg/*
│   │   ├───WhiskeylakeOpenBoardPkg/*
│   │   └───WhitleyOpenBoardPkg/*
│   └───Qemu/*
└───Silicon
    └───Intel
        ├───CoffeelakeSiliconPkg/*
        ├───KabylakeSiliconPkg/*
        ├───PurleyRefreshSiliconPkg/*
        ├───QuarkSocPkg/*
        ├───SimicsIch10Pkg/*
        ├───SimicsX58SktPkg/*
        ├───TigerlakeSiliconPkg/*
        ├───Vlv2DeviceRefCodePkg/*
        └───WhitleySiliconPkg/*

edk2-boards-arm
├───Platform
│   ├───96Boards/*
│   ├───AMD/*
│   ├───Ampere/*
│   ├───ARM/*
│   ├───BeagleBoard/*
│   ├───Hisilicon/*
│   ├───LeMaker/*
│   ├───Marvell/*
│   ├───NXP/*
│   ├───Phytium/*
│   ├───Qemu/*
│   ├───RaspberryPi/*
│   ├───Socionext/*
│   ├───SoftIron/*
│   └───SolidRun/*
└───Silicon
    ├───AMD/*
    ├───Ampere/*
    ├───ARM/*
    ├───Atmel/*
    ├───Broadcom/*
    ├───Hisilicon/*
    ├───Marvell/*
    ├───Maxim/*
    ├───NXP/*
    ├───Openmoko/*
    ├───Phytium/*
    ├───Qemu/*
    ├───Socionext/*
    ├───Synopsys/*
    └───TexasInstruments*

edk2-boards-risc-v
├───Platform
│   ├───RISC-V/*
│   └───SiFive/*
└───Silicon
    ├───RISC-V/*
    └───SiFive/*

edk2-boards-loongarch
└───Platform
    └───Loongson/*

[nate-desimone]

Unfortunately it seems like can't have nested replies on GitHub, specifically addressing @heatd:

If you go way back in the edk2 history to about ~2007, you will see that the original BaseTools used Apache Ant and Java. We dodged a bullet by moving BaseTools away from ant to Python since the Java lawsuits of the mid-2010s did not affect edk2.

Before edk2 came into existence, we actually did use plain Makefiles as you suggest for edk1. There was a lot of learning from edk1 that went into the design of edk2. The big problem we ran into with Makefiles is that there are many different implementations of make with different syntax. And there are many different C compilers with different command line parameters. Edk1 only works well with the Windows Server 2003 DDK C compiler and the Microsoft assembler (MASM) v6.2. One of the goals of edk2 was to be fully cross-platform, both for the OS environment used to compile and for the C compiler used. The level of abstraction BaseTools provides from the underlying toolchain is easy to take for granted for those who have not tried to write a cross-platform Makefile.

And BaseTools has a lot of features that are difficult to accomplish with Makefiles. For example, the PCD database and structured PCDs. The ability to override LibraryClasses is also extremely useful for firmware use cases. For example, it is easy for a silicon designer to provide a "default" implementation of SerialPortLib that uses the UART built-in to the SOC, and it is just as easy for the person porting to their motherboard design to override it with a platform specific UART device (for example, a UART built-in to the EC.)

I do agree with you though that just getting a build up and running from scratch is much harder than it should be. Approximately 5 years ago there was some early brainstorming on what an "edk3" might look like, but that is difficult to justify with how successful and established edk2 has become.

[nate-desimone]

@mhaeuser:

If OVMF and EmulatorPkg were moved, that would make the dependency tree a lot more sensical. However, this still does not fix Ext4Pkg's misplacement. FatPkg is in edk2, UDF drivers are in edk2, why would Ext4Pkg belong in edk2-platforms?

Because FAT32 is mandatory per the UEFI specification, whereas ext4 is not. Because of this, an ext4 driver looks a lot more like an "Option ROM". We used to have OpROMs in edk2 but they got moved edk2-platforms/Drivers/*

[heatd]

Unfortunately it seems like can't have nested replies on GitHub, specifically addressing @heatd:

If you go way back in the edk2 history to about ~2007, you will see that the original BaseTools used Apache Ant and Java. We dodged a bullet by moving BaseTools away from ant to Python since the Java lawsuits of the mid-2010s did not affect edk2.

Before edk2 came into existence, we actually did use plain Makefiles as you suggest for edk1. There was a lot of learning from edk1 that went into the design of edk2. The big problem we ran into with Makefiles is that there are many different implementations of make with different syntax. And there are many different C compilers with different command line parameters. Edk1 only works well with the Windows Server 2003 DDK C compiler and the Microsoft assembler (MASM) v6.2. One of the goals of edk2 was to be fully cross-platform, both for the OS environment used to compile and for the C compiler used. The level of abstraction BaseTools provides from the underlying toolchain is easy to take for granted for those who have not tried to write a cross-platform Makefile.

Right. In no way shape or form am I suggesting we use raw make and try to write something that works across gmake/bmake/nmake - that's simply unrealistic. You have plenty of current solutions that are indeed cross platform, fast, well maintained (Bazel, gn, CMake, meson, etc). As is, BaseTools is simply a very baroque build system maintained by firmware engineers :)

And BaseTools has a lot of features that are difficult to accomplish with Makefiles. For example, the PCD database and structured PCDs. The ability to override LibraryClasses is also extremely useful for firmware use cases. For example, it is easy for a silicon designer to provide a "default" implementation of SerialPortLib that uses the UART built-in to the SOC, and it is just as easy for the person porting to their motherboard design to override it with a platform specific UART device (for example, a UART built-in to the EC.)

Again, I'm not saying there should be a switch to Makefiles. LibraryClasses is useful but also makes it incredibly hard to figure out what uses what, without using report files, etc.

Large projects with more traditional build systems (Linux, coreboot, BSDs, etc) have shown that you don't need 30000 levels of abstraction and overriding to have a generic enough build system that it just works. I don't see how a plain old declarative build system (or build system syntax) wouldn't work here.

I do agree with you though that just getting a build up and running from scratch is much harder than it should be. Approximately 5 years ago there was some early brainstorming on what an "edk3" might look like, but that is difficult to justify with how successful and established edk2 has become.

Is it? edk2 had a good run, but can you do better?

[heatd]

@mhaeuser:

If OVMF and EmulatorPkg were moved, that would make the dependency tree a lot more sensical. However, this still does not fix Ext4Pkg's misplacement. FatPkg is in edk2, UDF drivers are in edk2, why would Ext4Pkg belong in edk2-platforms?

Because FAT32 is mandatory per the UEFI specification, whereas ext4 is not. Because of this, an ext4 driver looks a lot more like an "Option ROM". We used to have OpROMs in edk2 but they got moved edk2-platforms/Drivers/*

@nate-desimone But edk2 also contains a large number of non-UEFI-essential packages. What about PrmPkg? What about IntelFsp*Pkg which are quite literally in the middle of edk2.git despite being Intel specific. NetworkPkg? EmbeddedPkg? RedfishPkg? I could go on all day. None of these are required for EFI spec compliance. So what's the criteria here?

It's worth noting that Ext4Pkg is not a hardware driver and as such is fully independent from the hardware, platform, etc it's running on. So why is it in edk2-platforms? The only difference is that FatPkg is required for EFI compliance. And, as far as I know, if so, it should be added to MdePkg while Ext4Pkg gets added to edk2/.

The whole UsbNetworkPkg to edk2 deal is really pissing me off here. It seems that the only criteria used here has been "Project developed by the oh-so-great-OEMs-and-IBVs" vs "Tinpot GSoC project". And I'll let you know the code quality in my "tinpot project" is much higher than whatever crap the IBVs want to push upstream this time.

(At least the IBVs are figuring out there's an "upstream" they can dump code into. Baby steps...)

[nate-desimone]

@heatd - The FAT32 driver wouldn't go into MdePkg, it would be MdeModulePkg since it is part of implementing the spec as opposed to pure definitions of the spec itself. It was kept separate from MdeModulePkg because of legal/licensing reasons and because the amount of code is large.

NetworkPkg and RedfishPkg are reference implementations of UEFI spec defined protocols, so similar justification. PRM is still a UEFI forum draft spec, but theoretically that will end up in ACPI at some point.

You are right though that there are definitely some packages in there that seem "extra": ArmPkg, DynamicTablesPkg, EmbeddedPkg, PcAtChipsetPkg, FmpDevicePkg, etc.

I wasn't keeping track of that UsbNetworkingPkg stuff... yeah I would argue that looks like it should be in edk2-platforms as well. Personally, I don't consider your ext4 driver to be some "Tinpot GSoC project," I've been hoping someone would implement that for many years now so truly and sincerely I thank you for doing that work. I have been advocating for making it included in all MinPlatform based builds by default actually. Personally, I treat your contributions the same as all the ones I review. I'm really sorry about all this and I'll see what I can do.

[iworam]

We talked about requirements for a package to be unit tested. Can we make a list of those requirements? I have done most of the work to get all the Features/Intel packages building, but I want the checklist so I can call a package done and get it added to CI.

I was thinking package builds with:
VS2019
GCC5
CLANGPDB

And then I need a list of the existing CI tests that are expected to work, if someone knows that.

Is there anything else?
DEBUG/RELEASE/NOOPT
32/64b

[spbrogan]

For Core CI you can read about most of the tests here:
https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#current-pytool-test-capabilities
There have been a couple of additional tests added if you look thru the plugins listed here:
https://github.com/tianocore/edk2/tree/master/.pytool/Plugin

[spbrogan]

A github action has been setup and is working for Core CI.
https://github.com/spbrogan/edk2-platforms/blob/master/.github/workflows/Windows-VS-CI.yml
https://github.com/spbrogan/edk2-platforms/actions

Again, feedback is welcome and would be appreciated. PRs to enable more feature packages would be great.