Stuart Feedback #4369

makubacki · 2023-05-09T14:10:22Z

makubacki
May 9, 2023
Collaborator

A tool called Stuart is integrated with edk2 to help provide a consistent build and dependency management interface across different user environments.

Quick references:

Repo that hosts Stuart - edk2-pytool-extensions
User documentation area - User Documentation
TianoCore Specific Stuart Build Instructions - How to Build with Stuart

Feedback has been received on numerous occasions that Stuart:

Does not integrate well into some developer workflows - particularly on Linux
Has dependencies (like .NET) that are complicated to install - especially on various Linux distros
Has a CLI that is difficult to understand/use
Has a configuration process that is difficult to understand/use
Has a poor implementation
Is redundant/unnecessary

and so on...

This discussion was created to gather constructive feedback that can be channeled into actionable improvements. The tool aspires to provide a Python-based build environment that can accommodate common developer flows in the edk2 project, and this feedback will hopefully help bring it closer to achieving that.

I understand it can be frustrating to use something that may have fallen short of expectations. However, please try to keep feedback constructive, concise, and respectful.

ghost · 2023-05-09T14:53:35Z

ghost
May 9, 2023

One issue for me is having to pass the toolchain using TOOL_CHAIN_TAG= instead of -t (related: -t appears to have been redefined to mean "target" instead of "toolchain").
Also, specifying the build type using TARGET={RELEASE,NOOPT,DEBUG} doesn't appear to be documented.

Errors aren't always propagated:

(venv) bcran@cube:~/src/uefi/edk2$ stuart_update -c .pytool/CISettings.py
...
SECTION - 	Updated/Verified 3 dependencies
SECTION - Second pass update of environment
ERROR - Could not find appropriate folder for mu-uncrustify-release. Host(os='Linux', arch='x86', bit='64')
ERROR - Could not find appropriate folder for mu_nasm. Host(os='Linux', arch='x86', bit='64')
ERROR - Could not find appropriate folder for edk2-acpica-iasl. Host(os='Linux', arch='x86', bit='64')
ERROR - Could not find appropriate folder for mu-uncrustify-release. Host(os='Linux', arch='x86', bit='64')
ERROR - Could not find appropriate folder for mu_nasm. Host(os='Linux', arch='x86', bit='64')
ERROR - Could not find appropriate folder for edk2-acpica-iasl. Host(os='Linux', arch='x86', bit='64')
ERROR - Could not find appropriate folder for mu-uncrustify-release. Host(os='Linux', arch='x86', bit='64')
ERROR - Could not find appropriate folder for mu_nasm. Host(os='Linux', arch='x86', bit='64')
ERROR - Could not find appropriate folder for edk2-acpica-iasl. Host(os='Linux', arch='x86', bit='64')
Updating. Done
SECTION - 	Updated/Verified 3 dependencies
SECTION - Summary
PROGRESS - Success

Also, when using Edk2ToolsBuild.py, if I remove the brotli submodule code:

(venv) bcran@cube:~/src/uefi/tmp/edk2$ python3 BaseTools/Edk2ToolsBuild.py -t GCC5
SECTION - Init Self Describing Environment
SECTION - Loading Plugins
SECTION - Start Invocable Tool
Traceback (most recent call last):
  File "/home/bcran/src/uefi/tmp/edk2/BaseTools/Edk2ToolsBuild.py", line 177, in <module>
    main()
  File "/home/bcran/src/uefi/tmp/edk2/BaseTools/Edk2ToolsBuild.py", line 173, in main
    Edk2ToolsBuild().Invoke()
  File "/home/bcran/src/uefi/tmp/edk2/venv/lib/python3.11/site-packages/edk2toolext/base_abstract_invocable.py", line 236, in Invoke
    retcode = self.Go()
              ^^^^^^^^^
  File "/home/bcran/src/uefi/tmp/edk2/BaseTools/Edk2ToolsBuild.py", line 148, in Go
    raise Exception("Failed to build.")
Exception: Failed to build.

Looking in BaseTools/BaseToolsBuild/BASETOOLS_BUILD.txt there are no ERROR lines, even at:

INFO - BrotliCompress.c:20:10: fatal error: ./brotli/c/common/constants.h: No such file or directory
INFO -    20 | #include "./brotli/c/common/constants.h"
INFO -       |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO - compilation terminated.

7 replies

makubacki May 10, 2023
Collaborator Author

Thanks for checking. An issue here is that the GNUmakefile errors need to be recognized and logged as ERROR. I filed a bug to track it - tianocore/edk2-pytool-extensions#539.

makubacki May 10, 2023
Collaborator Author

One issue for me is having to pass the toolchain using TOOL_CHAIN_TAG= instead of -t (related: -t appears to have been redefined to mean "target" instead of "toolchain"). Also, specifying the build type using TARGET={RELEASE,NOOPT,DEBUG} doesn't appear to be documented.

You can eventually see some help:

(venv3.10) ➜  edk2 git:(master) ✗ stuart_ci_build -c .pytool/CISettings.py --help
usage: stuart_ci_build [-h] [-p PACKAGELIST] [-a REQUESTED_ARCH] [-t REQUESTED_TARGET] [-force_piptools | -no_piptools]
                       [--build-config BUILD_CONFIG] [--verbose]

options:
  -h, --help            show this help message and exit
  -p PACKAGELIST, --pkg PACKAGELIST, --pkg-dir PACKAGELIST
                        Optional - A package or folder you want to update (workspace relative).Can list multiple by doing -p
                        <pkg1>,<pkg2> or -p <pkg3> -p <pkg4>
  -a REQUESTED_ARCH, --arch REQUESTED_ARCH
                        Optional - CSV of architecutres requested to update. Example: -a X64,AARCH64
  -t REQUESTED_TARGET, --target REQUESTED_TARGET
                        Optional - CSV of targets requested to update. Example: -t DEBUG,NOOPT
  -force_piptools, --fpt
                        Force the system to use pip tools
  -no_piptools, --npt   Force the system to not use pip tools
  --build-config BUILD_CONFIG
                        Provide shell variables in a file
  --verbose, --VERBOSE, -v
                        verbose

CLI Env Guide:
  <key>=<value>              - Set an env variable for the pre/post build process
  <key>                      - Set a non-valued env variable for the pre/post build process
  BLD_*_<key>=<value>        - Set a build flag for all build types
                               (key=value will get passed to build process)
  BLD_*_<key>                - Set a non-valued build flag for all build types
  BLD_<TARGET>_<key>=<value> - Set a build flag for build type of <target>
                               (key=value will get passed to build process for given build type)
  BLD_<TARGET>_<key>         - Set a non-valued build flag for a build type of <target>

That also works for other commands supported by .pytool/CISettings.py like stuart_ci_setup.

Note that I had to know to use stuart_ci_build instead of stuart_build (because this is a "CI" file - .pytool/CISettings.py). If I had tried to use stuart_build:

(venv3.10) ➜  edk2 git:(master) ✗ stuart_build -c .pytool/CISettings.py --help 
Unable to use .pytool/CISettings.py as a BuildSettingsManager
Did you mean to use a different kind of invocable?
Traceback (most recent call last):
  File "/home/mikuback/src/edk2/venv3.10/lib/python3.10/site-packages/edk2toolext/edk2_invocable.py", line 360, in ParseCommandLineOptions
    self.PlatformSettings = locate_class_in_module(
TypeError: 'NoneType' object is not callable

...<Python traceback continues on for a while>

The text "Did you mean to use a different kind of invocable?" is a hint but probably not useful to someone looking for help.

Did finding this information contribute to the issue you were having? Is the information helpful?

makubacki May 10, 2023
Collaborator Author

Issue filed for -t confusion: tianocore/edk2-pytool-extensions#540

Javagedes May 12, 2023
Collaborator

tianocore/edk2-pytool-extensions#540 has been resolved. Additional error logging has been added when either TOOL_CHAIN_TAG or TARGET has not been set, along with a graceful exit. This will be available when v0.23.3 is released.

ghost May 12, 2023

Thanks!

ghost · 2023-05-16T15:49:39Z

ghost
May 16, 2023

@makubacki I was wondering if you could take a look at my stuart_pr_eval failure at tianocore/edk2-pytool-extensions#550 ? I don't know why it's not working.

1 reply

makubacki May 16, 2023
Collaborator Author

I replied. Feel free to follow up there if more help is needed.

osteffenrh · 2023-11-21T11:40:03Z

osteffenrh
Nov 21, 2023

Does not integrate well into some developer workflows - particularly on Linux

A wish of mine would be an "offline" mode, where it does not download the toolchain (gcc, nasm, etc) but uses the tools it can find in PATH. That is useful for Linux distribution specific work where we want to use the tools of the distro.

I started looking into this a while back but then got distracted.

Has dependencies (like .NET) that are complicated to install - especially on various Linux distros

I get that .NET is convenient in come cases and it is easily available on most Linux environments, but it is a large dependency compared to what it is used for in the end (downloading dependencies via nuget, I think? Or does it do other things too?). It adds quite some size to the container images that we use in the (Linux) CI.

2 replies

makubacki Nov 22, 2023
Collaborator Author

Offline mode seems reasonable, I've heard it before. We're also reducing .NET usage. @Javagedes will follow up.

spbrogan Dec 6, 2023
Collaborator

@osteffenrh - the online/offline and even .NET dependency comes down to how a platform uses stuart. For example, if you don't leverage any nuget package based dependencies then you don't actually need .NET or nuget. As @makubacki mentioned because this has been a pain point for some edk2 developers we are trying to move edk2 away from nuget based dependencies.

The platform controls the dependencies and tools it uses. You can control this through the scopes. You could easily create an option for an edk2 platform (like ovmf, armvirt, emulatorpkg) or your own platform that disables the scopes and makes it work with tools on the path. I don't believe this is a scalable or sustainable practice in that an end user has no idea what versions of tools are supported and thus we didn't do that when setting up the CI platforms in Edk2. One of the best value points for pytools is that it lets you define a repeatable and reproducible environment using artifacts that are checked into the source tree. But regardless, pytools shouldn't need any changes to support offline mode. Again, I am not a fan of this but can understand it could be ideal or useful for some projects and pytools in general should not require online usage if a "platform" doesn't require it. Finally, since I am not sure we have built a platform using all external tools (in a long time) it is possible something has creeped in. If so please let us know so we can fix it.

Javagedes · 2024-02-15T23:56:58Z

Javagedes
Feb 15, 2024
Collaborator

As requested from @mdkinney, We've added some stuart CLI usability improvements and will leave them open for a few days for anyone to leave comments.

Review each PR to see the changes, but overall they do the following:

When using the -h flag, Print the available options for -t (target), -p (package), and -a (architecture) if a configuration file has been provided.
Return a clean error if you provide an invalid command line argument (to include -<flag>, --<flag>, or /<flag>)
Allow running a subset of tests with stuart_ci_build by setting -d or --disable-all and then turning back on the plugin you wish to run via the command line with <Plugin>=run

tianocore/edk2-pytool-extensions#753
tianocore/edk2-pytool-extensions#754
tianocore/edk2-pytool-extensions#755

0 replies

mikebeaton · 2024-09-25T12:47:09Z

mikebeaton
Sep 25, 2024

@makubacki @Javagedes -

Was there any update on moving edk2 away from NuGet based dependencies?

I agree with @osteffenrh that it seems .NET (e.g. on Linux mono-complete, which is large) is used just to download NuGet dependencies, which in context are basically versioned zip files, making .NET a very heavyweight dependency to support this.

I don't know if there's been any thought to adding GitHub releases as a supported dependency type in https://github.com/tianocore/edk2-pytool-extensions/tree/master/edk2toolext/environment/extdeptypes ? These can be arbitrarily named releases, containing arbitrarily named zip files (not necessarily builds of source in the associated repo) which I believe is about the right amount of structure to conveniently replace the way nasm, iasl and so on are currently downloaded from NuGet. I've seen other projects use this as a place to store versions of dependencies.

EDIT 1: I guess GitHub release files can be fetched already, with the existing web dependency type (haven't yet tested, but it looks like it should be fine), but perhaps it still might be convenient to make explicit the support for named files within named releases in GitHub releases, with a new dependency type, rather then just folding this into a URL?

EDIT 2: A release can be from an arbitrarily named tag on a repo, I think it must contain the (auto-generated) zip of the repo source at that tag, but can have any other arbitrary added zip files - I'm suggesting naming the release in the former way, then adding the iasl, masm, etc. zip files in the latter way - probably from a repo without much source, perhaps just a README, for this purpose. This is a very rough-and-ready demo of the idea: https://github.com/mikebeaton/edk2_ci/releases/tag/edk2-release-test-v0.1

4 replies

Javagedes Sep 25, 2024
Collaborator

@mikebeaton Hello, as you mentioned, this can already be done with the web dependency exactly as requested as above, however it is not the most ergonomic. We have an example of this in mu_basecore, as we pre-compile our basetools rather than compile them locally:

https://github.com/microsoft/mu_basecore/blob/dev/202405/BaseTools/Bin/basetoolsbin_ext_dep.yaml

  "scope": "global",
  "id": "edk-tools-bin",
  "type": "web",
  "name": "Mu-Basetools",
  "source": "https://github.com/microsoft/mu_basecore/releases/download/v2023020002.0.3/basetools-v2023020002.0.3.tar.gz",
  "version": "v2023020002.0.3",
  "sha256": "7260efe6e4e734ac66b24d6dc5f4908bf7aa444a7827cb405a66131b7b75e86f",
  "internal_path": "/basetools/",
  "compression_type": "tar",
  "flags": ["set_shell_var", "set_path", "host_specific"],
  "var_name": "EDK_TOOLS_BIN"

I'd be happy to create an external dependency specific to github releases, what are you envisioning in terms of ergonomics compared to the web dependency?

I could see us converting the source to a repository and reducing it to microsoft/mu_basecore (tianocore/edk2 for edk2) and then building the final download path as:

path = f'https://github.com/{repository}/releases/download/{version}/{name}-{version}.tar.gz'

But I'd hope for a bit more ergonomic improvement between web dependencyies and github release dependencies if possible

Looking forward to your feedback. I can spend some time to get it up and running once we come up with something good.

Javagedes Sep 25, 2024
Collaborator

The issue with this, however, is that it does not work well in our current use case, which is where we re-package a different tool with a version we wish to preserve, in a release that does not match that version.
e.g. the github release is edk2-release-test-v0.1 but the zipped files are a different version (20190215.0.0 and 2.15.5). At that point I kind of feel like the web dependency would be better, as you specify the source path directly.

mikebeaton Sep 25, 2024

Hi!

How about:

"type": "github-release",
"repo": "tianocore/edk2-ci",
"release": "mu_nasm_2.15.5",
"file": "mu_nasm_2.15.5.zip",
.
.
.

Which is then otherwise the same as a web dependency with source:

https://github.com/tianocore/edk2-ci/releases/download/mu_nasm_2.15.5/mu_nasm_2.15.5.zip

Equally:

"type": "github-release",
"repo": "tianocore/edk2-ci",
"release": "iasl.20190215.0.0",
"file": "iasl.20190215.0.0.zip",
.
.
.

Which acts like a web dependency with source:

https://github.com/tianocore/edk2-ci/releases/download/iasl.20190215.0.0/iasl.20190215.0.0.zip

Then the dependency format allows fetching an arbitrary file from an arbitrary release, but the proposed repo https://github.com/tianocore/edk2-ci keeps to one tag-plus-release, per version, per tool?

Javagedes Sep 25, 2024
Collaborator

Yes that seems good to me. I'll prototype it.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stuart Feedback #4369

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 14 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Stuart Feedback #4369

makubacki May 9, 2023 Collaborator

Replies: 5 comments · 14 replies

ghost May 9, 2023

makubacki May 10, 2023 Collaborator Author

makubacki May 10, 2023 Collaborator Author

makubacki May 10, 2023 Collaborator Author

Javagedes May 12, 2023 Collaborator

ghost May 12, 2023

ghost May 16, 2023

makubacki May 16, 2023 Collaborator Author

osteffenrh Nov 21, 2023

makubacki Nov 22, 2023 Collaborator Author

spbrogan Dec 6, 2023 Collaborator

Javagedes Feb 15, 2024 Collaborator

mikebeaton Sep 25, 2024

Javagedes Sep 25, 2024 Collaborator

Javagedes Sep 25, 2024 Collaborator

mikebeaton Sep 25, 2024

Javagedes Sep 25, 2024 Collaborator

makubacki
May 9, 2023
Collaborator

Replies: 5 comments 14 replies

ghost
May 9, 2023

makubacki May 10, 2023
Collaborator Author

makubacki May 10, 2023
Collaborator Author

makubacki May 10, 2023
Collaborator Author

Javagedes May 12, 2023
Collaborator

ghost
May 16, 2023

makubacki May 16, 2023
Collaborator Author

osteffenrh
Nov 21, 2023

makubacki Nov 22, 2023
Collaborator Author

spbrogan Dec 6, 2023
Collaborator

Javagedes
Feb 15, 2024
Collaborator

mikebeaton
Sep 25, 2024

Javagedes Sep 25, 2024
Collaborator

Javagedes Sep 25, 2024
Collaborator

Javagedes Sep 25, 2024
Collaborator