-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Buildable Source Package Support #1954
Comments
If we were able to merge in the fpm-cookery "recipe" functionality, then we could use FPM to build any kind of source package, not just SRPM's. |
From the parent issue:
these can still be built with rpmbuild, though, and empty %build sections is how I’ve used rpm specs for many years even prior to fpm. I do understand this may seem an unusual use, but the tooling didn’t complain (rpmbuild, mrepo, etc). Can you describe your workflow and what tools are used? I’ll share one case from myself: My first job where I used CentOS (~2007ish), we built rpm’s almost entirely with empty %build sections. This deployed dozens of applications to production systems and the workflow was mostly, as I recall:
For my scenario, I think the build steps (what rpm might call %prep, %build, etc) were run before the rpmbuild process. IIRC, this allowed us more freedom to use the developer’s build tooling and was a little easier to integrate than trying to replicate the build tooling/process in the rpm spec. |
We have our own internal Perl building tools, which have not yet been publicly released on CPAN etc. My current understanding is that we simply need the ability to utilize "recipes" from fpm-cookery in order to correctly populate |
Ahh this helps. Thanks! And, I'm not opposed -- I'm trying to think how this might fit into fpm's workflow, internals, and use cases. As for integration with fpm-cookery (or a similar recipe system approach), at this time, I'm not quite sure how it would produce a result you wanted. In my mental model of fpm and how it works, it's unclear to me what the path towards this solution would look like, but such things can still be imaginable and possible ;) The proposal in bernd/fpm-cookery#8 would have likely treated fpm-cookery recipes in fpm as an input type. In fpm, the "build" steps are part of the
After In the fpm-cookery integration issue (fpm-cookery#8), the actual steps performed by fpm-cookery (if integrated in fpm as Additionally, the proposal in #1526 is a fairly minimal change that tells rpmbuild to emit an srpm while it builds the binary rpm at the same time. The major use case I expect for #1526 is for folks who have internal policy requirements to use srpm and would basically be working around a policy this way -- that is, maybe their software production pipeline uses Using fpm-cookery with fpm to produce a SPEC file with a %build/%prep that matches what fpm-cookery executes? I'm not sure what changes would be required for that. What of non |
@jordansissel |
fpm-cookery does not support outputting source based packages, however if we were to add a similar recipe feature to FPM then we could support any source based package format. For any source based target a
The recipe file should allow you to specify everything that an SRPM spec file and Debian DSC file allow you to specify. The other source based package formats I have seen (Gentoo and Slackware) provide the same functionality as SRPM's and Source Deb's, just with a different flavor. I believe that we can encapsulate the general information/behavior of source based packages into a class such as FPM::Package::Source that all concrete source based packages could inherit from. This would allow FPM to be extended to support any kind of source based package. We could use the same strategy that fpm-cookery uses, where recipe files are just Ruby code that define a class that inherits from a recipe base class. We should try to use the exact same syntax and vocabulary that fpm-cookery uses so that it is easier to use its ideas, and so users of fpm-cookery can seemlessly use FPM to build source packages. It is important to point out that some modifications to existing code will need to be made. One example is that some package types Does this sound reasonable to you @jordansissel? If so I would be eager to start trying to solve the problem. |
Thanks for the great work as usual, @NicholasBHubbard ! |
I'm still struggling to understand what the implementation would look like. I understand the goals, or at least, I think I do, to have some general-purpose recipe which describes "how to build" or, roughly, an equivalent to rpm's %build, %prep, etc, including BuildRequires and other information used by If we're limiting out scope to only fpm and fpm-cookery as possible tools to use, alone or together, then fpm-cookery alone might be the right place, but it's not without adding more code. In the text below, I'll use rpm concepts like Build and Install stepsfpm-cookery can provide "build" steps like in this redis example, but for the most part, the build steps are not shell, they're ruby. Looking at the code for the redis example:
Only one expression, I think, causes a separate process to be invoked, Build-time dependenciesfpm-cookery has a concept of build dependencies, and it installs them using Puppet as invoked directly from Ruby. It also uses fixed strings for the build dependencies, but it requires recipe authors to specify the correct dependency names for each platform because Red Hat and Debian (and its derivatives) use different names for the same things, such as "libffi-dev" in Ubuntu and "libffi-devel" in Fedora. An example of this is found in the fpm-cookery ruby recipe Language-specific packagesI haven't checked deeply, but it's worth noting that fpm-cookery invokes fpm for a large number of tasks, such as packaging things from cpan, npm, and rubygems. When this happens, fpm-cookery doesn't have much knowledge of /how/ the package is built or installed, so it wouldn't necessarily be able to generate an appropriate %build or %install step in shell. As mentioned previously, even fpm doesn't keep an internal record of the steps required -- it doesn't intrinsically know the differences between any of the following:
The first 3 steps would be necessary to gather the package metadata (name, version, etc) and for fetching the source file(s). fpm doesn't expose these steps, nor does it have an API for representing these steps. The 4th item (build, install, clean) are a mixture of ruby and system calls, and there's no mechanism (yet) to expose these steps as with category labels like prep, build, install, clean, etc. The final item often requires installing the package itself to make that determination (as in cpan), but sometimes fpm discovers this by checking the package metadata (as in gem). In fpm's gem support, the metadata and download steps are done separate from the prep/build/install/clean steps, but even the install step isn't just "gem install" as fpm will do extra steps patching files which is written in Ruby and would need to be converted to shell. Unless we don't require shell scripts for this. It could be pure-ruby, I suppose. Nothing preventing an rpm spec from having something like the following:
All that said, I agree in order to make this work some possibly significant structural changes would be required. It feels like fpm would need to be taught a few new tricks as well as changing internally how some of these steps are executed. Copying files in Ruby would need shell equivalents (and, if not shell, whatever equivalent step syntax in any supported source package format, if we are thinking that far ahead). |
If y'all want to try a minimal prototype that would demonstrate fpm+fpm-cookery working together to produce an srpm, that could be a good first step. Short of a prototype, we could also talk through the changes required and walk through those together. |
@jordansissel |
I have created a prototype (#1963) that is meant to show how we could produce source-based packages. This prototype is somewhat hacked together with the only goal being to produce a real SRPM. I overestimated the value of fpm-cookery for this problem, and it will not be relevant going forward. I created a subclass of FPM::Package called FPM::SourcePackage that concrete source-based packages can inherit from. I then created a FPM::Recipe class that produces a generic Recipe object by parsing an inputted recipe file. To initialize a FPM::SourcePackage you must pass a file path to a recipe file, that is parsed into a FPM::Recipe object which is an instance variable of the FPM::SourcePackage class. This recipe object can be used by concrete source-based packages to write a spec file. The recipe syntax I used is a simple tag based syntax where you create section tags such as [install], [build],[download_url], etc. We can extend this syntax to support any kind of feature necessary for producing source-based packages. Do you think that this strategy will work for allowing FPM to be capable of producing any kind of source-based package? Are there any major problems you see? |
Currently only supports cpan-to-srpm Ideas for #1954
Did some thinking beyond my comment on #1963 yesterday, and here's an idea I came up with - #1967 The interface should feel familiar to anyone already using fpm:
The internals of this prototype are quite messy, but they do enough to get the job done. I'll annotate #1967 about any large internal concerns I have about the prototype and possible steps forward. Back to our srpm, we can use the rpm with rpmbuild --rebuild:
By default, rpmbuild puts things into
The RPM SPEC
|
Are y'all thinking about targeting this feature to systems other than Red Hat's platform? Debian,etc? Are we thinking newer systems like flatpak, snap, appimage, etc? |
@jordansissel As for the Red Hat question, the answer is yes we are targeting all possible packaging systems that FPM can or ever will support. Nothing we are doing is Red Hat specific, it just so happens that RPM was the first package format that I personally chose to start working with a few years ago when you and I first met here online. We have already started preparing for DEB and snap etc etc... So whatever source package solution we choose, it needs to work with DEB and any other packaging system that supports "source packages". |
Soon fpm will support unbuildable SRPMs:
#237
#1657
#1952
What we really need is support for buildable SRPMs:
#237 (comment)
bernd/fpm-cookery#8 (comment)
This is a new issue created for adding buildable SRPM support, per request of @jordansissel:
#1657 (comment)
The text was updated successfully, but these errors were encountered: