-
-
Notifications
You must be signed in to change notification settings - Fork 406
CI: Fix Makefiles for parallel builds #2825
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
jimklimov
merged 72 commits into
networkupstools:master
from
jimklimov:fix-makefile-docs-check-changelog
Mar 1, 2025
Merged
CI: Fix Makefiles for parallel builds #2825
jimklimov
merged 72 commits into
networkupstools:master
from
jimklimov:fix-makefile-docs-check-changelog
Mar 1, 2025
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
❌ Build nut 2.8.2.2778-master failed (commit 693f88eb04 by @jimklimov) |
…s and renders once (for unmodified sources) Signed-off-by: Jim Klimov <[email protected]>
… once (for unmodified sources) Signed-off-by: Jim Klimov <[email protected]>
… recipes were hopefully rectified Signed-off-by: Jim Klimov <[email protected]>
…en do "all-recursive" and no longer retry "doc" just-in-case Signed-off-by: Jim Klimov <[email protected]>
On the one hand, code builds are well-parallelized, easy to avoid extra work when re-making, and are more useful in typical development cycles. The ChangeLog HTML/PDF generation is a time-consuming hog when enabled, and with automake's "all-recursive" implementation handling SUBDIRS only sequentially, it blocks the useful part of build for a long time when we e.g. `git commit` the code changes during development. Signed-off-by: Jim Klimov <[email protected]>
…fter "all" Signed-off-by: Jim Klimov <[email protected]>
… of automake SUBDIRS and how it hurts parallel builds of NUT Signed-off-by: Jim Klimov <[email protected]>
…), and detail the dependencies Signed-off-by: Jim Klimov <[email protected]>
…CS_NO_MAN option to simplify parallel work from top-level Makefile while allowing legacy "cd docs && make" to still do the right thing Signed-off-by: Jim Klimov <[email protected]>
…es more visibly Signed-off-by: Jim Klimov <[email protected]>
Signed-off-by: Jim Klimov <[email protected]>
…ual libs Signed-off-by: Jim Klimov <[email protected]>
Signed-off-by: Jim Klimov <[email protected]>
Signed-off-by: Jim Klimov <[email protected]>
…Windows (target-dependent) Signed-off-by: Jim Klimov <[email protected]>
Signed-off-by: Jim Klimov <[email protected]>
…)built Signed-off-by: Jim Klimov <[email protected]>
Signed-off-by: Jim Klimov <[email protected]>
…common.la and nut_version.h Signed-off-by: Jim Klimov <[email protected]>
…itionals in scripts/Windows/Makefile.am (note HAVE_MINGW_RESGEN too) Signed-off-by: Jim Klimov <[email protected]>
…r quicker SUBDIRS_ALL_LIBS_LOCAL Signed-off-by: Jim Klimov <[email protected]>
… not just "all" per se In automake, "all: all-recursive" seems to be always defined, so we must be a dependency of that, otherwise we ended up building BOTH the default "all-recursive" and the optimized parallel sub-make, so they wrote into same files and crashed. Signed-off-by: Jim Klimov <[email protected]>
Signed-off-by: Jim Klimov <[email protected]>
… if transitive Signed-off-by: Jim Klimov <[email protected]>
|
✅ Build nut 2.8.2.2798-master completed (commit 1e93f74b80 by @jimklimov) |
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 1, 2025
…e -j` [networkupstools#2825] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 1, 2025
…e -j` [networkupstools#2825] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 1, 2025
…e -j` [networkupstools#2825] Signed-off-by: Jim Klimov <[email protected]>
This was referenced Mar 2, 2025
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 12, 2025
…otools [networkupstools#2842, networkupstools#2825] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 12, 2025
…etworkupstools#2825, networkupstools#2842] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 12, 2025
…etworkupstools#2825, networkupstools#2842] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to sshambar/nut
that referenced
this pull request
Mar 12, 2025
…kupstools#2825, networkupstools#2842] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to sshambar/nut
that referenced
this pull request
Mar 12, 2025
…hain [networkupstools#2825, networkupstools#2842] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to sshambar/nut
that referenced
this pull request
Mar 12, 2025
…ILT_SOURCES so it is made early [networkupstools#2825, networkupstools#2842] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 15, 2025
… SUBDIRS of main Makefile; avoid two definitions of "all" goal there [networkupstools#2842, networkupstools#2825] Signed-off-by: Jim Klimov <[email protected]>
nut-travis
pushed a commit
that referenced
this pull request
Mar 15, 2025
Signed-off-by: Jim Klimov <[email protected]>
nut-travis
pushed a commit
that referenced
this pull request
Mar 15, 2025
Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 16, 2025
Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to sshambar/nut
that referenced
this pull request
Mar 16, 2025
…orkupstools#2825] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 17, 2025
…ols#2825] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 17, 2025
…ols#2825] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Apr 18, 2025
…(generate sources via nut-scanner-deps/tools only once) [networkupstools#2825] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Apr 18, 2025
…(generate sources via nut-scanner-deps/tools only once) [networkupstools#2825] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Jul 28, 2025
…nd it has work to do only once [follows up from PR networkupstools#2825] In practice we do call it as many times as there are listed dependency files, but it only generates them on the first run and quicky bails out later on. This should help avoid writing into same files in parallel, ending up with garbled or incomplete files when we want to use them. Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Jul 28, 2025
…ipe [follows up from PR networkupstools#2825] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Jul 28, 2025
…ncies [follows up from PR networkupstools#2825, highlights why issue networkupstools#2584 better be solved] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Jul 28, 2025
…nd it has work to do only once [follows up from PR networkupstools#2825] In practice we do call it as many times as there are listed dependency files, but it only generates them on the first run and quicky bails out later on. This should help avoid writing into same files in parallel, ending up with garbled or incomplete files when we want to use them. Signed-off-by: Jim Klimov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
CI
Entries related to continuous integration infrastructure (here CI = tools + scripts + recipes)
documentation
packaging
portability
We want NUT to build and run everywhere possible
refactor/fightwarn
PR or issue proposal to improve code maintainability without functional changes, or to fix warnings
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Standard automake
SUBDIRSallow for nested projects with every directory being buildable alone as well, however the recursive builds are effectively sequential (iterating over the listed directories one by one). This is nicely deterministic for sequential builds, but not so nice for parallel builds with unbalanced workload in the directories (noted whenChangeLog.(pdf|html)takes minutes to build while the other cores are quiet).A commonly recommended solution is to collapse everything to one Makefile which is not too feasible for us.
So this PR goes another way, to explicitly define the web of known dependencies between directories (or specifically the header files and our numerous helper, mock and client libraries that can be used in other directories too). This ends up with a highly parallelisable build routine, where docs and code can consume CPU simultaneously and less wall-clock time gets spent overall. However sequential builds, as well as a dozen of less time-critical recursive goals like
checkorinstallof built artifacts, are handled by exactly same code as before (whatever automake generates).This PR also revises interaction between root, docs and docs/man Makefiles, to minimize duplicate work that might be done to build or check documentation - this is another notable time hog, so sad to do it e.g. every time we pass a
make check(sometimes several times per build).Hopefully NUT CI farm turnaround will be a bit faster now.
UPDATE: Hopefully the later commits also definitively resolve the mix of our desire to build a new set of
ChangeLog*products when git metadata changes, but to not roll rebuilding them in ordinary cycles that visit these files several times (make distchecketc.)