Skip to content

Conversation

@jimklimov
Copy link
Member

@jimklimov jimklimov commented Feb 26, 2025

Standard automake SUBDIRS allow for nested projects with every directory being buildable alone as well, however the recursive builds are effectively sequential (iterating over the listed directories one by one). This is nicely deterministic for sequential builds, but not so nice for parallel builds with unbalanced workload in the directories (noted when ChangeLog.(pdf|html) takes minutes to build while the other cores are quiet).

A commonly recommended solution is to collapse everything to one Makefile which is not too feasible for us.

So this PR goes another way, to explicitly define the web of known dependencies between directories (or specifically the header files and our numerous helper, mock and client libraries that can be used in other directories too). This ends up with a highly parallelisable build routine, where docs and code can consume CPU simultaneously and less wall-clock time gets spent overall. However sequential builds, as well as a dozen of less time-critical recursive goals like check or install of built artifacts, are handled by exactly same code as before (whatever automake generates).

This PR also revises interaction between root, docs and docs/man Makefiles, to minimize duplicate work that might be done to build or check documentation - this is another notable time hog, so sad to do it e.g. every time we pass a make check (sometimes several times per build).

Hopefully NUT CI farm turnaround will be a bit faster now.

UPDATE: Hopefully the later commits also definitively resolve the mix of our desire to build a new set of ChangeLog* products when git metadata changes, but to not roll rebuilding them in ordinary cycles that visit these files several times (make distcheck etc.)

@jimklimov jimklimov added packaging CI Entries related to continuous integration infrastructure (here CI = tools + scripts + recipes) refactor/fightwarn PR or issue proposal to improve code maintainability without functional changes, or to fix warnings portability We want NUT to build and run everywhere possible labels Feb 26, 2025
@jimklimov jimklimov added this to the 2.8.3 milestone Feb 26, 2025
@AppVeyorBot
Copy link

…s and renders once (for unmodified sources)

Signed-off-by: Jim Klimov <[email protected]>
… once (for unmodified sources)

Signed-off-by: Jim Klimov <[email protected]>
… recipes were hopefully rectified

Signed-off-by: Jim Klimov <[email protected]>
…en do "all-recursive" and no longer retry "doc" just-in-case

Signed-off-by: Jim Klimov <[email protected]>
On the one hand, code builds are well-parallelized, easy to avoid extra
work when re-making, and are more useful in typical development cycles.
The ChangeLog HTML/PDF generation is a time-consuming hog when enabled,
and with automake's "all-recursive" implementation handling SUBDIRS only
sequentially, it blocks the useful part of build for a long time when we
e.g. `git commit` the code changes during development.

Signed-off-by: Jim Klimov <[email protected]>
… of automake SUBDIRS and how it hurts parallel builds of NUT

Signed-off-by: Jim Klimov <[email protected]>
…), and detail the dependencies

Signed-off-by: Jim Klimov <[email protected]>
…CS_NO_MAN option to simplify parallel work from top-level Makefile while allowing legacy "cd docs && make" to still do the right thing

Signed-off-by: Jim Klimov <[email protected]>
…common.la and nut_version.h

Signed-off-by: Jim Klimov <[email protected]>
…itionals in scripts/Windows/Makefile.am (note HAVE_MINGW_RESGEN too)

Signed-off-by: Jim Klimov <[email protected]>
…r quicker SUBDIRS_ALL_LIBS_LOCAL

Signed-off-by: Jim Klimov <[email protected]>
… not just "all" per se

In automake, "all: all-recursive" seems to be always defined,
so we must be a dependency of that, otherwise we ended up
building BOTH the default "all-recursive" and the optimized
parallel sub-make, so they wrote into same files and crashed.

Signed-off-by: Jim Klimov <[email protected]>
@AppVeyorBot
Copy link

@jimklimov jimklimov merged commit b7f3f95 into networkupstools:master Mar 1, 2025
30 checks passed
@jimklimov jimklimov deleted the fix-makefile-docs-check-changelog branch March 1, 2025 09:35
jimklimov added a commit to jimklimov/nut that referenced this pull request Mar 1, 2025
jimklimov added a commit to jimklimov/nut that referenced this pull request Mar 1, 2025
jimklimov added a commit to jimklimov/nut that referenced this pull request Mar 1, 2025
jimklimov added a commit to jimklimov/nut that referenced this pull request Mar 12, 2025
jimklimov added a commit to sshambar/nut that referenced this pull request Mar 12, 2025
jimklimov added a commit to sshambar/nut that referenced this pull request Mar 12, 2025
jimklimov added a commit to jimklimov/nut that referenced this pull request Mar 15, 2025
… SUBDIRS of main Makefile; avoid two definitions of "all" goal there [networkupstools#2842, networkupstools#2825]

Signed-off-by: Jim Klimov <[email protected]>
nut-travis pushed a commit that referenced this pull request Mar 15, 2025
nut-travis pushed a commit that referenced this pull request Mar 15, 2025
jimklimov added a commit to jimklimov/nut that referenced this pull request Mar 16, 2025
jimklimov added a commit to sshambar/nut that referenced this pull request Mar 16, 2025
jimklimov added a commit to jimklimov/nut that referenced this pull request Mar 17, 2025
jimklimov added a commit to jimklimov/nut that referenced this pull request Mar 17, 2025
jimklimov added a commit to jimklimov/nut that referenced this pull request Apr 18, 2025
…(generate sources via nut-scanner-deps/tools only once) [networkupstools#2825]

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this pull request Apr 18, 2025
…(generate sources via nut-scanner-deps/tools only once) [networkupstools#2825]

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this pull request Jul 28, 2025
…nd it has work to do only once [follows up from PR networkupstools#2825]

In practice we do call it as many times as there are listed
dependency files, but it only generates them on the first run
and quicky bails out later on. This should help avoid writing
into same files in parallel, ending up with garbled or incomplete
files when we want to use them.

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this pull request Jul 28, 2025
jimklimov added a commit to jimklimov/nut that referenced this pull request Jul 28, 2025
…ncies [follows up from PR networkupstools#2825, highlights why issue networkupstools#2584 better be solved]

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this pull request Jul 28, 2025
…nd it has work to do only once [follows up from PR networkupstools#2825]

In practice we do call it as many times as there are listed
dependency files, but it only generates them on the first run
and quicky bails out later on. This should help avoid writing
into same files in parallel, ending up with garbled or incomplete
files when we want to use them.

Signed-off-by: Jim Klimov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI Entries related to continuous integration infrastructure (here CI = tools + scripts + recipes) documentation packaging portability We want NUT to build and run everywhere possible refactor/fightwarn PR or issue proposal to improve code maintainability without functional changes, or to fix warnings

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants