Skip to content

Conversation

@tlaurion
Copy link
Collaborator

@tlaurion tlaurion commented Feb 22, 2022

coreboot-4.11 patches: remove unwanted .orig artifacts that seems to be making CircleCI fail in the past days.
EDIT: Not sure what happened there with coreboot 4.11 cache. New CircleCI builds have recreated a new caches (coreboot, all modules cache while reusing musl-cross-make cache) and the problem disappeared.

Heads build system is re-extracting coreboot archives (and coreboot archives alone!?) and reapplies patches on each coreboot build iteration.That might need to change, since creating files from patches is nothing wrong.... But re-extracting archives that were previously validated (.module-version_verify?!) is not really supposed to happen, no is repatching the content of the archives. In the present case, coreboot 4.11 was creating 4 files out of patches/coreboot-4.11/0001-Add-Heads-TPM-measured-boot-support.patch. Two of which were not supposed to exist. CircleCI catched this. But we still may have a problem with the two other valid file created, which if patch is reapplied, will still exist in cache passed from one board build to another...

CircleCI optimizes building time by providing cache mechanisms and forces its users to build a target under an hour.
This is to force Open Source projects (free tier) to not be leechers of the free tier.

Hopefully, this is CircleCI having a maximum of 3 automatically entered input (it fails on the 3rd)... And this fix will permit src/security/tpm/sha1.c and src/security/tpm/sha1.h to be skipped if existing. Below, we see that CircleCI fills patch prompts with EOF 2 times, and then waits for input on third prompt and then timeouts.

Here is the failing log trace from https://app.circleci.com/pipelines/github/tlaurion/heads/990/workflows/f2a430fd-dc8c-4e95-abe3-364a0e825533/jobs/4914/parallel-runs/0/steps/0-103:

Exerpt of that log:

if [ -d patches/coreboot-4.11 ] && [ -r patches/coreboot-4.11 ] ; then for patch in patches/coreboot-4.11/*.patch ; do echo "Applying patch file : $patch " ; ( cd /root/project/build/coreboot-4.11/ ; patch -p1 ) < $patch || exit 1 ; done ; fi
Applying patch file : patches/coreboot-4.11/0000-cpu-x86-smm-Use-PRIxPTR-to-print-uintptr_t.patch
patching file src/cpu/x86/smm/tseg_region.c
Applying patch file : patches/coreboot-4.11/0001-Add-Heads-TPM-measured-boot-support.patch
patching file src/Kconfig
The next patch would create the file src/Kconfig.orig,
which already exists!  Assume -R? [n] EOF
Apply anyway? [n] EOF
Skipping patch.
1 out of 1 hunk ignored
patching file src/include/program_loading.h
patching file src/lib/cbfs.c
patching file src/lib/hardwaremain.c
Hunk #2 succeeded at 549 (offset 8 lines).
patching file src/lib/rmodule.c
patching file src/security/tpm/Makefile.inc
The next patch would create the file src/security/tpm/sha1.c,
which already exists!  Assume -R? [n] make: *** [Makefile:507: /root/project/build/coreboot-4.11/.canary] Hangup

context deadline exceeded

…be making CircleCI fail in the past days.

Heads build system is reextracting archives and reapplying patches on each iteration.
CircleCI optimizes building time by providing cache mechanisms and forces users to build a target under an hour.
This is to force Open Source projects (free tier) to not be leechers of the free tier.

In the past days, CircleCI bails on building coreboot 4.11 boards because some files being cached are already being present (created files from patches).
In those, two files were unwanted artifacts, recreated on top of coreboot 4.11 extracted original files (undesired .orig files), while bailing on the creating of src/security/tpm/sha1.c from patches/coreboot-4.11/0001-Add-Heads-TPM-measured-boot-support.patch.

Hopefully, this is CircleCI having a maximum of 3 automatically entered input (it fails on the 3rd)... And this fix will permit src/security/tpm/sha1.c and src/security/tpm/sha1.h to be skipped if existing.
Below, we see that CircleCI fills patch prompts with EOF 2 times, and then waits for input and then timeouts.

Here is the failing log trace from https://app.circleci.com/pipelines/github/tlaurion/heads/990/workflows/f2a430fd-dc8c-4e95-abe3-364a0e825533/jobs/4914/parallel-runs/0/steps/0-103:

Exerpt of that log:
if [ -d patches/coreboot-4.11 ] && [ -r patches/coreboot-4.11 ] ; then for patch in patches/coreboot-4.11/*.patch ; do echo "Applying patch file : $patch " ; ( cd /root/project/build/coreboot-4.11/ ; patch -p1 ) < $patch || exit 1 ; done ; fi
Applying patch file : patches/coreboot-4.11/0000-cpu-x86-smm-Use-PRIxPTR-to-print-uintptr_t.patch
patching file src/cpu/x86/smm/tseg_region.c
Applying patch file : patches/coreboot-4.11/0001-Add-Heads-TPM-measured-boot-support.patch
patching file src/Kconfig
The next patch would create the file src/Kconfig.orig,
which already exists!  Assume -R? [n] EOF
Apply anyway? [n] EOF
Skipping patch.
1 out of 1 hunk ignored
patching file src/include/program_loading.h
patching file src/lib/cbfs.c
patching file src/lib/hardwaremain.c
Hunk #2 succeeded at 549 (offset 8 lines).
patching file src/lib/rmodule.c
patching file src/security/tpm/Makefile.inc
The next patch would create the file src/security/tpm/sha1.c,
which already exists!  Assume -R? [n] make: *** [Makefile:507: /root/project/build/coreboot-4.11/.canary] Hangup

context deadline exceeded
@tlaurion
Copy link
Collaborator Author

tlaurion commented Feb 22, 2022

@MrChromebox please approve.

As said above, we might have a bigger problem. By watching builds more and more over time, it seems that coreboot is the only one not respecting initial archive validation only (archive checksums) and touching packages/.coreboot-blobs-4.11_verify in the present case. Instead of checking if packages/.coreboot-blobs-4.11_verify and packages/.coreboot-4.11_verify exists, coreboot is reextracts itself, its toolchain and its re-patches files. But since all the modules are already built, this is not taking too long so we never really realized.

But still, it should not happen. This was not always the case.
It doesn't happen for any other built module, but it happens for coreboot.

This might be another quest for Makefile kung fu. Will recheck the logic of .module-version_verify and try to attempt to check why its not respected for coreboot. But as of now, this is just a realization. And the present fix is still needed, but might not fix the problem: we should not reextract archives being already verified, even less repatch them...

If your eyes see something, this is a more profound needed fix: https://github.com/osresearch/heads/blob/master/Makefile#L386-L409

EDIT: works as expected. Complete logs of rebuilds to not show re-extraction, repatch and rebuilds outside of coreboot toolstack. And there, Heads build system is patching coreboot buildsystem when needed, but only the later is reran. So all good.

The issue in CircleCI is unknown; not sure why the coreboot 4.11 cache was screwed.
Nonetheless, this PR fixes unwanted patching.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants