Python 3.6 compilation fails on AppVeyor: libeay.lib was created with an older compiler

[vstinner]

BPO	34575
Nosy	@pfmoore, @vstinner, @tjguk, @zware, @zooba, @tirkarthi
PRs	[3.6] bpo-34575: Build with only VS2015 on AppVeyor #9066

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-09-21.22:13:34.033>
created_at = <Date 2018-09-04.09:31:27.274>
labels = ['build', 'OS-windows']
title = 'Python 3.6 compilation fails on AppVeyor: libeay.lib was created with an older compiler'
updated_at = <Date 2018-09-21.22:13:34.032>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-09-21.22:13:34.032>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-21.22:13:34.033>
closer = 'vstinner'
components = ['Build', 'Windows']
creation = <Date 2018-09-04.09:31:27.274>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 34575
keywords = ['patch']
message_count = 14.0
messages = ['324576', '324593', '324613', '324616', '324617', '324618', '324619', '324621', '324622', '324625', '324626', '324633', '324635', '326049']
nosy_count = 6.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'zach.ware', 'steve.dower', 'xtreak']
pr_nums = ['9066']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue34575'
versions = ['Python 3.6']

[vstinner]

Example on my PR:
#9057

https://ci.appveyor.com/project/python/cpython/build/3.6build21480/job/lge7r4qknx0t0tlv

LINK : fatal error C1047: The object or library file 'C:\projects\cpython\PCBuild\win32\libeay.lib' was created with an older compiler than other objects; rebuild old objects and libraries [C:\projects\cpython\PCbuild\_hashlib.vcxproj]
LINK : fatal error LNK1257: code generation failed [C:\projects\cpython\PCbuild\_hashlib.vcxproj]
LINK : fatal error C1047: The object or library file 'C:\projects\cpython\PCBuild\win32\libeay.lib' was created with an older compiler than other objects; rebuild old objects and libraries [C:\projects\cpython\PCbuild\_ssl.vcxproj]
LINK : fatal error LNK1257: code generation failed [C:\projects\cpython\PCbuild\_ssl.vcxproj]
Command exited with code 1

[zooba]

Works on VSTS build. I suspect AppVeyor is caching files between builds, but I'm not familiar enough with the configuration to fix that.

[tirkarthi]

If it's due to cache then there is a REST API to clear the cache manually and someone with access to Appveyor credentials can give it a try to clear the cache and see if it's fixed.

Ref : appveyor/ci#985

Log in and execute the below from the console as per appveyor/ci#985 (comment)

$.ajax({
url: 'https://ci.appveyor.com/api/projects/\<username>/<project>/buildcache',
type: 'DELETE'})

A workaround : https://help.appveyor.com/discussions/questions/1310-delete-cache

Thanks

[vstinner]

REST API for the AppVeyor build cache:
https://www.appveyor.com/docs/build-cache/#rest-api

Authentication of the REST API:
https://www.appveyor.com/docs/api/#authentication

I tried:

$ curl -H "Authorization: Bearer <MY_PRIVATE_TOKEN>" -X "DELETE" https://ci.appveyor.com/api/projects/python/cpython/buildcache; echo

{"message":"You do not have required permissions to perform this action."}

Oh. I'm not allowed to clear the build cache :-(

I'm logged as "python" and my role is "Super User". If a Super User is not allowed to clear the cache, who is allowed to do that? Maybe my URL is wrong? The documentation says:

DELETE https://ci.appveyor.com/api/projects/{accountName}/{projectSlug}/buildcache

[pfmoore]

I don't see an appveyor.yml file in the CPython repository. How are
the appveyor builds configured? I was going to take a look, but need
to see the config :-(
On Wed, 5 Sep 2018 at 10:32, STINNER Victor <report@bugs.python.org> wrote:

STINNER Victor <vstinner@redhat.com> added the comment:

REST API for the AppVeyor build cache:
https://www.appveyor.com/docs/build-cache/#rest-api

Authentication of the REST API:
https://www.appveyor.com/docs/api/#authentication

I tried:

$ curl -H "Authorization: Bearer <MY_PRIVATE_TOKEN>" -X "DELETE" https://ci.appveyor.com/api/projects/python/cpython/buildcache; echo

{"message":"You do not have required permissions to perform this action."}

Oh. I'm not allowed to clear the build cache :-(

I'm logged as "python" and my role is "Super User". If a Super User is not allowed to clear the cache, who is allowed to do that? Maybe my URL is wrong? The documentation says:

DELETE https://ci.appveyor.com/api/projects/{accountName}/{projectSlug}/buildcache

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue34575\>

---

[vstinner]

Ah! If I user my personal account which has a different token, I'm able to list my roles using the REST API.

So it seems that I lack some permissions on the "python" account of AppVeyor. Who owns this account?

[tirkarthi]

Another way will be to invalidate the cache by manually updating cleanup-cache.txt in the end. But this requires a commit that has to be reverted later to enable caching and I don't think it's worthy to pollute git history for this.

https://help.appveyor.com/discussions/questions/1310-delete-cache#comment_36916917

You can have any file as dependency that will be triggering cache invalidation, even "cleanup-cache.txt" say in the root of your repo:
cache:
- my_cached_folder -> cleanup-cache.txt
Whenever CRC32 of cleanup-cache.txt is changed the cache is invalidated, i.e. not restored in the beginning of the build.

@paul.moore It's here : https://github.com/python/cpython/blob/master/.github/appveyor.yml

Thanks

[tirkarthi]

One another way will be to set environment variable to skip the cache restore and trigger a build to see if cache is the actual problem. I don't know how configurable environment variables are from the UI to trigger a specific build with the set of environment variable values.

https://www.appveyor.com/docs/build-cache/#skipping-cache-operations-for-specific-build

Skipping cache operations for specific build

You can skip cache restore or save stages with the following tweak environment variables:

APPVEYOR_CACHE_SKIP_RESTORE - set to true to disable cache restore
APPVEYOR_CACHE_SKIP_SAVE - set to true to disable cache update

Thanks

[vstinner]

APPVEYOR_CACHE_SKIP_RESTORE - set to true to disable cache restore
APPVEYOR_CACHE_SKIP_SAVE - set to true to disable cache update

I see these as well, but it would only be a temporary solution and may make build much slower, whereas AppVeyor is already a bottleneck in our workflow. AppVeyor only give us two jobs in parallel which take 10-30 min, whereas it's common that we have much more new pull requests per hour.

[tirkarthi]

Ah ok, I thought disabling the cache with APPVEYOR_CACHE_SKIP_RESTORE as true will trigger a successful build and in the end it will store the new set of artifacts from the recent compiler to the cache and then we can toggle the environment variable (APPVEYOR_CACHE_SKIP_RESTORE) so that for the next build artifacts built from old compiler are replaced with the newer ones resembling a cache clear. Sorry I might be wrong here and it's a workaround that I thought will work as in the other CI systems. I will be happy if it's solved through API then with the right credentials.

Thanks

[pfmoore]

One thought - appveyor.yml says that the cache of externals depends on PCBuild. Could it be that appveyor is caching PCBuild to verify whether it's changed (and hence whether to use the cached externals)? I know that sounds bizarre, but I'm not entirely sure why externals should be dependent on the *whole* of PCBuild anyway - maybe we should change it to depend only on PCBuild/get_externals.bat?

Clutching at straws here, there's definitely nothing obvious I can see though.

[vstinner]

Discussion on python-dev:
https://mail.python.org/pipermail/python-dev/2018-September/155075.html

[zware]

New changeset 635461f by Zachary Ware in branch '3.6':
[3.6] bpo-34575: Build with only VS2015 on AppVeyor (GH-9066)
635461f