ctypes.util.find_library("c") no longer makes sense

[zooba]

BPO	23606
Nosy	@amauryfa, @abalkin, @pitrou, @tjguk, @alex, @meadori, @zware, @eryksun, @zooba

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2016-03-12.16:30:06.139>
created_at = <Date 2015-03-08.04:41:09.947>
labels = ['ctypes', 'type-bug']
title = 'ctypes.util.find_library("c") no longer makes sense'
updated_at = <Date 2017-03-10.14:29:01.529>
user = 'https://github.com/zooba'

bugs.python.org fields:

activity = <Date 2017-03-10.14:29:01.529>
actor = 'alex'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2016-03-12.16:30:06.139>
closer = 'steve.dower'
components = ['ctypes']
creation = <Date 2015-03-08.04:41:09.947>
creator = 'steve.dower'
dependencies = []
files = []
hgrepos = []
issue_num = 23606
keywords = []
message_count = 25.0
messages = ['237510', '237511', '237533', '237687', '237689', '237731', '237773', '237780', '237781', '238095', '238339', '238349', '248838', '251759', '258254', '258255', '258282', '258295', '260079', '261659', '261660', '261666', '289361', '289365', '289366']
nosy_count = 11.0
nosy_names = ['amaury.forgeotdarc', 'belopolsky', 'pitrou', 'tim.golden', 'alex', 'cgohlke', 'meador.inge', 'python-dev', 'zach.ware', 'eryksun', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23606'
versions = ['Python 3.5']

[zooba]

With the changes to the CRT on Windows, it no longer makes any sense to call find_library("c") or ("m") as there is no single C Runtime DLL. The new structure has a grouped and layered approach that is better for versioning, so we now link to "api-ms-win-crt-*-l1-1-0.dll" where "*" is something like "filesystem", "heap", "locale", "math", "string", etc. and the "l1-1-0" part is a version.

I don't know what the intended purpose of this function is, so I can't suggest a good replacement other than very fast deprecation in 3.4 and raising an error in 3.5. Any better suggestions?

[eryksun]

Shouldn't find_library("c") return "ucrtbase.dll" or "ucrtbased.dll" (debug)?

Introducing the Universal CRT
http://blogs.msdn.com/b/vcblog/archive/2015/03/03/introducing-the-universal-crt.aspx

[zooba]

That was my original thought, but it's going to lose all of its named exports and effectively become useless for this.

[eryksun]

The api-ms-win-crt-* DLLs forward their exports to ucrtbase.dll, which currently uses named exports. When is it planned to remove the named exports?

>>> crt = CDLL('ucrtbase')
>>> filesystem = CDLL('api-ms-win-crt-filesystem-l1-1-0')  
>>> math = CDLL('api-ms-win-crt-math-l1-1-0')

    >>> c_void_p.from_buffer(crt._stat64)
    c_void_p(8791677890384)
    >>> c_void_p.from_buffer(filesystem._stat64)
    c_void_p(8791677890384)

    >>> c_void_p.from_buffer(crt.fabs)          
    c_void_p(8791678417616)
    >>> c_void_p.from_buffer(math.fabs)
    c_void_p(8791678417616)

[zooba]

AIUI, by the time Windows 10 or Visual Studio 2015 releases (since it's now a Windows component, it's technically on a different schedule, even though the main developer is still working against Visual Studio's schedule) and probably sooner (VS 2015 RC is most likely).

It's always possible that they'll decide not to remove them, but the current plan is that loading ucrtbase.dll directly will not be helpful, since it would prevent Windows from making breaking changes to the API that the api-*.dll files would mask.

[eryksun]

Will manual forwarding eventually be replaced by the loader's ApiSetMap (in the process PEB), i.e. will ucrtbase join ntdll, sechost, kernelbase, and kernel32 in apisetschema.dll? In this case will the CRT API sets only exist in winsxs, i.e. will they no longer be hard linked in System32?

What about debug builds? python35_d.dll links directly to ucrtbased.dll:

C:\Program Files\Python35>dumpbin /dependents python35_d.dll | find /i ".dll"
Dump of file python35_d.dll
    WS2_32.dll
    KERNEL32.dll
    ADVAPI32.dll
    VCRUNTIME140D.dll
    ucrtbased.dll

[zooba]

Good question, I'm not sure. Right now, the api-* DLLs use forwarding rather than apisetschema, but that could change. Since they support back to XP though, I suspect it probably won't the api-* DLLs will stay as they are with ucrtbase exporting ordinals rather than names.

ucrtbased is probably going to stay as it is. When you install a debug runtime you're voluntarily giving up well-defined versioning. You'll only have that file installed if you have VS 2015 or later though - it won't be part of the updates for everyone.

[python-dev]

New changeset 86c9ef950288 by Steve Dower in branch 'default':
Issue bpo-23606: Disable ctypes.util.find_library("c") on Windows so tests are skipped while we figure out how best to approach the CRT change
https://hg.python.org/cpython/rev/86c9ef950288

[zooba]

That change should get the buildbots passing again, as it will now skip those tests. The "appcrt%d" return value was blatantly incorrect anyway.

[python-dev]

New changeset fabbe6093567 by Steve Dower in branch 'default':
Issue bpo-23606: Temporarily suppress test for CRT name.
https://hg.python.org/cpython/rev/fabbe6093567

[eryksun]

Say I need to use ctypes to call _wsopen_s to open a file without write sharing. If I read you correctly, you're saying I'll need to know it's exported by api-ms-win-crt-stdio-l1-1-0.dll? Does the 'l1-1-0' suffix reflect a version number that will be incremented over time? If so, how about adding an API that maps 'stdio' to the version that Python is linked against?

[zooba]

Pretty much, except the entry point DLL version won't increment unless there's a breaking change to the API. So you have to know where it's from, but (AIUI) the l1-0-0 file will always be available. At some point it may turn into a wrapper rather than a redirect, but that doesn't really matter.

I'm not sure there's any better way to provide access to the functions other than adding them to msvcrtmodule. It's more work, but far more friendly. I hope we have a pretty low bar for adding functions to that module.

[zooba]

I've been ignoring this because I wasn't assigned...

Here's the options. If we make it load ucrtbase.dll directly (which does still have named exports, and also uses the API schema on Windows 10):

• some code that uses it will need updating (due to API changes since VC9/VC10)
• code may need to change depending on OS updated (which could change the ucrtbase exports)
• much existing code using this will probably work

If we leave it as is:

• all existing uses will obviously fail, nothing subtle
• users will either substitute the name themselves, find an equivalent stdlib function, or use a supported Windows API

Having just written that out, I still think not supporting it is best, but I do need to write it up in the porting info still. Any other thoughts?