Kay Hayen: Nuitka Release 0.6.19
This is to inform you about the new stable release ofNuitka. It is the extremely compatible Python compiler,“download now”.
This release adds support for 3.10 while also adding very many new optimization, and doing a lot of bug fixes.
Bug Fixes
Calls toimportlib.import_modulewith expressions that need releases, i.e. are not constant values, could crash the compilation. Fixed in 0.6.18.1 already.
After a fix for the previous release, modules that fail to import are attempted again when another import is executed. However, during this initialization for top level module in--modulemode, this was was done repeatedly, and could cause issues. Fixed in 0.6.18.1 already.
Standalone: Ignore warning given bypatchelfon Linux with at least newer OpenSUSE. Fixed in 0.6.18.1 already.
Fix, need to avoid computing large values out of<<operation as well. Fixed in 0.6.18.2 already.
Standalone: Ignore warning given bypatchelfon Linux about a workaround being applied.
Fix, calls toimportlib.import_modulewere not correctly creating code for dynamic argument values that need to be released, causing the compilation to report the error. Fixed in 0.6.18.1 already.
MSYS2: Fix, the console scripts are actually good for it as opposed to CPython, and the batch scripts should not be installed. Fixed in 0.6.18.2 already.
Setuptools: Added support older version ofsetuptoolsin metabuildintegration of Nuitka.
Fix, calls toimportlib.import_modulewith 2 arguments that are dynamic, were not working at all. Fixed in 0.6.18.2 already.
Windows: Compiling with MinGW64 withoutccachewas not working due to issues in Scons. Fixed in 0.6.18.2 already.
Fix, thereprbuilt-in was falsely annotated as producing astrvalue, but it can be also derived orunicodein Python2.
Fix, attribute nodes were not considering the value they are looking up on. Now that more values will know to have the attributes, that was causing errors. Fixed in 0.6.18.2 already.
Fix, left shifting can also produce large values and needs to be avoided in that case, similar to what we do for multiplications already. Fixed in 0.6.18.2 already.
UI: The new option--disable-ccachedidn’t really have the intended effect. Fixed in 0.6.18.3 already.
UI: The progress bar was causing tearing and corrupted outputs, when outputs were made, now using propertqdmAPI for doing it, this has been solved. Fixed in 0.6.18.4 already.
Fix, the constant valuesys.version_infodidn’t yet have support for its type to be also a compile time constant in e.g. tuples. Fixed in 0.6.18.4 already.
Onefile: Assertions were not disabled, and on Windows with MinGW64 this lead to including the C filenames of thezstdinline copy files and obviously less optimal code. Fixed in 0.6.18.4 already.
Standalone: Added support forbottle.extloading extensions to resolve at compile time. Fixed in 0.6.18.5 already.
Standalone: Added support forseedirrequired data file. Fixed in 0.6.18.5 already.
MSYS2: Failed to link when using the static libpython, which is also now the default forMSYS2. Fixed in 0.6.18.5 already.
Python3.6+: Fix, the intended finalizer of compiledasyncgenwas not present and in fact associated to help type. This could have caused corruption, but that was also very unlikely. Fixed in 0.6.18.5 already.
Python3: Fix, need to set__file__before executing modules, as some modules, e.g. newer PyWin32 use them to locate things during their initialization already.
Standalone: Handle all PyWin32 modules that need the special DLLs and not just a few.
Fix, some.pthfiles create module namespaces with__path__that does not exist, ignore these in module importing.
Python2.6-3.4: Fix, modules with an error could use their module name after it was released.
Distutils: When providing arguments, the method suggested in the docs is not compatible with all other systems, e.g. notsetuptools_rustfor which a two elements tuple form needs to be used for values. Added support for that and documented its use as well in the User Manual.
Python3.7+: Do no longer allow deleting cell values, this can lead to corruption and should be avoided, it seems unlikely outside of tests anyway.
Standalone: Added support for more ciphers and hashes withpycryptodomeandpycryptodomex, while also only including Ciphers when needed.
Distutils: Was not including modules or packages only referenced in the entry point definition, but not in the list of packages. That is not compatible and has been fixed.
Fix, must not expose the constants blob from extension modules, as loading these into a compiled binary can cause issues in this case.
Standalone: Added support for including OpenGL and SSL libraries withPySide2andPySide6packages.
Windows: Fix, thecmdfiles created for uninstalled Python and accelerated programs to find the Python installation were not passing command line arguments.
Windows: Executing modules with--runwas not working properly due to missing escaping of file paths.
Fix, parsing.pyifiles that make relative imports was not resolving them correctly.
Python3: Fix, when disabling the console on Windows, make sure the file handles still work and are notNone.
Windows: Fix, need to claim all OS versions of Windows as supported, otherwise e.g. high DPI features are not available.
New Features
Programs that are to be executed with the-mflag, can now be compiled with--python-flag=-mand will then behave in a compatible way, i.e. load the containing package first, and have a proper__package__value at run time.
We now can write XML reports with information about the compilation. This is initially for use in PGO tests, to decide if the expected forms of inclusions have happened and should grow into a proper reporting tool over time. At this point, the report is not very useful yet.
Added support for Python 3.10, onlymatchstatements are not completely supported. Variations with|matches that also assign are not allowed currently.
Windows: Allow using--clangwith--mingw64to e.g. use theclang.exethat is contained in the Nuitka automatic download rather thangcc.exe.
Added support for Kivy. Works through a plugin that is automatically enabled and needs no other inputs, detecting everything from using Kivy at compile time.
Added initial support for Haiku OS, a clone of BeOS with a few differences in their Python installation.
Added experimental plugintriothat works around issues with that package.
Optimization
Also trust hard imports made on the module level in function level code, this unlocks many more static optimization e.g. withsys.version_infowhen the import and the use are not on the same level.
For the built-in type method calls with generic implementation, we now do faster method descriptor calls. These avoid creating a temporaryPyCFunctionobject, that the normal call slot would, this should make these calls faster. Checking them for compiled function, etc. was only wasteful, so this makes it more direct.
Loop and normal merge traces were keeping assignments made before the loop or inside a branch, that was otherwise unused alive. This should enable more optimization for code with branches and loops. Also unused loop traces are now recognized and removed as well.
Avoiding merges of escaped traces with the unescaped trace, there is no point in them. This was actually happening a lot and should mean a scalability improvement and unlock new optimization as well.
Avoid escaping un-init traces. Unset values need not be considered as potentially modified as that cannot be done.
Thestrshape is now detected through variables, this enables many optimization on the function level.
Added manystroperation nodes.
These are specifically all methods with no arguments, as these are very generic to add, introduced a base class for them, where we know they all have no effect or raise, as these functions are all guaranteed to succeed and can be served by a common base class.
This covers thestr.capitalize,str.upper,str.lower,str.swapcase,str.title,str.isalnum,str.isalpha,str.isdigit,str.islower,str.isupper,str.isspace, andstr.istitlefunctions.
For static optimizationstr.findandstr.rfindwere added, as they are e.g. used in asys.version.find(...)style in theosmodule, helping to decide to not considerOS/2only modules.
Then, support forstr.indexandstr.rindexwas added, as these are very similar tostr.findforms, only that these may raise an exception.
Also add support forstr.splitandstr.rsplitwhich will be used sometimes for code needed to be compile time computed, to e.g. detect imports.
Same goes forendswithandstartswith, the later is e.g. popular withsys.platformchecks, and can remove a lot of code from compilation with them now being decided at compile time.
Added trust forsys.builtin_module_namesas well. Theosmodule is using it to make platform determinations.
When writing constant values, esp.tuple,list, ordictvalues, an encoding of “last value” has been added, avoiding the need to repeat the same value again, making many values more compact.
When starting Nuitka, it usually restarts itself with information collected in a mode without thesitemodule loaded, and with hash randomization disabled, for deterministic behaviour. There is a option to prevent this from happening, where the goal is to avoid it, e.g. in testing, say for the coverage taking, but that meant to parse the options twice, which also loads a lot of code.
Now only a minimal amount of code is used, and the options are parsed only on the restart, and then an error is raised when it notices, it was not allowed to do so. This also makes code a lot cleaner.
Specialized comparison code for Python2longand Python3intcode, making these operations much faster to use.
Specialized comparison code for Python2unicodeand Python3strcode, making these operations much faster to use, currently only==and!=are fully accelerated, the other comparisons will follow.
Enable static libpython with Python3 Debian packages too. As with Python2, this will improve the performance of the created binary a lot and reduce size for standalone distribution.
Comparisons withinandnotinalso consider value traces and go through variables as well where possible. So far only the rich comparisons andisandisnotdid that.
Create fixed import nodes in re-formulations rather than__import__nodes, avoiding later optimization doing that, and of course that’s simpler code too.
Python 3.10: Added support foruniontypes as compiled time constants.
Modules are now fully optimized before considering which modules they are in turn using, this avoids temporary dependencies, that later turn out unused, and can shorten the compilation in some cases by a lot of time.
On platforms without a static link library, in LTO mode, and with gcc, we can use the-O3mode, which doesn’t work forlibpython, but that’s not used there. This also includes fake static libpython, as used by MinGW64 and Anaconda on Windows.
Theanti-bloatplugin now also handles newersklearnand knows more about the standard library, and its runners which it will exclude from compilation if use for it. Currently that is not the default, but it should become that.
Organizational
Migrated the Nuitka blog from Nikola to Sphinx based ABlog and made the whole site render with Sphinx, making it a lot more usable.
Added a small presentation about Nuitka on the Download page, to make sure people are aware of core features.
Thegiplugin is now always on. The copying of thetypelibwhengiis imported is harmless and people can disable the plugin if that’s not needed.
Thematplotlibplugin is new and also always on. It previously was part of thenumpyplugin, which is doing too many unrelated things. Moving this one out is part of a plan to split it up and have it on by default without causing issues.
MSYS2: DetectingMinGWandPOSIXflavors of this Python. For theMinGWflavor of MSYS2, the option--mingw64is now the default, before it could attempt to use MSVC, which is not going to work for it. And also the Tcl and Tk installations of it are being detected automatically for thetk-interplugin.
Added Windows version to Nuitka version output, so we have this for bug reports.
User Manual: Added example explaining how to access values from your code in Nuitka project options.
UI: For Python flavors where we expect a static libpython, the error message will now point out how to achieve it for each flavor.
UI: Disable progress bar when--show-sconsis used, it makes capturing the output from the terminal only harder.
UI: Catch error of specifying both--msvc=and--mingw64options.
Distutils: Improved error messages when usingsetuptoolsorbuildintegration and failing to provide packages to compile.
Plugins: Removed now unused feature to rename modules on import, as it was only making the code more complex, while being no more needed after recently adding a place for meta path based importers to be accounted for.
Twitter: Use embedded Tweet in Credits, and regular follow button in User Manual.
Warnings about imports not done, are now only given when optimization can not remove the usage, and no options related to following have been given.
Added Windows version to--versionoutput of Nuitka. This is to more clearly recognize Windows 10 from Windows 11 report, and also the odd Windows 7 report, where tool chain will be different.
In Visual Code, the default Python used is now 3.9 in the “Linux” C configuration. This matches Debian Bullseye.
Nicer outputs from check mode of the auto-format as run for CI testing, displays problematic files more clearly.
Remove broken links to old bug tracker that is no longer online from the Changelog.
UI: When hitting CTRL-C during initial technical import detection, no longer ask to submit a bug report with the exception stack, instead exit cleanly.
Windows: Enable LTO mode for MinGW64 and other gcc by default. We require a version that can do it, so take advantage of that.
For cases, where code generation of a module takes long, make sure its name is output when CTRL-C is hit.
Windows: Splash screen only works with MSVC, added error indicator for MinGW64 that states that and asks for porting help.
Cleanups
Generate all existing C code for generic builtin type method calls automatically, and use those for method attribute lookups, making it easier to add more.
ChangedTkIntermodule to data file providing interface, yielding the 2 directories in question, with a filter fordemos.
The importing code got a major overhaul and no longer works with relative filenames, or filenames combined with package names, and module names, but always only with module names and absolute filenames. This cleans up some of the oldest and most complex code in Nuitka, that had grown to address various requirements discovered over time.
Major cleanup of Jinja2 template organisation.
Renamed all C templates from.j2to.c.j2for clarity, this was not done fully consistent before. Also move all C templates tonuitka.codegenpackage data, it will be confusing to make a difference between ones used during compile time and for the static generation, and the lines are going to become blurry.
Added Jinja2 new macroCHECK_OBJECTSto avoid branches on argument count in the call code templates. More of these things should be added.
Cleanup of code that generates header declarations, there was some duplication going on, that made it hard to generate consistent code.
Removednuitka.finalizatios.FinalizationBase, we only have one final visitor that does everything, and that of course makes a lot of sense for its performance.
Major cleanup of the Scons C compiler configuration setup. Moved things to the dedicate function, and harmonized it more.
Resolved deprecation warnings given by with--python-debugfor Nuitka.
Tests
Started test suite for Python PGO, not yet completely working though, it’s not yet doing what is needed though.
Added generated test that exercises str methods in multiple variations.
Revivedreflectedtest suite, that had been removed, because of Nuitka special needs. This one is not yet passing again though, due to a few details not yet being as compatible as needed.
Added test suite for CPython 3.10 and enable execution of tests with this version on GitHub actions.
Summary
This release is another big step forward.
The amount of optimization added is again very large, some of which yet again unlocks more static optimization of module imports, that previously would have to be considered implicit. Now analyzing these on the function level as well, we can start searching for cases, where it could be done, but is not done yet.
After starting withdict, method optimization has focused onstrwhich is esp. important for static optimization of imports. The next goal will here be to coverlistwhich are important for run time performance and currently not yet optimized. Future releases will progress there, and also add more types.
The C type specialization for Python3 has finally progressed, such that it is also covering thelongandunicodeand as such not limited to Python2 as much. The focus now needs to turn back to not working withPyObject*for these types, but e.g. with+=1to make it directly work withCLONGrather thanLONGfor which structural changes in code generation will be needed.
For scalability, theanti-bloatwork has not yet progressed as much as to be able to enable it by default. It needs to be more possible to disable it where it causes problems, e.g. when somebody really wants to includepytestand test frameworks generally, that’s something that needs to be doable. Compiling withoutanti-bloatplugin is something that is immediately noticeable in exploding module amounts. It is very urgently recommended to enable it for your compilations.
The support for Windows has been further refined, actually fixing a few important issues, esp. for the Qt bindings too.
This release adds support for 3.10 outside of very specialmatchstatements, bringing Nuitka back to where it works great with recent Python. Unfortunatelyorderedsetis not available for it yet, which means it will be slower than 3.9 during compilation.
Overall, Nuitka is closing many open lines of action with this. Thesetuptoolssupport has yet again improved and at this point should be very good.
