Kay Hayen: Nuitka Release 1.0
This is to inform you about the new stable release ofNuitka. It is the extremely compatible Python compiler,“download now”.
This release contains a large amount of new features, while consolidating what we have with many bug fixes. Scalability should be dramatically better, as well as new optimization that will accelerate some code quite a bit. See the summary, how this release is paving the way forward.
Bug Fixes
Python3: Fix,bytes.decodewith onlyerrorsargument given was not working. Fixed in 0.9.1 already.
MSYS2: Fix, the accelerate mode.cmdfile was not working correctly. Fixed in 0.9.1 already.
Onefile: Fix, the bootstrap when waiting for the child, didn’t protect against signals that interrupt this call. This only affected users of the non-public--onefile-tempdiroption on Linux, but with that becoming the default in 1.0, this was discovered. Fixed in 0.9.1 already.
Fix,pkg_resourcescompile time generatedDistributionvalues could cause issues with code that put it into calls, or in tried blocks. Fixed in 0.9.1 already.
Standalone: Added implicit dependencies ofXlibpackage. Fixed in 0.9.1 already.
macOS: Fix, the package configuration forwxhad become invalid when restructuring the Yaml with code and schema disagreeing on allowed values. Fixed in 0.9.1 already.
Fix: Thestr.formatwith a single positional argument didn’t generate proper code and failed to compile on the C level. Fixed in 0.9.1 already.
Fix, the type shape ofstr.countresult was wrong. Fixed in 0.9.1 already.
UI: Fix, the warning about collision of just compiled package and original package in the same folder hiding the compiled package should not apply to packages without an__init__.pyfile, as those donottake precedence. Fixed in 0.9.2 already.
Debugging: Fix, the fallback tolldbfromgdbwhen using the option--debuggerwas broken on anything but Windows. Fixed in 0.9.2 already.
Python3.8: The moduleimportlib.metadatawas not recognized before 3.9, but actually 3.8 already has it, causing the compile time resolution of package versions to not work there. Fixed in 0.9.3 already.
Standalone: Fix, at least on macOS we should also scan from parent folders of DLLs, since they may contain sub-directories in their names. This is mostly the case, when using frameworks. Fixed in 0.9.2 already.
Standalone: Added package configuration forPyQt5to require onefile bundle mode on macOS, and recommend to disable console for PyQt6. This is same as we already do forPySide2andPySide6. Fixed in 0.9.2 already.
Standalone: Removed stray macOS onefile bundle package configuration forpicklemodule which must have been added in error. Fixed in 0.9.2 already.
UI: Catch user error of attempting to compile the__init__.pyrather than the package directory. Fixed in 0.9.2 already.
Fix, hard name import nodes failed to clone, causing issues in optimization phase. Fixed in 0.9.2 already.
Fix, avoid warnings given with gcc 11. Fixed in 0.9.2 already.
Fix, dictionary nodes where the operation itself has no effect, e.g.dict.copywere not properly annotating that their dictionary argument could still cause a raise and have side effects, triggering an assertion violation in Nuitka. Fixed in 0.9.2 already.
Standalone: Addedpynputimplicit dependencies on Linux. Fixed in 0.9.2 already.
Fix, boolean condition checks on variables converted immutable constant value assignments to boolean values, leading to incorrect code execution. Fixed in 0.9.2 already.
Python3.9: Fix, could crash on generic aliases with non-hashable values. Fixed in 0.9.3 already.
Python3: Fix, an iteration oversys.version_infowas falsely optimized into a tuple, which is not always compatible. Fixed in 0.9.3 already.
Standalone: Added support forxgboostpackage. Fixed in 0.9.3 already.
Standalone: Added data file fortext_unidecodepackage. Fixed in 0.9.4 already.
Standalone: Added data files forswagger_ui_bundlepackage. Fixed in 0.9.4 already.
Standalone: Added data files forconnexionpackage. Fixed in 0.9.4 already.
Standalone: Added implicit dependencies forsklearn.utilsandrapidfuzz. Fixed in 0.9.4 already.
Python3.10: Fix, the reformulation ofmatchstatements could create nodes that are used twice, causing code generation to assert. Fixed in 0.9.4 already.
Fix, module objects removed fromsys.modulesbut still used could lack a reference to themselves, and therefore crash due to working on a released module variables dictionary. Fixed in 0.9.5 already.
Fix, the MSVC compiles code generated for SciPy 1.8 wrongly. Added a workaround for that code to avoid triggering it. Fixed in 0.9.6 already.
Fix, calls tostr.formatwhere the result is not used, could crash the compiler during code generation. Fixed in 0.9.6 already.
Standalone: For DLLs on macOS and Anaconda, also consider thelibdirectory of the root environment, as some DLLs are otherwise not found.
Fix, allownonlocalandglobalfor__class__to be used on the class level.
Fix,xrangewith large values didn’t work on all platforms. This affected at least Python2 on macOS, but potentially others as well.
Windows: When scanning for installed Pythons to e.g. run Scons or onefile compression, it was attempting to use installations that got deleted manually and could crash.
macOS: Fix, DLL conflicts are now resolved by checking the version information too, also all cases that previously errored out after a conflict was reported, will now work.
Fix, conditional expressions whose statically decided condition picking a branch will raise an exception could crash the compilation.
Windows: Make sure we set C level standard file handles too
At least newer subprocess was affected by this, being unable to provide working handles to child processes that pass their current handles through, and also this should help DLL code to use it as level.
Standalone: Added support forpyqtgraphdata files.
Standalone: Added support fordipyby anti-bloat removal of its testing framework that wants to do unsupported stuff.
UI: Could still give warnings about modules not being followed, where that was not true.
Fix,--include-modulewas not working for non-automatic standard library paths.
New Features
Onefile: Recognize a non-changing path from--onefile-tempdir-specand then use cached mode. By default a temporary folder is used in the spec value, make it delete the files afterwards.
The cached mode is not necessarily faster, but it is not going to change files already there, leaving the binaries there intact. In the future it may also become faster to execute, but right now checking the validity of the file takes about as long as re-creating it, therefore no gain yet. The main point, is to not change where it runs from.
Standalone: Added option to exclude DLLs. You can npw use--noinclude-dllsto exclude DLLs by filename patterns.
The may e.g. come from Qt plugins, where you know, or experimented, that it is not going to be used in your specific application. Use with care, removing DLLs will lead to very hard to recognize errors.
Anaconda: UseCondaCCfrom environment variables for Linux and macOS, in case it is installed. This can be done with e.g.condainstallgcc_linux-64on Linux orcondainstallclang_osx-64on macOS.
Added new option--nowarn-mnemonicto disable warnings that use mnemonics, there is currently not that many yet, but it’s going to expand. You can use this to acknowledge the ones you accept, and not get that warning with the information pointer anymore.
Added method for resolving DLL conflicts on macOS too. This is using version information and picks the newer one where possible.
Added option--user-package-configuration-filefor user provided Yaml files, which can be used to provide package configuration to Nuitka, to e.g. add DLLs, data files, do some anti-bloat work, or add missing dependencies locally. The documentation for this does not yet exist though, but Nuitka contains a Yaml schema in themisc/nuitka-package-config-schema.jsonfile.
Addednuitka-project-elseto avoid repeating conditions in Nuitka project configuration, this can e.g. be used like this:
Previously, the inverted condition had to be used in anothernuitka-project-ifwhich is no big deal, but less readable.
Added support for deep copying uncompiled functions. There is now a section in the User Manual that explains how to clone compiled functions. This allows a workaround like this:
Plugins: Added explicit deprecation status of a plugin. We now have a few that do nothing, and are just there for compatibility with existing users, and this now informs the user properly rather than just saying it is not relevant.
Fix, some Python installations crash when attempting to import modules, such asoswith aModuleNameobject, because we limit string operations done, and e.g. refuse to do.startswithwhich of course, other loaders that your installation has added, might still use.
Windows: In case of not found DLLs, we can still examine the run time of the currently compiling Python process of Nuitka, and locate them that way, which helps for some Python configurations to support standalone, esp. to find CPython DLL in unusual spots.
Debian: Workaround forlib2to3data files. These are from stdlib and therefore the patched code from Debian needs to be undone, to make these portable again.
Optimization
Scalability: Avoid merge traces of initial variable versions, which came into play when merging a variable used in only one branch. These are useless and only made other optimization slower or impossible.
Scalability: Also avoid merge traces of merge traces, instead flatten merge traces and avoid the duplication doing so. There were pathological cases, where this reduced optimization time for functions from infinite to instant.
For comparison helpers, switch comparison where possible, such that there are only 3 variants, rather than 6. Instead the boolean result is inverted, e.g. changing>=intonot<effectively. Of course this can only be done for types, where we know that nothing special, i.e. no method overloads of__gte__is going on.
For binary operations that are commutative with the selected types, in mixed type cases, swap the arguments during code generation, such that e.g.long_a+float_bis actually computed asfloat_b+long_a. This again avoids many helpers. It also can be done for*with integers and container types.
In cases, where a comparison (or one of the few binary operation where we consider it useful), is used in a boolean context, but we know it is impossible to raise an exception, a C boolean result type is used rather than anuitka_boolwhich is now only used when necessary, because it can indicate the exception result.
Anti-Bloat: More anti-bloat work was done for popular packages, covering also uses ofsetuptools_scm,noseandnose2package removals and warnings. There was also a focus on makingmmvc,tensorflowandtifffilecompile well, removing e.g. the uses of the tensorflow testing framework.
Faster comparison ofintvalues with constant values, this uses helpers that work with Clongvalues that represent a single “digit” of a value, or ones that use the full value space of Clong.
Faster comparison offloatvalues with constant values, this uses helpers that work with Cfloatvalues, avoiding the useless Python level constant objects.
Python2: Comparison ofintandlongnow has specialized helpers that avoids converting theintto alongthrough coercion. This takes advantage of code to compare Clongvalues (which are at the core of Python2intobjects, withlongobjects.
For binary operation on mixed types, e.g.int*bytesthe slot of the first function was still considered, and called to give aPy_NotImplementedreturn value for no good reason. This also applies to mixed operations ofint,long, andfloattypes, and forstrandunicodevalues on Python2.
Added missing helper for**operation with floats, this had been overlooked so far.
Added dedicated nodes forctypes.CDLLwhich aims to allow us to detect used DLLs at compile time in the future, and to move closer to support its bindings more efficiently.
Added specialized nodes fordict.popitemas well. With this, now all of the dictionary methods are specialized.
Added specialized nodes forstr.expandtabs,str.translate,str.ljust,str.rjust,str.center,str.zfill, andstr.splitlines. While these are barely performance relevant, this completes allstrmethods, exceptremoveprefixandremovesuffixthat are Python3.9 or higher.
Added type shape for result ofstr.indexoperation as well, this was missing so far.
Optimizestr,bytesanddictmethod calls through variables.
Optimize calls through variables containing e.g. mutable constant values, these will be rare, because they all become exceptions.
Optimize calls through variables containing built-in values, unlocking optimization of such calls, where it is assigned to a local variable.
For generated attribute nodes, avoid local doing import statements on the function level. While these were easier to generate, they can only be slow at run time.
For thestrbuilt-in annotate its value as derived fromstr, which unfortunately does not allow much optimization, since that can still change many things, but it was still a missing attribute.
For variable value release nodes, specialize them by value type as well, enhancing the scalability, because e.g. parameter variable specific tests, need not be considered for all other variable types as well.
Organizational
Plugins: Major changes to the Yaml file content, cleaning up some of the DLL configuration to more easy to use.
The DLL configuration has two flavors, one from code and one from filename matching, and these got separated into distinct items in the Yaml configuration. Also how source and dest paths get provided got simplified, with a relative path now being used consistently and with sane defaults, deriving the destination path from where the module lives. Also what we called patterns, are actually prefixes, as there is still the platform specific DLL file naming appended.
Plugins: Move mode checks to dedicated plugin calledoptions-nannythat is always enabled, giving also much cleaner Yaml configuration with a new section added specifically for these. It controls advice on the optional or required use of--disable-consoleand the like. Some packages, e.g.wxare known to crash on macOS when the console is enabled, so this advice is now done with saner configuration.
Plugins: Also for all Yaml configuration sub-items where is now a consistentwhenfield, that allows checking Python version, OS, Nuitka modes such as standalone, and only apply configuration when matching this criterion, with that the anti-bloat options to allow certain bloat, should now have proper effect as well.
The use ofAppImageon Linux is no more. The performance for startup was always slower, while having lost the main benefit of avoiding IO at startup, due to new cached mode, so now we always use the same bootstrap binary as on macOS and Windows.
UI: Do not display implicit reports reported by plugins by default anymore. These have become far too many, esp. with the recent stdlib work, and often do not add any value. The compilation report will become where to turn to find out why a module in included.
UI: Ask the user to install the ordered set package that will actually work for the specific Python version, rather than making him try one of two, where sometimes only one can work, esp. with Python 3.10 allowing only one.
GitHub: More clear wording in the issue template thatpython-mnuitka--versionoutput is really required for support to given.
Attempt to use Anacondaccachebinary if installed on non-Windows. This is esp. handy on macOS, where it is harder to get it.
Windows: Avoid byte-compiling the inline copy of Scons that uses Python3 when installing for Python2.
Added experimental switches to disable certain optimization in order to try out their impact, e.g. on corruption bugs.
Reports: Added included DLLs for standalone mode to compilation report.
Reports: Added control tags influencing plugin decisions to the compilation report.
Plugins: Make theimplicit-importsdependency section in the Yaml package configuration a list, for consistency with other blocks.
Plugins: Added checking of tags such from the package configuration, so that for things dependent on python version (e.g.python39_or_higher,before_python39), the usage of Anaconda (anaconda) or certain OS (e.g.macos), or modes (e.g.standalone), expressions inwhencan limit a configuration item.
Quality: Re-enabled string normalization from black, the issues with changes that are breaking to Python2 have been worked around.
User Manual: Describe using a minimal virtualenv as a possible help low memory situations as well.
Quality: The yaml auto-format now properly preserves comments, being based onruamel.yaml.
Nuitka-Python: Added support for the Linux build with Nuitka-Python for our own CPython fork as well, previously only Windows was working, amd macOS will follow later.
The commit hook when installed from git bash was working, but doing so fromcmd.exedidn’t find a proper path for shell from thegitlocation.
Debugging: A lot of experimental toggles were added, that allow control over the use of certain optimization, e.g. use of dict, list, iterators, subscripts, etc. internals, to aid in debugging in situations where it’s not clear, if these are causing the issue or not.
Added support for Fedora 36, which requires some specific linker options, also recognize Fedora based distributions as such.
Removed long deprecated option--noinclude-matplotlibfrom numpy plugin, as it hasn’t had an effect for a long time now.
Visual Code: Added extension for editing Jinja2 templates. This one even detects that we are editing C or Python and properly highlights accordingly.
Cleanups
Standalone: Major cleanup of the dependency analysis for standalone. There is no longer a distinction between entry points (main binary, extension modules) and DLLs that they depend on. The OS specific parts got broken out into dedicated modules as well and decisions are now taken immediately.
Plugins: Split the Yaml package configuration files into 3 files. One contains now Python2 only stdlib configuration, and another one general stdlib.
Plugins: Also cleanup thezmqplugin, which was one the last holdouts of now removed plugin method, moving parts to the Yaml configuration. We therefore no longer haveconsiderExtraDllswhich used to work on the standalone folder, but instead only plugin code that provides included DLL or binary objects fromgetExtraDllswhich gives Nuitka much needed control over DLL copying. This was a long lasting battle finally won, and will allow many new features to come.
UI: Avoid changing whitespace in warnings, where we have intended line breaks, e.g. in case of duplicate DLLs. Went over all warnings and made sure to either avoid new-lines or have them, depending on wanted output.
Iterator end check code now uses the same code as rich comparison expressions and can benefit from optimization being done there as well.
Solved TODO item about code generation time C types to specify if they have error checking or not, rather than hard coding it.
Production of binary helper function set was cleaned up massively, but still needs more work, comparison helper function set was also redesigned.
Changing the spelling of our container package to become more clear.
Usednamedtupleobjects for storing used DLL information for more clear code.
Added spellchecker ignores for all attribute and argument names of generated fixed attribute nodes.
In auto-format make sure the imports float to the top. That very much cleans up generated attribute nodes code, allowing also to combine the many ones it makes, but also cleans up some of our existing code.
The package configuration Yaml files are now sorted according to module names. This will help to avoid merge conflicts during hotfixes merge back to develop and automatically group related entries in a sane way.
Moved large amounts of code producing implicit imports to Yaml configuration files.
Changed thetensorflowplugin to Yaml based configuration, making it a deprecated do nothing plugin, that only remains there for a few releases, to not crash existing build scripts.
Lots of spelling cleanups, e.g. renamingnuitka.codegentonuitka.code_generationfor clarity.
Tests
Added generated test to coverbytesmethod. This would have found the issue withdecodepotentially.
Enhanced standalone test forctypeson Linux to actually have something to test.
Summary
This release improves on many things at once. A lot of work has been put into polishing the Yaml configuration that now only lacks documentation and examples, such that the community as a whole should become capable of adding missing dependencies, data files, DLLs, and even anti-bloat patches.
Then a lot of new optimization has been done, to close the missing gaps withdictandstrmethods, but before completinglistwhich is already a work in progress pull request, andbytes, we want to start and generate the node classes that form the link or basis of dedicated nodes. This will be an area to work on more.
The many improvements to existing code helpers, and them being able to pick target types for the arguments of comparisons and binary operations, is a pre-cursor to universal optimization of this kind. What is currently only done for constant values, will in the future be interesting for picking specific C types for use. That will then be a huge difference from what we are doing now, where most things still have to usePyObject*based types.
Scalability has again seen very real improvements, memory usage of Nuitka itself, as well as compile time inside Nuitka are down by a lot for some cases, very noticeably. There is never enough of this, but it appears, in many cases now, large compilations run much faster.
For macOS specifically, the new DLL dependency analysis, is much more capable or resolving conflicts all by itself. Many of the more complex packages with some variants of Python, specifically Anaconda will now be working a lot better.
And then, of course there is the big improvement for Onefile, that allows to use cached paths. This will make it more usable in the general case, e.g. where the firewall of Windows hate binaries that change their path each time they run.
Future directions will aim to make the compilation report more concise, and given reasons and dependencies as they are known on the inside more clearly, such that is can be a major tool for testing, bug reporting and analysis of the compilation result.
