Compiling against master branch fails; error: expected expression SET_DECODE_ERROR

[CuriousLearner]

BPO	32197
Nosy	@vstinner, @serhiy-storchaka, @CuriousLearner
PRs	bpo-32197: Try to fix a compiler error introduced in bpo-32030. #4679 bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. #4681 Revert "bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (#4681)" #4694

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-12-04.13:06:28.532>
created_at = <Date 2017-12-02.12:50:00.715>
labels = ['build', '3.7']
title = 'Compiling against master branch fails; error: expected expression SET_DECODE_ERROR'
updated_at = <Date 2017-12-04.13:43:34.053>
user = 'https://github.com/CuriousLearner'

bugs.python.org fields:

activity = <Date 2017-12-04.13:43:34.053>
actor = 'CuriousLearner'
assignee = 'none'
closed = True
closed_date = <Date 2017-12-04.13:06:28.532>
closer = 'vstinner'
components = []
creation = <Date 2017-12-02.12:50:00.715>
creator = 'CuriousLearner'
dependencies = []
files = []
hgrepos = []
issue_num = 32197
keywords = ['patch']
message_count = 9.0
messages = ['307427', '307429', '307443', '307444', '307452', '307559', '307562', '307563', '307566']
nosy_count = 3.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'CuriousLearner']
pr_nums = ['4679', '4681', '4694']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue32197'
versions = ['Python 3.7']

[CuriousLearner]

The current master branch at commit af5a895

fails while building the interpreter.

The following is the traceback while running make after ./configure --with-pydebug

gcc -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I. -I./Include -DPy_BUILD_CORE -o Modules/main.o Modules/main.c
Modules/main.c:904:20: error: expected expression
return SET_DECODE_ERROR("PYTHONEXECUTABLE environment "
^
Modules/main.c:46:5: note: expanded from macro 'SET_DECODE_ERROR'
do { \
^
1 error generated.
make: *** [Modules/main.o] Error 1

[CuriousLearner]

I used git bisect to track down the commit that broke the build.

Seems like this is the SHA: ebac19d

ebac19d is the first bad commit
commit ebac19d
Author: Victor Stinner <victor.stinner@gmail.com>
Date: Fri Dec 1 20:09:52 2017 +0100

bpo-32030: Don't call _PyPathConfig_Fini() in Py_FinalizeEx() (bpo-4667)

Changes:

* _PyPathConfig_Fini() cannot be called in Py_FinalizeEx().
  Py_Initialize() and Py_Finalize() can be called multiple times, but
  it must not "forget" parameters set by Py_SetProgramName(),
  Py_SetPath() or Py_SetPythonHome(), whereas _PyPathConfig_Fini()
  clear all these parameters.
* config_get_program_name() and calculate_program_full_path() now
  also decode paths using Py_DecodeLocale() to use the
  surrogateescape error handler, rather than decoding using
  mbstowcs() which is strict.
* Change _Py_CheckPython3() prototype: () => (void)
* Truncate a few lines which were too long

:040000 040000 de4ec929ff3fcd3c6455d8dae4d16e47fdd32ad5 8b289341f1b5e68757ac3b111a7f11808f5de1ec M Include
:040000 040000 ecddc93b37f74818d85f6d8e8ab0e0c57e893c03 c51971a3af2b6f163e625505265277410bb9eedd M Modules
:040000 040000 b92d30ac6375b7dde865a878e701684dd400e742 7d5d8810407060f32c9cbce20dcfc6f9d425524f M PC
:040000 040000 537f9e0e6a27b41178fe7b828a8da99c63cc2d9b b14940c66180023a3704ad3949367db3e82ca832 M Python

[CuriousLearner]

I tried to resolve the issue by trying to figure out the syntax for the SET_DECODE_ERROR macro.

I tried to call a function from the macro, and eventually do the checking in that function.

Second way I tried was to replace statement for macro instead of using do-while for placing the if-else clause.

I was stuck at the issue that at the point when the compiler tries to replace the macro with actual code, pymain wasn't defined.

If someone could provide a pointer on this, it would be really helpful and I'll try to work on a patch.

One simplest possible solution would be not use macro and simply use the code at the 3 places where it is being used. But I'll defer this anti-DRY approach if we can have a better solution.

[CuriousLearner]

Even a simple use of ternary operator instead of using if-else inside do-while to handle the macro like:

#define SET_DECODE_ERROR(NAME, LEN) \
    pymain->err = ((LEN) == (size_t)-2) \
    ? _Py_INIT_USER_ERR("cannot decode " #NAME) \
    : _Py_INIT_NO_MEMORY()

, throws the error undeclared identifier pymain:

gcc -c -Wno-unused-result -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I. -I./Include -DPy_BUILD_CORE -o Modules/main.o Modules/main.c
Modules/main.c:899:20: error: use of undeclared identifier 'pymain'
return SET_DECODE_ERROR("PYTHONEXECUTABLE environment "
^
Modules/main.c:46:5: note: expanded from macro 'SET_DECODE_ERROR'
pymain->err = ((LEN) == (size_t)-2) \
^
1 error generated.
make: *** [Modules/main.o] Error 1

I'm still trying to figure out a solution for this.

[serhiy-storchaka]

New changeset 13badcb by Serhiy Storchaka in branch 'master':
bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (bpo-4681)
13badcb

[vstinner]

New changeset 31a8393 by Victor Stinner in branch 'master':
Revert "bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (bpo-4681)" (bpo-4694)
31a8393

[vstinner]

Oh sorry, I missed the macOS failure and I wasn't available last week-end for quickly fix it. So thanks Serhiy for the revert ;-) I reapplied my changes with the following fix. test_embed randomly failed, but it's unrelated (I was able to reproduce the bug on master, with the revert):
https://bugs.python.org/issue20891#msg307550

The fix:

diff --git a/Modules/main.c b/Modules/main.c
index 84706e1e290..4095259b88c 100644
--- a/Modules/main.c
+++ b/Modules/main.c
@@ -901,8 +901,8 @@ config_get_program_name(_PyMainInterpreterConfig *config)
         size_t len;
         wchar_t* program_name = Py_DecodeLocale(p, &len);
         if (program_name == NULL) {
-            return SET_DECODE_ERROR("PYTHONEXECUTABLE environment "
-                                    "variable", len);
+            return DECODE_LOCALE_ERR("PYTHONEXECUTABLE environment "
+                                     "variable", (Py_ssize_t)len);
         }
         config->program_name = program_name;
     }
@@ -916,8 +916,8 @@ config_get_program_name(_PyMainInterpreterConfig *config)
             size_t len;
             wchar_t* program_name = Py_DecodeLocale(pyvenv_launcher, &len);
             if (program_name == NULL) {
-                return SET_DECODE_ERROR("__PYVENV_LAUNCHER__ environment "
-                                        "variable", len);
+                return DECODE_LOCALE_ERR("__PYVENV_LAUNCHER__ environment "
+                                         "variable", (Py_ssize_t)len);
             }
             config->program_name = program_name;
         }

[vstinner]

I tested manually that compilation and tests succeed on macOS.

[CuriousLearner]

Hey Victor, thanks for the update.

I was confused about how the macro was replaced in the previous version.