BUG: segfault when using datetime.datetime.replace on Timestamp

[johnmreynolds]

• I have checked that this issue has not already been reported.

This seems to be similar to much older bugs 14621 and 7825, but I can't see an open one,

• I have confirmed this bug exists on the latest version of pandas.

Occurs with 1.2.4 which seems to be the current one I can get.

• (optional) I have confirmed this bug exists on the master branch of pandas.

Not done.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

>>> import pandas as pd
>>> import datetime
>>> datetime.datetime.replace(pd.Timestamp('2021-01-01'), tzinfo=None)
Segmentation fault

Problem description

Unexpected segmentation fault, which shouldn't occur with basic function usage.

Expected Output

It's not entirely clear whether this should work, but it shouldn't crash.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.7.9.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : None.None

pandas : 1.2.4
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.3
setuptools : 50.3.0.post20201006
Cython : None
pytest : 6.2.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 1.3.7
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.8.6 (dt dec pq3 ext lo64)
jinja2 : 2.11.2
IPython : 7.18.1
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fsspec : 0.8.4
fastparquet : 0.4.1
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : 3.0.0
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : None
numba : 0.51.2

Also occurs on python 3.8.8.

[DylanDmitri]

Can confirm it segfaults on not only datetime.replace but also date.replace and time.replace. Will investigate further.

---

More context:

• At the end of the datetime file, if I import pandas I can replace just fine.
• Only when importing the method does it segfault.
• The segfault is on viewing the returned object not calling the replace method.
• Does not have to do with slots

[mzeitlin11]

Thanks for reporting this @johnmreynolds and @DylanDmitri for looking into it! Investigations certainly welcome - there are other issues with segfaults related to timestamp which may be related, for example #33931

[joelgibson]

This is a bug in CPython I think (or at least, the CPython .py implementation of datetime.datetime.replace disagrees with the .c implementation).

Firstly, the segfault is occurring whenever the _freq property of the _Timestamp object is being accessed. (The repr method indirectly accesses _freq, which is causing the segfault in interactive mode, but this can be replaced by a direct access to _freq to hit the same segfault). This suggests that _freq is either null or uninitialised, which is strange since as far as I can tell, the only place where an _Timestamp object is created is in crate_timestamp_from_ts, which certainly sets _freq to a not-garbage value.

The python implementation of datetime.datetime.replace is fairly innocuous, ending in a call to type(self)(...), so when a pd.Timestamp object is passed in, a new pd.Timestamp object is constructed in the usual way - in this case it calls pd.Timestamp.__new__(...). Actually, if you copy-paste the Python implementation of datetime.datetime.replace, and call that instead of the module version, everything works and there is no segfault. The plot thickens...

However the real implementation of datetime.datetime.replace is the datetime_replace function in _datetimemodule.c. (This can be seen in the imports at the very end of datetime.py). We can see that datetime_replace is calling datetime_new, which eventually ends up allocating a _Timestamp object (but not newwing it), and then treating it like a datetime object (which seems to work mostly, I guess due to similar object layouts). What happens is most fields of the actual _Timestamp object get initialised almost by accident by the datetime module, before being returned.

I tried making datetime_replace act more like datetime.replace by replacing the call to datetime_new with a call to the __new__ method on the correct type:

// clone = datetime_new(Py_TYPE(self), tuple, NULL);
// Replaced by
clone = Py_TYPE(self)->tp_new(Py_TYPE(self), tuple, NULL);

which fixes the segfault in the OP.

[mroeschke]

Closing as this appears to be an upstream issue