Pandas rolling returns NaN when infinity values are involved

Question

When using rolling on a series that contains inf values the result contains NaN even if the operation is well defined, like min or max. For example:

import numpy as np
import pandas as pd

s = pd.Series([1, 2, 3, np.inf, 5, 6])
print(s.rolling(window=3).min())

This gives:

0    NaN
1    NaN
2    1.0
3    NaN
4    NaN
5    NaN
dtype: float64

while I expected

Computing the minimum of the series directly works as expected:

s.min()  # 1.0

What is the reason for additional NaN values being introduced?

Python 3.8.1, pandas 1.0.2

ALollz · Accepted Answer · 2020-03-19 23:29:49Z

6

np.inf is explicitly converted to np.NaN in pandas/core/window/rolling.py

# Convert inf to nan for C funcs
inf = np.isinf(values)
if inf.any():
    values = np.where(inf, np.nan, values)

How to represent inf or -inf in Cython with numpy? gives information on why they had to do this.

You'd find the exact same behavior if you had NaN instead of np.inf. It can be difficult to get your output because min_counts will throw away those intermediate groups because they lack sufficient observations. One clean "hack" is to replace inf with the biggest value you can, which should be rather safe taking 'min'.

import numpy as np
s.replace(np.inf, np.finfo('float64').max).rolling(3).min()

#0    NaN
#1    NaN
#2    1.0
#3    2.0
#4    3.0
#5    5.0
#dtype: float64

edited Mar 19, 2020 at 23:29

answered Mar 19, 2020 at 23:19

ALollz

59.7k7 gold badges74 silver badges97 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

a_guest Over a year ago

Given the link to the source code I see why this result emerges but I don't understand why it is converted in the first place. inf is defined in the floating point standard so why would it cause any trouble? I read the question you linked but couldn't find the relevant information.

ALollz Over a year ago

@a_guest I don't know enough about cython to say exactly why this is done, perhaps there's a better link somewhere else I can see if I can find.

Collectives™ on Stack Overflow

Pandas rolling returns NaN when infinity values are involved

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related