[C++][Python] Large strings cause ArrowInvalid: offset overflow while concatenating arrays

[asfimport]

When working with medium-sized datasets that have very long strings, arrow fails when trying to operate on the strings. The root is the combine_chunks function.

Here is a minimally reproducible example

import numpy as np
import pyarrow as pa

# Create a large string
x = str(np.random.randint(low=0,high=1000, size=(30000,)).tolist())
t = pa.chunked_array([x]*20_000)
# Combine the chunks into large string array - fails
combined = t.combine_chunks()

I get the following error

--------------------------------------------------------------------------- ArrowInvalid Traceback (most recent call last) /var/folders/x6/00594j4s2yv3swcn98bn8gxr0000gn/T/ipykernel_95780/4128956270.py in <module> ----> 1 z=t.combine_chunks()
~/.venv/lib/python3.7/site-packages/pyarrow/table.pxi in pyarrow.lib.ChunkedArray.combine_chunks() 
~/.venv/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.concat_arrays() ~/Documents/Github/dataquality/.venv/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status() ~.venv/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status() 
ArrowInvalid: offset overflow while concatenating arrays 

With smaller strings or smaller arrays this works fine.

x = str(np.random.randint(low=0,high=1000, size=(10,)).tolist())
t = pa.chunked_array([x]*1000)
combined = t.combine_chunks()

The first example that fails takes a few minutes to run. If you'd like a faster example for experimentation, you can use vaex to generate the chunked array much faster. This will throw the identical error and will run about 1 second.

import vaex
import numpy as np

n = 50_000
x = str(np.random.randint(low=0,high=1000, size=(30_000,)).tolist())
df = vaex.from_arrays(
    id=list(range(n)),
    y=np.random.randint(low=0,high=1000,size=n)
)
df["text"] = vaex.vconstant(x, len(df))
# text_chunk_array is now a pyarrow.lib.ChunkedArray
text_chunk_array = df.text.values
x = text_chunk_array.combine_chunks() 

Reporter: Ben Epstein

Related issues:

• [C++] Take kernel can't handle ChunkedArrays that don't fit in an Array #25822 (relates to)
• [Python] combine_chunks fails on column of table, but does not error on table itself #28850 (relates to)
• [C++] Allow automatic String -> LargeString promotions when concatenating tables #23539 (relates to)
• [Python] pyarrow.concat_arrays segfaults if a resulting StringArray's capacity overflows #26180 (supercedes)

_{Note: This issue was originally created as ARROW-17828. Please see the migration documentation for further details.}

[asfimport]

David Li / @lidavidm:
It's because Arrow string arrays have a 2 GiB limit due to the representation (each element is represented by 32-bit signed offsets into a backing buffer); you can cast to large_string first to avoid this (this uses 64-bit offsets) but I think the error here could be much improved (it should at least tell you what to do)

[asfimport]

Ben Epstein:
@lidavidm thanks for the reply!

 

Is there a way for me to check this before casting? Ie instead of trying, catching the failure, casting, and trying again, is there a (fast) computation I can run to see if i'm over the size limit?

[asfimport]

David Li / @lidavidm:
Hmm, off the top of my head, there isn't such a helper; you could do it by adding up lengths of the buffers but it'd be a bit annoying to account for the offsets.

Array.nbytes is an overapproximation but you could sum that up for all the chunks (overapproximation because it includes the space taken up by the validity buffer and the offsets themselves)

[asfimport]

David Li / @lidavidm:
For posterity: labeling good-first-issue in case someone wants to improve the error message here, also linked some related issues

[asfimport]

David Li / @lidavidm:
It's also possible we could convert from string to large_string for you; not sure which behavior would be more surprising to people

[asfimport]

Ben Epstein:
I (personally) would prefer you convert to large string and emit a warning (converting string to large_string as column XX is over the arrow string size limit).

 

I'm using vaex for all my arrow work, so using your suggestion here's how i'm handling it (in cast others find themselves here)

import pyarrow as pa
import vaex
import numpy as np
from vaex.dataframe import DataFrame

n = 50_000
x = str(np.random.randint(low=0,high=1000, size=(30_000,)).tolist())
# Create a df with a string too large
df = vaex.from_arrays(
    id=list(range(n)), 
    y=np.random.randint(low=0,high=1000,size=n)
)
df["text"] = vaex.vconstant(x, len(df))

# byte limit for arrow strings
# because 1 character = 1 byte, the total number of characters in the 
# column in question must be less than the size_limit
size_limit = 2*1e9
def validate_str_cols(df: DataFrame) -> DataFrame:
    for col, dtype in zip(df.get_column_names(), df.dtypes):
        if dtype == str and df[col].str.len().sum() >= size_limit:
            df[col] = df[col].to_arrow().cast(pa.large_string())
    return df

# text is type string
print(df.dtypes)
df = validate_str_cols(df)
# test is type large_string
print(df.dtypes)
# works!
y = df.text.values.combine_chunks()

[asfimport]

David Li / @lidavidm:
That might be reasonable, though I'm not sure if we can emit a Python warning in C++ (would be an interesting thing to do, and would probably be useful elsewhere). Maybe @jorisvandenbossche has opinions.

In general the large_X vs X type handling could use some consideration, I think

[pramodsatya]

Hi @asfimport, I would like to work on this issue. Could you please assign it to me?

[kou]

Could you add a comment that contains only "take" like #33849 (comment) to here?
See also: https://arrow.apache.org/docs/dev/developers/bug_reports.html#issue-assignment

[leprechaunt33]

Current work around I've developed for vaex in general with this pyarrow related error on dataframes for which the technique mentioned above does not work (for materialisation of pandas array from a joined multi file data frame where I was unable to set the arrow data type on the column):

• catch the ArrowInvalid exception, create blank pandas data frame with required columns and iterate the columns in the vaex data frame to materialize them one by one within the pandas df.
• If ArrowInvalid is caught again, evaluate the rogue column with an evaluate_iterator() with prefetch and suitable chunk_size that will not exceed the bounds of the string, working off maximum expected record size, and collate the pyarrow StringArray/ChunkedArray data
• Continue iterating columns, typically only one or two columns will need the chunked treatment.

[pramodsatya]

take