[C++] An "if_else" kernel to combine two arrays based on a mask

[asfimport]

(from discussion in ARROW-9489 with @maartenbreddels)

A general "where" kernel like np.where (https://numpy.org/doc/stable/reference/generated/numpy.where.html) seems a generally useful kernel to have, and could also help mimicking some other python (setitem-like) operations.

The concrete use case in ARROW-9489 is to basically do a fill_null(array[string], array[string]) which could be expressed as where(is_null(arr), arr2, arr).

Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Niranda Perera / @nirandaperera

Related issues:

• [C++] Add additional type support for if_else kernel (is a parent of)
• [C++][Python] if_else kernel doesn't support upcasting (is a parent of)
• [C++] Add a 'choose' kernel/scalar compute function (relates to)
• [C++] Add "replace" kernel (is related to)
• [C++] Add "replace" kernel (supercedes)
• [R] Add bindings for ifelse() and if_else() (is depended upon by)

PRs and other links:

• GitHub Pull Request #10377
• GitHub Pull Request #10410

_{Note: This issue was originally created as ARROW-10640. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
A boolean entry can be true, false or null. Shouldn't the kernel combine three arrays? (of course, you're free to pass the same input twice).

[asfimport]

Maarten Breddels / @maartenbreddels:
Another idea would be to have a 'choose' like kernel https://numpy.org/doc/stable/reference/generated/numpy.choose.html 

choose(array[int], a0, a1, ... an) where the first argument should be indices referring to array aX. where could then be implemented by casting the boolean to an int.

Something that might also be interesting would be to 'zip' the arguments. Something like this:

 

condition # boolean array
ar        # some array of length N
assert len(condition) == len(ar)

ar1 = ar.filter(condition)
ar2 = ar.filter(pc.invert(condition))
x1 = expensive_calculation1(ar1)
x2 = expensive_calculation2(ar2)
assert len(x1) + len(x2) == len(ar)
x = pc.zip(condition, x1, x2) # length N again

 This would allow for a vectorized ternary like operation equivalent to

[expensive_calculation1(ar[i]) if condition[i] else expensive_calculation2(ar[i]) for i in range(len(ar))]

 

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:

A boolean entry can be true, false or null. Shouldn't the kernel combine three arrays? (of course, you're free to pass the same input twice).

Yes, that's indeed a possibility. Now, I think we could also decide that a null in the mask (or the indices) becomes a null in the output (I suppose the 3-arrays approach is more generally flexible, but might also be less performant compared to simply propagating the validity bitmask?)

—

np.choose indeed sounds as a generalization of where. No idea how much it is used though, and if it would be worth the added complexity compared to np.where (although it might not be that more complex)

The "zip" seems useful as well. Basically you could say it is the same as "where", but where the arrays only have those number of elements as needed to fill in the output array. In that sense, could also have such a generalized zip (or interleave), similarly as "choose" generalizes "where": zip(array[int], x1, x2, ..., xn) using an array of indices instead of a boolean mask.

[asfimport]

Maarten Breddels / @maartenbreddels:
Yes, that would maybe be the 'ultimate' variant, but I think the integer variation would be quite difficult to implement compared to the booleans, which only has to deal with 2 arrays.

[asfimport]

Neal Richardson / @nealrichardson:
In R this is called ifelse https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/ifelse

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
And in SQL, this is related to a "CASE WHEN" clause: https://www.postgresqltutorial.com/postgresql-case/ (although here you can provide multiple boolean conditions, and it's also using expressions instead of actual boolean masks, but that's something to optimize only when there is a query engine)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Note there is some discussion about this on #9024

[asfimport]

David Li / @lidavidm:
I think to summarize the discussion (including from the PR), we can call this if_else which uses a Boolean mask to select from one of two input arrays/scalars.

A null mask value outputs a null.

A True mask value outputs the LHS scalar/corresponding value from the LHS array.

A False mask value outputs the RHS scalar/corresponding value from the RHS value.

Hence:

if_else([True, False, null], 1, 0) -> [1, 0, null]
if_else([True, False, null], null, 0) -> [null, 0, null]
if_else([True, False, null], 1, null) -> [1, null, null

We can file separate issues for CASE WHEN/choose.

[asfimport]

David Li / @lidavidm:
Issue resolved by pull request 10410
#10410