Forwarding arguments to the right nested functions in python

Question

The dask apply_along_axis function has the following signature:

dask.array.apply_along_axis(func1d, axis, arr, *args, dtype=None, shape=None, **kwargs)

In my case, func1d has the following signature:

my_fun(arr, x, xp, propagate=True)

I need to specify dtype and shape arguments (their default to None does not fit my case). I'm not familiar with *args and **kwargs syntaxes (I come from C++ with template parameter packs syntaxes), and even if I've tried a number call forms, I always get something along my_fun() got an unexpected keyword argument 'shape'.

How do I call this function to pass the right arguments to the right functions?

Repoducible example:

import dask
import dask.array as da
import numpy as np

def my_fun(data, x, xp):
    return data

new_array = np.zeros((100,100,100))
big_array=da.from_array(new_array, chunks=(100,100,100))
x="foo"
xp="fee"
interpolated = da.apply_along_axis(func1d=my_fun, axis=0, arr=big_array, shape=big_array.shape, dtype=big_array.dtype, x=x, xp=xp).compute()

Returns:

Traceback (most recent call last):
  File "parallelDask.py", line 28, in <module>
    interpolated = da.apply_along_axis(func1d=my_fun, axis=0, arr=big_array, shape=big_array.shape, dtype=big_array.dtype, x=x, xp=xp).compute()
  File "/home/becheler/dev/virtual_environments/crumbs-env/lib/python3.8/site-packages/dask/array/routines.py", line 304, in apply_along_axis
    test_result = np.array(func1d(test_data, *args, **kwargs))
TypeError: my_fun() got an unexpected keyword argument 'shape'

EDIT: I think it comes pretty close to this issue but I could not figure out how to solve the problem.

is my_fun actually masked_interpolation? can you show us the full traceback and ideally create a minimal reproducible example so we can fully understand/debug the issue? at minimum it would be really helpful if you could provide one of the calls that produced the error. thanks! — Michael Delgado
– Michael Delgado, Commented Mar 28, 2022 at 20:22
So I think I solved it (not sure!). The signature of the my_fun functor has to end with (*args, **kwargs), so the test in apply_along_axis can pass to it the shape and dtype without complaining (even if they are not used in the body of my_fun). If what I say holds, then it's quite a weird behavior I would never have expected. I guess it works for most of the use cases because usual functions like np.sum define args and kwargs in their signature. May be worth to add to the doc for those who want to write new functors? — WaterFox
– WaterFox, Commented Mar 29, 2022 at 2:28
@ArnaudBecheler The example in your question is working fine for me, without errors. :/ Could you please share your dask and numpy versions? — pavithraes
– pavithraes, Commented Mar 29, 2022 at 12:01

pavithraes · Accepted Answer · 2022-03-30 13:12:52Z

1

Updating the Dask version fixes this issue. We've verified this works with Dask 2022.3.0 -- latest Dask version at the time of answering. :)

answered Mar 30, 2022 at 13:12

pavithraes

8046 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Forwarding arguments to the right nested functions in python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related