Executing a single task multiple times w/ different params (aka 'parameterization')

[tschijnmo]

Hello everyone!

I recently found this nice invoke package and really like it. But there is a small problem for me if there is a recommended way to repeat a task with different parameters? For example, if we have got an input file template input_template, and we have got a set of parameters [1, 2, 3, 4, 5]. Maybe sometimes we might want to make a directory for each of the parameters, render the template in each of the directories for their value of the parameter, do something, and then loop over the subdirectories again to get something. Could you please help me if there is a recommended way to finish these kinds of tasks in invoke? If not, some pointers on some addition to invoke would be deeply appreciated. Then maybe I can finish it and make a pull request. Thank you! I could see that this can be achieved by making the processing for any of the parameters a task. And then make an overall task requiring a list of the individual tasks with different parameter values. But I am not quite sure if it is an elegant and correct way to do this.

PS If the looping over the subdirectories could be reused for both the file generation and result reading part, that would be a better one.

[bitprophet]

This is definitely a thing we want/need as it supports higher use cases like those in the SSH Fabric lib (2.0 is building on top of Invoke), e.g. "run this task over this list of remote hosts".

In that sense this is kind of related to #63 (also kinda #170).

[bitprophet]

Poking at this now, FTR noting that I had some (pretty old, haven't reviewed em yet) notes about this issue in some "TBD docs" which I just moved to a THOUGHTS file: https://github.com/pyinvoke/invoke/blob/ccd2bae21a94ceef93cba34441cf575d4ddb7ff9/THOUGHTS.rst#parameterizing-tasks

[bitprophet]

Also see fabric/fabric#635 which I'll close and link here, preferring to move "fabric 2.0" tickets involving Invoke to this repo.

The tl;dr of that was basically just "make it easy to call a task N times with some iterable of contexts", and while at the time that ticket got written there was no Invoke/Contexts, I even used the word 'contexts'. Seems clear to me at this point that we really are just talking about unifying Context with the args/kwargs of a callable, and running through it multiple times.

Can't remember w/o checking it over but current Executor may be able to do this already, in which case we're really only talking about a method of 'expanding' some shorthand (at the CLI level, presumably, since at the library level it is literally just calling Executor methods - possibly with some more added for useful shorthand/syntactic sugar).

[bitprophet]

Also implied by those earlier Fabric tickets (incl fabric/fabric#636 and a source level TODO in Invoke) is the need for this to alter how Executor.execute currently returns values, it's a simple dict keyed on task, which straight up doesn't work for non-deduped task runs (which parameterized tasks certainly count as).