[C++][Dataset][Compute] Replace UnionDataset with Union ExecNode

[asfimport]

UnionDataset allows Fragments of multiple schemas and differing file formats to be scanned together as a single Dataset. This is useful functionality but makes the Dataset interface somewhat difficult to reason about since it must be general enough to accommodate UnionDataset.

After ARROW-11928 it will probably be more natural to support unioning of datasets through a subclass of ExecNode. Reconciliation of differing schemas can then be trivially handled by a full ProjectNode.

Note this would obviate both ARROW-11001 and ARROW-11749. In addition, Dataset could be simplified to a concrete class containing a set of compatibly typed/formatted Fragments.

Reporter: Ben Kietzman / @bkietz
Assignee: Alexander Ocsa / @aocsa

Related issues:

• [C++] Query engine umbrella issue (is a child of)
• [C++][Dataset] Remove UnionDataset in favor of UnionExecNode (is related to)

PRs and other links:

• GitHub Pull Request #10927

_{Note: This issue was originally created as ARROW-11981. Please see the migration documentation for further details.}

[asfimport]

Ben Kietzman / @bkietz:
[~davidli] @westonpace

[asfimport]

Weston Pace / @westonpace:
It does raise the question of whether dataset scanning itself should be an exec node?

[asfimport]

David Li:
@bkietz Sorry, you've got the wrong  [~davidli]. I am not associated with this project! :D

[asfimport]

Weston Pace / @westonpace:
@lidavidm

[asfimport]

David Li / @lidavidm:
(I usually go by lidavidm since my name is rather common, but I wonder if I shouldn't just change my name sometimes…)

To Weston's question - I think that's the plan, to have datasets as just another note in the pipeline, and to move projection/filtering into the pipeline and out of the datasets code.

[asfimport]

David Li / @lidavidm:
Issue resolved by pull request 10927
#10927