Function Repository Resource:

IncrementalParallelSelect

Source Notebook

Incrementally select elements in parallel from a list or infinite range that satisfy a predicate

Contributed by: Roman E. Maeder

ResourceFunction["IncrementalParallelSelect"][list,crit]

computes Select[list,crit] in parallel and returns partial results when aborted.

ResourceFunction["IncrementalParallelSelect"][n,crit]

computes Select[Range[n,Infinity],crit] in parallel until aborted.

Details and Options

A computation can be aborted with EvalutationAbortEvaluation or with the stop button in the progress display.
Using the stop button lets pending evaluations run to completion; an abort terminates immediately.
When a computation is aborted, the results found so far are returned.
ResourceFunction["IncrementalParallelSelect"][n,crit] runs until aborted, computing crit[n], crit[n+1], …, with no upper limit.
ResourceFunction["IncrementalParallelSelect"] takes the DistributedContexts option like other parallel functions.
The temporary progress display shows the number of kernels used, the number of elements already tested, and the number of elements for which crit[e] is True. It also shows the elapsed time and the average time for each computation.

Examples

Basic Examples (2) 

Search for Mersenne primes in the range 1 to 1200:

In[1]:=
ResourceFunction["IncrementalParallelSelect"][Range[1200], PrimeQ[2^Prime[#] - 1] &]
Out[1]=
Image

Search for Mersenne primes with no upper limit, until aborted:

In[2]:=
ResourceFunction["IncrementalParallelSelect"][1, PrimeQ[2^Prime[#] - 1] &]
Image
Out[2]=
Image

Scope (2) 

The "Results" opener in the temporary progress display allows one to see the results as they are found:

In[3]:=
ResourceFunction["IncrementalParallelSelect"][Range[1200], PrimeQ[2^Prime[#] - 1] &]
Image
Out[3]=
Image

When aborting a computation with the stop button, the number of evaluations that are still running is displayed:

In[4]:=
ResourceFunction["IncrementalParallelSelect"][2200, PrimeQ[2^Prime[#] - 1] &]
Image
Out[4]=
Image

Aborting with EvalutationAbortEvaluation terminates more quickly:

In[5]:=
ResourceFunction["IncrementalParallelSelect"][1200, PrimeQ[2^Prime[#] - 1] &]
Out[5]=
Image

Options (2) 

DistributedContexts (2) 

By default, definitions in the current context are distributed automatically:

In[6]:=
remote[_] := (Pause[0.1]; PrimeQ[$KernelID])
In[7]:=
ResourceFunction["IncrementalParallelSelect"][Range[20], remote]
Out[7]=
Image

Do not distribute any definitions of functions. In this case, the definition of local is applied only after the results are returned to the local kernel:

In[8]:=
local[x_] := (Pause[0.1]; PrimeQ[$KernelID])
In[9]:=
ResourceFunction["IncrementalParallelSelect"][Range[20], local, DistributedContexts -> None]
Out[9]=
Image

Properties and Relations (2) 

IncrementalParallelSelect returns the same result as Select, or its parallelized, non-incremental version:

In[10]:=
Parallelize[Select[Range[1200], PrimeQ[2^Prime[#] - 1] &]]
Out[10]=
Image

The standard parallel version cannot return partial results when aborted:

In[11]:=
Parallelize[Select[Range[1200], PrimeQ[2^Prime[#] - 1] &]]
Out[11]=
Image

Neat Examples (1) 

Find memorable primes:

In[12]:=
memorableNumber[n_] := With[{seq = IntegerString /@ Range[n]},
  FromDigits@StringJoin@Join[seq, Rest[Reverse[seq]]]
  ]
In[13]:=
memorableNumber[10]
Out[13]=
Image
In[14]:=
ResourceFunction["IncrementalParallelSelect"][1, PrimeQ@*memorableNumber]
Image
Out[14]=
Image
In[15]:=
Short[memorableNumber[2446]]
Out[15]=
Image

Publisher

Roman Maeder

Version History

  • 1.0.0 – 11 January 2022

Related Resources

Author Notes

IncrementalParallelSelect is best for experiments where each evaluation takes a long time, or the extent of the input range is not yet known.

For long input ranges and short evaluation times, Parallelize[Select[…]] incurs less overhead and may be faster.

License Information