bug(Shiny): Performance degradation in package rewrite

Describe the bug

rstudio/promises#124 clearly describes how promises::future_promise() is performing poorly with versions of {future} >= 1.40.0. promises::future_promise() is behaving as a regular future::future() who has ran out of workers and holding the responses until the last worker is started.

Reproduce example

A reproducible example using R code: https://gist.github.com/schloerke/c2628c0959dfc094b5b0de5150561378

Current behavior:

Expected behavior

Expected request timing:

Performing a git bisect, ea1c169 is the first commit who is experiencing the unexpected behavior.

Session information

Please share your session information after the error has occurred so that we also see which packages and versions are involved;

> sessionInfo()
❯❯ sessioninfo::session_info()
─ Session info ────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.5.0 (2025-04-11)
 os       macOS Sequoia 15.5
 system   aarch64, darwin20
 ui       X11
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       America/New_York
 date     2025-06-24
 pandoc   3.1 @ /opt/homebrew/bin/pandoc
 quarto   1.5.57 @ /Applications/quarto/bin/quarto

─ Packages ────────────────────────────────────────────────────────────────────────────────────────────────────────
 package      * version     date (UTC) lib source
 cachem         1.1.0       2024-05-16 [1] CRAN (R 4.5.0)
 callr          3.7.6       2024-03-25 [1] CRAN (R 4.5.0)
 chromote     * 0.5.1       2025-04-24 [1] CRAN (R 4.5.0)
 cli            3.6.5       2025-04-23 [1] CRAN (R 4.5.0)
 clisymbols     1.2.0       2017-05-21 [1] CRAN (R 4.5.0)
 codetools      0.2-20      2024-03-31 [2] CRAN (R 4.5.0)
 colorout       1.3-2       2025-05-29 [1] Github (jalvesaq/colorout@572ab10)
 crayon         1.5.3       2024-06-20 [1] CRAN (R 4.5.0)
 desc           1.4.3       2023-12-10 [1] CRAN (R 4.5.0)
 devtools       2.4.5       2022-10-11 [1] CRAN (R 4.5.0)
 digest         0.6.37      2024-08-19 [1] CRAN (R 4.5.0)
 dplyr        * 1.1.4       2023-11-17 [1] CRAN (R 4.5.0)
 ellipsis       0.3.2       2021-04-29 [1] CRAN (R 4.5.0)
 farver         2.1.2       2024-05-13 [1] CRAN (R 4.5.0)
 fastmap        1.2.0       2024-05-15 [1] CRAN (R 4.5.0)
 fs             1.6.6       2025-04-12 [1] CRAN (R 4.5.0)
 future       * 1.34.0-9361 2025-06-24 [1] local
 generics       0.1.4       2025-05-09 [1] CRAN (R 4.5.0)
 ggplot2      * 3.5.2       2025-04-09 [1] CRAN (R 4.5.0)
 git2r          0.36.2      2025-03-29 [1] CRAN (R 4.5.0)
 globals        0.18.0      2025-05-08 [1] CRAN (R 4.5.0)
 glue           1.8.0       2024-09-30 [1] CRAN (R 4.5.0)
 gtable         0.3.6       2024-10-25 [1] CRAN (R 4.5.0)
 htmltools      0.5.8.1     2024-04-04 [1] CRAN (R 4.5.0)
 htmlwidgets    1.6.4       2023-12-06 [1] CRAN (R 4.5.0)
 httpuv         1.6.16      2025-04-16 [1] CRAN (R 4.5.0)
 httr2        * 1.1.2       2025-03-26 [1] CRAN (R 4.5.0)
 jsonlite     * 2.0.0       2025-03-27 [1] CRAN (R 4.5.0)
 later          1.4.2       2025-04-08 [1] CRAN (R 4.5.0)
 lifecycle      1.0.4       2023-11-07 [1] CRAN (R 4.5.0)
 listenv        0.9.1       2024-01-29 [1] CRAN (R 4.5.0)
 magrittr       2.0.3       2022-03-30 [1] CRAN (R 4.5.0)
 memoise        2.0.1       2021-11-26 [1] CRAN (R 4.5.0)
 memuse         4.2-3       2023-01-24 [1] CRAN (R 4.5.0)
 mime           0.13        2025-03-17 [1] CRAN (R 4.5.0)
 miniUI         0.1.2       2025-04-17 [1] CRAN (R 4.5.0)
 pak            0.9.0       2025-05-27 [1] local
 parallelly     1.45.0      2025-06-02 [1] CRAN (R 4.5.0)
 patchwork    * 1.3.0       2024-09-16 [1] CRAN (R 4.5.0)
 pillar         1.10.2      2025-04-05 [1] CRAN (R 4.5.0)
 pkgbuild       1.4.8       2025-05-26 [1] CRAN (R 4.5.0)
 pkgconfig      2.0.3       2019-09-22 [1] CRAN (R 4.5.0)
 pkgload        1.4.0       2024-06-28 [1] CRAN (R 4.5.0)
 processx       3.8.6       2025-02-21 [1] CRAN (R 4.5.0)
 profvis        0.4.0       2024-09-20 [1] CRAN (R 4.5.0)
 promises     * 1.3.3       2025-05-29 [1] CRAN (R 4.5.0)
 prompt         1.0.2.9000  2025-05-29 [1] Github (gaborcsardi/prompt@17bd0e1)
 ps             1.9.1       2025-04-12 [1] CRAN (R 4.5.0)
 purrr        * 1.0.4       2025-02-05 [1] CRAN (R 4.5.0)
 R6             2.6.1       2025-02-15 [1] CRAN (R 4.5.0)
 ragg           1.4.0       2025-04-10 [1] CRAN (R 4.5.0)
 rappdirs       0.3.3       2021-01-31 [1] CRAN (R 4.5.0)
 RColorBrewer   1.1-3       2022-04-03 [1] CRAN (R 4.5.0)
 Rcpp           1.0.14      2025-01-12 [1] CRAN (R 4.5.0)
 remotes        2.5.0.9000  2025-05-29 [1] Github (r-lib/remotes@bcd35d5)
 rlang          1.1.6       2025-04-11 [1] CRAN (R 4.5.0)
 rprojroot      2.0.4       2023-11-05 [1] CRAN (R 4.5.0)
 scales         1.4.0       2025-04-24 [1] CRAN (R 4.5.0)
 sessioninfo    1.2.3       2025-02-05 [1] CRAN (R 4.5.0)
 shiny          1.10.0      2024-12-14 [1] CRAN (R 4.5.0)
 systemfonts    1.2.3       2025-04-30 [1] CRAN (R 4.5.0)
 textshaping    1.0.1       2025-05-01 [1] CRAN (R 4.5.0)
 tibble         3.3.0       2025-06-08 [1] CRAN (R 4.5.0)
 tidyr          1.3.1       2024-01-24 [1] CRAN (R 4.5.0)
 tidyselect     1.2.1       2024-03-11 [1] CRAN (R 4.5.0)
 urlchecker     1.0.1       2021-11-30 [1] CRAN (R 4.5.0)
 usethis        3.1.0       2024-11-26 [1] CRAN (R 4.5.0)
 vctrs          0.6.5       2023-12-01 [1] CRAN (R 4.5.0)
 websocket      1.4.4       2025-04-10 [1] CRAN (R 4.5.0)
 withr          3.0.2       2024-10-28 [1] CRAN (R 4.5.0)
 xtable         1.8-4       2019-04-21 [1] CRAN (R 4.5.0)

 [1] /Users/barret/Library/R/arm64/4.5/library
 [2] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library
 * ── Packages attached to the search path.

> future::futureSessionInfo()
*** Package versions
future 1.34.0.9361, parallelly 1.45.0, parallel 4.5.0, globals 0.18.0, listenv 0.9.1

*** Allocations
availableCores():
           system /proc/self/status 
               12                12 
availableWorkers():
$system
 [1] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
[10] "localhost" "localhost" "localhost"


*** Settings
- future.plan=<not set>
- future.fork.multithreading.enable=<not set>
- future.globals.maxSize=<not set>
- future.globals.onReference=<not set>
- future.resolve.recursive=<not set>
- future.rng.onMisuse=<not set>
- future.wait.timeout=<not set>
- future.wait.interval=<not set>
- future.wait.alpha=<not set>
- future.startup.script=FALSE

*** Backends
Number of workers: 9
List of future strategies:
1. cluster:
   - args: function (..., workers = "<socket cluster with 9 nodes on host ‘localhost’>", envir = parent.frame())
   - tweaked: TRUE
   - call: future::plan(future::cluster, workers = cl)

*** Basic tests
Main R session details:
    pid     r sysname release
1 24313 4.5.0  Darwin  24.5.0
                                                                                                 version nodename
1 Darwin Kernel Version 24.5.0: Tue Apr 22 19:54:25 PDT 2025; root:xnu-11417.121.6~2/RELEASE_ARM64_T6020  host001
  machine   login    user effective_user
1   arm64 user002 user001        user001

Thanks for all this detailed report and troubleshooting. It's very helpful. I agree, there's definitely something funky going on here. I cannot jump on this one right now, but if you have time, and since you're already set up, could you please run these benchmarks with also:

plan(future.callr::callr)
plan(future.mirai::mirai_multisession)

? I suspect (=hope) that you won't see the same issue with those. If not, then it gives more evidence that it is specific to the multisession/cluster backend. If specific multisession/cluster, the delay in launching futures could be due to the conservative pre-launch round-trip validation of each worker that I introduced in future (>= 1.40.0). That part can probably be relaxed. What's more concerning is the latency in collecting the result from the future - but as usual, there must be a simple explanation at the end of the day.

👀

plan(future.callr::callr)
plan(future.mirai::mirai_multisession)

Updated reprex: https://gist.github.com/schloerke/e69189d375aa16239936ac245ba187b8

I included the options(future.debug=TRUE) logs for all of the runs. There are a few extra log lines that I added to try and help coordinate where the app is and where the future debug log is. There is a Creating future promise for work 1 : 2025-06-25 15:58:23.26957 style logs to prove that promises::future_promise() is being created and the workers should not be overloaded during the test.

Unfortunately, all plan types behaved poorly: https://gist.github.com/schloerke/e69189d375aa16239936ac245ba187b8?permalink_comment_id=5638711#gistcomment-5638711

Thanks for this. Hmm, that's interesting. I'll see what I can reproduce and find using my own profiling machinery (https://www.jottr.org/2022/06/23/future-user2022-slides/)

I think the culprit in future (>= 1.40.0 & <= 1.58.0) is in the pre-validation of cluster and multisession workers:

future/R/backend_api-11.ClusterFutureBackend-class.R

Lines 1125 to 1129 in ec2446b

    
           res <- tryCatch({ 
        
             suppressWarnings({ 
        
               clusterCall(cl = cl, function(x) x, truth)[[1]] 
        
             }) 
        
           }, error = identity)

that takes place before a future is launched. That anonymous function function(x) x will carry along the local environment of the calling function, which resulted in a 0.1-0.2 second overhead for each(!) future launched. This results in:

Replacing it with a identity avoids this overhead resulting in a 0.0 second check. This single fix results in:

My conclusion is therefore that this affected only the cluster/multisession futures, but none of the other backends, i.e. future.callr and future.mirai were indeed not affected(1).

Please retry with:

remotes::install_github("futureverse/future", ref = "develop")

and let me know if it behaves as expected.

(1). I think there's something iffy with your latest reprex (https://gist.github.com/schloerke/e69189d375aa16239936ac245ba187b8). After a bit of debugging, I realized I could not obtain the expected behavior even with promises 1.32.0 and future 1.34.0. I then went back to your original reprex (https://gist.github.com/schloerke/c2628c0959dfc094b5b0de5150561378) and I could reproduce original results.

@HenrikBengtsson

@HenrikBengtsson

I think it might be related but we have been seeing odd patterns in CPU usage with future: https://github.com/topepo/bootstrap-parallel-benchmark

The speed-ups are low and I see 1-2 CPUs active at a time and jumping cores a lot. See the first graph on the readme.

@topepo

@topepo , thanks. I'm on a mobile right now. Could you please retry with the version in the develop branch?

@HenrikBengtsson

@HenrikBengtsson I'll give my setup a try later tonight. Am currently tied up at SciPy during the day this week

@HenrikBengtsson

@HenrikBengtsson I see a big improvement! Thank you!

Todays runs of 1.34.0, 1.58.0, and latest develop branch.

Rscript test-system.R 1.34.0

Rscript test-system.R 1.58.0

Rscript test-system.R sha (f366428)

There still is a small delay in response time in the latest versions (which I believe is caused by the extra calls on the foreground process).

v1.34.0 responds back to the user after a consistent amount of time (good / no batching).
In your v1.58.9004, it also responds in a consistent amount of time (good).
However, in my 1.58.9019 the responses are batched together.

While better, something is definitely waiting for computation to finish (unexpected).

Testing against your PR @ 9f87a5a (v1.58.9004),

... v1.58.9004 returns the response quicker than v1.58.9019.

but none of the other backends, i.e. future.callr and future.mirai were indeed not affected(1).

I tried my second reprex and with the latest develop branch @ v1.58.9019, future.callr and future.mirai both return slowly.

Rscript test-system.R sha promises-future-callr

Rscript test-system.R sha promises-future-mirai

Both packages still seem to have a similar overhead to the identity fix that you gave above.

To reiterate the goals:

There should be minimal time between the end of the future work and the response
The responses should not be batched (grouped vertically) after each future is done computing
Ideally, the square (response sent) and the triangle (response received) should be on top of each other.
For this example, there should be 3 batches of 9 parallel workers. The execution should appear in 3 groups, not linearly spaced out execution.

Reprex, future debug logs, and images (as I have trouble with the gist getting all the items):

b-debug-future-7-10-2025.zip

Related post showing improvements in future v1.58.9019 sans httpuv/shiny: rstudio/promises#143 (comment) .

@jcheng5

There's should be nothing in future that causes this type of buffering/delaying. Could it be that https://github.com/rstudio/promises/blob/834b0ad1213ee3305d91f587aff4b551d0e889c1/R/future_promise.R#L233-L238:

      # If we are not waiting on someone else, we can do work now
      while ((private$queue$size() > 0) && private$can_proceed()) {
        # Do work right away
        private$reset_delay()
        private$do_work()
      }

"starves out" the responses from being "sent"? It's a bit tricky for me to follow the code and what is evaluated when with the later in action. While the above loop is running, will onFullfilled() of the promises be called, or are those only being called after this loop? If they're somehow buffered/postponed by the promises package, then it make sense that the responses are "sent" all at the same time. I can see how the extent of this artifact would depends on the wall time of each task in combination of the number of workers and exactly when the later loop and polling takes place. For instance, if I decrease the polling interval for promises from 0.1s (100ms) to 0.01s, and replace the above while loop with:

      # If we are not waiting on someone else, we can do work now
      if ((private$queue$size() > 0) && private$can_proceed()) {
        # Do work right away
        private$reset_delay()
        private$do_work() 
        return()
      }

I get:

whereas with promises 1.3.3, I get what you get:

Now, I'm not sure it is safe to just replace "launch all promises" with a "launch one promise" as I did. For instance, it could be that it you might end up in a deadlock somewhere. Maybe @jcheng5 can comment on the current design.

Also, it is could very well be that this has been the case all along, but it was only noticed because of the 0.1-0.2s launch delay bug introduced in future 1.34.0 when you and others started to benchmark it this way.

/cc @jcheng5

Just a nerdy note that following the conversation of extremely knowledgeable people debugging across different important projects like this makes it very fun and inspiring to be part of the R community. Thank you everyone.

@HenrikBengtsson

Could it be that rstudio/promises@834b0ad/R/future_promise.R#L233-L238 "starves out" the responses from being "sent"?

Yes, the batching makes sense if the main process is starved, then the promises will all batch up and respond all at the same time. I do not know who is the "culprit"

it is could very well be that this has been the case all along

I'm wondering if there is work being done (don't know who) that adds just enough overhead to starve it out, rather than sneaking in between different responses.

@HenrikBengtsson
Q: Is it possible for two separate R sessions to share the same future workers (therefore two sessions are independently competing for workers)?
If it is only ever a 1:1 relationship, then your code change is a great idea! If there can be many R sessions to a single collective set of workers, then we need to safeguard against this.

Ex:
There is a 6 worker cluster is available. Proc1 grabs 2 workers and Proc2 grabs 4 workers. Then, let's say more work comes in for Proc2, I'd like for future_promise to only make a single later call to periodically ask the cluster if it is available (even if there is k queued tasks). The last thing I'd like is for all queued k tasks to individually & repeatedly ask the server if it is free.

If the code is changed to an if, then Proc2 should never grow beyond 4 workers even when there are tasks in the queue and there are available workers. This situation would hold until a new task was submitted and a single worker would be reengaged as a worker is available.

If the code is maintained as a while, then Proc2 would gain back as many workers as it can grab until the cluster declares itself full. This feels like the correct solution but it only has merit if two separate R process can share a set of workers.

Could it be that rstudio/promises@834b0ad/R/future_promise.R#L233-L238:

      # If we are not waiting on someone else, we can do work now
      while ((private$queue$size() > 0) && private$can_proceed()) {
        # Do work right away
        private$reset_delay()
        private$do_work()
      }

The section of code is there to flush the full queue of tasks. Every call to $do_work() should make one more worker busy, updating the value of $can_proceed().

$can_proceed() is built upon future::nbrOfFreeWorkers(). I have observed non-trivial execution time (~0.11s with plan(multisession) ) when all workers are currently busy. (It is fast when at least 1 worker is free (< 0.0001s).)

With every finishing task who adds a single task to the last available worker (so length(tasks) - (2 * length(workers) ) ; the first set of tasks are all available and the last set of tasks all have available workers) this would make sense why it is slowing down. By having the first set and last set of tasks perform faster, it would make sense that the $can_proceed() calls would explain the steeper increase of delay in the middle set of tasks rather than at the start/end.

With inspiration from rstudio/promises#143 (comment), I have an actual small reprex using pak::pak("rstudio/promises#146")! This reprex is explored in rstudio/promises#146

library(promises)
library(future)
plan(multisession, workers = 9)
proms <- lapply(seq_len(27), function(i) { future_promise({Sys.sleep(1.5); paste0("Hello, world!-", i)}) })
p <- promise_all(.list = proms)
p %...>% {
  message("\n")
  for (i in .) message(i)
}
while (!later::loop_empty()) {
  later::run_now(0.1)
}

I am going heads down on {otel} integration into {promises} / {shiny} for a few weeks. Please grant me patience if I'm slow to respond.

$can_proceed() is built upon future::nbrOfFreeWorkers(). I have observed non-trivial execution time (~0.11s with plan(multisession) ) when all workers are currently busy. (It is fast when at least 1 worker is free (< 0.0001s).)

Good point. This was due to me addressing #774 a little bit too quickly; I updated the develop version to nbrOfFreeWorkers() to attempt free up a worker if none were available. That's neat if you don't call it too frequently, but if you do, the overhead adds up. I've reverted this in future 1.58.0-9024.

I hope this helps.

Is it possible for two separate R sessions to share the same future workers (therefore two sessions are independently competing for workers)?

Workers of multicore, multicore, callr and mirai_multisession are all owned by the R session that sets them up.

If it is only ever a 1:1 relationship, then your code change is a great idea! If there can be many R sessions to a single collective set of workers, then we need to safeguard against this.

... in contrast, batchtools_slurm and likes are of course competing for slots on a job queue that is shared among processes and users. I'm not sure, but I can imagine that one could set up a mirai queue that is shared across processes, and maybe across users too. If so, this discussion would affect future.mirai::mirai_cluster, bare-bone mirai::mirai() futures, and anything that uses a shared worker queue. But, starvation and prioritization in such environments are something that those environments' policies handle and is out of reach for each user and process, i.e. its a moot issue.

If the code is changed to an if, then Proc2 should never grow beyond 4 workers even when there are tasks in the queue and there are available workers. This situation would hold until a new task was submitted and a single worker would be reengaged as a worker is available.

For shared worker queues, one can limit the number of queued tasks via the workers argument, e.g.

plan(future.batchtools::batchtools_slurm, workers = 4)

would cause future() to block if there are already 4 workers on the Slurm queue (running or not). So, in that sense, one can limit the number of queue workers per process. For example, some Slurm environments might limit each user to 128 job on the queue - attempts to submit more will cause sbatch to fail. The workers argument allows the user to stay below this limit, but also distribute those 128 slots across multiple processes of theirs.

I hope this second part also helps.

PS. I might submit the next version of future soon-ish.

future 1.67.0 is on CRAN (https://future.futureverse.org/news/index.html)

@HenrikBengtsson

Rerunning with 1.67.0

Future v1.34.0

Future v1.67.0

There still is a minor increase in delay when returning the value.

But for the purpose of the original Issue, I believe this issue and rstudio/promises#124 are resolved.

Thank you @HenrikBengtsson for your help!

Awesome. Thanks. This was a great troubleshooting journey for me and it helped me identify defaults of the core Future API that I thought were not critical.

Navigation Menu

Repository navigation

bug(Shiny): Performance degradation in package rewrite #798

Future v1.34.0

Future v1.67.0

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	res <- tryCatch({
	suppressWarnings({
	clusterCall(cl = cl, function(x) x, truth)[[1]]
	})
	}, error = identity)

bug(Shiny): Performance degradation in package rewrite #798

Description

Activity

HenrikBengtsson commented on Jun 25, 2025

schloerke commented on Jun 25, 2025

schloerke commented on Jun 25, 2025

HenrikBengtsson commented on Jun 25, 2025

HenrikBengtsson commented on Jul 7, 2025

topepo commented on Jul 9, 2025

HenrikBengtsson commented on Jul 9, 2025

schloerke commented on Jul 9, 2025

schloerke commented on Jul 10, 2025

schloerke commented on Jul 10, 2025

HenrikBengtsson commented on Jul 13, 2025

scottkosty commented on Jul 13, 2025

schloerke commented on Jul 18, 2025

schloerke commented on Jul 18, 2025

HenrikBengtsson commented on Jul 21, 2025

HenrikBengtsson commented on Jul 29, 2025

schloerke commented on Aug 10, 2025

Future v1.34.0

Future v1.67.0

HenrikBengtsson commented on Aug 10, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions