Using HTcondor with future.batchtools

can one of the backends (batchtools_*) be used to send jobs to HTCondor? or would it need to be done with a batchtools_custom?

I don't think so, but I also know nothing about HTCondor - you probably have to roll your own custom config setup.

There's an open issue over at batchtools for adding built-in support for HTCondor (mlr-org/batchtools#68). The best would be to get it implemented there first. Then it'll be straightforward to include a corresponding future.batchtools backend.

is there an example of an implementation of batchtools_custom?

It's used to with "cluster functions" (a batchtools concept), e.g.

cf <- batchtools::makeClusterFunctionsInteractive(external = TRUE)
plan(batchtools_custom, cluster.workers = cf)

## Create explicit future
f <- future({
  cat("PID:", Sys.getpid(), "\n")
  42L
})
v <- value(f)
print(v)

(I've added this to example("batchtools_custom") for the next release)

So, to if you can create proper batchtools cluster functions for HTCondor, then you can use the above.

Add a simple example(future_custom) [#29]

@yonicd

@yonicd were you able to figure out how to use batchtools with HTCondor (and then with the future package) in the end? I am working with someone who uses HTCondor and would love to be able to use future.

I have not used Condor for a while now (un)fortunately, sorry

I have not used Condor for a while now (un)fortunately, sorry

No worries, thanks a lot for the reply!

This isn’t future based, but worked well enough with the condor hpc cluster https://github.com/yuliasidi/condor

Thank you! I am set on using future, but if someday I have the time to jump down the rabbit hole that might be helpful for figuring out a patch to batchtools.

Navigation Menu

Repository navigation

Using HTcondor with future.batchtools #29

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Using HTcondor with future.batchtools #29

Description

Activity

HenrikBengtsson commented on Aug 9, 2018

yonicd commented on Aug 9, 2018

HenrikBengtsson commented on Aug 12, 2018

scottkosty commented on Feb 28, 2024

yonicd commented on Feb 29, 2024

scottkosty commented on Feb 29, 2024

yonicd commented on Feb 29, 2024

scottkosty commented on Feb 29, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions