Package: taskqueue 0.2.0

Image

Bangyou Zheng

taskqueue: Task Queue for Parallel Computing Based on PostgreSQL

Implements a task queue system for asynchronous parallel computing using 'PostgreSQL' <https://www.postgresql.org/> as a backend. Designed for embarrassingly parallel problems where tasks do not communicate with each other. Dynamically distributes tasks to workers, handles uneven load balancing, and allows new workers to join at any time. Particularly useful for running large numbers of independent tasks on high-performance computing (HPC) clusters with 'SLURM' <https://slurm.schedmd.com/> job schedulers.

Authors:Bangyou Zheng [aut, cre]

taskqueue_0.2.0.tar.gz
taskqueue_0.2.0.zip(r-4.6)taskqueue_0.2.0.zip(r-4.5)taskqueue_0.2.0.zip(r-4.4)
taskqueue_0.2.0.tgz(r-4.5-any)
taskqueue_0.2.0.tar.gz(r-4.6-any)taskqueue_0.2.0.tar.gz(r-4.5-any)
taskqueue_0.2.0.tgz(r-4.5-emscripten)
taskqueue.pdf |taskqueue.html✨
taskqueue/json (API)

# Install 'taskqueue' in R:
install.packages('taskqueue', repos = c('https://byzheng.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/byzheng/taskqueue/issues

Pkgdown/docs site:https://taskqueue.bangyou.me

On CRAN:

Conda:

parallel-computingpostgresqltask-queue

4.60 score 1 stars 4 scripts 31 exports 62 dependencies

Last updated from:d70b0496d1. Checks:9 OK. Indexed: yes.

TargetResultTotal timeArtifact
linux-devel-x86_64OK137
source / vignettesOK223
linux-release-x86_64OK133
macos-release-arm64OK118
macos-oldrel-arm64OK95
windows-develOK117
windows-releaseOK94
windows-oldrelOK102
wasm-releaseOK115

Exports:db_cleandb_connectdb_disconnectdb_initis_db_connectproject_addproject_deleteproject_getproject_listproject_resetproject_resource_addproject_resource_add_jobsproject_resource_getproject_resource_log_deleteproject_startproject_statusproject_stopresource_addresource_getresource_listshiny_apptask_addtask_cleantask_gettask_resettask_statustaskqueue_optionstaskqueue_resettq_applyworkerworker_slurm

Dependencies:askpassbase64encbitbit64blobbslibcachemclicommonmarkcpp11credentialscurlDBIdigestfarverfastmapfontawesomefsgenericsggplot2gluegtablehmshtmltoolshttpuvisobandjquerylibjsonlitelabelinglaterlifecyclelubridatemagrittrmemoisemimeopensslotelpkgconfigplogrpromisesR6rappdirsRColorBrewerRcpprlangRPostgresS7sassscalessettingsshinysourcetoolssshstringistringrsystimechangevctrsviridisLitewhiskerwithrxtable

Getting Started with taskqueue

Rendered fromgetting-started.Rmdusingknitr::rmarkdownon Dec 04 2025.

Last update: 2025-11-24
Started: 2025-11-24

PostgreSQL Setup for taskqueue

Rendered frompostgresql-setup.Rmdusingknitr::rmarkdownon Dec 04 2025.

Last update: 2025-11-24
Started: 2025-11-24

Simple Workflow with tq_apply

Rendered fromsimple-workflow.Rmdusingknitr::rmarkdownon Dec 04 2025.

Last update: 2025-11-24
Started: 2025-11-24

SSH Setup for taskqueue

Rendered fromssh-setup.Rmdusingknitr::rmarkdownon Dec 04 2025.

Last update: 2025-11-24
Started: 2025-11-24

Readme and manuals

Help Manual

Help pageTopics
Check absolute path for system.check_absolute_path
Check absolute path for Linux.check_linux_absolute_path
Check absolute path for Windows.check_windows_absolute_path
Clean All Tables and Definitions from Databasedb_clean
Connect to PostgreSQL Databasedb_connect
Disconnect from PostgreSQL Databasedb_disconnect
Initialize PostgreSQL Database for taskqueuedb_init
A Wrapper function for DBI interfacedb_sql
Test Database Connectionis_db_connect
Create a New Projectproject_add
Delete a Projectproject_delete
Get Project Informationproject_get
List All Projectsproject_list
Reset a Projectproject_reset
Assign a Resource to a Projectproject_resource_add
Manage SLURM Job List for Project Resourceproject_resource_add_jobs
Get resources of a projectproject_resource_get
Delete Log Files for a Project Resourceproject_resource_log_delete
Start a Projectproject_start
Display Project Statusproject_status
Stop a Projectproject_stop
Add a New Computing Resourceresource_add
Get Information for a Specific Resourceresource_get
List All Computing Resourcesresource_list
Launch Shiny App for Monitoring Projectsshiny_app
Check whether a table is existedtable_exist
Add Tasks to a Projecttask_add
Remove All Tasks from a Projecttask_clean
Get Detailed Task Informationtask_get
Reset Task Status to Idletask_reset
Get Task Status Summarytask_status
Set or Get taskqueue Optionstaskqueue_options
Reset taskqueue Options to Defaultstaskqueue_reset
Apply a Function with Task Queue (Simplified Workflow)tq_apply
Execute Tasks as a Workerworker
Create a worker on slurm clusterworker_slurm
Image

  • Image