HyperShell is an elegant, cross-platform, high-throughput computing utility for processing shell commands over a distributed, asynchronous queue. It is a highly scalable workflow automation tool for many-task scenarios.

Built on Python and tested on Linux, macOS, and Windows.

Other tools may offer similar functionality in some places but not within a single tool and not with the flexibility, ergonomics, and scalability provided by HyperShell.

Design elements include but are not limited to:

• Client-server: Run the server in stand-alone mode with SQLite or PostgreSQL.

  Scale clients elastically as needed (even down to zero).
• Cross-platform: trivial to install, run on any platform where Python runs.

  Mix platforms within a running cluster (Server on Linux, Clients on Windows).
• Staggered launch: Come up gradually to balance the workload.

  Scale to 1000+ nodes, 250k+ workers without crashing the server.
• Database in-the-loop: persist task metadata across runs.

  Fault-tolerant by default. Automated retries. Task history.
• User-defined tags: annotate tasks with key:value tags.

  Manage catalogs of large collections of tasks with ease.

HyperShell is primarily a command-line program. Most users will operate the hs cluster command (hsx for short) in a start-to-finish workflow scenario much like people tend to do with alternatives like xargs, GNU Parallel, or HPC-specific tools like ParaFly or TaskFarmer (NERSC-only) or Launcher (TACC).

seq 1000000 | hsx -t 'echo {}' -N64 --ssh 'a[00-32].cluster' > task.out

Documentation is available at hypershell.readthedocs.io. For basic usage information on the command line use: hs --help. For a more comprehensive usage guide on the command line you can view the manual page with man hs.

Join the Discord server to post questions, discuss your project, share with the community, keep in touch with announcements and upcoming events!

HyperShell is an open-source project developed on GitHub. If you find bugs or issues with the software please create an Issue. Contributions are welcome in the form of Pull requests for bug fixes, documentation, and minor feature improvements.

We've added a Code of Conduct recently, adapted from the Contributor Covenant, version 2.0.

HyperShell is released under the Apache Software License (v2).

If HyperShell has helped in your research please consider citing us.

@inproceedings{lentner_2022,
    author = {Lentner, Geoffrey and Gorenstein, Lev},
    title = {HyperShell v2: Distributed Task Execution for HPC},
    year = {2022},
    isbn = {9781450391610},
    publisher = {Association for Computing Machinery},
    url = {https://doi.org/10.1145/3491418.3535138},
    doi = {10.1145/3491418.3535138},
    booktitle = {Practice and Experience in Advanced Research Computing},
    articleno = {80},
    numpages = {3},
    series = {PEARC '22}
}