dask-hpcconfig: database of cluster configurations#
dask, distributed and deployment packages like dask-jobqueue or dask-kubernetes have a dazzling amount of settings allowing to fine-tune a cluster.
However, as useful as that is for experts, new users easily get confused by the steps necessary to create a cluster. For dask-jobqueue:
copy this configuration file to that directory and remember the name
install the appropriate deployment package
instantiate the appropriate class using that name, e.g.
dask_jobqueue.PBSCluster
dask-hpcconfig
aims to make this much easier by providing a
unified API and by shipping a pre-configured set of cluster
definitions. This will change the list above to:
install the appropriate deployment package
call
dask_hpcconfig.cluster()
with the desired cluster configuration name
where dask_hpcconfig.cluster()
will tell the name of the
package to install if it is still missing.