Skip to content

hpcman

Usage:

$ hpcman [OPTIONS] COMMAND [ARGS]...

Options:

  • --help: Show this message and exit.

Commands:

  • cluster: Get cluster information
  • conda: Manage cluster, lab, and user conda setups
  • lab: Get information about lab configuration
  • queue: Manage queuing system jobs, and get queue...
  • user: Get and/or set user env variables,...
  • version: Print version number and exit.

hpcman cluster

Get cluster information

Usage:

$ hpcman cluster [OPTIONS] COMMAND [ARGS]...

Options:

  • --help: Show this message and exit.

Commands:

  • disk: Get cluster disk-free usage
  • paths: Get cluster paths for standard...
  • software: Look for installed software that may not...

hpcman cluster disk

Get cluster disk-free usage

Usage:

$ hpcman cluster disk [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman cluster paths

Get cluster paths for standard software/database installs

Usage:

$ hpcman cluster paths [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman cluster software

Look for installed software that may not be in $PATH

Usage:

$ hpcman cluster software [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman conda

Manage cluster, lab, and user conda setups

Usage:

$ hpcman conda [OPTIONS] COMMAND [ARGS]...

Options:

  • --help: Show this message and exit.

Commands:

  • status: Check status of user conda setup

hpcman conda status

Check status of user conda setup

Usage:

$ hpcman conda status [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman lab

Get information about lab configuration

Usage:

$ hpcman lab [OPTIONS] COMMAND [ARGS]...

Options:

  • --help: Show this message and exit.

Commands:

  • info: Get path of potential nfs drives for your lab
  • manager: Get lab manager information

hpcman lab info

Get path of potential nfs drives for your lab

Usage:

$ hpcman lab info [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman lab manager

Get lab manager information

Usage:

$ hpcman lab manager [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman queue

Manage queuing system jobs, and get queue status

Usage:

$ hpcman queue [OPTIONS] COMMAND [ARGS]...

Options:

  • --help: Show this message and exit.

Commands:

  • avail: Alias: hqavail.
  • error: Get error queues, and explain state
  • launch: Launch interactive browser job type on a node
  • stat: Alias: hqstat.
  • submit: Alias: hqsub.
  • usage: Get past queue usage information

hpcman queue avail

Alias: hqavail. Get available queues

Usage:

$ hpcman queue avail [OPTIONS]

Options:

  • -u, --user TEXT: User name to check. Use '*' to check all users. [default: davised]
  • -q, --queue TEXT: Queue to check. By default, all queues are checked.
  • --slurm / --no-slurm: Check slurm jobs. [default: no-slurm]
  • --sge / --no-sge: Check sge queues. [default: sge]
  • --gpu: Hides non gpu queues.
  • --hide-full / --no-hide-full: Hide full queues. [default: hide-full]
  • --help: Show this message and exit.

Produces a table similar to the qstat output.

hpcman queue error

Get error queues, and explain state

Usage:

$ hpcman queue error [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman queue launch

Launch interactive browser job type on a node

Usage:

$ hpcman queue launch [OPTIONS] COMMAND:{serve|jupyter}

Arguments:

  • COMMAND:{serve|jupyter}: Command to launch. [required]

Options:

  • --force: Force the submission of the job even if run is already present.
  • -q, --queue TEXT: The queue to use. Provide '*' to explicitly select any available queue. [required]
  • -p, -P, --procs INTEGER: Number of processors to request. [default: 1]
  • -f, --free TEXT: Free memory to request on the machine to run this job. (100M, 1G, 4G, 32G etc.) [default: 4G]
  • -m, --max TEXT: Maximum memory this job may use (kill if exceeded). (100M, 1G, 4G, 32G etc.)
  • -F, --filesize TEXT: Kill the job if any created file exceeds this size. (100M, 1G, 4G, 32G etc.) [default: 500G]
  • --path TEXT: The $PATH to use for the submitted commands. [default: ($PATH environment variable of current shell)]
  • -r, --runname TEXT: The run name and log output directory name. [default: (sge.{command})]
  • --help: Show this message and exit.

hpcman queue stat

Alias: hqstat. Get job status

Usage:

$ hpcman queue stat [OPTIONS]

Options:

  • -u, --user TEXT: User name to check. Use '*' to check all users. [default: davised]
  • -q, --queue TEXT: Queue to check. By default, all queues are checked.
  • --slurm / --no-slurm: Check slurm jobs. [default: no-slurm]
  • --sge / --no-sge: Check sge queues. [default: sge]
  • -w, --watch: Watch for changes to the queue status
  • --help: Show this message and exit.

Produces a table similar to the qstat output.

hpcman queue submit

Alias: hqsub. Submit queueing job "COMMAND". Multiple lines of commands supported using piped input from STDIN. If providing as argument and there are spaces in the command, use quotes to provide the command, either '' or "" will work! Do not use -c '', just provide the command as argument.

Usage:

$ hpcman queue submit [OPTIONS] "COMMAND"

Arguments:

  • "COMMAND": Command to submit. Reading from stdin assumed. Provide '-' to explicitly use stdin. [required]

Options:

  • -r, --runname TEXT: The run name and log output directory name. Provide '-' for automatic name generation. [required]
  • -q, --queue TEXT: The queue to use. Provide '*' to explicitly select any available queue. [required]
  • -t, --type [batch|array]: Type of job to submit. Note: Default depends on inputs. batch is default except when reading from stdin, then array is default. [default: (batch; array when reading stdin)]
  • -p, -P, --procs INTEGER: Number of processors to request. Match with threads/cores used in command. Exposed as $NPROCS in the submit script. [default: 1]
  • -f, --free TEXT: Free memory to request on the machine to run this job. (100M, 1G, 4G, 32G etc.) [default: 4G]
  • -m, --max TEXT: Maximum memory this job may use (kill if exceeded). (100M, 1G, 4G, 32G etc.)
  • -F, --filesize TEXT: Kill the job if any created file exceeds this size. (100M, 1G, 4G, 32G etc.) [default: 500G]
  • --force: Force the submission of the job even if --runname is already present. (as SGE_Array default)
  • --path TEXT: The $PATH to use for the submitted commands. [default: ($PATH environment variable of current shell)]
  • --hold TEXT: Hold the job until specified jobs (either job name or job ID) are completed. If multiple, must be comma-delimited.
  • --hold-auto: Hold the execution of the job until all other jobs in dir have finished. (Uses .hpcman_jobnums file)
  • -b, --concurrency INTEGER: Array job maximum task concurrency. ('batch size'; unused in batch mode) [default: 50]
  • --queuetype [SGE|SLURM]: Type of queuing system. [default: (SLURM)]
  • --validate-path / --no-validate-path: Validate the provided $PATH value. Turn off at own risk. You may need to disable if you want to use the default $PATH of your bash shell, or if you want to do some testing. [default: validate-path]
  • --local-drive [pertask|shared]: Use the local drive for the submitted job. If set to pertask, uses $TMPDIR (/tmp) as the prefix for the tempdir (mktemp -d -p). If set to shared, uses $TMPDIR/$USER/$DIRNAME ($TMPDIR/davised/cqls-hpcman) as the tempdir, where $DIRNAME is the name of the current directory.
  • --local-prefix TEXT: Override the default prefix of the tempdir. See the help for --local-drive.
  • --mirror-type [link|copy]: Type of mirroring to local drive. Only used if --local-drive is set. [default: (link)]
  • --copy-results / --no-copy-results: Copy final results back to submission directory. Defaults to True if --local-drive pertask and False if --local-drive shared.
  • --dry-run: Do not actually submit the job, but set up the submit directory,
  • --watch: Run hpcman queue stat --watch to monitor the status of the jobs.
  • --conda: Enable use of conda (e.g. conda activate ...).
  • --conda-exe FILE: Path to conda executable to enable. [default: /local/cluster/miniconda3_base/bin/conda]
  • --help: Show this message and exit.

Emulates some functions of SGE_Batch and SGE_Array. Functions have been intentionally removed and/or modified in some cases to enable new features and improve reliability. Please submit a support ticket request at https://shell.cqls.oregonstate.edu/support to provide feedback on this software.

[magenta bold]-= Examples =-[/magenta bold]

• Batch job:

 hqsub -q '*' -r - 'blastp -db nr -query ...'

• Batch job, watching for job submission:

 hqsub -q '*' -r - 'blastp -db nr -query ...' --watch

• Batch job with multiple processors (use $NPROCS in command):

 hqsub -q '*' -r - -p 8 'blastp -db nr -num_threads $NPROCS -query ...'

• Array job with multiple processors, reading commands from stdin:

 bash submit_jobs.sh | hqsub -q '*' -r - -p 8 -t array -

• Using conda activate:

 hqsub --conda -q '*' -r - 'conda activate bakta; which bakta'

• Using /data drive:

 hqsub --local-drive pertask -q '*' -r - ...

hpcman queue usage

Get past queue usage information

Usage:

$ hpcman queue usage [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman user

Get and/or set user env variables, preferred program versions, and quota

Usage:

$ hpcman user [OPTIONS] COMMAND [ARGS]...

Options:

  • --help: Show this message and exit.

Commands:

  • check: Compare settings to system defaults
  • enable: Enable configuration of environment...
  • info: Get configuration info for user
  • quota: Check user disk quota
  • set: Set configuration info for user
  • ssh: Check sshkey setup and provide help

hpcman user check

Compare settings to system defaults

Usage:

$ hpcman user check [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman user enable

Enable configuration of environment variables using hpcman

Usage:

$ hpcman user enable [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman user info

Get configuration info for user

Usage:

$ hpcman user info [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman user quota

Check user disk quota

Usage:

$ hpcman user quota [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman user set

Set configuration info for user

Usage:

$ hpcman user set [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman user ssh

Check sshkey setup and provide help

Usage:

$ hpcman user ssh [OPTIONS]

Options:

  • --help: Show this message and exit.

hpcman version

Print version number and exit.

Uses the get_version() function specified in the __init__.py file.

Instead of using a --version flag, this command will print the version.

Returns: None

Usage:

$ hpcman version [OPTIONS]

Options:

  • --help: Show this message and exit.