...
Parallel processing is a computational method that uses multiple computers, or 'cores', to solve computationally intensive problems much faster than is achievable on a single computer. By breaking a problem down into subsections and solving those simultaneously, parallel processing can drastically speed up the runtime of certain programs. However, it's worth noting that some programs can not benefit from parallel computing.
What programs can I use parallel processing with?
...
For R programs, there are two main ways to do this: Rslurm and doParallel. We recommend Rslurm is recommended for most tasks, but both packages allow for parallel processing of R code.
How do I run Rstudio on the cluster?
Start Xquartz
Open the terminal window
Run the command: ssh -Y command.dmz.carleton.edu
Sign in with your password.
Run the command: QMLSCENE_DEVICE=softwarecontext rstudio
This will probably throw some errors, but should still open up Rstudio. If Rstudio opens but is all white, check to make sure you included the -Y in your command. If the commands are correct but Rstudio fails to open at all, contact Mike Tie for help.
How do I use Rslurm?
Rslurm is an R library that allows users to run R jobs through the Slurm Workload Manager. As a result, Rslurm allows you to manage your R jobs in the Carleton Research Users Group Cluster (or CRUG). It is designed for parallel processing and will only work with parallelized programs. One advantage to Rslurm is that it uses the slurm Slurm manager, which means that users don't need to reserve cluster space beforehand and can instead add their jobs to a queue. Jobs will then be run once the job dequeues and there are enough cores available for it to run. For more information on this process, read the CRUG documentation.
To use Rslurm, you must first create a function and a data frame of parameters values that the function can be applied to. In the data frame, each column must correspond to a specific parameter of the function, and each row must correspond to a separate function call. Rslurm works by dividing the data frame up into multiple segments, and then running the function on each of those segments simultaneously by using multiple cores. This is done using the slurm_apply(f, params) command, where f is the function and params is the data frame. The slurm_apply command returns a slurm job which can be called by other commands later. Slurmslurm_apply can take other parameters as well, including the number of nodes used and the number of cores acpus per node.
To see the current status of their your job, users can use the print_job_status(slr_job) command, where slr_job is a slurm job. command. To return the results of the job, use the get_slurm_out command. A single instance of a function can also be run using the slurm_call(f, param) command. For another _call command. After a job has finished, any unnecessary temporary files can be cleaned up with the cleanup_files command. Slurm jobs are best canceled through the terminal window using the scancel(jobname) command, rather than the cancel_slurm command provided by rslurm.
For a more detailed explanation of how to use Rslurm, notation about all relevant Rslurm functions, and examples of code you can use, please visit this linkthe Rslurm documentation.
Rslurm Example:
Here's what the example is doing at important lines of code:
1) We import the rslurm library.
4) We create a data frame pars of parameters with a length of a thousand.
7) We create a function ftest that takes in the parameters stored in the data frame.
11) We run the slurm_apply command, which takes in the function ftest, the parameters, a jobname, and how many nodes we're using. It returns a slurm job object which we call in subsequent functions.
12) We print the job status with print_job_status, which displays as a table in the console.
13) We get the result of the job with get_slurm_out and store it as res.
15) We cleanup the extra files our job created using the cleanup_files command.
How do I use doParallel?
doParallel allows you to run some jobs in R over multiple processing cores in CRUG. doParallel is a a “parallel back-end” for the foreach R package, so it will let you execute for loops in parallel. doParallel will not work for R jobs that are not based on for loops; if you need to run such jobs in parallel, refer to the Rslurm section of this page.
You MUST have both doParallel and foreach loaded in your code to use their functionality! However, you will need to do more than simply importing the doParallel library. You must use the package's registerDoParallel function before trying to use doParallel with your job, specifying either the number of cores or the cluster you would like to use (these are doParallel's workers). For example, the code registerDoParallel(4) would cause your job to run over four cores, and registerDoParallel(c1) would run the job over the previously defined cluster c1. If you do not give registerDoParallel an argument, the result depends on what computer system you're using; on Windows, you will automatically get three workers, while Unix-like systems will give you a number of workers that is equal to around about half your total available cores.
You may also assign or change the number of cores after using registerDoParallel using the option command. For example, option(cores=2) would change your job to run on two cores. You can check how many workers your job is currently using with getDoParWorkers().
For another explanation of how to use doParallel along with examples of code and documentation about the rest of the package's functionality, please visit this link.
doParallel Example:
Here's what the example is doing at important lines of code:
1) We import the doParallel library
4/5) We create a lot of trials of R data for testing
Note: The middle part here (7-15) tests the baseline performance on one core before we use multiple cores as a benchmark test of the speed-up from parallel processing.
17) We create a cluster to run our job in parallel over
18) We register the cluster with doParallel
21) We make our for loop to run R commands on the data
27) We stop the cluster when our for loop is finished
What if I need help using parallel processing or I have other questions?
If you need assistance with doParallel, Rslurm or , the concept of parallel processing, or other related questions, please contact Mike Tie.
...