... | ... | @@ -5,23 +5,13 @@ We have a prebuilt version of 5.4.1. To use it |
|
|
bash$ module load env/cades-cnms
|
|
|
bash$ module load vasp
|
|
|
```
|
|
|
Look at the example pbs_scripts in
|
|
|
```shell_session
|
|
|
bash$ ls /software/user_tools/current/cades-cnms/pbs_samples
|
|
|
```
|
|
|
|
|
|
Note if you are batching jobs you need to look at the example for that. It does not work like at NERSC
|
|
|
## Multiple mpiruns within a Job
|
|
|
It does not work like with srun or aprun on the "real" Cray's at NERSC or OLCF
|
|
|
|
|
|
The most robust option is to give each mpirun in your job its own pbs nodefile to work with and pay attention to the binding and mapping you are using, there is an example in the repo like this.
|
|
|
See: [Multiple Runs in a Job](VASP_OneJob_Parallel_Runs/vasp_batched.pbs)
|
|
|
|
|
|
More simply this mpirun idiom should work but you need to be careful and check it in an interactive job first.
|
|
|
for example you have 512 cores allocated
|
|
|
to run three separate batch jobs
|
|
|
```shell_session
|
|
|
bash$ mpirun -x OMP_NUM_THREADS=1 --mca btl self,sm,openib --map-by slot:SPAN --bind-to core
|
|
|
--mca btl_openib_receive_queues S,9216,256,128,32:S,65536,256,128,32 --mca orte_base_help_aggregate 0
|
|
|
-n 128 -wdir ./First_VASP_Dir $VASP 2>&1 >vasp.out :
|
|
|
-wdir ./SecondVaspDir -n 128 $VASP 2>&1 >vasp.out :
|
|
|
-wdir ./ThirdVaspDir -n 256 $VASP 2>&1 >vasp.out &
|
|
|
```
|
|
|
up to the : it is just as you would always do an mpirun, after the colon(s) list the options specific to the next program and its invocation. |
|
|
\ No newline at end of file |
|
|
one instance of mpirun within a pbs job does not know what the other mpirun is doing so they need to be given different segments of the allocation to deal with otherwise they will attempt to run on top of each other.
|
|
|
The allocation is available in a file format from $PBS_NODEFILE within a job. There is one line per slot per node.
|
|
|
|
|
|
The : syntax for Multiple Job Multiple Data isn't worth the trouble (perhaps MPI_WORLD clashes?) |
|
|
\ No newline at end of file |