README.md

# This Challenge

This example and challenge are designed to show you an example of how to submit gpu jobs on the CADES condos. They are not designed to teach you fully how to use OpenACC. For that and other GPU porting techniques please register for NVIDA’s GPU class that runs one day a month for the next few months. Register at https://www.olcf.ornl.gov/calendar/cuda-shared-memory/

The birthright nodes have 2 GPUS each. We will use one of them to do a vector addition for this challenge.

For this example and challenge you will need to have done the Globus challenge to get the files.
Otherwise, please login to or-slurm-login01 and cp -r /lustre/or-hydra/cades-birthright/world-shared/GPU-Basics .

## VecAdd.c

VecAdd.c adds two vectors A and B to produce C, where Ci = Ai + Bi. 

To see the code 
```
$ cd GPU-Basics
$ vi VecAcc.c

```

It uses OpenACC directives, called pragmas, to copy the two vectors, a and b, to the GPU add them them, and 
 copy the result back to the CPU. If you want to see a more complete explanation
of this see the CADES self-guided tutorial in the SLRUM Migration section of our Condo User Guide:
https://support.cades.ornl.gov/user-documentation/_book/condos/slurm/openacc-tutorial.html



```
#include <stdio.h>
#include <stdlib.h>
#include <math.h>

int main( int argc, char* argv[] )
{

    // Size of vectors
    int n = 10000;
    
    // Input vectors
    double *restrict a;
    double *restrict b;
    // Output vector 
    double *restrict c;
    
    // Size, in bytes, of each vector
    size_t bytes = n*sizeof(double);
    
    // Allocate memory for each vector
    a = (double*)malloc(bytes); 
    b = (double*)malloc(bytes); 
    c = (double*)malloc(bytes);
    
    // Initialize content of input vectors, vector a[i] = sin(i)^2 vector b[i] = cos(i)^2
    int i;
    for(i=0; i<n; i++) {
        a[i] = sin(i)*sin(i);
        b[i] = cos(i)*cos(i);
    }   
    
    // sum component wise and save result into vector c
    #pragma acc kernels copyin(a[0:n],b[0:n]), copyout(c[0:n])
    for(i=0; i<n; i++) {
        c[i] = a[i] + b[i];
    }   
    
    // Sum up vector c and print result divided by n, this should equal 1 within error 
    double sum = 0.0;
    for(i=0; i<n; i++) {
        sum += c[i];
    }   
    sum = sum/n;
    printf("final result: %f\n", sum);
    
    // Release memory
    free(a);
    free(b);
    free(c);
    
    return 0;
}   
```

## Compiling VecAdd.c with OpenAcc

The PGI compiler is the only CADES compiler that has full OpenACC support.
To set up your programming environment:

```
$ module purge
$ module load pgi/19.4

```
To compile this code with the pgcc compiler and OpenAcc use the `-acc` flag.

The pgi compiler’s `-Minfo` flag allows you to see what the compiler is doing with your code

## Challenge part 1:


Set up your programming environment and fill in the blanks to compile the code with the -acc flag and the - Minfo flag.
```

$ pgcc -___ -_____ VecAdd.c -o VecAdd.o

```

Did the compiler generate any code for the GPU? How can you tell?

Run the code. Your vector should sum to a value of 1.

```
$ ./VecAdd.o
```
## GPU Batch Scripting


Standard SBATCH directives for CADES Birthright: 
```
#SBATCH -A birthright
#SBATCH -p gpu
#SBATCH --gres=gpu:2
```
You must use --gres=gpu:2 to tell the job to use the node's GPUs.

We will also use NVprof, NVIDIA's built-in profiler. It will show you that your code is running on the GPU and also give you performance information about the code. 

You must be on a compute node to use NVprof and you must load the cuda module.

To use nvprof inside your batch script :
```
. . . 

#SBATCH --mail-user=<your_email>

module purge
module load pgi/19.4
module load cuda


nvprof ./VecAdd.o
```

### Challenge 2 Submit a GPU Job with SLRUM

Below is a batch script to run the VecAdd.o, that you compiled in challenge 1. Note that we are using NVProf. Fill in the blanks using your knowledge of the SLURM batch directives
and the CADES software environment. If your compilation failed copy the VecAcc.o file from the “answers” folder into your work area to do this part. 

In your work area the sbatch script is called gpu.sbatch.
Edit the script with vi by filling in the blanks as described below. 
```
$ vi gpu.sbatch 

```

**gpu.sbatch **

```
#!/bin/bash

#SBATCH -A _______
#SBATCH -p _______
#SBATCH ___________
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -c 1
#SBATCH -J gpu-test-job
#SBATCH --mem=00
#SBATCH -t 10:00
#SBATCH -o ./%j-gpu-output.txt
#SBATCH -e ./%j-gpu-error.txt
#SBATCH --mail-type=FAIL
#SBATCH --mail-user=<your_email>

module purge
module load _____
module load _____

nvprof ./VecAdd.o

```
To see where you stand in the Queue issue:
```
squeue -u <userID>

``` 
A job state of “PD” means your job is waiting in the queue. 
A job state of “R” means that it is running. 

To see details about your job, issue
```
scontrol show job <jobID> 

```
After the code runs, open your gpu-error.txt file and see if the code ran on the GPU.
How long did it spend copying data to the GPU?