Commit 5d1319ca authored by Stansberry, Dale's avatar Stansberry, Dale
Browse files

- Updates to readme

parent d8c78aa0
Pipeline #3838 skipped
......@@ -14,11 +14,13 @@ Note: Spark 1.6.x + Hadoop 2.6.x (stand-alone) must be installed and the SPARK_H
To configure a spark job, run with the following parameters:
-s <file> : The PBS script file to generate'
-a <account> : Name of account to charge'
-n <num> : Number of nodes'
-w <time> : Maximum walltime'
-d <path> : Spark deployment directory'
-s <file> : The PBS script file to generate
-a <account> : Name of account to charge
-n <num> : Number of nodes*
-w <time> : Maximum walltime
-d <path> : Spark deployment directory
*Note: number of nodes must be 2 or greater (tasks are not run on master node)
The deployment directory must be unique for each Spark batch job being executed and should be located in a scratch space (Spark uses this directory to write temporary files). After running, the specified deployment directory will be created (or re-initialized if it already exists) and template configuration files/scripts will copied into the "templates" subdirectory under the deployment directory. If needed, these template files may be modified before the Spark job is submitted. When the job is submitted, these template files will be copied into per-node configuration directories and used by Spark to configure worker nodes.
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment