Commit 5d1319ca authored by Stansberry, Dale's avatar Stansberry, Dale
Browse files

- Updates to readme

parent d8c78aa0
Pipeline #3838 skipped
...@@ -14,11 +14,13 @@ Note: Spark 1.6.x + Hadoop 2.6.x (stand-alone) must be installed and the SPARK_H ...@@ -14,11 +14,13 @@ Note: Spark 1.6.x + Hadoop 2.6.x (stand-alone) must be installed and the SPARK_H
To configure a spark job, run spark_setup.py with the following parameters: To configure a spark job, run spark_setup.py with the following parameters:
-s <file> : The PBS script file to generate' -s <file> : The PBS script file to generate
-a <account> : Name of account to charge' -a <account> : Name of account to charge
-n <num> : Number of nodes' -n <num> : Number of nodes*
-w <time> : Maximum walltime' -w <time> : Maximum walltime
-d <path> : Spark deployment directory' -d <path> : Spark deployment directory
*Note: number of nodes must be 2 or greater (tasks are not run on master node)
The deployment directory must be unique for each Spark batch job being executed and should be located in a scratch space (Spark uses this directory to write temporary files). After running spark_setup.py, the specified deployment directory will be created (or re-initialized if it already exists) and template configuration files/scripts will copied into the "templates" subdirectory under the deployment directory. If needed, these template files may be modified before the Spark job is submitted. When the job is submitted, these template files will be copied into per-node configuration directories and used by Spark to configure worker nodes. The deployment directory must be unique for each Spark batch job being executed and should be located in a scratch space (Spark uses this directory to write temporary files). After running spark_setup.py, the specified deployment directory will be created (or re-initialized if it already exists) and template configuration files/scripts will copied into the "templates" subdirectory under the deployment directory. If needed, these template files may be modified before the Spark job is submitted. When the job is submitted, these template files will be copied into per-node configuration directories and used by Spark to configure worker nodes.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment