CONTRIBUTING.md

# Contributing

This documentation provides guidelines for new PyARC developers to learn about development practices. Thank you for your contributions!

## Table of Contents

1. [Developer Code of Conduct](#code) - Best practices recommendations for new developers
2. [Version Control Workflow](#version-control-workflow) - Adding new codes or features to PyARC
3. [Unit Tests](#unit-tests) - Unit-tests implementation
4. [Integration Steps](#new-code-integration-steps) - Description of steps for new code integration
    * [Input Structure](#input-structure) - Definition of input structure
    * [New Code Addition](#new-code-addition) - Workflow definition
    * [Tutorial](#tutorial) - Tutorial development
5. [Code Integration Example](#code-integration-example) - Example of code integration
    * [Schema](#schema) - Developing in the Schema
    * [PyMyCode Example](#pymycode) - Structure of PyCode
        * [Pre Check](#precheck) - Early check logic
        * [Pre Run](#prerun) - Pre-run logic
        * [Execution](#execution) - Execution logic
        * [Post Run](#postrun) - Post-run logic
    * [Integration in PyARC](#pyarc-integration) - Integration of PyCode into PyARC
    * [Unit Test](#unit-test) - Example of unit-test
6. [Additional resources](#resources) - More Gitlab resources

# Developer Code of Conduct

The code of conduct for developer interactions can be found
[here](./CODE_OF_CONDUCT.md). Please report any unacceptable behavior to
pyarc@anl.gov.

# Version control workflow

Please follow the traditional collaborative development guidelines:
1. Work from Gitlab, in a branch off of `development`, and frequently push your
   changes to the repository to simplify review. Create issues and reference
   them when appropriate.
2. Once you have completed your development, complete a Merge Request (from your
   branch to `development`) and request review of your code from another
   developer. Merged changes will be included in new versions of PyARC by
   merging the `development` branch into the `master` branch.
3. Strive for high quality code rather than quick demonstration. Improving code
   afterward is always more time consuming than improving it upfront.

If you do not have access to push branches and/or merge the PyARC repository,
please contact either pyarc@anl.gov to request access.

# Unit-tests

Unit tests are automatically run everytime changes are pushed to the repository.
They are also a very important tool for developers who can manually run them
during the development work to verify his code development did not affect other
parts of PyARC.

Execution of unit tests:
* all unit tests: `Workbench-path/rte/entry.sh -m unittest discover -v`
* specific unit test: `Workbench-path/rte/entry.sh -m unittest discover -p test_pyarc_mytest.py`

Unit tests must be setup to test:
* pre-check in [test_pyarc_precheck](/test/test_pyarc_precheck.py): a few unit-tests may be required to check the `precheck` logic and verify the error messages returned.
* pre-run in [test_pyarc_prerun](/test/test_pyarc_prerun.py): typically 5-10 unit tests may be needed to test out verious options of the code. In this process, the generated input by PyARC is compared line-by-line with a reference one located in `test/data/`.
* post-run in [test_pyarc_postrun](/test/test_pyarc_postrun.py): typically, 1-2 unit tests may be needed to test out post-processing logic. In this process, the generated summary files will be compared line-by-line with a reference one located in `test/data/`.
* execution in [test_pyarc_run](/test/test_pyarc_run.py): 1 unit test is typically used to test out the execution. In this process, a few results from the calculation will be compared with reference values.
* tutorial in [test_pyarc_run_tutorial](/test/test_pyarc_run_tutorial.py): it is important to automatically verify the proper execution of the tutorial results to verify it is up-to-date.

# New Code Integration Steps

Please review the following integration steps. They should generally be
completed in the following order. Each and every one of these steps needs to be
completed whether you are integrating a new code to PyARC or adding an option
within a code (some of the steps may be straightforward in this case).

## Input Structure

For development requiring changes in input definition (new input option), the
first step is to define the input structure specifying for each proposed option:
* Required/optional, dependencies with other options?
* Range of appearance
* Types of value, ranges of their values or list of acceptable values
* etc.

This input definition is then used to update the [schema](etc/arc.sch) and
develop templates (provided to the used during autocompletion), as discussed in
[Schema](#schema). **It is important to thoroughly test your input from the
Workbench before moving to the next step.**

## New Code Addition

The workflow definition of new code integration is implemented within a class
that contains the following methods:

* pre-check: list of initial checks that can be done on the input. Such additional tests may be required due to limitation in Workbench input interpretor
* pre-run: define the pre-processing logic, extracting the information from the PyARC input and translate it into the native code input.
* execution: describe the execution logic. It is usually very straightforward (build temporary directory and execute the input there before cleaning up).
* post-run: define the post-processing logic, extracting the information from the output and translate it into the suggested PyARC `*.summary` and `*.extended.summary` files.

If pre-processing or execution failures are observed, any remaining steps in the
code should be skippped and an error message should be provided to the user.

The Workflow can be made more complex by calling sub-codes (like MCC3 is being
called when running PERSENT). In this case, prerun of MCC3 is called from prerun
of PERSENT.

## Tutorial

Here is the link to existing [Tutorials](/tutorial/README.md#tutorial). Code
integration is completed only when tutorial is developed with proper
documentation. User input files (based on ABTR core design if possible, to be
consistent with other tutorial) needs to be accompanied by expected summary
files, and by a `README.md` file detailing the input and output.

# Code Integration Example

In this example we'll integrate a code named `MyCode` into PyARC. We will define
`PyMyCode.py` file and describe its key components.

## Schema

The following input example needs to be develop to understand the different
options that should be defined in the PyARC schema.

```javascript
mycode ( block_id ) {  # optional, can appear several time - no limit. Block ID is required, and take any string
        my_value = 5 # required, integer between 1 to 10 (included), default = 1
        my_list = [ val1 val2 val3 val4 val5 ] # required, takes string, number of string specified by my_value, each is unique
    }
```

This input definition can be translated into [__schema__](etc/arc.sch) as
following. More information about the SON input structure is available in
[WASPON](https://code.ornl.gov/neams-workbench/wasp/-/blob/master/waspson/README.md#standard-object-notation-son)
documentation, and more information on schema language structure is available in
[HIVE](https://code.ornl.gov/neams-workbench/wasp/-/blob/master/wasphive/README.md#hive)
documentation.

```javascript
mycode{
    Description = "[optional] my block description"
    MinOccurs=0
    MaxOccurs=NoLimit
    InputTmpl="mycode"
    id{
        MinOccurs=1
        MaxOccurs=1
        ValType=String
    }
    my_value{
        Description = "[required] description of my_value"
        MinOccurs=1
        MaxOccurs=1
        ValType=Int
        MinValInc=1
        MaxValInc=10
        InputTmpl="flagtypes"
        InputDefault=1
    }
    my_list{
        Description = "[required] description of my_list"
        InputTmpl="sonarray"
        MinOccurs=1
        MaxOccurs=1
        ChildUniqueness  = [ value ]
        value{
            MaxOccurs="../../my_value"
            MaxOccurs="../../my_value"
            ValType=String
        }
    }
}
```

This should be associated with a `mycode.tmpl` located in `etc/templates` that
would look like bellow. The template will be provided back to the user when they
use the autocompletion function within the Workbench.

```javascript
mycode ( block_id ) {
    my_value = 1
    my_list = [ list_val ]
}
```

## PyMyCode

Below is a summary of the structure of [PyMyCode](PyMyCode.py) that is being
developed. It takes 5 blocks. First, the `__init__` block defines variables
within the MyCode class. Then, the `workflow` block provides the workflow,
calling for `prerun`, `run`, and `postrun` procedures. The `precheck` procedure
is called much earlier, during the early steps of `PyARC`.

```javascript
class MyCode(object):
    """Defines mycode integration procedures"""
    def __init__(self, user_object, do_prerun = False, do_run = False, do_postrun = False):
        self.user_object = user_object
        self.do_prerun = do_prerun
        self.do_run = do_run
        self.do_postrun = do_postrun
        self.list_mycode_inp_path = []
        self.list_mycode_out_path = []

    def workflow(self):
        """Defines the workflow for mycode calculations"""
        self.user_object.print_timelines("mycode calculation")
        mycode = self.user_object.calculations_input().mycode
        for step_calc in range(len(mycode.depletion_steps.value)):
            num_time_step = int(str(mycode.depletion_steps.value[step_calc]))
            time_step = self.user_object.list_time_steps[num_time_step]
            if self.do_prerun:
                mycode_crash = self.prerun(time_step)
                if mycode_crash:
                    self.do_run = False
                    self.do_postrun = False
            if self.do_run:
                mycode_crash = self.run(time_step)
                if mycode_crash: self.do_postrun = False
            if self.do_postrun:
                self.postrun(time_step)

    def precheck(self):
        """Perform various early checks on the input and associated files - crash before any pre-processing on any codes if mycode_crash is True"""
        mycode_crash = False
        return mycode_crash

    def prerun(self, time_step):
        """Write mycode input file"""
        return mycode_crash

    def run(self, timestep):
        """ Procedure that executes mycode inputs and returns the output """
        return mycode_crash

    def postrun(self, timestep):
        """Extracts results from mycode output file"""
```

### Precheck

Below is an extended example of the `precheck` block. The main role of this
block are to detect issues and crash the code with error message before PyARC
start processing any input (from any codes). The objective if to complete
additional checks on the user input that the Workbench input logic cannot
handle. Example of checks are listed below:

* Check executable file is available
* Perform complex geometry checks (outer radius is larger than inner radius, etc.)
* etc.

```javascript
    def precheck(self):
        """Perform various early checks on the input and associated files - crash before any pre-processing on any codes if mycode_crash is True"""
        mycode_crash = False

        # check 1 - executable is available
        mycode_exec = "%s/mycode.x"%self.user_object.arc_executable
        if os.path.isfile(mycode_exec) is False:
            mycode_crash = False # should be True, but kept to false since the executable is not meant to exist!
            self.user_object.write_err_lines("ERROR: Executable %s does not exist"%mycode_exec)

        return mycode_crash
```

### Prerun

Below is an extended example of the `prerun` block. The main role of this block are:
* create `mycode.inp` input file
* extract the information from the PyARC `.son` input file, process it (if needed)
* write the `mycode.inp` input file
* optionally: detect issues and crash the code with error message

```javascript
    def prerun(self):
        """Write MyCode input file"""
        calculations = self.user_object.calculations_input()
        mycode = calculations.mycode
        mycode_crash = False
        self.user_object.print_timelines("   Pre-run  MyCode")
        mycode_input_file  = open(self.mycode_inp_path, "w")

        # TODO: write mycode input logic - here is an example highlighting how to access the data from the PyARC input
        mycode_input_file.writelines("mycode - V0.0.0")
        if calculations.mycode is not None:
            for it, mycode_block in enumerate(calculations.mycode):
                mycode_id = str(mycode_block.id).lower()
                mycode_input_file.writelines("\n  Block: %s"%mycode_id)
                num_val = int(str(mycode_block.my_value.value).lower())
                mycode_input_file.writelines("\n  Block Values: %i"%num_val)
                for val in range(num_val):
                    item = str(mycode_block.my_list.value[val])
                    mycode_input_file.writelines("\n    Block Item %i: %s"%(val, item))

        mycode_input_file.close()
        append_lines(self.mycode_inp_path, self.user_object.main_input_file_name)
        return mycode_crash
```

### Execution

Below is an extended example of the `run` block. The main role of this block are:
* create the temporary directory `tmp_mycode` to run the code
* execute the code
* save the `mycode.out` output file
* detect code crash and return error message
* cleanup the directory

```javascript
    def run(self):
        """ Procedure that executes MyCode inputs and returns the output """
        mycode_crash = False
        self.user_object.print_timelines("   Run  MyCode")
        input_name = self.mycode_inp_path

        mycode_path = "%s/mycode.x"%self.user_object.arc_executable
        if os.path.exists("tmp_mycode") == False:
            os.mkdir("tmp_mycode")
        os.chdir("tmp_mycode")

        os.system("%s < %s > %s"%(mycode_path, self.mycode_inp_path, self.mycode_out_path)) # example of execution logic

        mycode_output_file = open(self.mycode_out_path, "w")
        mycode_output_file.writelines("MyCode output")
        mycode_output_file.close()
        shutil.copy(self.mycode_out_path, "../")

        append(self.mycode_out_path, self.user_object.output_path)

        lines1 = open(self.mycode_out_path, "r").readlines()
        for l in range(len(lines1)):
            if "FATAL ERROR MESSAGE FROM MyCode" in lines1[l]:
                self.user_object.write_err_lines("ERROR: MyCode simulation has crashed")
                error_msg = ""
                error_lines = min(len(lines1), 20)
                for j in range(error_lines):
                    error_msg += "\n"+lines1[l+(j-error_lines+2)]
                self.user_object.write_err_lines(error_msg)
                mycode_crash = True

        os.chdir("..")
        shutil.rmtree("tmp_mycode")
        return mycode_crash
```

### Postrun

Below is an extended example of the `postrun` block. The main role of this block are:
* read the `mycode.out` output file and extract information
* create and populate the `mycode.summary` file with the main information that is the most useful to the user (results per assembly or areas)
* create and populate the `mycode.extended.summary` filed with more information (results for each region)

```javascript
    def postrun(self):
        """Extracts results from MyCode output file"""
        self.user_object.print_timelines("   Post-run MyCode")

        output = self.mycode_out_path + ".summary"
        out = open(output, "w")
        out.writelines("\n\n************************ MyCode SUMMARY ***************************\n")

        output_ex = self.mycode_out_path + ".extended.summary"
        outex = open(output_ex, "w")
        outex.writelines("\n\n************************ MyCode SUMMARY  ***************************\n")


        mycode_output = open(self.mycode_out_path, "r").readlines()

        # TODO: implement post-processing logic

        self.user_object.results['mycode'] = 0 # provide some results that we think users will always be interested. This will be used for unit-tests (check results do not change) and can be extracted from Python execution

        out.writelines("\n")
        out.close()
        append(output, self.user_object.output_summary_path)

        outex.writelines("\n")
        outex.close()
        append(output_ex, self.user_object.output_extended_summary_path)
```

## PyARC Integration

In [PyARC](PyARC.py) and [PyARCUserObject](PyARCUserObject.py), check where `mycode` is being called for to understand how to bring it in the overall PyARC workflow. The following lines in particular provide where in the PyARC workflow is MyCode being executed (at the end, in our case).

```javascript
if self.user_object.invoke_mycode():
    self.mycode.workflow()
```

## Unit Test

See the example below that is in [test_pyarc](test/test_pyarc_input_generation.py). The input example in [arcbench_test0_mycode.son](test/data/arcbench_test0_mycode.son) is being executed only to est the **prerun** logic. The automatically generated `mycode.inp` input is compared line-by-line with the reference [test0_arcbench_test0_mycode.inp](test/data/test0_arcbench_test0_mycode.inp) input that was stored by the developer.

```javascript
    def test_PyARC_input0_mycode(self):
        """ Fake unit test to monstrate unit-tests logic applied to MyCode Fake integrated code """
        this = PyARC.PyARC()
        input_file = self.current_directory+"/data/arcbench_test0_mycode.son"
        options = PyARC.RunOptions()
        options.input = input_file
        od = self.current_directory
        wd = self.current_directory+"/arcbench_test0_mycode"
        this = PyARC.PyARC()
        this.user_object.do_run = False
        this.user_object.do_postrun = False
        this.execute(["-i",input_file,"-w",wd,"-o",od])

        generated_input = this.working_dirs[input_file]+"/mycode.inp"
        reference_input = self.current_directory+"/data/test0_arcbench_test0_mycode.inp"
        self.assertTrue(self.diff_file(generated_input, reference_input))

        shutil.rmtree(wd)
        os.remove(input_file.replace(".son",".isotxs_R_0"))
        os.remove(input_file.replace(".son",".inp"))
        os.remove(input_file.replace(".son",".zip"))
        os.remove(input_file.replace(".son",".timeline.out"))
        os.remove(input_file.replace(".son",".summary"))
        os.remove(input_file.replace(".son",".extended.summary"))
        os.remove(input_file.replace(".son",".user_info.out"))
```

# Additional Resources

Useful git commands:

```shell
    git pull # pull new changes from the PyARC's gitlab repository into the current branch.
    git status # check on the status of changed files in the local repository.
    git add <filename> # Add file "filename" to staged changes of the repository.
    git commit –m “my commit message” # Commit staged changes with the specified message.
    git push origin <branch_name> # Push a branch to the gitlab repository.
    git rm <filename> # Stage the removal of "filename" for the next commit.
```

For additional git commands and explanations, please see this git
[cheatsheet](https://training.github.com/downloads/github-git-cheat-sheet.pdf).

For information on effective communication on Gitlab (opening issues, merge
requests, performing code reviews, etc.), here is a link to
[Gitlab training](https://www.linkedin.com/learning/learning-gitlab-2/version-control-and-more?u=74472898).