Commit 7d3965fa authored by William A. Wieselquist's avatar William A. Wieselquist
Browse files

Add version 1.0

parents
Instructions
------------
If you are a criticality safety expert who has used ICSBEP benchmarks, please
help us and participate in this activity! If you are not, please forward this
to criticality safety experts in your organization.
Download the sg8-v1.0-feedback-form.xlsx and add to it your ratings, comments,
and suggestions for improvement for ICSBEP benchmarks you have used. Additional
instructions are inside the excel sheet.
Send the form to Will Wieselquist (SG-8 lead) at ww5@ornl.gov to be
incorporated into this repository.
Forms are due December 5, 2021 to be incorporated in the final report. Each
form will be placed with last name of the contributor in the feedback directory.
Forms may be updated numerous times. Please don't hesitate to send me numerous
updates. Any updates can be reincorporated.
Contents
--------
This repository will contain the data by individual in the `feedback` and then
combined into one `summary.csv` file for analysis via `scripts/sg8-v1-post.py`.
For example, to add the contents of two feedback forms to the summary.csv,
one wold execute:
```
python3 scripts/sg8-v1-post.py feedback/Marshall.xlsx summary.csv
python3 scripts/sg8-v1-post.py feedback/Percher.xlsx summary.csv
```
There is a directory with some test files at `scripts/tests` to practice
using the sg-v1-post.py script.
```
$ python3 ../sg8-v1-post.py test-case1.xlsx practice.csv
Reading data from: test-case1.xlsx
Scanning for all cases: 4952it [00:00, 13275.35it/s]
Processing rows: 4952it [00:00, 14081.22it/s]
Updating existing: practice.csv
Done.
```
This produces a simple CSV as follows.
```
Case,Rev,Author,Rating,ModelIssue,BiasIssue,UncertaintyIssue,ImprovementsRecommended,AdditionalNotes
HEU-COMP-FAST-001-001,1,J. Doe,3.0,True,False,True,,
HEU-COMP-FAST-001-002,1,J. Doe,2.0,False,False,False,,
HEU-COMP-FAST-002-001,2,J. Doe,4.0,False,True,False,,
HEU-COMP-FAST-003-001,1,J. Doe,2.0,False,False,False,Do better,Try harder
HEU-COMP-FAST-003-002,1,J. Doe,2.0,False,False,False,Do better,Try harder
HEU-COMP-FAST-003-021,1,J. Doe,2.0,False,False,False,Do better,Try harder
HEU-COMP-FAST-003-022,1,J. Doe,2.0,False,False,False,Do better,Try harder
HEU-COMP-FAST-003-023,1,J. Doe,2.0,False,False,False,Do better,Try harder
HEU-COMP-INTER-003-001,0,J. Doe,3.0,False,False,False,,
HEU-COMP-INTER-003-002,0,J. Doe,3.0,False,False,False,,
HEU-COMP-INTER-003-003,0,J. Doe,3.0,False,False,False,,
```
What is SG-8
------------
Subgroup 8 (SG-8) is an OECD/NEA Working Party for Nuclear Criticality Safety (WPNCS)
sub group charged with developing a methodology for collecting and disseminating
feedback on evaluations from qualified experts to better serve users of the
ICSBEP benchmarks.
Over twenty-five years of benchmarking activity, the expectations and review
rigor required for ICSBEP has evolved, the benchmarks are being used for
unanticipated scenarios, tools and computational power exist to solve more
complex problems, and new practitioners are entering the field.
A need has been recognized to preserve expert knowledge and judgement regarding
the suitability of ICSBEP evaluations to common uses such as modern code validation,
nuclear data evaluation, and nuclear data adjustment.
Technical Significance
----------------------
With the increasing rigor of the ICSBEP review process, there exists a
disparity between earlier and modern benchmarks in terms of uncertainty
quantification and more realistic modelling of the configurations. For example,
earlier benchmarks may quote unrealistic uncertainties that are then used to
set safety limits or assess nuclear data evaluations. Additionally, there are
benchmarks that have clear consistency problems, internally across cases or
compared to other, similar benchmarks.
import pandas as pd
import sys
import os
from collections import defaultdict
from tqdm import tqdm
# set to true to generate debugging output
debug = False
# function to parse a full case name like
# HEU-COMP-FAST-002-001 and return the last 001
# ---
def case_parse(full_case):
if full_case == "(REJECTED)":
case0 = -1
elif full_case == "":
case0 = 0
else:
case0 = int(full_case.split("-")[-1])
return case0
# function to take the full case name like
# HEU-COMP-FAST-002-001 and a special sequence of
# cases to add like "5-9,13" and expand it to the
# case list of 1,5,6,7,8,9,13. The keyword 'ALL'
# implies all cases found in scanning the excel
# sheet.
def case_list(full_case, add_cases, all_cases):
case0 = case_parse(full_case)
cases = list()
if add_cases == "":
cases.append(case0)
else:
for c in add_cases.split(","):
if c == "ALL":
cases.extend(all_cases)
else:
c_list = c.split("-")
if len(c_list) > 1:
cases.extend(list(range(int(c_list[0]), int(c_list[1]) + 1)))
else:
cases.append(int(c_list[0]))
return list(set(cases))
# reads a specially formatted Excel data file with benchmark feedback
def process_excel_file(input_file):
# get excel sheet data
header = pd.read_excel(input_file, nrows=11, header=None)
c_author = header.loc[5, 2]
c_email = header.loc[5, 4]
# get ratings data
print("Reading data from:", input_file)
data = pd.read_excel(input_file, skiprows=13, usecols="B:I")
data = data.fillna("")
# print data for debugging
if debug:
print(data)
# create the ALL case range
all_cases = defaultdict(list)
for index, row in tqdm(
data.iterrows(), desc="Scanning for all cases", ascii=False, ncols=75
):
iden = row["Identifier"]
full_case = row["Case"]
case0 = case_parse(full_case)
all_cases[iden].append(case0)
# print all cases for debugging
if debug:
print(all_cases)
# create a list element for each records row
records = list()
for index, row in tqdm(data.iterrows(), "Processing rows", ascii=False, ncols=75):
# only add data with a rating
c_rating = row["Rating"]
if c_rating != "":
# extract data
iden = str(row["Identifier"])
full_case = str(row["Case"])
c_rev = int(row["Rev."])
add_cases = str(row["Add Cases*"])
c_issue_keys = set(str(row["Issue Keys**"]).split(","))
c_improvements = str(row["Improvements Recommended"] or "")
c_notes = str(row["Additional Notes"] or "")
# expand to a case list (repeating the above data for each case)
for case in case_list(full_case, add_cases, all_cases[iden]):
# reconstruct full_case identifier
if case == -1:
c_full_case = iden + "-REJ" # rejected
else:
c_full_case = "{}-{:03d}".format(iden, case)
# create a record with all the data
records.append(
{
"Case": str(c_full_case),
"Rev": int(c_rev),
"Author": str(c_author),
"Rating": "{:2.1f}".format(c_rating),
"ModelIssue": "M" in c_issue_keys,
"BiasIssue": "B" in c_issue_keys,
"UncertaintyIssue": "U" in c_issue_keys,
"ImprovementsRecommended": str(c_improvements),
"AdditionalNotes": str(c_notes),
}
)
if debug:
print(*records, sep="\n")
# convert records data to a data frame
df = pd.DataFrame(data=records)
return df
def main():
# input file from which we read data
input_file = sys.argv[1]
# archive file we merge new data into
archive_file = sys.argv[2]
if input_file.lower().endswith((".xlsx")):
df = process_excel_file(input_file)
else:
df = pd.read_csv(input_file)
# if the archive file exists, append to it.
if os.path.exists(archive_file):
print("Updating existing:", archive_file)
# use a custom index to ensure the combine happens as expected
df_index = ["Case", "Rev", "Author"]
df.set_index(df_index, inplace=True)
# read the old data and update the index as well
dfa = pd.read_csv(archive_file)
dfa.set_index(df_index, inplace=True)
# combine first (also called "upsert" which is a combination of
# updating existing and inserting). the data frame with priority
# should call combine first.
df = df.combine_first(dfa)
# reset the index so the CSV output is complete
df.reset_index(inplace=True)
updated = open(archive_file, "w")
updated.write(df.to_csv(index=False, header=True))
updated.close()
print("Done.")
main()
Case,Rev,Author,Rating,ModelIssue,BiasIssue,UncertaintyIssue,ImprovementsRecommended,AdditionalNotes
HEU-COMP-FAST-001-001,1,J. Doe,3.0,True,False,True,,
HEU-COMP-FAST-001-002,1,J. Doe,2.0,False,False,False,,
HEU-COMP-FAST-002-001,2,J. Doe,4.0,False,True,False,,
HEU-COMP-FAST-003-001,1,J. Doe,2.0,False,False,False,Do better,Try harder
HEU-COMP-FAST-003-002,1,J. Doe,2.0,False,False,False,Do better,Try harder
HEU-COMP-FAST-003-021,1,J. Doe,2.0,False,False,False,Do better,Try harder
HEU-COMP-FAST-003-022,1,J. Doe,2.0,False,False,False,Do better,Try harder
HEU-COMP-FAST-003-023,1,J. Doe,2.0,False,False,False,Do better,Try harder
HEU-COMP-INTER-003-001,0,J. Doe,3.0,False,False,False,,
HEU-COMP-INTER-003-002,0,J. Doe,3.0,False,False,False,,
HEU-COMP-INTER-003-003,0,J. Doe,3.0,False,False,False,,
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment