remove cache (a0949811) · Commits · sns-hfir-scse / diffraction / nomad_calibration

.cache/outputgrouping.xml

deleted100644 → 0

+0 −154

File deleted.

Preview size limit exceeded, changes collapsed.

.gitignore

+1 −1

Original line number	Diff line number	Diff line
		.offsets/
		.cache/
		.cache/*

README.md

+62 −1

Original line number	Diff line number	Diff line
		# nomad_calibration
		# Calibration routine for NOMAD at SNS, ORNL

		The calibration routine takes the input configuration in the `cal_config.json`
		file and can be executed from the command line simply as,

		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
		./nom_cal
		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

		> For the moment, the path is not added to the systemwise environment variable
		yet, and therefore one needs to change directory from the command line to the
		current directory before executing the command above.

		Here is a typical input configuration file (in JSON format),

		```json
		{
		"Diamond": "/SNS/NOM/IPTS-31346/nexus/NOM_192520.nxs.h5",
		"Instrument": "NOM",
		"Date": "2023-06-14",
		"SampleEnv": "Shifter",
		"OutputDir": "/SNS/NOM/shared/CALIBRATION/2023_1_1A_Group_CAL",
		"DiaLattParam": 3.5671299351,
		"GenShadowMask": true,
		"GroupMethod": "KMEANS_ED",
		"SaveInitCalTable": true,
		"Quiet": true,
		"ArbCalFile": "arb_starting_cal.h5",
		"MaskFile": "outputmask.xml"
		}
		```

		The diamond lattice parameter in the example here (see the `DiaLattParam` entry) is
		for diamond at ambient condition and one can refer to the following paper for
		detailed information,

		> DOI: 10.1103/PhysRevLett.104.085901

		The `GenShadowMask` entry is for controlling whether to generate the masks for
		the shadowed region of the detectors. The `GroupMethod` entry specifies the
		method to be used for grouping pixels initially before the calibration.
		Currently, the grouping of pixels is only used for identifying those pixels
		which are far away from others and thus will be treated as outliers and masked
		out. The `SaveInitCalTable` flag is for specifying whether to save the
		intermediate calibration where all pixels are lined up without further
		calibration. The `Quiet` flag controls whether to print out logs during the
		running. By suppressing the console log output, one can save a lot of wall time.
		The `ArbCalFile` entry specifies an arbitrary starting calibration file. This is
		to be used as the input calibration file for running the `PDCalibration`
		algorithm during the calibration run. It indeed can be any arbitrary
		pre-existing calibration file, even one is running across different sample
		environments, since anyhow it is only used as the initial starting point. If one
		prefers to use their own mask file, they can specify their own version of mask
		file in the `MaskFile` entry and then one needs to put the mask file to the
		`inputs` directory.

		---

		Yuanpeng Zhang @ 2023-09-05 Tue 05:09 PM EDT
		SNS-HFIR, ORNL

		---

nom_cal_dev

+36 −164

Original line number	Diff line number	Diff line
		#!/bin/bash

		if [ "$#" -lt 1 ]; then
		echo ""
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo "Error > No control file provided. Hence we have to stop..."
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo ""
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo "Info > Usage: ./run_me control_file"
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		exit
		fi

		input=$1

		if [ ! -f "$input" ]; then
		echo ""
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo "Error> Control file '$input' not found. Hence we have to stop..."
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo ""
		exit
		fi

		echo ""
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		while IFS= read -r line
		do
		echo $line \| grep -iq "diamond" && a=1 \|\| a=0
		if [ $a = 1 ] ; then
		dia_file=$(echo $line \| awk -F'::' '{print $2}' \| xargs)
		echo "Diamond file: $dia_file"
		fi
		echo $line \| grep -iq "date" && a=1 \|\| a=0
		if [ $a = 1 ] ; then
		date=$(echo $line \| awk -F'::' '{print $2}' \| xargs)
		echo "Date: $date"
		fi
		echo $line \| grep -iq "instrument" && a=1 \|\| a=0
		if [ $a = 1 ] ; then
		inst=$(echo $line \| awk -F'::' '{print $2}' \| xargs)
		echo "Instrument: $inst"
		fi
		echo $line \| grep -iq "sample environment" && a=1 \|\| a=0
		if [ $a = 1 ] ; then
		sam_env=$(echo $line \| awk -F'::' '{print $2}' \| xargs)
		echo "Sample environment: $sam_env"
		fi
		echo $line \| grep -iq "output directory" && a=1 \|\| a=0
		if [ $a = 1 ] ; then
		out_dir=$(echo $line \| awk -F'::' '{print $2}' \| xargs)
		working_dir="/SNS/NOM/shared/CALIBRATION/"
		cd $working_dir

		dia_file=$(python utils/load.py cal_config.json)
		gen_mask_file=$(python utils/mg_config.py cal_config.json)
		running_dir="/SNS/NOM/shared/CALIBRATION/.cache/"
		group_method=$(python utils/gm_config.py cal_config.json)
		out_dir=$(python utils/od_config.py cal_config.json)

		# echo settings
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo "Working directory: $working_dir"
		echo "Calibrant file: $dia_file"
		echo "Generate shadow mask file: $gen_mask_file"
		echo "Running directory: $running_dir"
		echo "Grouping method: $group_method"
		echo "Output directory: $out_dir"
		fi
		echo $line \| grep -iq "group method" && a=1 \|\| a=0
		if [ $a = 1 ] ; then
		group_method=$(echo $line \| awk -F'::' '{print $2}' \| xargs)
		echo "Group method: $group_method"
		fi
		echo $line \| grep -iq "mask file" && a=1 \|\| a=0
		if [ $a = 1 ] ; then
		mask_file=$(echo $line \| awk -F'::' '{print $2}' \| xargs)
		echo "Mask file: $mask_file"
		fi
		echo $line \| grep -iq "generate mask" && a=1 \|\| a=0
		if [ $a = 1 ] ; then
		gen_mask_file=$(echo $line \| awk -F'::' '{print $2}' \| xargs)
		echo "Genarated mask file: $gen_mask_file"
		fi
		done < $input
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo ""

		if [ -z $dia_file ]; then
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo "Error > No diamond file specified in the control file!"
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo ""
		exit
		fi
		if [ -z $date ]; then
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo "Error > No date info specified in the control file!"
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo ""
		exit
		fi
		if [ -z $inst ]; then
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo "Error > No instrument specified in the control file!"
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo ""
		exit
		fi
		if [ -z $sam_env ]; then
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo "Error > No sample environment specified in the control file!"
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo ""
		exit
		fi
		if [ -z $out_dir ]; then
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo "Error > No output directory specified in the control file!"
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo ""
		exit
		fi
		if [ -z $group_method ]; then
		group_method="KMEANS_ED"
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo "Info > No group method specified in the control file!"
		echo "Info > Will default to 'KMEANS_ED' method."
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo ""
		fi
		if [ -z $mask_file ]; then
		mask_file="mask.in"
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo "Info > No mask file specified in the control file!"
		echo "Info > Will default to 'mask.in' file as input mask."
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"
		echo ""
		fi

		running_dir="/SNS/NOM/shared/CALIBRATION/Group_calib_running/"
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"

		# Generate mask
		if [ $gen_mask_file ]; then
		mantidpython /SNS/NOM/shared/CALIBRATION/Group_calib_scripts/mask_gen/nom_shadow_mask_gen.py $dia_file
		if [ $gen_mask_file = "yes" ]; then
		mantidpython $working_dir/inputs/mask_gen/nom_shadow_mask_gen.py $dia_file
		rm -rf $running_dir/manual_mask_cache/gen_mask/$gen_mask_file
		Rscript /SNS/NOM/shared/CALIBRATION/Group_calib_scripts/mask_gen/find_dip.R $gen_mask_file
		cat $running_dir/$mask_file $running_dir/manual_mask_cache/gen_mask/$gen_mask_file > $running_dir/mask_combined_cache.in
		Rscript $working_dir/inputs/mask_gen/find_dip.R $gen_mask_file
		cat $working_dir/inputs/mask.in \
		$running_dir/manual_mask_cache/gen_mask/$gen_mask_file \
		> $running_dir/mask_combined_cache.in
		else
		cat $running_dir/$mask_file > $running_dir/mask_combined_cache.in
		cat $working_dir/inputs/mask.in > $running_dir/mask_combined_cache.in
		fi

		cd $running_dir
		@@ -146,33 +42,18 @@ cp autogrouping_config_template.json autogrouping_config.json
		sed -i "s%TO_REPLACE_CALIBRANT%${dia_file}%" autogrouping_config.json
		sed -i "s%TO_REPLACE_GM%${group_method}%" autogrouping_config.json
		sed -i "s%TO_REPLACE_MF%mask_combined_cache.in%" autogrouping_config.json
		cp group_calib_template.json group_calib.json
		sed -i "s%TO_REPLACE_CALIBRANT%$dia_file%" group_calib.json
		sed -i "s%TO_REPLACE_DATE%$date%" group_calib.json
		sed -i "s%TO_REPLACE_INS%$inst%" group_calib.json
		sed -i "s%TO_REPLACE_ENV%$sam_env%" group_calib.json
		sed -i "s%TO_REPLACE_DIR%$out_dir%" group_calib.json

		# Grouping
		# . /opt/anaconda/etc/profile.d/conda.sh # un-comment for prod version
		. /opt/anaconda/etc/profile.d/conda.sh
		# In case the `dev` environment is not working properly,
		# one can roll back to the `prod` version by switching
		# the commented line below.
		# conda activate mantidtotalscattering-dev # un-comment for prod version
		conda activate mantidtotalscattering-dev
		# conda activate mantidtotalscattering

		# -----> Comment the following lines for prod version
		source /SNS/users/y8z/Dev/mantid_total_scattering/.venv/bin/activate
		python /SNS/NOM/shared/Dev/mantid/build/bin/AddPythonPath.py
		cd /SNS/users/y8z/Dev/mantid_total_scattering
		python setup.py develop
		cd -
		# <----- Comment out the lines above for prod version

		python autogrouping.py

		# conda deactivate # un-comment for prod version
		deactivate # comment out for prod version
		conda deactivate

		if [ "$group_method" = "KMEANS_ED" ]; then
		sed -i '3d' outputgrouping.xml
		@@ -182,23 +63,14 @@ fi

		# External call to translate the mask file in plain text form to
		# its XML form.
		eval "$(conda shell.bash hook)"
		conda activate ~y8z/miniconda/envs/nom_calib
		python group_calib_prep.py
		conda deactivate

		# Calibration
		# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
		# Sometimes the nightly version of Mantid may not be
		# working properly, in which case we then need
		# to comment out the following line and
		# uncomment the line down below it.
		# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
		# cp outputgrouping_physical.xml outputgrouping.xml
		mantidpythonnightly group_calib.py
		# mantidpython group_calib.py
		cd $working_dir

		cd $out_dir
		# Calibration
		mantidpython $working_dir/utils/nom_cal.py $working_dir

		echo ""
		echo "+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+"

readme.txt

deleted100644 → 0

+0 −39

Original line number	Diff line number	Diff line
		The new calibration routine has been moved to a central place, located at,

		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
		/SNS/NOM/shared/CALIBRATION/Group_calib_scripts
		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

		Going into the directory above and run something like,

		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
		./run_me 2022_1_1A/control_172183.dat
		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

		will do the calibration job as usual.

		Here following is a typical example of the control file,

		+-+-+-+-+-+-+-+-+-+-+-+-+-+DO NOT INCLUDE ME+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
		diamond file :: /SNS/NOM/IPTS-28922/nexus/NOM_172183.nxs.h5
		instrument :: NOMAD
		date :: 2022-05-19
		sample environment :: shifter
		mask file :: mask.in
		generate mask :: shadow_mask_cryostream_172183.in
		output directory :: /SNS/NOM/shared/CALIBRATION/2022_1_1A_Group_CAL
		+-+-+-+-+-+-+-+-+-+-+-+-+-+DO NOT INCLUDE ME+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

		IMPORTANTLY, the 'mask file' entry specifies the input manual mask -
		which can come from anywhere. Basically, it is a fixed mask that we
		believe those included pixels are bad. The 'generate mask' entry is for
		storing the automatically generated mask which corresponds to the shadow
		region of the detectors. Usually, those shadowed detectors still have
		normal Bragg peaks so they won't be masked out during the peak fitting
		step. We just implemented an algorithm to automatically recognize
		those shadowed pixels using a convolution algorithm.

		============================
		Yuanpeng Zhang @ May-27-2022
		SNS-HFIR, ORNL
		============================