Commit 4c4655cf authored by Kennedy, Joseph H's avatar Kennedy, Joseph H
Browse files

LEX-0.1-rc1

A first release candidate of the LIVVkit Extensions repository to
demonstrate our extensions functionality. It lincludes the CESM Greenland analyses for
cloud cover, energy balance, and surface mass balance, as well the
CISM-Albany dynamics analyses, which will be documented in the Evans et
al. (2018) LIVVkit validation paper.

This relase candidate:
  Provides basic documentation
  All required data to run on any machine
parents b6958ea4 a692cf5d
.. figure:: https://raw.githubusercontent.com/wiki/LIVVkit/LIVVkit/imgs/livvkit.png
:alt:
:alt:
LIVVkit Extensions
==================
LIVVkit Extensions (LEX)
========================
Note: this repository uses git's Large File Storage, you'll need to
install git-lfs.
This repository holds a collection of extension to`LIVVkit <https://livvkit.github.io/Docs/index.html>`_
for validation and custom analyses of ice sheet models and their associated Earth
system models.
Warning: git-lfs doesn't seem to work with non-default remote names, so
make sure you keep the default remote name ``origin``.
LEX was first described in [Evans2018]_; to reproduce the analyses there, see the
`Reproducing Evans et al. (2018)`_ section.
.. [Evans2018] Evans, K.J., J.H. Kennedy, D. Lu, M.M. Forrester, S. Price, J. Fyke,
A.R. Bennett, M.J. Hoffman, I. Tezaur, C.S. Zender, and M. Vizcaino (In Review).
LIVVkit 2.1: Automated and extensible ice sheet model validation.
*Geoscientific Model Development.*
Dependencies
------------
Because Validation analyses are particularly data intensive, this extensions repo
uses `git-lfs <https://git-lfs.github.com>`_ (Git Large File Support) in order to
distribute the required data. ``git-lfs`` can be installed either before or after
cloning this repository, but it will be needed *before* downloading the required
data. You can determine if you have ``git-lfs`` installed on your system by running
this command in POSIX compliant shells (`bash`, `zsh`, `fish`, etc.):
.. code:: bash
command -v git-lfs
or this command in `csh` variants:
.. code:: csh
which git-lfs
If ``git-fls`` is not installed, you can install it by following the instructions here:
https://git-lfs.github.com
Likewise, you will need to have `LIVVkit installed <https://livvkit.github.io/Docs/index.html>`_
to run these extensions. You can install it by running this command:
.. code:: bash
pip install --user livvkit
Basic Usage
-----------
First, install the required `Dependencies`_, then clone and enter this repository:
.. code:: bash
git lfs clone https://code.ornl.gov/LIVVkit/lex.git
cd lex
**Warning:** Because validation analyses are particularly data intensive, this
repository is rather large (~ GBs currently), and uses git-lfs (git large
file support) [#]_.
Each extension will have an associated JSON configuration file which will describe
the extension's analysis code, data locations, and options. To see a list of
available extensions, you can run this command:
.. code:: bash
find . -iname "*.json"
To execute any of these extensions, point ``livv`` (LIVVkit's command line interface)
to any of these extensions via the ``-e/--extension`` option (or the ``-V/--validate``).
For example, to run the minimal example extension, place the output website in the
`vv_test` directory, and serve the output website you'd run this command:
.. code:: bash
livv -e example/example.json -o vv_test -s
*Note:* All the extension configurations files assume you are working from the
top level ``lex`` directory. You *can* run any of these extensions from any
directory, but you will need to edit the paths in the JSON configuration files so
that ``livv`` can find the required files.
Likewise, you can also apply these analyses to any new model run [#]_ py adjusting
the paths to point to your model run.
.. [#] You may find `this tutorial by Atlassian useful <https://www.atlassian.com/git/tutorials/git-lfs>`_.
.. [#] This assumes the new data files conform to the format of the included data
files. That is, an extension that analyses output from the CISM-Albany ice
sheet model will likely be able to analyze any similar CISM-Albany simulation,
but likely would *not* be able to analyze output from the PISM ice sheet
model without "massaging" the PISM files into a CISM-Albany like structure, or
adjusting the extension. *This is a problem we are working on for future LEX
releases.*
Reproducing Evans et al. (2018)
-------------------------------
If all the required `Dependencies`_ are installed, and you've cloned the repository
into the directory ``lex``, you can reproduce all the figures and tables in
[Evans2018]_ by running this command from within the ``lex`` directory:
.. code:: bash
livv --validate smb/smb_icecores.json \
energy/energy_cesm.json \
clouds/clouds_cesm.json \
dynamics/dynamics_cisma.json \
-o vv_evans2018 -s
Developing a custom extension
-----------------------------
See the `LIVVkit documentation <https://livvkit.github.io/Docs/extend.html>`_ for
details on how to develop an extension. Briefly, a absolute minimum working example
is provided by the ``examples/`` extension, which should be copied to provide the
basis for your new extension. All extensions are required to contain a minimal working
example set of data such that they can be run an executed on any machine.
For extensions that require data for which re-host permission cannot be granted,
they must include documentation on how to acquire and use the data as well as either
a small set of processed data or a set of "fake" example data.
Issues, questions, comments, etc.?
----------------------------------
If you would like to suggest features, request tests, discuss contributions,
report bugs, ask questions, or contact us for any reason, use the
`LIVVkit issue tracker <https://github.com/LIVVkit/LIVVkit/issues>`_.
Want to send us a private message?
**Joseph H. Kennedy**
:github: @jhkennedy
:email: kennedyjh@ornl.gov
**Katherine J. Evans**
:github: @kevans32
:email: evanskj@ornl.gov
*Note: If you're emailing us, we recommend CC-ing all of us.*
\ No newline at end of file
......@@ -10,15 +10,12 @@ from livvkit.util import elements as el
describe = """CESM_cldhgh_ANN plot."""
def make_plot(config=None, out_path='.',
cloud_path='/lustre/atlas1/cli115/world-shared/4ue/obs_data/',
cism_path='/lustre/atlas1/cli115/world-shared/4ue/',
cesm_path='/lustre/atlas1/cli115/world-shared/4ue/b.e10.BG20TRCN.f09_g16.002/'):
def make_plot(config, out_path='.'):
# ---------------- Data source in TITAN ------------------------
f_cism = os.path.join(cism_path, 'Greenland_5km_v1.1_SacksRev_c110629.nc')
f_cesm_atm_climo_jja = os.path.join(cesm_path, 'postproc/atm/climos/b.e10.BG20TRCN.f09_g16.002_JJA_climo.nc')
f_isccp = os.path.join(cloud_path, 'ISCCP_ANN_climo.nc')
f_cloudsat = os.path.join(cloud_path, 'CLOUDSAT_ANN_climo.nc')
f_cism = os.path.join(config['cism_data'], 'Greenland_5km_v1.1_SacksRev_c110629.nc')
f_cesm_atm_climo_jja = os.path.join(config['cesm_atm_climos'], 'b.e10.BG20TRCN.f09_g16.002_JJA_climo.nc')
f_isccp = os.path.join(config['cloud_data'], 'ISCCP_ANN_climo.nc')
f_cloudsat = os.path.join(config['cloud_data'], 'CLOUDSAT_ANN_climo.nc')
# --------------------------------------------------------------
img_list = []
......@@ -190,12 +187,7 @@ def make_plot(config=None, out_path='.',
img_elem = el.image('CESM_cldhgh_ANN',
' '.join(describe.split()),
img_link)
if config:
img_elem['Height'] = config['image_height']
img_elem['Height'] = config['image_height']
img_list.append(img_elem)
return img_list
if __name__ == '__main__':
make_plot()
......@@ -10,15 +10,12 @@ from livvkit.util import elements as el
describe = """CESM_cldlow_ANN plot."""
def make_plot(config=None, out_path='.',
cloud_path='/lustre/atlas1/cli115/world-shared/4ue/obs_data/',
cism_path='/lustre/atlas1/cli115/world-shared/4ue/',
cesm_path='/lustre/atlas1/cli115/world-shared/4ue/b.e10.BG20TRCN.f09_g16.002/'):
def make_plot(config, out_path='.'):
# ---------------- Data source in TITAN ------------------------
f_cism = os.path.join(cism_path, 'Greenland_5km_v1.1_SacksRev_c110629.nc')
f_cesm_atm_climo_jja = os.path.join(cesm_path, 'postproc/atm/climos/b.e10.BG20TRCN.f09_g16.002_JJA_climo.nc')
f_isccp = os.path.join(cloud_path, 'ISCCP_ANN_climo.nc')
f_cloudsat = os.path.join(cloud_path, 'CLOUDSAT_ANN_climo.nc')
f_cism = os.path.join(config['cism_data'], 'Greenland_5km_v1.1_SacksRev_c110629.nc')
f_cesm_atm_climo_jja = os.path.join(config['cesm_atm_climos'], 'b.e10.BG20TRCN.f09_g16.002_JJA_climo.nc')
f_isccp = os.path.join(config['cloud_data'], 'ISCCP_ANN_climo.nc')
f_cloudsat = os.path.join(config['cloud_data'], 'CLOUDSAT_ANN_climo.nc')
# --------------------------------------------------------------
img_list = []
......@@ -190,12 +187,7 @@ def make_plot(config=None, out_path='.',
img_elem = el.image('CESM_cldlow_ANN',
' '.join(describe.split()),
img_link)
if config:
img_elem['Height'] = config['image_height']
img_elem['Height'] = config['image_height']
img_list.append(img_elem)
return img_list
if __name__ == '__main__':
make_plot()
......@@ -10,15 +10,12 @@ from livvkit.util import elements as el
describe = """CESM_cldtot_ANN plot."""
def make_plot(config=None, out_path='.',
cloud_path='/lustre/atlas1/cli115/world-shared/4ue/obs_data/',
cism_path='/lustre/atlas1/cli115/world-shared/4ue/',
cesm_path='/lustre/atlas1/cli115/world-shared/4ue/b.e10.BG20TRCN.f09_g16.002/'):
def make_plot(config, out_path='.'):
# ---------------- Data source in TITAN ------------------------
f_cism = os.path.join(cism_path, 'Greenland_5km_v1.1_SacksRev_c110629.nc')
f_cesm_atm_climo_jja = os.path.join(cesm_path, 'postproc/atm/climos/b.e10.BG20TRCN.f09_g16.002_JJA_climo.nc')
f_isccp = os.path.join(cloud_path, 'ISCCP_ANN_climo.nc')
f_cloudsat = os.path.join(cloud_path, 'CLOUDSAT_ANN_climo.nc')
f_cism = os.path.join(config['cism_data'], 'Greenland_5km_v1.1_SacksRev_c110629.nc')
f_cesm_atm_climo_jja = os.path.join(config['cesm_atm_climos'], 'b.e10.BG20TRCN.f09_g16.002_JJA_climo.nc')
f_isccp = os.path.join(config['cloud_data'], 'ISCCP_ANN_climo.nc')
f_cloudsat = os.path.join(config['cloud_data'], 'CLOUDSAT_ANN_climo.nc')
# --------------------------------------------------------------
img_list = []
......@@ -190,12 +187,7 @@ def make_plot(config=None, out_path='.',
img_elem = el.image('CESM_cldtot_ANN',
' '.join(describe.split()),
img_link)
if config:
img_elem['Height'] = config['image_height']
img_elem['Height'] = config['image_height']
img_list.append(img_elem)
return img_list
if __name__ == '__main__':
make_plot()
......@@ -10,9 +10,7 @@ from livvkit.util import elements as el
describe = """yearly_cycle_cldhgh plot."""
def make_plot(config=None, out_path='.',
cloud_path='/lustre/atlas1/cli115/world-shared/4ue/obs_data/',
cesm_path='/lustre/atlas1/cli115/world-shared/4ue/b.e10.BG20TRCN.f09_g16.002/'):
def make_plot(config, out_path='.'):
img_list = []
......@@ -24,18 +22,18 @@ def make_plot(config=None, out_path='.',
for month in months:
# CESM1
f_cesm = os.path.join(cesm_path, "postproc", "atm", "climos",
f_cesm = os.path.join(config['cesm_atm_climos'],
"b.e10.BG20TRCN.f09_g16.002_{:02d}_aavg_climo.nc".format(month))
ncid1 = Dataset(f_cesm, mode='r')
model_cld = ncid1.variables['CLDHGH'][0]
# CLDSAT
f_cloudsat = os.path.join(cloud_path, "CLOUDSAT_{:02d}_aavg_climo.nc".format(month))
f_cloudsat = os.path.join(config['cloud_data'], "CLOUDSAT_{:02d}_aavg_climo.nc".format(month))
ncid2 = Dataset(f_cloudsat, mode='r')
cldsat_cld = ncid2.variables['CLDHGH'][0]
# ISCCP
f_isccp = os.path.join(cloud_path, "ISCCP_{:02d}_aavg_climo.nc".format(month))
f_isccp = os.path.join(config['cloud_data'], "ISCCP_{:02d}_aavg_climo.nc".format(month))
ncid3 = Dataset(f_isccp, mode='r')
isccp_cld = ncid3.variables['CLDHGH'][0]
......@@ -74,8 +72,3 @@ def make_plot(config=None, out_path='.',
img_list.append(img_elem)
return img_list
if __name__ == '__main__':
plt.switch_backend('agg')
make_plot()
......@@ -10,9 +10,7 @@ from livvkit.util import elements as el
describe = """yearly_cycle_cldlow plot."""
def make_plot(config=None, out_path='.',
cloud_path='/lustre/atlas1/cli115/world-shared/4ue/obs_data/',
cesm_path='/lustre/atlas1/cli115/world-shared/4ue/b.e10.BG20TRCN.f09_g16.002/'):
def make_plot(config, out_path='.'):
img_list = []
......@@ -25,18 +23,18 @@ def make_plot(config=None, out_path='.',
for month in months:
# ---------------- Data source at OLCF ------------------------
# CESM1
input_file1 = os.path.join(cesm_path, "postproc", "atm", "climos",
input_file1 = os.path.join(config['cesm_atm_climos'],
"b.e10.BG20TRCN.f09_g16.002_{:02d}_aavg_climo.nc".format(month))
ncid1 = Dataset(input_file1, mode='r')
model_cld = ncid1.variables['CLDLOW'][0]
# CLDSAT
f_cloudsat = os.path.join(cloud_path, "CLOUDSAT_{:02d}_aavg_climo.nc".format(month))
f_cloudsat = os.path.join(config['cloud_data'], "CLOUDSAT_{:02d}_aavg_climo.nc".format(month))
ncid2 = Dataset(f_cloudsat, mode='r')
cldsat_cld = ncid2.variables['CLDLOW'][0]
# ISCCP
f_isccp = os.path.join(cloud_path, "ISCCP_{:02d}_aavg_climo.nc".format(month))
f_isccp = os.path.join(config['cloud_data'], "ISCCP_{:02d}_aavg_climo.nc".format(month))
ncid3 = Dataset(f_isccp, mode='r')
isccp_cld = ncid3.variables['CLDLOW'][0]
......@@ -75,8 +73,3 @@ def make_plot(config=None, out_path='.',
img_list.append(img_elem)
return img_list
if __name__ == '__main__':
plt.switch_backend('agg')
make_plot()
......@@ -10,9 +10,7 @@ from livvkit.util import elements as el
describe = """yearly_cycle_cldtot plot."""
def make_plot(config=None, out_path='.',
cloud_path='/lustre/atlas1/cli115/world-shared/4ue/obs_data/',
cesm_path='/lustre/atlas1/cli115/world-shared/4ue/b.e10.BG20TRCN.f09_g16.002/'):
def make_plot(config, out_path='.'):
img_list = []
......@@ -25,18 +23,18 @@ def make_plot(config=None, out_path='.',
for month in months:
# ---------------- Data source at OLCF ------------------------
# CESM1
f_cesm = os.path.join(cesm_path, "postproc", "atm", "climos",
f_cesm = os.path.join(config['cesm_atm_climos'],
"b.e10.BG20TRCN.f09_g16.002_{:02d}_aavg_climo.nc".format(month))
ncid1 = Dataset(f_cesm, mode='r')
model_cld = ncid1.variables['CLDTOT'][0]
# CLDSAT
f_cloudsat = os.path.join(cloud_path, "CLOUDSAT_{:02d}_aavg_climo.nc".format(month))
f_cloudsat = os.path.join(config['cloud_data'], "CLOUDSAT_{:02d}_aavg_climo.nc".format(month))
ncid2 = Dataset(f_cloudsat, mode='r')
cldsat_cld = ncid2.variables['CLDTOT'][0]
# ISCCP
f_isccp = os.path.join(cloud_path, "ISCCP_{:02d}_aavg_climo.nc".format(month))
f_isccp = os.path.join(config['cloud_data'], "ISCCP_{:02d}_aavg_climo.nc".format(month))
ncid3 = Dataset(f_isccp, mode='r')
isccp_cld = ncid3.variables['CLDTOT'][0]
......@@ -75,8 +73,3 @@ def make_plot(config=None, out_path='.',
img_list.append(img_elem)
return img_list
if __name__ == '__main__':
plt.switch_backend('agg')
make_plot()
......@@ -2,6 +2,9 @@
"clouds_cesm" : {
"module" : "clouds/clouds_cesm.py",
"references" : "clouds/clouds_cesm.bib",
"cesm_atm_climos": "data/cesm/atm",
"cism_data": "data/cism/glissade",
"cloud_data": "data/clouds",
"image_height" : 300
}
}
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
This source diff could not be displayed because it is stored in LFS. You can view the blob instead.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment