LEX-0.1-rc1

A first release candidate of the LIVVkit Extensions repository to
demonstrate our extensions functionality. It lincludes the CESM Greenland analyses for
cloud cover, energy balance, and surface mass balance, as well the
CISM-Albany dynamics analyses, which will be documented in the Evans et
al. (2018) LIVVkit validation paper.

This relase candidate:
  Provides basic documentation
  All required data to run on any machine

README.rst

 .. figure:: https://raw.githubusercontent.com/wiki/LIVVkit/LIVVkit/imgs/livvkit.png
-   :alt: 
+   :alt:
 
-LIVVkit Extensions
-==================
+LIVVkit Extensions (LEX)
+========================
 
-Note: this repository uses git's Large File Storage, you'll need to
-install git-lfs.
+This repository holds a collection of extension to`LIVVkit <https://livvkit.github.io/Docs/index.html>`_
+for validation and custom analyses of ice sheet models and their associated Earth
+system models.
 
-Warning: git-lfs doesn't seem to work with non-default remote names, so
-make sure you keep the default remote name ``origin``.
+LEX was first described in [Evans2018]_; to reproduce the analyses there, see the
+`Reproducing Evans et al. (2018)`_ section.
+
+.. [Evans2018] Evans, K.J., J.H. Kennedy, D. Lu, M.M. Forrester, S. Price, J. Fyke,
+   A.R. Bennett, M.J. Hoffman, I. Tezaur, C.S. Zender, and M. Vizcaino (In Review).
+   LIVVkit 2.1: Automated and extensible ice sheet model validation.
+   *Geoscientific Model Development.*
+
+Dependencies
+------------
+
+Because Validation analyses are particularly data intensive, this extensions repo
+uses `git-lfs <https://git-lfs.github.com>`_ (Git Large File Support) in order to
+distribute the required data. ``git-lfs`` can be installed either before or after
+cloning this repository, but it will be needed *before* downloading the required
+data. You can determine if you have ``git-lfs`` installed on your system by running
+this command in POSIX compliant shells (`bash`, `zsh`, `fish`, etc.):
+
+.. code:: bash
+
+    command -v git-lfs
+
+or this command in `csh` variants:
+
+.. code:: csh
+
+    which git-lfs
+
+If ``git-fls`` is not installed, you can install it by following the instructions here:
+
+https://git-lfs.github.com
+
+
+Likewise, you will need to have `LIVVkit installed <https://livvkit.github.io/Docs/index.html>`_
+to run these extensions. You can install it by running this command:
+
+.. code:: bash
+
+    pip install --user livvkit
+
+
+
+Basic Usage
+-----------
+
+First, install the required `Dependencies`_, then clone and enter this repository:
+
+.. code:: bash
+
+    git lfs clone https://code.ornl.gov/LIVVkit/lex.git
+    cd lex
+
+**Warning:** Because validation analyses are particularly data intensive, this
+repository is rather large (~ GBs currently), and uses git-lfs (git large
+file support) [#]_.
+
+Each extension will have an associated JSON configuration file which will describe
+the extension's analysis code, data locations, and options. To see a list of
+available extensions, you can run this command:
+
+.. code:: bash
+
+    find . -iname "*.json"
+
+To execute any of these extensions, point ``livv`` (LIVVkit's command line interface)
+to any of these extensions via the ``-e/--extension`` option (or the ``-V/--validate``).
+For example, to run the minimal example extension, place the output website in the
+`vv_test` directory, and serve the output website you'd run this command:
+
+.. code:: bash
+
+    livv -e example/example.json -o vv_test -s
+
+
+*Note:* All the extension configurations files assume you are working from the
+top level ``lex`` directory. You *can* run any of these extensions from any
+directory, but you will need to edit the paths in the JSON configuration files so
+that ``livv`` can find the required files.
+
+Likewise, you can also apply these analyses to any new model run [#]_ py adjusting
+the paths to point to your model run.
+
+.. [#] You may find `this tutorial by Atlassian useful <https://www.atlassian.com/git/tutorials/git-lfs>`_.
+
+.. [#] This assumes the new data files conform to the format of the included data
+   files. That is, an extension that analyses output from the CISM-Albany ice
+   sheet model will likely be able to analyze any similar CISM-Albany simulation,
+   but likely would *not* be able to analyze output from the PISM ice sheet
+   model without "massaging" the PISM files into a CISM-Albany like structure, or
+   adjusting the extension. *This is a problem we are working on for future LEX
+   releases.*
+
+
+Reproducing Evans et al. (2018)
+-------------------------------
+
+If all the required `Dependencies`_ are installed, and you've cloned the repository
+into the directory ``lex``, you can reproduce all the figures and tables in
+[Evans2018]_ by running this command from within the ``lex`` directory:
+
+.. code:: bash
+
+    livv --validate smb/smb_icecores.json \
+                    energy/energy_cesm.json \
+                    clouds/clouds_cesm.json \
+                    dynamics/dynamics_cisma.json \
+                    -o vv_evans2018 -s
+
+
+
+Developing a custom extension
+-----------------------------
+
+See the `LIVVkit documentation <https://livvkit.github.io/Docs/extend.html>`_ for
+details on how to develop an extension. Briefly, a absolute minimum working example
+is provided by the ``examples/`` extension, which should be copied to provide the
+basis for your new extension. All extensions are required to contain a minimal working
+example set of data such that they can be run an executed on any machine.
+
+For extensions that require data for which re-host permission cannot be granted,
+they must include documentation on how to acquire and use the data as well as either
+a small set of processed data or a set of "fake" example data.
+
+
+Issues, questions, comments, etc.?
+----------------------------------
+
+If you would like to suggest features, request tests, discuss contributions,
+report bugs, ask questions, or contact us for any reason, use the
+`LIVVkit issue tracker <https://github.com/LIVVkit/LIVVkit/issues>`_.
+
+Want to send us a private message?
+
+**Joseph H. Kennedy**
+:github: @jhkennedy
+:email: kennedyjh@ornl.gov
+
+**Katherine J. Evans**
+:github: @kevans32
+:email: evanskj@ornl.gov
+
+*Note: If you're emailing us, we recommend CC-ing all of us.*
\ No newline at end of file

clouds/clouds/cesm_cldhgh_ann.py

@@ -10,15 +10,12 @@ from livvkit.util import elements as el
 describe = """CESM_cldhgh_ANN plot."""
 
 
-def make_plot(config=None, out_path='.',
-              cloud_path='/lustre/atlas1/cli115/world-shared/4ue/obs_data/',
-              cism_path='/lustre/atlas1/cli115/world-shared/4ue/',
-              cesm_path='/lustre/atlas1/cli115/world-shared/4ue/b.e10.BG20TRCN.f09_g16.002/'):
+def make_plot(config, out_path='.'):
     # ---------------- Data source in TITAN ------------------------
-    f_cism = os.path.join(cism_path, 'Greenland_5km_v1.1_SacksRev_c110629.nc')
-    f_cesm_atm_climo_jja = os.path.join(cesm_path, 'postproc/atm/climos/b.e10.BG20TRCN.f09_g16.002_JJA_climo.nc')
-    f_isccp = os.path.join(cloud_path, 'ISCCP_ANN_climo.nc')
-    f_cloudsat = os.path.join(cloud_path, 'CLOUDSAT_ANN_climo.nc')
+    f_cism = os.path.join(config['cism_data'], 'Greenland_5km_v1.1_SacksRev_c110629.nc')
+    f_cesm_atm_climo_jja = os.path.join(config['cesm_atm_climos'], 'b.e10.BG20TRCN.f09_g16.002_JJA_climo.nc')
+    f_isccp = os.path.join(config['cloud_data'], 'ISCCP_ANN_climo.nc')
+    f_cloudsat = os.path.join(config['cloud_data'], 'CLOUDSAT_ANN_climo.nc')
     # --------------------------------------------------------------
 
     img_list = []
@@ -190,12 +187,7 @@ def make_plot(config=None, out_path='.',
     img_elem = el.image('CESM_cldhgh_ANN',
                         ' '.join(describe.split()),
                         img_link)
-    if config:
-        img_elem['Height'] = config['image_height']
+    img_elem['Height'] = config['image_height']
     img_list.append(img_elem)
 
     return img_list
-
-
-if __name__ == '__main__':
-    make_plot()

clouds/clouds/cesm_cldlow_ann.py

@@ -10,15 +10,12 @@ from livvkit.util import elements as el
 describe = """CESM_cldlow_ANN plot."""
 
 
-def make_plot(config=None, out_path='.',
-              cloud_path='/lustre/atlas1/cli115/world-shared/4ue/obs_data/',
-              cism_path='/lustre/atlas1/cli115/world-shared/4ue/',
-              cesm_path='/lustre/atlas1/cli115/world-shared/4ue/b.e10.BG20TRCN.f09_g16.002/'):
+def make_plot(config, out_path='.'):
     # ---------------- Data source in TITAN ------------------------
-    f_cism = os.path.join(cism_path, 'Greenland_5km_v1.1_SacksRev_c110629.nc')
-    f_cesm_atm_climo_jja = os.path.join(cesm_path, 'postproc/atm/climos/b.e10.BG20TRCN.f09_g16.002_JJA_climo.nc')
-    f_isccp = os.path.join(cloud_path, 'ISCCP_ANN_climo.nc')
-    f_cloudsat = os.path.join(cloud_path, 'CLOUDSAT_ANN_climo.nc')
+    f_cism = os.path.join(config['cism_data'], 'Greenland_5km_v1.1_SacksRev_c110629.nc')
+    f_cesm_atm_climo_jja = os.path.join(config['cesm_atm_climos'], 'b.e10.BG20TRCN.f09_g16.002_JJA_climo.nc')
+    f_isccp = os.path.join(config['cloud_data'], 'ISCCP_ANN_climo.nc')
+    f_cloudsat = os.path.join(config['cloud_data'], 'CLOUDSAT_ANN_climo.nc')
     # --------------------------------------------------------------
 
     img_list = []
@@ -190,12 +187,7 @@ def make_plot(config=None, out_path='.',
     img_elem = el.image('CESM_cldlow_ANN',
                         ' '.join(describe.split()),
                         img_link)
-    if config:
-        img_elem['Height'] = config['image_height']
+    img_elem['Height'] = config['image_height']
     img_list.append(img_elem)
 
     return img_list
-
-
-if __name__ == '__main__':
-    make_plot()

clouds/clouds/cesm_cldtot_ann.py

@@ -10,15 +10,12 @@ from livvkit.util import elements as el
 describe = """CESM_cldtot_ANN plot."""
 
 
-def make_plot(config=None, out_path='.',
-              cloud_path='/lustre/atlas1/cli115/world-shared/4ue/obs_data/',
-              cism_path='/lustre/atlas1/cli115/world-shared/4ue/',
-              cesm_path='/lustre/atlas1/cli115/world-shared/4ue/b.e10.BG20TRCN.f09_g16.002/'):
+def make_plot(config, out_path='.'):
     # ---------------- Data source in TITAN ------------------------
-    f_cism = os.path.join(cism_path, 'Greenland_5km_v1.1_SacksRev_c110629.nc')
-    f_cesm_atm_climo_jja = os.path.join(cesm_path, 'postproc/atm/climos/b.e10.BG20TRCN.f09_g16.002_JJA_climo.nc')
-    f_isccp = os.path.join(cloud_path, 'ISCCP_ANN_climo.nc')
-    f_cloudsat = os.path.join(cloud_path, 'CLOUDSAT_ANN_climo.nc')
+    f_cism = os.path.join(config['cism_data'], 'Greenland_5km_v1.1_SacksRev_c110629.nc')
+    f_cesm_atm_climo_jja = os.path.join(config['cesm_atm_climos'], 'b.e10.BG20TRCN.f09_g16.002_JJA_climo.nc')
+    f_isccp = os.path.join(config['cloud_data'], 'ISCCP_ANN_climo.nc')
+    f_cloudsat = os.path.join(config['cloud_data'], 'CLOUDSAT_ANN_climo.nc')
     # --------------------------------------------------------------
 
     img_list = []
@@ -190,12 +187,7 @@ def make_plot(config=None, out_path='.',
     img_elem = el.image('CESM_cldtot_ANN',
                         ' '.join(describe.split()),
                         img_link)
-    if config:
-        img_elem['Height'] = config['image_height']
+    img_elem['Height'] = config['image_height']
     img_list.append(img_elem)
 
     return img_list
-
-
-if __name__ == '__main__':
-    make_plot()

clouds/clouds/yearly_cycle_cldhgh.py

@@ -10,9 +10,7 @@ from livvkit.util import elements as el
 describe = """yearly_cycle_cldhgh plot."""
 
 
-def make_plot(config=None, out_path='.',
-              cloud_path='/lustre/atlas1/cli115/world-shared/4ue/obs_data/',
-              cesm_path='/lustre/atlas1/cli115/world-shared/4ue/b.e10.BG20TRCN.f09_g16.002/'):
+def make_plot(config, out_path='.'):
 
     img_list = []
 
@@ -24,18 +22,18 @@ def make_plot(config=None, out_path='.',
 
     for month in months:
         # CESM1
-        f_cesm = os.path.join(cesm_path, "postproc", "atm", "climos",
+        f_cesm = os.path.join(config['cesm_atm_climos'],
                               "b.e10.BG20TRCN.f09_g16.002_{:02d}_aavg_climo.nc".format(month))
         ncid1 = Dataset(f_cesm, mode='r')
         model_cld = ncid1.variables['CLDHGH'][0]
 
         # CLDSAT
-        f_cloudsat = os.path.join(cloud_path, "CLOUDSAT_{:02d}_aavg_climo.nc".format(month))
+        f_cloudsat = os.path.join(config['cloud_data'], "CLOUDSAT_{:02d}_aavg_climo.nc".format(month))
         ncid2 = Dataset(f_cloudsat, mode='r')
         cldsat_cld = ncid2.variables['CLDHGH'][0]
 
         # ISCCP
-        f_isccp = os.path.join(cloud_path, "ISCCP_{:02d}_aavg_climo.nc".format(month))
+        f_isccp = os.path.join(config['cloud_data'], "ISCCP_{:02d}_aavg_climo.nc".format(month))
         ncid3 = Dataset(f_isccp, mode='r')
         isccp_cld = ncid3.variables['CLDHGH'][0]
 
@@ -74,8 +72,3 @@ def make_plot(config=None, out_path='.',
     img_list.append(img_elem)
 
     return img_list
-
-
-if __name__ == '__main__':
-    plt.switch_backend('agg')
-    make_plot()

clouds/clouds/yearly_cycle_cldlow.py

@@ -10,9 +10,7 @@ from livvkit.util import elements as el
 describe = """yearly_cycle_cldlow plot."""
 
 
-def make_plot(config=None, out_path='.',
-              cloud_path='/lustre/atlas1/cli115/world-shared/4ue/obs_data/',
-              cesm_path='/lustre/atlas1/cli115/world-shared/4ue/b.e10.BG20TRCN.f09_g16.002/'):
+def make_plot(config, out_path='.'):
 
     img_list = []
 
@@ -25,18 +23,18 @@ def make_plot(config=None, out_path='.',
     for month in months:
         # ---------------- Data source at OLCF ------------------------
         # CESM1
-        input_file1 = os.path.join(cesm_path, "postproc", "atm", "climos",
+        input_file1 = os.path.join(config['cesm_atm_climos'],
                                    "b.e10.BG20TRCN.f09_g16.002_{:02d}_aavg_climo.nc".format(month))
         ncid1 = Dataset(input_file1, mode='r')
         model_cld = ncid1.variables['CLDLOW'][0]
 
         # CLDSAT
-        f_cloudsat = os.path.join(cloud_path, "CLOUDSAT_{:02d}_aavg_climo.nc".format(month))
+        f_cloudsat = os.path.join(config['cloud_data'], "CLOUDSAT_{:02d}_aavg_climo.nc".format(month))
         ncid2 = Dataset(f_cloudsat, mode='r')
         cldsat_cld = ncid2.variables['CLDLOW'][0]
 
         # ISCCP
-        f_isccp = os.path.join(cloud_path, "ISCCP_{:02d}_aavg_climo.nc".format(month))
+        f_isccp = os.path.join(config['cloud_data'], "ISCCP_{:02d}_aavg_climo.nc".format(month))
         ncid3 = Dataset(f_isccp, mode='r')
         isccp_cld = ncid3.variables['CLDLOW'][0]
 
@@ -75,8 +73,3 @@ def make_plot(config=None, out_path='.',
     img_list.append(img_elem)
 
     return img_list
-
-
-if __name__ == '__main__':
-    plt.switch_backend('agg')
-    make_plot()

clouds/clouds/yearly_cycle_cldtot.py

@@ -10,9 +10,7 @@ from livvkit.util import elements as el
 describe = """yearly_cycle_cldtot plot."""
 
 
-def make_plot(config=None, out_path='.',
-              cloud_path='/lustre/atlas1/cli115/world-shared/4ue/obs_data/',
-              cesm_path='/lustre/atlas1/cli115/world-shared/4ue/b.e10.BG20TRCN.f09_g16.002/'):
+def make_plot(config, out_path='.'):
 
     img_list = []
 
@@ -25,18 +23,18 @@ def make_plot(config=None, out_path='.',
     for month in months:
         # ---------------- Data source at OLCF ------------------------
         # CESM1
-        f_cesm = os.path.join(cesm_path, "postproc", "atm", "climos",
+        f_cesm = os.path.join(config['cesm_atm_climos'],
                               "b.e10.BG20TRCN.f09_g16.002_{:02d}_aavg_climo.nc".format(month))
         ncid1 = Dataset(f_cesm, mode='r')
         model_cld = ncid1.variables['CLDTOT'][0]
 
         # CLDSAT
-        f_cloudsat = os.path.join(cloud_path, "CLOUDSAT_{:02d}_aavg_climo.nc".format(month))
+        f_cloudsat = os.path.join(config['cloud_data'], "CLOUDSAT_{:02d}_aavg_climo.nc".format(month))
         ncid2 = Dataset(f_cloudsat, mode='r')
         cldsat_cld = ncid2.variables['CLDTOT'][0]
 
         # ISCCP
-        f_isccp = os.path.join(cloud_path, "ISCCP_{:02d}_aavg_climo.nc".format(month))
+        f_isccp = os.path.join(config['cloud_data'], "ISCCP_{:02d}_aavg_climo.nc".format(month))
         ncid3 = Dataset(f_isccp, mode='r')
         isccp_cld = ncid3.variables['CLDTOT'][0]
 
@@ -75,8 +73,3 @@ def make_plot(config=None, out_path='.',
     img_list.append(img_elem)
 
     return img_list
-
-
-if __name__ == '__main__':
-    plt.switch_backend('agg')
-    make_plot()

clouds/clouds_cesm.json

@@ -2,6 +2,9 @@
     "clouds_cesm" : {
         "module" : "clouds/clouds_cesm.py",
         "references" : "clouds/clouds_cesm.bib",
+        "cesm_atm_climos": "data/cesm/atm",
+        "cism_data": "data/cism/glissade",
+        "cloud_data": "data/clouds",
         "image_height" : 300
     }
 }