Commit 88f40f13 authored by Luca Cinquini's avatar Luca Cinquini
Browse files

Updating publishing instructions for a modularized data node.

parent 06051f28

_sources/data_publishing.txt

+15 −12
Original line number Diff line number Diff line
@@ -2,18 +2,20 @@ 
Data Publishing

***************



*Tested with ESGF_VERSION=1.1*

*Tested with ESGF_VERSION=dev_1.4*



Description

===========



This page instructs on how to use the esgf-publisher Python client to

publish a sample dataset and file to the ESGF data node running as a

Docker container. These instructions are meant to be executed inside the

data node container::

publish a sample dataset and file to the ESGF **tds** and **index** node

containers. These publishing instructions are meant to be executed inside the

**publisher** container. Additionally, the **postgres** contrainer is needed

to store publishing metadata, and the **orp** container is needed to enforce access

control (for publishing and downloading data)::



  docker-compose up 

  docker exec -it -u 0 data-node /bin/bash

  docker exec -it -u 0 publisher /bin/bash



One-Time Setup

==============
@@ -50,14 +52,14 @@ Generate the mapfile listing the dataset and files to be published:: 
  cd /esg/config/esgcet

  source ${CDAT_HOME}/bin/activate esgf-pub 

  esgprep mapfile --project test /esg/data/test

  ls -l test.test.map

  ls -l mapfiles/test.test.map



Step 2

======



Publish to the postgres database::

  

  esgpublish --project test --map test.test.map --service fileservice

  esgpublish --project test --map mapfiles/test.test.map --service fileservice

  esglist_datasets test



Step 3
@@ -65,7 +67,7 @@ Step 3 


Publish to the TDS::



  esgpublish --project test --map test.test.map --service fileservice --noscan --thredds

  esgpublish --project test --map mapfiles/test.test.map --service fileservice --noscan --thredds



Note: this operation will use the credentials contained in the *esg.ini*

file to invoke the TDS re-initialization URL:
@@ -79,7 +81,7 @@ from the TDS main catalog page: 


and downloadable using any openid, password combination that is trusted

by the data-node. The authorization required for downloading the file is

specified in the access policy file: */esg/config/esgf_policies_local.xml*

specified inside the **orp** container in the access policy file: */esg/config/esgf_policies_local.xml*

as an XML statement of the form::



   <policy resource=".*test.*" attribute_type="AUTH_ONLY" attribute_value="" action="Read"/>
@@ -95,20 +97,21 @@ the identity provider container. In the meantime, the following 
workarounds can be adopted.



Obtain a short-term X509 certificate from any other trusted ESGF

identity provider, and copy it into the data-node container in the location

identity provider, and copy it into the **publisher** container in the location

referenced by the file esg.ini::



  cp certificate-file /root/.globus/certificate-file



Then, disable specific authorization for publishing test data, requiring only

Then, disable the specific authorization for publishing test data, requiring only

the availability of an X509 certificate. Edit the file: */esg/config/esgf_policies_local.xml*

inside the **orp** container

and insert the following policy statement (as XML)::



  <policy resource=".*test.*" attribute_type="ANY" attribute_value="" action="Write"/>



At this point, you can issue the publishing command::



  esgpublish --project test --map test.test.map --service fileservice --noscan --publish

  esgpublish --project test --map mapfiles/test.test.map --service fileservice --noscan --publish



After about a minute, the dataset and file should be returned when

querying the "slave" Solr index:

data_publishing.html

+15 −12
Original line number Diff line number Diff line
@@ -51,15 +51,17 @@ 
            

  <div class="section" id="data-publishing">

<h1>Data Publishing<a class="headerlink" href="#data-publishing" title="Permalink to this headline"></a></h1>

<p><em>Tested with ESGF_VERSION=1.1</em></p>

<p><em>Tested with ESGF_VERSION=dev_1.4</em></p>

<div class="section" id="description">

<h2>Description<a class="headerlink" href="#description" title="Permalink to this headline"></a></h2>

<p>This page instructs on how to use the esgf-publisher Python client to

publish a sample dataset and file to the ESGF data node running as a

Docker container. These instructions are meant to be executed inside the

data node container:</p>

publish a sample dataset and file to the ESGF <strong>tds</strong> and <strong>index</strong> node

containers. These publishing instructions are meant to be executed inside the

<strong>publisher</strong> container. Additionally, the <strong>postgres</strong> contrainer is needed

to store publishing metadata, and the <strong>orp</strong> container is needed to enforce access

control (for publishing and downloading data):</p>

<div class="highlight-python"><div class="highlight"><pre>docker-compose up

docker exec -it -u 0 data-node /bin/bash

docker exec -it -u 0 publisher /bin/bash

</pre></div>

</div>

</div>
@@ -95,14 +97,14 @@ wget -O sftlf.nc http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/externals/s 
<div class="highlight-python"><div class="highlight"><pre>cd /esg/config/esgcet

source ${CDAT_HOME}/bin/activate esgf-pub

esgprep mapfile --project test /esg/data/test

ls -l test.test.map

ls -l mapfiles/test.test.map

</pre></div>

</div>

</div>

<div class="section" id="step-2">

<h2>Step 2<a class="headerlink" href="#step-2" title="Permalink to this headline"></a></h2>

<p>Publish to the postgres database:</p>

<div class="highlight-python"><div class="highlight"><pre>esgpublish --project test --map test.test.map --service fileservice

<div class="highlight-python"><div class="highlight"><pre>esgpublish --project test --map mapfiles/test.test.map --service fileservice

esglist_datasets test

</pre></div>

</div>
@@ -110,7 +112,7 @@ esglist_datasets test 
<div class="section" id="step-3">

<h2>Step 3<a class="headerlink" href="#step-3" title="Permalink to this headline"></a></h2>

<p>Publish to the TDS:</p>

<div class="highlight-python"><div class="highlight"><pre>esgpublish --project test --map test.test.map --service fileservice --noscan --thredds

<div class="highlight-python"><div class="highlight"><pre>esgpublish --project test --map mapfiles/test.test.map --service fileservice --noscan --thredds

</pre></div>

</div>

<p>Note: this operation will use the credentials contained in the <em>esg.ini</em>
@@ -125,7 +127,7 @@ from the TDS main catalog page:</p> 
</ul>

<p>and downloadable using any openid, password combination that is trusted

by the data-node. The authorization required for downloading the file is

specified in the access policy file: <em>/esg/config/esgf_policies_local.xml</em>

specified inside the <strong>orp</strong> container in the access policy file: <em>/esg/config/esgf_policies_local.xml</em>

as an XML statement of the form:</p>

<div class="highlight-python"><div class="highlight"><pre>&lt;policy resource=&quot;.*test.*&quot; attribute_type=&quot;AUTH_ONLY&quot; attribute_value=&quot;&quot; action=&quot;Read&quot;/&gt;

</pre></div>
@@ -139,19 +141,20 @@ that is able to issue short term X509 certificates for users managed by 
the identity provider container. In the meantime, the following

workarounds can be adopted.</p>

<p>Obtain a short-term X509 certificate from any other trusted ESGF

identity provider, and copy it into the data-node container in the location

identity provider, and copy it into the <strong>publisher</strong> container in the location

referenced by the file esg.ini:</p>

<div class="highlight-python"><div class="highlight"><pre>cp certificate-file /root/.globus/certificate-file

</pre></div>

</div>

<p>Then, disable specific authorization for publishing test data, requiring only

<p>Then, disable the specific authorization for publishing test data, requiring only

the availability of an X509 certificate. Edit the file: <em>/esg/config/esgf_policies_local.xml</em>

inside the <strong>orp</strong> container

and insert the following policy statement (as XML):</p>

<div class="highlight-python"><div class="highlight"><pre>&lt;policy resource=&quot;.*test.*&quot; attribute_type=&quot;ANY&quot; attribute_value=&quot;&quot; action=&quot;Write&quot;/&gt;

</pre></div>

</div>

<p>At this point, you can issue the publishing command:</p>

<div class="highlight-python"><div class="highlight"><pre>esgpublish --project test --map test.test.map --service fileservice --noscan --publish

<div class="highlight-python"><div class="highlight"><pre>esgpublish --project test --map mapfiles/test.test.map --service fileservice --noscan --publish

</pre></div>

</div>

<p>After about a minute, the dataset and file should be returned when

doctrees/data_publishing.doctree

+2.05 KiB (21 KiB)

File changed.

No diff preview for this file type.

View original file
View changed file

doctrees/environment.pickle

+120 B (33.8 KiB)

File changed.

No diff preview for this file type.

View original file
View changed file

searchindex.js

+1 −1

File changed.

Preview size limit exceeded, changes collapsed.