add some tips

# Sphinx build info version 1

# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.

config: 5181bd2b8280e663a350317fc93a53af

config: 4d3597b4651c71301cf359fad5392948

tags: 645f666f9bcd5a90fca523b33c5a78b7

*  Docker version 17.09.0-ce, build afdb6d4

Docker container managment

==========================

List the running containers::

  docker ps

List all the containers (including the stopped ones)::

  docker ps -a

Remove all the container (running or not)::

  docker ps -aq | xargs docker rm --force

Inspect/connect to a running container (ex: 67e4c31da0ee)::

  # Get the container ID from the docker ps command.

  docker exec -it 67e4c31da0ee bash

Docker volume managment

=======================

List the volumes::

  docker volume ls

Remove dangling volumes of container::

  docker volume ls -f "dangling=true" | xargs docker volume rm 

Docker image managment

======================

List the images::

  docker image ls # or docker images

Remove dangling images::

  docker image ls -f "dangling=true" | xargs docker image rm 

  docker image ls -q | xargs docker image rm --force

Remove image according to a given pattern (e.g. the esgfhub devel images)::

Remove image according to a given pattern (ex: the esgfhub devel images)::

  docker images "esgfhub/*:devel" -q | xargs docker image rm

Remove imges tagged "<none>"::

Remove images tagged "<none>"::

  docker image ls | grep "^<none>" | awk '{print $3}' | xargs docker image rm --force

Inspect the contains of an image (esgfhub/esgf-http:devel image)::

Inspect/connect to an image (ex: esgfhub/esgf-http:devel image)::

  docker run --rm -it --entrypoint=/bin/bash esgfhub/esgf-http:devel

 No newline at end of file

***************

esgf-cog Image

***************

This document contains miscellaneous details on how the *esgf-cog* Docker container is configured and operated.

In the following, the environment variable *ESGF_VERSION* must be set to the desired image version, for example::

  export ESGF_VERSION=latest

Starting the container

======================

To pre-download the CoG image::

  docker pull esgfhub/esgf-cog:${ESGF_VERSION}

To start the container standalone::

  docker run -ti -p 8000:8000 --name cog esgfhub/esgf-cog:${ESGF_VERSION}

* will run CoG through the Django development server, running on port 8000

* will use hostname=localhost

* will skip ESGF setup, i.e. it will use a sqllite database (not the ESGF postgres database), and will not install any ESGF configuration files

* CoG will be accessible at the URL: http://localhost:8000/

* will need to login at the URL: http://localhost:8000/login2/ with the default username ("rootAdmin") and password ("changeit")

To start the container standalone, but using a specific hostname::

  docker run -ti -p 8000:8000 --name cog -e ESGF_HOSTNAME=${ESGF_HOSTNAME} esgfhub/esgf-cog:${ESGF_VERSION} ${ESGF_HOSTNAME} false true

* CoG will be accessible at the URL: http://${ESG_HOSTNAME}:8000/

To start the container standalone, but using source code from a local directory, for development purposes::

  docker run -ti -p 8000:8000 --name cog -v ${COG_INSTALL_DIR}:/usr/local/cog/cog_install esgfhub/esgf-cog:${ESGF_VERSION}

* for example ${COG_INSTALL_DIR}=/Users/cinquini/Documents/workspace/cog

* will again use hostname=localhost

* will again run CoG through the Django server

To run CoG as part of the ESGF software stack, with postgres database and Apache front-end::

  # initialize the $ESGF_CONFIG directory containing all ESGF node configuration

  cd .../esgf-docker

  scripts/esgf_node_init.sh

  # start CoG and its dependency containers

  docker-compose up esgf-cog esgf-postgres esgf-httpd

* will run CoG within the Apache httpd server (and container) through mod_wsgi

* the cog container is kept running as a data container, exposing the CoG source directory *COG_INSTALL_DIR=/usr/local/cog/cog_install* to the httpd container

After the first startup, the containers can be restarted without re-initializing them as such::

  export INIT=false

  docker-compose up -d esgf-postgres

  docker-compose up esgf-cog esgf-httpd

To run the full ESGF software stack::

  cd .../esgf-docker

  docker-compose up

To start the container standalone::

  docker run -ti -p 8983:8983 -p 8984:8984 --name esgf-solr esgfhub/esgf-solr:${ESGF_VERSION}

To start the container using an existing pre-populated index::

  docker run -ti -p 8983:8983 -p 8984:8984 --name esgf-solr -v ${SOLR_INDEX_DIR}:/esg/solr-index  esgfhub/esgf-solr:${ESGF_VERSION}

where the *SOLR_INDEX_DIR* directory must contain the master and slave indexes::

  ls -l $SOLR_INDEX_DIR

  master-8984

  slave-8983

To start the container using docker-compose::

  docker-compose -f docker-stack.yml up esgf-solr

or you can start the container as part of the whole ESGF stack::

To start the container as part of the whole ESGF stack::

  docker stack -c docker-stack.yml esgf-stack

   supervisorctl update

The instructions above will revert the actions taken when the shard was created.

Other Admin Commands

====================

To optimize an index (i.e. reduce the number of segments to 1, which optimnizes query performance)::

   http://localhost:8984/solr/datasets/update?optimize=true

Repeat for all cores (datasets, files, aggregations) for the *master* and *local* shards only (i.e. do not

execute for the *slave* and *replica* shards).

To migrate the Lucene index to the latest version::

  cd /usr/local/solr/server/solr-webapp/webapp/WEB-INF/lib

  java -cp lucene-core-<version>.jar:lucene-backward-codecs-<version>.jar org.apache.lucene.index.IndexUpgrader -verbose /esg/solr-index/master-8984/datasets/index/

Again repeat for all cores (datasets, files, aggregations) for the *master* and *local* shards only, not for the *slave* or *replicas*.

   docker_swarm_setup_on_linux.rst

   versioning_guideline.rst

   git_workflow.rst

   esgf_cog.rst

   esgf_solr.rst

   testing_guide.rst

   docker_tips.rst