Loading docs/_docs/developer/architecture.md +16 −14 Original line number Diff line number Diff line Loading @@ -8,22 +8,24 @@ toc: false The following text-diagram shows the inheritance hierarchy of the ESGF Docker containers: ``` centos:6 -> esgf-postgres centos/postgresql-96-centos7 -> esgfhub/postgres -> esgfhub/postgres-esgcet nginx -> esgf-proxy nginx -> esgfhub/proxy solr:5.5 -> esgf-solr solr:5.5 -> esgfhub/solr alpine -> esgf-configure alpine -> esgfhub/configure python:2.7-slim -> esgf-django -> esgf-auth -> esgf-cog -> esgf-slcs python:2.7-slim -> esgfhub/django -> esgfhub/auth -> esgfhub/cog -> esgfhub/slcs openjdk:8-jre -> tomcat:8 -> esgf-tomcat -> esgf-idp-node -> esgf-index-node -> esgf-orp -> esgf-tds openjdk:8-jre -> tomcat:8 -> esgfhub/tomcat -> esgfhub/idp-node -> esgfhub/index-node -> esgfhub/orp -> esgfhub/tds continuumio/miniconda -> esgfhub/publisher ``` ## Configuration Loading Loading @@ -59,8 +61,8 @@ allows us to install the minimum amount of software required to run an applicati in the final image, reducing container bloat and potential attack surface. A build image can include dependencies that are only required when building that are not required in the final image - for example, the build image for the [esgf-slcs](https://github.com/cedadev/esgf-docker/blob/master/slcs/Dockerfile) that are not required in the final image - for example, the build image for [esgfhub/slcs](https://github.com/ESGF/esgf-docker/blob/master/slcs/Dockerfile) installs `build-essential` (for GCC) and the header files for OpenSSL, neither of which are required to actually run the application. The resulting [Python wheels](https://pythonwheels.com/) are copied into the application image, Loading @@ -73,6 +75,6 @@ as a way to share files with other images using the `COPY --from=<image>` syntax This is mainly to work-around the fact that Docker does not allow symlinks in a build context. In particular, the `esgf-configure` container is a mixin container, that exists In particular, the `esgfhub/configure` image is a mixin image that exists purely to allow other images to `COPY` the default templates for `/esg/config` and some useful scripts. docs/_docs/developer/comparison.md +7 −5 Original line number Diff line number Diff line Loading @@ -48,9 +48,7 @@ as it means: With this implementation of ESGF Docker, each Tomcat and Django application runs entirely in its own container. The `esgf-httpd` container has been replaced with a pure-HTTP proxy using Nginx. When deployed using Kubernetes, the proxy will not be used at all, with each application managing its own [Ingress resources](https://kubernetes.io/docs/concepts/services-networking/ingress/). a pure-HTTP proxy using Nginx. ## Container dependencies Loading @@ -63,12 +61,16 @@ source installations, as it can be error-prone. Instead, this implementation takes the decision to use officially supported and maintained base images where possible. This means that it now uses the following images from the Docker Hub Library as the basis for `esgf-*` containers: images from the Docker Hub Library as the basis for `esgfhub/*` containers: * `centos/postgresql-96-centos7` * `nginx` * `solr:5.5` * `alpine` * `python:2.7-slim` * `openjdk:8-{jre,jdk}` * `tomcat:8` * `python:2.7-slim` * `continuumio/miniconda` In the original implementation, there was also an `esgf-node` image that served as a base image for all other ESGF containers. As well as containing supervisor, Loading docs/_docs/developer/contributing.md 0 → 100644 +117 −0 Original line number Diff line number Diff line --- title: Contributing category: Developers order: 2.3 toc: false --- Contribution to `esgf-docker` happens via the GitHub repository - [https://github.com/ESGF/esgf-docker](https://github.com/ESGF/esgf-docker) - following a feature-branch-style workflow. ## Protected branches The `esgf-docker` repository has two protected branches: * `master` contains only code that has been tested. The intention is that any commit to `master` could be deployed in production. This is the branch against which releases are tagged. * `devel` contains the latest development code. The intention is that each commit to `devel` works, but is not as well tested as `master`. These branches are only committed to using pull requests. [Travis](https://travis-ci.org/ESGF/esgf-docker) is configured to automatically build pull requests when they are submitted, and a pull request should not be merged without a successful build. Travis is also configured to automatically build, tag and push each commit to the `master` and `devel` branches, using the `latest` and `devel` Docker tags respectively. Each commit is also given a unique Docker tag that combines the closest tag and the commit hash, allowing deployments to target a specific commit. ## Making a contribution These steps describe the process to follow when making a contribution. They assume that you do **not** have write access to the main `esgf-docker` repository. Those with write access to the main `esgf-docker` repository are free to work in branches in that repository if they wish. * Unless you are working on an existing issue, create a Github issue [in the main esgf-docker repository](https://github.com/ESGF/esgf-docker/issues) describing the change you intend to make. * If this is your first contribution, create a GitHub fork of the main `esgf-docker` repository in your own workspace and check it out to your development machine. ``` git clone git@github.com:<github username>/esgf-docker ``` * Ensure you have pulled the latest changes from the `devel` branch. ``` git checkout devel git pull upstream ``` * Create a new branch from `devel`. The preferred format for the branch name is `issue/<issue number>/<descriptive slug>` - for example, the branch that created this page is called `issue/46/contributing-docs`. ``` git checkout -b <branch name> ``` * Make your changes in this branch, referencing the issue from above using the `ESGF/esgf-docker#<issue number>` syntax in commit messages. ``` git add [-A | list of files] git commit -m "ESGF/esgf-docker#<issue number> <commit message>" ``` * Make sure all your changes are pushed to your GitHub fork. ``` git push ``` * Once you have completed and tested your changes, submit a pull request from the branch in your GitHub fork targeting the `devel` branch in the main `esgf-docker` repository. * Iterate with the reviewer until your pull request is accepted and merged to `devel`. If `devel` has changed since you created your branch, this may involve [rebasing](https://git-scm.com/docs/git-rebase) to the current `HEAD` of `devel` and fixing any merge conflicts. ## Committing to master The following describe the process of committing to the `master` branch. This will generally only happen for a release. The steps assume you have write access to the main `esgf-docker` repository. Releases should have the format `{major}.{minor}.{patch}[.{alpha|beta}.{increment}]`, e.g. `2.0.1`, `2.1.0.alpha.0` or `3.0.0.beta.2`. * Create an integration branch from the current `devel`. The preferred format for naming of integration branches is `integration/<release>`. ``` git checkout devel git pull git checkout -b integration/<release> ``` * [Rebase](https://git-scm.com/docs/git-rebase) the integration branch onto `master` and fix any merge conflicts. ``` git rebase master ``` * Perform the required testing for a release, making any required fixes. Once you are satisfied, submit a pull request targeting `master`. * Merge the pull request onto `master`. * Tag `master` with the release. ``` git checkout master git pull git tag <release> git push --tags ``` * Rebase `devel` to the current `HEAD` of `master`. If `devel` has changed since the integration branch was created, this may involve fixing any merge conflicts. ``` git checkout master git pull git checkout devel git pull git rebase master git push ``` docs/_docs/usage/configuration.md +12 −26 Original line number Diff line number Diff line Loading @@ -13,12 +13,8 @@ be generated using the scripts described in this page. * `$ESGF_CONFIG` * `certificates` * `esg-hostcert-bundle.p12` PKCS12-encoded bundle containing host certificate and private key for SAML signing * `esg-trust-bundle.jks` JKS-encoded bundle containing trusted CA certificates for Java apps * `esg-trust-bundle.pem` PEM-encoded bundle containing trusted CA certificates for Python apps PEM-encoded bundle containing trusted CA certificates * `hostcert` * `hostcert.crt` PEM-encoded host certificate, including all intermediate CA certificates, used for SSL Loading @@ -40,10 +36,6 @@ be generated using the scripts described in this page. Password for dbsuper user of main ESGF database * `database-publisher-password` Password for esgcet user of main ESGF database * `java-hostcert-bundle-password` Password for PKCS12-encoded host certificate bundle * `java-trust-bundle-password` Password for JKS-encoded trust bundle * `rootadmin-password` Password for rootAdmin node admin account * `shared-cookie-secret-key` Loading @@ -61,7 +53,7 @@ described above can be automatically generated. For a production installation, valid certificates signed by a trusted CA should be obtained for the host certificate and SLCS CA, and placed into the directory structure at the correct location. The scripts for generating configuration have been bundled up into the `cedadev/esgf-setup` The scripts for generating configuration have been bundled up into the `esgfhub/setup` container image, meaning that the only requirement on the host system is Docker. ### Generating secrets Loading @@ -70,9 +62,9 @@ To generate random secrets for all the passwords and secret keys required for an ESGF Docker installation, just run the following commands: ```sh $ export ESGF_HOSTNAME=local.esgf.org $ export ESGF_CONFIG=/path/to/empty/config/directory $ docker run -v "$ESGF_CONFIG":/esg -e ESGF_HOSTNAME cedadev/esgf-setup generate-secrets export ESGF_HOSTNAME=local.esgf.org export ESGF_CONFIG=/path/to/empty/config/directory docker-compose run esgf-setup generate-secrets ``` This will generate the contents of the `secrets` directory from the above structure. Loading @@ -90,7 +82,7 @@ To generate self-signed certificates for the host and SLCS CA, just run this command: ```sh $ docker run -v "$ESGF_CONFIG":/esg -e ESGF_HOSTNAME cedadev/esgf-setup generate-test-certificates docker-compose run esgf-setup generate-test-certificates ``` This will generate the `certificates/hostcert` and `certificates/slcsca` directories Loading @@ -99,22 +91,16 @@ created manually. ### Creating trust bundles The three trust bundle files, `esg-hostcert-bundle.p12`, `esg-trust-bundle.jks` and `esg-trust-bundle.pem`, are automatically generated from existing certificates. `esg-hostcert-bundle.p12` is obviously generated from the host certificate and private key in `certificates/hostcert`. The two trust bundles contain the exact same certificates, just in different formats. First, the certificates from `certificates/esg_trusted_certificates.tar` are included, if it exists. This tar file can be downloaded from an ESGF distribution The trust bundle `esg-trust-bundle.pem` is automatically generated from existing certificates. First, the certificates from `certificates/esg_trusted_certificates.tar` are included, if it exists. This tar file can be downloaded from an ESGF distribution site, and should contain a single directory called `esg_trusted_certificates` containing all the trusted CAs for the federation. If the SLCS CA is self-signed, it is also added to the trust bundles so that the installation can trust its own it is also added to the trust bundle so that the installation can trust its own certificates. The command to create the trust bundles is: The command to create the trust bundle is: ```sh $ docker run -v "$ESGF_CONFIG":/esg -e ESGF_HOSTNAME cedadev/esgf-setup create-trust-bundles docker-compose run esgf-setup create-trust-bundle ``` docs/_docs/usage/quick-start.md +2 −2 Original line number Diff line number Diff line Loading @@ -17,7 +17,7 @@ on the host system. First, clone the repository: ```sh git clone https://github.com/cedadev/esgf-docker.git git clone https://github.com/ESGF/esgf-docker.git cd esgf-docker ``` Loading Loading @@ -55,7 +55,7 @@ This means you need to make sure that the `ESGF_CONFIG` directory is in your home directory. </div> ## Pull the container images from Docker Hub ## Pull the container images from Docker Hub You only need to pull the images from Docker Hub when they have changed, or if you are deploying for the first time: Loading Loading
docs/_docs/developer/architecture.md +16 −14 Original line number Diff line number Diff line Loading @@ -8,22 +8,24 @@ toc: false The following text-diagram shows the inheritance hierarchy of the ESGF Docker containers: ``` centos:6 -> esgf-postgres centos/postgresql-96-centos7 -> esgfhub/postgres -> esgfhub/postgres-esgcet nginx -> esgf-proxy nginx -> esgfhub/proxy solr:5.5 -> esgf-solr solr:5.5 -> esgfhub/solr alpine -> esgf-configure alpine -> esgfhub/configure python:2.7-slim -> esgf-django -> esgf-auth -> esgf-cog -> esgf-slcs python:2.7-slim -> esgfhub/django -> esgfhub/auth -> esgfhub/cog -> esgfhub/slcs openjdk:8-jre -> tomcat:8 -> esgf-tomcat -> esgf-idp-node -> esgf-index-node -> esgf-orp -> esgf-tds openjdk:8-jre -> tomcat:8 -> esgfhub/tomcat -> esgfhub/idp-node -> esgfhub/index-node -> esgfhub/orp -> esgfhub/tds continuumio/miniconda -> esgfhub/publisher ``` ## Configuration Loading Loading @@ -59,8 +61,8 @@ allows us to install the minimum amount of software required to run an applicati in the final image, reducing container bloat and potential attack surface. A build image can include dependencies that are only required when building that are not required in the final image - for example, the build image for the [esgf-slcs](https://github.com/cedadev/esgf-docker/blob/master/slcs/Dockerfile) that are not required in the final image - for example, the build image for [esgfhub/slcs](https://github.com/ESGF/esgf-docker/blob/master/slcs/Dockerfile) installs `build-essential` (for GCC) and the header files for OpenSSL, neither of which are required to actually run the application. The resulting [Python wheels](https://pythonwheels.com/) are copied into the application image, Loading @@ -73,6 +75,6 @@ as a way to share files with other images using the `COPY --from=<image>` syntax This is mainly to work-around the fact that Docker does not allow symlinks in a build context. In particular, the `esgf-configure` container is a mixin container, that exists In particular, the `esgfhub/configure` image is a mixin image that exists purely to allow other images to `COPY` the default templates for `/esg/config` and some useful scripts.
docs/_docs/developer/comparison.md +7 −5 Original line number Diff line number Diff line Loading @@ -48,9 +48,7 @@ as it means: With this implementation of ESGF Docker, each Tomcat and Django application runs entirely in its own container. The `esgf-httpd` container has been replaced with a pure-HTTP proxy using Nginx. When deployed using Kubernetes, the proxy will not be used at all, with each application managing its own [Ingress resources](https://kubernetes.io/docs/concepts/services-networking/ingress/). a pure-HTTP proxy using Nginx. ## Container dependencies Loading @@ -63,12 +61,16 @@ source installations, as it can be error-prone. Instead, this implementation takes the decision to use officially supported and maintained base images where possible. This means that it now uses the following images from the Docker Hub Library as the basis for `esgf-*` containers: images from the Docker Hub Library as the basis for `esgfhub/*` containers: * `centos/postgresql-96-centos7` * `nginx` * `solr:5.5` * `alpine` * `python:2.7-slim` * `openjdk:8-{jre,jdk}` * `tomcat:8` * `python:2.7-slim` * `continuumio/miniconda` In the original implementation, there was also an `esgf-node` image that served as a base image for all other ESGF containers. As well as containing supervisor, Loading
docs/_docs/developer/contributing.md 0 → 100644 +117 −0 Original line number Diff line number Diff line --- title: Contributing category: Developers order: 2.3 toc: false --- Contribution to `esgf-docker` happens via the GitHub repository - [https://github.com/ESGF/esgf-docker](https://github.com/ESGF/esgf-docker) - following a feature-branch-style workflow. ## Protected branches The `esgf-docker` repository has two protected branches: * `master` contains only code that has been tested. The intention is that any commit to `master` could be deployed in production. This is the branch against which releases are tagged. * `devel` contains the latest development code. The intention is that each commit to `devel` works, but is not as well tested as `master`. These branches are only committed to using pull requests. [Travis](https://travis-ci.org/ESGF/esgf-docker) is configured to automatically build pull requests when they are submitted, and a pull request should not be merged without a successful build. Travis is also configured to automatically build, tag and push each commit to the `master` and `devel` branches, using the `latest` and `devel` Docker tags respectively. Each commit is also given a unique Docker tag that combines the closest tag and the commit hash, allowing deployments to target a specific commit. ## Making a contribution These steps describe the process to follow when making a contribution. They assume that you do **not** have write access to the main `esgf-docker` repository. Those with write access to the main `esgf-docker` repository are free to work in branches in that repository if they wish. * Unless you are working on an existing issue, create a Github issue [in the main esgf-docker repository](https://github.com/ESGF/esgf-docker/issues) describing the change you intend to make. * If this is your first contribution, create a GitHub fork of the main `esgf-docker` repository in your own workspace and check it out to your development machine. ``` git clone git@github.com:<github username>/esgf-docker ``` * Ensure you have pulled the latest changes from the `devel` branch. ``` git checkout devel git pull upstream ``` * Create a new branch from `devel`. The preferred format for the branch name is `issue/<issue number>/<descriptive slug>` - for example, the branch that created this page is called `issue/46/contributing-docs`. ``` git checkout -b <branch name> ``` * Make your changes in this branch, referencing the issue from above using the `ESGF/esgf-docker#<issue number>` syntax in commit messages. ``` git add [-A | list of files] git commit -m "ESGF/esgf-docker#<issue number> <commit message>" ``` * Make sure all your changes are pushed to your GitHub fork. ``` git push ``` * Once you have completed and tested your changes, submit a pull request from the branch in your GitHub fork targeting the `devel` branch in the main `esgf-docker` repository. * Iterate with the reviewer until your pull request is accepted and merged to `devel`. If `devel` has changed since you created your branch, this may involve [rebasing](https://git-scm.com/docs/git-rebase) to the current `HEAD` of `devel` and fixing any merge conflicts. ## Committing to master The following describe the process of committing to the `master` branch. This will generally only happen for a release. The steps assume you have write access to the main `esgf-docker` repository. Releases should have the format `{major}.{minor}.{patch}[.{alpha|beta}.{increment}]`, e.g. `2.0.1`, `2.1.0.alpha.0` or `3.0.0.beta.2`. * Create an integration branch from the current `devel`. The preferred format for naming of integration branches is `integration/<release>`. ``` git checkout devel git pull git checkout -b integration/<release> ``` * [Rebase](https://git-scm.com/docs/git-rebase) the integration branch onto `master` and fix any merge conflicts. ``` git rebase master ``` * Perform the required testing for a release, making any required fixes. Once you are satisfied, submit a pull request targeting `master`. * Merge the pull request onto `master`. * Tag `master` with the release. ``` git checkout master git pull git tag <release> git push --tags ``` * Rebase `devel` to the current `HEAD` of `master`. If `devel` has changed since the integration branch was created, this may involve fixing any merge conflicts. ``` git checkout master git pull git checkout devel git pull git rebase master git push ```
docs/_docs/usage/configuration.md +12 −26 Original line number Diff line number Diff line Loading @@ -13,12 +13,8 @@ be generated using the scripts described in this page. * `$ESGF_CONFIG` * `certificates` * `esg-hostcert-bundle.p12` PKCS12-encoded bundle containing host certificate and private key for SAML signing * `esg-trust-bundle.jks` JKS-encoded bundle containing trusted CA certificates for Java apps * `esg-trust-bundle.pem` PEM-encoded bundle containing trusted CA certificates for Python apps PEM-encoded bundle containing trusted CA certificates * `hostcert` * `hostcert.crt` PEM-encoded host certificate, including all intermediate CA certificates, used for SSL Loading @@ -40,10 +36,6 @@ be generated using the scripts described in this page. Password for dbsuper user of main ESGF database * `database-publisher-password` Password for esgcet user of main ESGF database * `java-hostcert-bundle-password` Password for PKCS12-encoded host certificate bundle * `java-trust-bundle-password` Password for JKS-encoded trust bundle * `rootadmin-password` Password for rootAdmin node admin account * `shared-cookie-secret-key` Loading @@ -61,7 +53,7 @@ described above can be automatically generated. For a production installation, valid certificates signed by a trusted CA should be obtained for the host certificate and SLCS CA, and placed into the directory structure at the correct location. The scripts for generating configuration have been bundled up into the `cedadev/esgf-setup` The scripts for generating configuration have been bundled up into the `esgfhub/setup` container image, meaning that the only requirement on the host system is Docker. ### Generating secrets Loading @@ -70,9 +62,9 @@ To generate random secrets for all the passwords and secret keys required for an ESGF Docker installation, just run the following commands: ```sh $ export ESGF_HOSTNAME=local.esgf.org $ export ESGF_CONFIG=/path/to/empty/config/directory $ docker run -v "$ESGF_CONFIG":/esg -e ESGF_HOSTNAME cedadev/esgf-setup generate-secrets export ESGF_HOSTNAME=local.esgf.org export ESGF_CONFIG=/path/to/empty/config/directory docker-compose run esgf-setup generate-secrets ``` This will generate the contents of the `secrets` directory from the above structure. Loading @@ -90,7 +82,7 @@ To generate self-signed certificates for the host and SLCS CA, just run this command: ```sh $ docker run -v "$ESGF_CONFIG":/esg -e ESGF_HOSTNAME cedadev/esgf-setup generate-test-certificates docker-compose run esgf-setup generate-test-certificates ``` This will generate the `certificates/hostcert` and `certificates/slcsca` directories Loading @@ -99,22 +91,16 @@ created manually. ### Creating trust bundles The three trust bundle files, `esg-hostcert-bundle.p12`, `esg-trust-bundle.jks` and `esg-trust-bundle.pem`, are automatically generated from existing certificates. `esg-hostcert-bundle.p12` is obviously generated from the host certificate and private key in `certificates/hostcert`. The two trust bundles contain the exact same certificates, just in different formats. First, the certificates from `certificates/esg_trusted_certificates.tar` are included, if it exists. This tar file can be downloaded from an ESGF distribution The trust bundle `esg-trust-bundle.pem` is automatically generated from existing certificates. First, the certificates from `certificates/esg_trusted_certificates.tar` are included, if it exists. This tar file can be downloaded from an ESGF distribution site, and should contain a single directory called `esg_trusted_certificates` containing all the trusted CAs for the federation. If the SLCS CA is self-signed, it is also added to the trust bundles so that the installation can trust its own it is also added to the trust bundle so that the installation can trust its own certificates. The command to create the trust bundles is: The command to create the trust bundle is: ```sh $ docker run -v "$ESGF_CONFIG":/esg -e ESGF_HOSTNAME cedadev/esgf-setup create-trust-bundles docker-compose run esgf-setup create-trust-bundle ```
docs/_docs/usage/quick-start.md +2 −2 Original line number Diff line number Diff line Loading @@ -17,7 +17,7 @@ on the host system. First, clone the repository: ```sh git clone https://github.com/cedadev/esgf-docker.git git clone https://github.com/ESGF/esgf-docker.git cd esgf-docker ``` Loading Loading @@ -55,7 +55,7 @@ This means you need to make sure that the `ESGF_CONFIG` directory is in your home directory. </div> ## Pull the container images from Docker Hub ## Pull the container images from Docker Hub You only need to pull the images from Docker Hub when they have changed, or if you are deploying for the first time: Loading
