dhis2_clone

Clone a dhis2 installation from another server. Supported:

• Remote server: tomcat.
• Local server: tomcat or d2-docker.

This program will perform several steps on a running DHIS2 server to transform it into a clone of a remote one.

The steps are:

• Stop the running tomcat/d2-docker.
• Backup the DHIS2 war file and database.
• Copy the webapps from the remote server into the local machine.
• Empty the local database and fill it with a replica of the remote one.
• Start the tomcat/d2-docker again.

Any step can be individually switched off (--no-backups, --no-webapps, --no-db, --manual-restart, --use-backup).

Also, it can run some sql scripts on the database before starting the server again (--post-sql). This is useful in several scenarios. For example, to empty the tables that contain the data (while keeping the metadata), as one would want for a training server (see the empty_data_tables_228.sql example). Or it can be used to automatically upgrade from one version to another (say, 2.29 to 2.30 by running upgrade-230.sql).

If it is so specified in the configuration file, it will perform some postprocessing actions once the server is running again (see section below).

All the commands run to perform the different steps are printed on the screen (with color to help identify the steps), and the output of those commands too. If any step fails, the program will signal an error and stop all the processing at that point.

Instances of type d2-docker might change its image (typically, in an upgrade) in the cloning process, so there is the concept of "transformed" images. Use options --stop-transformed to use config entry local_docker_image_stop_transformed and --start-transformed to use config entry local_docker_image_start_transformed instead of the default value local_docker_image.

Setup

sudo apt install pip3
pip install -r requirements.txt

Usage

dhis2_clone.py [-h] [--db-local DB_LOCAL] [--db-remote DB_REMOTE] [--api-local-username API_LOCAL_USERNAME]
                    [--api-local-password API_LOCAL_PASSWORD] [--no-backups] [--no-webapps] [--no-db]
                    [--no-postprocess] [--no-preprocess] [--manual-restart] [--post-sql POST_SQL [POST_SQL ...]]
                    [--strict-sql] [--pre-api PRE_API] [--post-api POST_API] [--keep-temp] [--post-clone-scripts]
                    [--post-import] [--update-config] [--no-color] [--start-transformed] [--stop-transformed]
                    [--use-backup USE_BACKUP]
                    config

positional arguments:

config                                   file with configuration

options:

-h, --help                               show this help message and exit
--db-local DB_LOCAL                      db to be override
--db-remote DB_REMOTE                    db to be copied in the db-local
--api-local-username API_LOCAL_USERNAME  api local user
--api-local-password API_LOCAL_PASSWORD  api local password
--no-backups                             don't make backups
--no-webapps                             don't clone the webapps
--no-db                                  don't clone the database
--no-postprocess                         don't do postprocessing
--no-preprocess                          don't do preprocessing
--manual-restart                         don't stop/start tomcat
--post-sql POST_SQL [POST_SQL ...]       sql files to run post-clone
--strict-sql                             stop the sql script on first fail and show in the log
--pre-api PRE_API                        Pre Api calls compatible versions: 2.34 / 2.36 / 2.38 / 2.41 (default: 2.36)
--post-api POST_API                      Post Api calls compatible versions: 2.34 / 2.36 / 2.38 / 2.41 (default: 2.36)
--keep-temp                              Preserve temporary d2-docker files for cloning the instance
--post-clone-scripts                     execute all py and sh scripts in post_clone_scripts_dir
--post-import                            import to DHIS2 selected json files from post_process_import_dir
--update-config                          update the config file
--no-color                               don't use colored output
--start-transformed                      Override d2-docker image for start
--stop-transformed                       Override d2-docker image for stop
--use-backup USE_BACKUP                  Path to remote backup file to use instead of making a remote pg_dump

Configuration

To invoke the program you need to specify a configuration file, as in:

dhis2_clone config_training.json

An example configuration file is provided in this repository (configuration_example.json).

The sections in the configuration file are:

• local_type: Local instance type. "tomcat" or "d2-docker".
• local_docker_image". Docker image to use (example: "eyeseetea/dhis2-data:2.32-sierra-leone").
• local_docker_image_stop_transformed. If --stop-transformed options is used, then instead of using local_docker_image, it will use entry local_docker_image_stop_transformed. This "stop" action is then used in the following d2-docker commands: stop_tomcat -> d2-docker stop, backup_db -> d2-docker copy, get_db -> d2-docker create data. "local_docker_image_start_transformed": If --start-transformed options is used, then instead of using local_docker_image, it will use entry local_docker_image_start_transformed. This "start" action is then used in the following d2-docker commands: start_tomcat -> d2-docker start.
• local_docker_port: Docker port.
• local_docker_deploy_path: Docker instance deploy path namespace.
• backups_dir: directory where it will store the backups.
• backup_name: an identifier that it will append to the name of the war file and database backups.
• server_dir_local: base directory of the tomcat running in the local server. Although server_dir_local points to the Tomcat base directory, it is also used internally as a temporary directory by d2-docker. When running a d2-docker cloning operation, this temporary directory will be removed by default (unless you use the --keep-temp option).

If this path is set to a critical location (such as your home directory or any folder with important files), you risk losing data. Only the d2-docker temporary files are intended to be removed, but the behavior can be misleading. * server_dir_remote: base directory of the tomcat running in the

remote server.

• hostname_remote: name or IP of the machine containing the remote DHIS2 instance. The user running the script is assumed to have ssh access to that machine.

• db_local: URI conninfo string to connect to the local database.

• db_remote: URI conninfo string to connect to the remote database.

• war_local: name of the local war file (when connecting to the web server, this corresponds to the last part of the URL - for example, if it is dhis2-demo.war, the webserver will respond at https://.../dhis2-demo).

• war_remote: name of the remote war file.

• docker_temp_folder: Base folder for the docker creation temp folders.

• api_local: if some post-processing steps are applied, this section needs to define as params the username and password url, username and password to connect to the running DHIS2 system after the cloning.

• postprocess: list of blocks, each containing users (specified by selectUsernames and/or selectFromGroups) and an action to perform on them (activate to activate them, deleteOthers to keep them in exclusive, addRoles to specify a list of extra roles to give, or addRolesFromTemplate to give a reference username whose roles we want to add). Instead of a block, you can give a url, and the blocks contained in that url will be added to the list of blocks.

• preprocess: list of blocks, each containing actions for each department (specified in departments) All the rules will create a sql file to execute after launch tomcat. In the case of edit a department metadata, you should include the metadata type in selectMetadataType the department selectDepartament, and the action, for example anonymizeData or deleteData and the list of metadata: selectDatasets, selectTrackedEntityAttributes, selectDataElements, Valid options: dataSets, programs, trackerPrograms. Format: ["dataSets"] Examples:

  {

  "selectDepartament": "NTD", "selectMetadataType": ["trackerPrograms","eventPrograms","dataSets"], "selectDatasets": [

  "tnek2LjfuIm", "zna8KfLMXn4", "XBgvNrxpcDC", "WHPEpoVDFFv", "SAV16xEdCZW", "AAYgHGENgbF", "NKWbkXyfO5F", "oVxjBKA1Yzu", "S1UMweeoPsi", "s3iaozBY0dv", "JP4bMwvJ6oU", "U5ejGQdX4Ih"

  ], "action": "removeData"

  }

  selectEventProgram or selectTrackerProgram to filter the event programs or tracker programs.1 You could also filter by

  "selectOrgUnitAndDescendants": ["example_uid"], "selectDataElements": ["example_uid"], "selectOrgUnits": ["example_uid"],

  You can also remove organisationunits using the action: removeOrganisationUnitTree you must add the organisationunit uids: selectOrganisationUnit: ["uid","uid2"] or removeOrganisationUnitTreeByLevel (needs a level attribute, like level:3). Examples:

  {

  "selectDepartament": "All", "selectMetadataType": ["organisationUnits"], "selectOrganisationUnit": [ "hmZE3mVAZFf", "G3thRWUQAX9", "HfVjCurKxh2", "seHJdofSPcM" ], "action": "removeOrganisationUnitTree"

  }, {

  "selectDepartament": "All", "selectMetadataType": ["organisationUnits"], "level": 3, "action": "removeOrganisationUnitTreeByLevel"

  }

  To anonymizeData you should use the action "anonymizeData" and could add the following params

  {

  "selectDepartament": "NTD", "selectMetadataType": ["trackerPrograms"], "anonymizePhone": true, "anonymizeMail": true, "anonymizeOrgUnit": true, "anonymizeCoordinates": true, "selectTrackedEntityAttributes": ["oTvXfEywjT3", "n8E6WIyAwcC", "DwZNiXy5Daz", "FHw1NKy0PWY", "eQtZaLIO3XU",

  "na3ZJRtjpGH", "HkBG3DVELBM", "sKBh0kazOCk", "AAkZm4ZxFw7", "ENRjVGxVL6l", "aBaYLJryaMr", "iy884aJfYTc"],

  "action": "anonymizeData"

  }, {

  "selectDepartament": "HWF", "selectMetadataType": ["dataSets"], "action": "anonymizeData"

  },

  To anonimize users except some of them, you should fill the follow rule:

  {

  "selectDepartament": "ALL", "selectAdminUser": "newadmin", "excludeUsernames": [

  "oldadmin", "oldadmin2"

  ], "selectOldAdminUser": "oldadmin", "action": "anonymizeUsers" }

  to perform on them (activate to activate them, deleteOthers to keep them in exclusive, addRoles to specify a list of extra roles to give, or addRolesFromTemplate to give a reference username whose roles we want to add). Instead of a block, you can give a url, and the blocks contained in that url will be added to the list of blocks.

• removeUnlistedData: deletes ALL data for non-listed items in the departament list of programs/dataset uids. - For tracker/event programs: removes events, enrollments, and trackedEntityInstances from every program not present in the departament metadata list. - For aggregate datasets: removes dataValues from every dataset not present in the departament metadata list.

  Example:

  {

  "action": "removeUnlistedData" }

• showDataSummary: displays data counts grouped by program and dataset at the end of preprocess.sql execution.

  Example: {

  "action": "showDataSummary"

  }

Automatic cloning

You may want to run the cloning script periodically. For that, you can use the appropriate users's crontab:

crontab -e

For example, this will run the cloning for a training server every Saturday night at 22:00:

$ crontab -l
00 22 * * 6 /usr/local/bin/dhis2_clone --post-sql /usr/share/dhis2_clone/empty_data_tables.sql /usr/share/dhis2_clone/training.json >> /var/log/dhis2_clone.log 2>&1

Requirements

Python

This program depends on a few Python standard modules and also:

• psycopg2: to connect to the postgres database.
• requests: to make HTTP requests.

They are available already packaged in most distributions (normally called python-psycopg2 and python-requests).

Also, it relies on two more modules included here:

• process.py: includes all the post-processing logic.
• d2apy.py: handles communications with a DHIS2 server through its api.

System programs

Other than the standard system utilities, the program will need to have a local installation of:

• rsync (used with ssh to copy the remote webapps).
• ssh (used to copy the remote webapps and to launch the remote dump of the database to be cloned).
• psql (used to modify the local database).
• pg_dump (used to make a backup of the local database, and a dump of the remote one -- so this one needs to exist on hostname_remote too).
• zcat (used to read remote backup).

User permissions

The program assumes that it runs with permissions to:

• Read and write all the files in <server_dir_local>, and especially,

  • run the files <server_dir_local>/bin/startup.sh and

  <server_dir_local>/bin/shutdown.sh.

  • write on <server_dir_local>/webapps and <server_dir_local>/files.

• Write on <backups_dir>.

• Run ssh to connect to <hostname_remote>.

• Run psql and pg_dump on the local host, and on <hostname_remote> thru ssh.

• Read all the files in <hostname_remote>:<server_dir_remote> thru ssh.

• Have read and write access to the local database thru the db_local conninfo string, and read access to the remote one thru db_remote.

• Have read access to the remote backup file if --use-backup is used.

If it runs any kind of postprocessing (by having an api_local and postprocess section in the configuration file), it will also need permissions to:

• Access the running dhis2 instance thru the url, username and password present in the api_local section, and have permissions to change the users. Using --api-local-username, --api-local-password params

In any case, it does not assume permissions to:

• Delete and create databases.

Api versions

You can filter vi api version in the preprocess or proprocess, for example adding in the config file:

"pre_api": "2.36", "post_api": "2.34",

or adding as param pre-api/post-api apiversion