FileHooks

The fileHooks project comprises the backend of the CodeAbility Sharing Platform. The services GitLab and Elasticsearch are considered backend services. It is responsible for the data collection and preparation. This section describes the fileHooks used in GitLab and the infrastructure setup. Finally, some tips to handle errors are provided.

GitLab FileHooks

Currently, there is one fileHook for GitLab - doing both health check and indexing - to extend the functionality of GitLab. Details are provided in the following.

FileHook - trigger_project_update

This file hook does two tasks:

1. Health check and validation: It informs the user who modified the project via email if the metadata information is incomplete or invalid after a modification in a repository was conducted. Validation happens on the master-branch of all projects in the group sharing. It will mainly be triggered by push events, but also by moving or renaming a project. The check proceeds as follows: First, the root directory of the repository is checked for files named metadata.json, metadata.yaml, or metadata.yml. There must be exactly one such file, otherwise the check fails. Subsequently, the correctness of all metadata files is validated (also dependent metadata files, if it is a collection). If an error occurred, an email is sent to the user who pushed the changes.

2. It keeps the Elasticsearch index up-to-date by adding/updating/deleting files according to the triggered GitLab event. Only the main-branch (or master if main does not exist) and the group sharing (including subgroups and all subprojects) are indexed in Elasticsearch. Metadata files (metadata.json, metadata.yaml, or metadata.yml) at the project root are indexed in the alias metadata.

Warning

Note that GitLab does not trigger an event if a group is transferred! Those changes remain unnoticed in Elasticsearch! To prevent those inconsistencies, users should not transfer groups!

Infrastructure Setup

It is currently assumed that all services run on the same host as separate docker containers. The setup of the containers and the host server is discussed in the following. Lastly, the manual installation procedure for file hooks is given as a reference.

Container Setup

Subsequently, the setup for GitLab and Elasticsearch is shown. The setup of the Services GitLab search and MySQL are discussed in the section GitSearch Application.

To create all containers for the backend in production, a script situated in setup/ is provided. It takes a configuration file as the only argument. The configuration files can be found in setup/config/. For a local development setup, the file local can be used without further modification. For deployment on a server, the files development_template and production_template are provided. These configurations require secrets. Do not put secrets into these files but create a copy and put them into the copy. Files in the setup/config/ directory are ignored by git by default, so writing secrets into a copy prevents accidentally committing them.

The secrets which need to be added to the copy of the configuration file are:

• MAIL_USERNAME: The email user name to authenticate with at smtp.uibk.ac.at (KeePass @ artemis-support MailBox).

• MAIL_PASSWORD The password for the authentication at smtp.uibk.ac.at (KeePass @ artemis-support MailBox).

The following code block shows how to deploy the project in production.

cd setup
cp config/production_template config/production
$EDITOR config/production  # add the secrets to the copy
./setup-infrastructure.sh config/production

Similarly, the containers for the development backend can be created with:

cd setup
cp config/development_template config/development
$EDITOR config/development  # add the secrets to the copy
./setup-infrastructure.sh config/development

The following environment variables are set within the config files. No modification should be required for those if the correct config file is used.

• GITLAB_HOME: Directory where data generated by GitLab is persisted

• EXTERNAL_URL: External Url of the GitLab instance

• GITLAB_HOSTNAME: Hostname of GitLab

• ES_HOME: Directory where data generated by Elasticsearch is persisted

• FILEHOOKS_CONFIG_FILE: Name of the file in filehooks/conf/ used to configure the filehooks code

Environment variable	Production	Development
GITLAB_HOME	/mnt/qt-sharing-codeability/srv/gitlab	/mnt/qt-codeability-austria/sharing/srv/gitlab
ES_HOME	/mnt/qt-sharing-codeability/es	/mnt/qt-codeability-austria/sharing/es
EXTERNAL_URL	https://sharing-codeability.uibk.ac.at	https://sharing.codeability-austria.uibk.ac.at
GITLAB_HOSTNAME	sharing-codeability.uibk.ac.at	sharing.codeability-austria.uibk.ac.at
GITLAB_HOSTNAME	conf/production.ini	conf/staging.ini

Note

If the container is set up from scratch (there are no persisted data available), a password for the user root has to be specified using the web interface. For the development and production server, this password should be added to KeePass. Alternatively, the password can also be set directly in the GitLab container:

docker exec -it sharing_gitlab gitlab-rake 'gitlab:password:reset[root]'

Installing the Filehooks package

In the previous section, the container infrastructure is set up. When this is successfully done, the filehooks code needs to be installed in the GitLab container. There is another script in the setup directory for this job:

./install_filehooks_locally.sh --create-index

This script copies files from the repository into the GitLab container and sets up the code such that it is run whenever GitLab emits an event.

The --create-index flag causes the initial index to be created in ElasticSearch.

Local Development Setup

For development, it can be beneficial to have the setup running on the development machine.

In order to have access to the GitLab and ElasticSearch containers via http, modify setup/docker-compose.yml and enable the lines marked with the comment

# add this for your local testing setup

It might also be useful to remove the lines saying restart: always.

For local development the local config file can by used directly: Afterwards, the same setup procedure as for deployments can be used. The configuration for local development does not need to be copied and modified if the defaults are used.

cd setup
./setup-infrastructure.sh config/local

When this completes, the filehooks code needs to be installed as described previously:

./install_filehooks_locally.sh --create-index

At this point, GitLab should be reachable at http://localhost:10082 and ElasticSearch at http://localhost:9200.

To view the entire index check http://localhost:9200/metadata/_search.

The index will be updated whenever a repository or group in the GitLab group “sharing” is updated.

Execution logs of the filehooks code can be found in the GitLab container at /var/log/gitlab/gitlab-rails/trigger_project_update.log for normal logs and /var/log/gitlab/gitlab-rails/file_hook.log for crashes due to uncaught exceptions. A convenient way to observe these files is running

docker exec -it sharing_gitlab tail -f /var/log/gitlab/gitlab-rails/file_hook.log

for errors and similarly for the logs.

To deploy modifications of the code to GitLab, the relevant files need to be copied to the mounted volume. The script install_filehooks_locally.sh does this automatically when called without any arguments. This allows to quickly install new code without having to restart the container.

The GitLab container can be accessed interactively by running

docker exec -it sharing_gitlab /bin/bash

Server Setup

To make GitLab reachable from outside, conduct the following steps after connecting to the server via ssh:

1. Add the following snippet to /etc/apache2/sites-enabled/default-ssl.conf:

   # Sharing
   <VirtualHost *:443>
       SSLProxyEngine On
       AllowEncodedSlashes NoDecode
       SSLProxyVerify none
       SSLProxyCheckPeerCN off
       SSLProxyCheckPeerName off
       SSLProxyCheckPeerExpire off
   
       ##### Portainer #####
   
       RewriteRule ^/portainer$ /portainer/ [R,L]
       ProxyPass /portainer/ http://sharing-codeability.uibk.ac.at:9000/
       ProxyPassReverse /portainer/ http://sharing-codeability.uibk.ac.at:9000/
       ProxyPass /portainer/api/websocket/ http://sharing-codeability.uibk.ac.at:9000/api/websocket/
       ProxyPassReverse /portainer/api/websocket/ http://sharing-codeability.uibk.ac.at:9000/api/websocket/
   
       #####################
       ProxyPass / https://sharing-codeability.uibk.ac.at:10083/ nocanon
       ProxyPassReverse / https://sharing-codeability.uibk.ac.at:10083/
   
   
       ErrorLog ${APACHE_LOG_DIR}/error.sharing-codeability.uibk.ac.at.log
       CustomLog ${APACHE_LOG_DIR}/access.sharing-codeability.uibk.ac.at.log combined
   
       # Michael further tools settings
       Include /etc/apache2/codeAbility/sharing/*.conf
   </VirtualHost>
   

Note

Please review the configuration above carefully. Gitlab is very sensitive, when run behind a reverse proxy!

2. Add the following snippet to /etc/apache2/sites-enabled/000-default.conf:

   <VirtualHost *:80>
       # ...
       Redirect / https://sharing-codeability.uibk.ac.at
   </VirtualHost>
   

3. sudo systemctl restart apache2

Afterward, the containers can be started:

1. Navigate to /mnt/qt-sharing-codeability/file-hooks (prod).

2. Pull this repository for the latest updates

   • It could be necessary to git reset --hard the repository before, because some scripts replace confidential variables during installation.

3. Navigate to setup and set up the containers (see above)

Manual File Hook Setup

Note

Just for reference.

As a reference on how to add other file hooks to GitLab, the steps to install the file hook trigger_project_update.py are given below:

1. Install python requirements:

   pip3 install --upgrade setuptools
   pip3 install -r requirements.txt
   

2. Create API-Token with the user root and the scopes api, read_api, read_repository

3. Add the API-Token in conf.production.ini (section gitlab, key token)

4. Install the filehooks package

   pip3 install .
   

5. Install java

   apt-get install openjdk-8-jdk
   

6. Initialize the indices in elasticsearch using the script create_index.py. This script can be deleted once the indexes were created successfully.

   python3 create_index.py
   

7. Add the file trigger_project_update.py to /opt/gitlab/embedded/service/gitlab-rails/file_hooks in the GitLab container

8. Ensure that the script trigger_project_update.py has the permissions 755

9. Validate the file hooks:

   gitlab-rake file_hooks:validate
   

Infrastructure Update

Subsequently, a guide for updating GitLab and the filehooks is provided.

Update Guide GitLab

1. Navigate to the root directory of filehooks repository.

2. Create a backup of GitLab with the script setup/backup_sharing_gitlab.sh

3. Navigate to the parent directory of GITLAB_HOME and copy the mounted volume, e.g.,

   cp -a srv srv_2021_01_31
   

4. Change the GitLab version in the file setup/sendmail/Dockerfile.

5. Run the script setup/setup-infrastructure.sh

6. Wait & check if GitLab starts successfully. Usually it takes about 10 minutes until GitLab reaches the status healthy. It may very well be that the status is unhealthy for a period of time along the way.

7. Install the filehooks in GitLab using the script setup/install_filehooks_locally.sh.

8. Check if the filehooks work properly.

Note

When upgrading the GitLab version, follow the upgrade recommendations from GitLab.

Update Guide Filehooks

1. Check out the version of the code which should be deployed somewhere in the file system.

2. Run setup/install_filehooks_locally.sh.

Errors

In case a container crashes, it should start automatically. Consequently, it should not be necessary to start any container manually after the setup was executed successfully.

Warning

If the GitLab container crashes, the python-package filehooks is not re-installed automatically. (TODO check whether this is still true) Hence, new or changed files will not be added to elastic search. You have to install the filehooks (see update guide filehooks)! And do a complete reindexing, to ensure a consistent index.

Subsequently, the logging systems for GitLab and FileHooks are discussed.

GitLab

GitLab has an advanced logging system distributed over many log files. Details can be found in the GitLab documentation. For example, the command docker logs -f -n 10 sharing_gitlab can be used to inspect the logs.

FileHooks

• /var/log/gitlab/gitlab-rails/file_hook.log: Fatal errors (e.g., unexpected exceptions) are logged in this file.

• /var/log/gitlab/gitlab-rails/trigger_project_update.log: General logging information for the fileHook trigger_project_update.py are logged in this file.