GitSearch Application

This document presents the basic idea of the GitLab search service and the setup of the development environment.

Concept

Ideas

The main idea of the frontend of the CodeAbility Sharing Platform is providing a search engine which allows searching the projects provided in any repository that lies within the group sharing. The search engine should also allow users to search/filter for specific metadata (e.g., license(s), programming language(s)). Search results should be presented visually appealing so that users can easily find and use resources without any prior knowledge (especially Git). Moreover, the Sharing Platform should provide a starting and an imprint page, as well as data privacy pages.

Challenges

• How can we ensure that a user is allowed to see specific content? How can we synchronize user permissions with GitLab?

• How can we add additional statistics not recorded by GitLab, like the number of downloads and views?

Technology

The prototype uses a monolithic architecture and Spring Boot with Java as a backend and Angular as frontend technology. It was generated with JHipster v6.10.0. JWT-tokens are used for authentication. In production MySQL is utilized as database management system. H2 with disk-based persistence is employed in development. Maven is used for building the backend.

Production/Development environment

Before starting the search, please ensure that the backend is up and running! Afterward, the following steps can be executed to start the search.

1. Log in to the Docker registry

docker login docker.uibk.ac.at:443

2. Clone the GitLab search project

3. Navigate to the folder src/main/docker

4. Set the required environment variables

Environment variable	Production	Development
MYSQL_HOME	/mnt/qt-sharing-codeability/mysql	/mnt/qt-codeability-austria/sharing/mysql

5. Start the docker containers

docker-compose -f gitsearch.yml up

Note

When this setup is executed for the first time, the default password of the users admin (Sharing-Search - default - admin) and user (Sharing-Search - default - user) should be changed! The default password can be found in KeePass. The new password of admin and user should be added to KeePass (Sharing-Search - Dev - admin and Sharing-Search - Dev - user for development and Sharing-Search - Prod - admin and Sharing-Search - Prod - user for production).

Local development environment

A local development environment may help develop, test, and debug the application. The following parts should run on your development machine to set up a development environment locally.

Backend

To set up the entire backend of the CodeAbility Sharing Platform, execute the following steps.

1. Navigate to the folder setup

2. Open the ports and add the public network to the service elastic in the file docker-compose.yml:

ports:
    - '9200:9200'
    - '9300:9300'
networks:
  - backend
  - frontend

3. Execute the script setup-infrastructure.sh (TODO: add script documentation)

Note

This setup is somewhat resource intensive. For developing the GitLab search, it may be sufficient to run only an elasticsearch server and add data ‘manually’ to elasticsearch. For example, SSH port forwarding could be used ssh -L 9200:192.168.48.4:9200 <username>@codeability-austria.uibk.ac.at. For this to work, the port 9200 needs to be made available. In the default configuration it cannot be accessed this way. If you want to enable such access using the setup script, edit setup/docker-compose.yml as described previously. Note that such access via SSH requires being connected to UIBK’s internal network (possibly via VPN).

Frontend

1. Clone the GitLab search project

2. Start the spring backend of GitLab search in your IDE or by running ./mvnw. If this does not work, running ./mvnw clean && ./mvnw might fix it.

3. To enable automatic recompilation upon changes in the frontend code (Angular), run npm start in the root directory of the repository.

After the frontend and backend were started successfully, the services should be available on the following default ports.

Service	Port
GitLab search (no auto-update)	8080
GitLab search (auto-update)	9000
Elasticsearch (if enabled)	9200
GitLab (optional)	10082

If port 9200 is accessible, it can be useful to check http://localhost:9200/metadata/_search for debugging purposes. When data has been indexed by Elasticsearch it should show up there in JSON format. Note that GitLab will only tell Elasticsearch to index repositories in the group sharing. Only valid metadata is indexed.

Authentication and Integration with GitLab

It is important that the user of the search platform does not need to register / login twice for the CodeAbility repository

In the long run we should integrate with existing authentication systems as e.g. EduId (or even Google or Facebook).

For a short term solution we use the sharing platform GitLab as an authentication (and authorisation) provider.

Basic Concepts

GitLab offers an OAuth2 Service to share authentication and further to allow access to other resources. In the following we describe the configuration issues for GitLab.

Establish Preconditions

gitsearch must run behind an (apache) reverse proxy that forwards the original URL. In order to do so, add

SSLProxyEngine    On

ProxyPreserveHost On
ProxyRequests Off
RequestHeader set X-Forwarded-Proto "https"
ProxyPass / http://codeability-austria.uibk.ac.at:10084/
ProxyPassReverse / http://codeability-austria.uibk.ac.at:10084/

Configuration in GitLab

As an administrator choose “Admin Area” -> “Applications” and select “New Application” or edit for an existing Application. In the form enter

• any descriptive name

• as redirect URI the full URI of the gitsearch endpoint (e.g. https://dev-exchange.codeability-austria.uibk.ac.at/login/oauth2/code/oidc). You can specify serveral urls.

• select “Trusted” (to avoid nasty confirmation questions from gitlab)

• deselect “Confidential” (not necessary)

• select api, openid and read_user

After submission gitlab will have generated

• an application ID and

• an application secret

Configuration in gitsearch

Enter the ids from the last section in application{-prod | -dev | -staging}.properties under

security:
  oauth2:
    client:
      provider:
        oidc:
          issuer-uri: https://sharing.codeability-austria.uibk.ac.at
      registration:
        oidc:
          client-id: 149276ac11138d9ba72fb3cd12815e3fa2f372866df0eac0f7d1aae5fdffea24
          client-secret: 6f480635241f420a361581f4837594ea6f48f5ee6f515c1aa89f325dd922dbb0

The issuer-uri must be changed to the base url of your gitlab installation (e.g. https://sharing.codeability.uibk.ac.at)

For the “Entwicklungsserver” an additional staging profile was defined, in order to override certain production configuration settings. To activate it, edit the docker-compose file src/main/docker/gitsearch.yml

gitsearch-app:
  image: docker.uibk.ac.at:443/csar9407/gitsearch:development-8ad4fb188a8358c34355cebc0400ae731cdf966e
  #     image: docker.uibk.ac.at:443/csar9407/gitsearch:master-1bf1f21ceee62f335eeb4b070d93443cdbf4a14b
  container_name: sharing_search
  environment:
    - _JAVA_OPTIONS=-Xmx512m -Xms256m
    - SPRING_PROFILES_ACTIVE=prod,swagger,staging

Note

Remark, to register several oauth2 service providers, add them under different “oidc” tags.