.. _ref_git_search: GitSearch Application ===================== This document presents the basic idea of the GitLab search service and the setup of the development environment. Concept ------- Ideas ~~~~~ The main idea of the frontend of the CodeAbility Sharing Platform is providing a search engine which allows searching the projects provided in any repository that lies within the group ``sharing``. The search engine should also allow users to search/filter for specific metadata (e.g., license(s), programming language(s)). Search results should be presented visually appealing so that users can easily find and use resources without any prior knowledge (especially Git). Moreover, the Sharing Platform should provide a starting and an imprint page, as well as data privacy pages. Challenges ~~~~~~~~~~ - How can we ensure that a user is allowed to see specific content? How can we synchronize user permissions with GitLab? - How can we add additional statistics not recorded by GitLab, like the number of downloads and views? Technology ~~~~~~~~~~ The prototype uses a monolithic architecture and `Spring Boot `_ with Java as a backend and `Angular `_ as frontend technology. It was generated with `JHipster `_ v6.10.0. `JWT `_-tokens are used for authentication. In production `MySQL `_ is utilized as database management system. H2 with disk-based persistence is employed in development. Maven is used for building the backend. Production/Development environment ---------------------------------- Before starting the search, please ensure that the backend is up and running! Afterward, the following steps can be executed to start the search. 1. Log in to the Docker registry .. code-block:: bash docker login docker.uibk.ac.at:443 2. Clone the GitLab search project 3. Navigate to the folder ``src/main/docker`` 4. Set the required environment variables +----------------------+----------------------------------------+------------------------------------------------+ | Environment variable | Production | Development | +======================+========================================+================================================+ | MYSQL_HOME | /mnt/qt-sharing-codeability/mysql | /mnt/qt-codeability-austria/sharing/mysql | +----------------------+----------------------------------------+------------------------------------------------+ 5. Start the docker containers .. code-block:: bash docker-compose -f gitsearch.yml up .. note:: When this setup is executed for the first time, the default password of the users ``admin`` (``Sharing-Search - default - admin``) and ``user`` (``Sharing-Search - default - user``) should be changed! The default password can be found in KeePass. The new password of admin and user should be added to KeePass (``Sharing-Search - Dev - admin`` and ``Sharing-Search - Dev - user`` for development and ``Sharing-Search - Prod - admin`` and ``Sharing-Search - Prod - user`` for production). Local development environment ----------------------------- A local development environment may help develop, test, and debug the application. The following parts should run on your development machine to set up a development environment locally. Backend ~~~~~~~ To set up the entire backend of the CodeAbility Sharing Platform, execute the following steps. 1. Navigate to the folder ``setup`` 2. Open the ports and add the public network to the service ``elastic`` in the file ``docker-compose.yml``: .. code-block:: YAML ports: - '9200:9200' - '9300:9300' networks: - backend - frontend 3. Execute the script ``setup-infrastructure.sh`` (TODO: add script documentation) .. note:: This setup is somewhat resource intensive. For developing the GitLab search, it may be sufficient to run only an elasticsearch server and add data 'manually' to elasticsearch. For example, SSH port forwarding could be used ``ssh -L 9200:192.168.48.4:9200 @codeability-austria.uibk.ac.at``. For this to work, the port ``9200`` needs to be made available. In the default configuration it cannot be accessed this way. If you want to enable such access using the setup script, edit ``setup/docker-compose.yml`` as described previously. Note that such access via SSH requires being connected to UIBK's internal network (possibly via VPN). Frontend ~~~~~~~~ 1. Clone the GitLab search project 2. Start the spring backend of GitLab search in your IDE or by running ``./mvnw``. If this does not work, running ``./mvnw clean && ./mvnw`` might fix it. 3. To enable automatic recompilation upon changes in the frontend code (Angular), run ``npm start`` in the root directory of the repository. After the frontend and backend were started successfully, the services should be available on the following default ports. +--------------------------------+-------+ | Service | Port | +================================+=======+ | GitLab search (no auto-update) | 8080 | +--------------------------------+-------+ | GitLab search (auto-update) | 9000 | +--------------------------------+-------+ | Elasticsearch (if enabled) | 9200 | +--------------------------------+-------+ | GitLab (optional) | 10082 | +--------------------------------+-------+ If port ``9200`` is accessible, it can be useful to check ``http://localhost:9200/metadata/_search`` for debugging purposes. When data has been indexed by Elasticsearch it should show up there in JSON format. Note that GitLab will only tell Elasticsearch to index repositories in the group ``sharing``. Only valid metadata is indexed. Authentication and Integration with GitLab ------------------------------------------ It is important that the user of the search platform does not need to register / login twice for the CodeAbility repository In the long run we should integrate with existing authentication systems as e.g. EduId (or even Google or Facebook). For a short term solution we use the sharing platform GitLab as an authentication (and authorisation) provider. Basic Concepts ~~~~~~~~~~~~~~ GitLab offers an OAuth2 Service to share authentication and further to allow access to other resources. In the following we describe the configuration issues for GitLab. Establish Preconditions ^^^^^^^^^^^^^^^^^^^^^^^ gitsearch must run behind an (apache) reverse proxy that forwards the original URL. In order to do so, add .. code-block:: SSLProxyEngine On ProxyPreserveHost On ProxyRequests Off RequestHeader set X-Forwarded-Proto "https" ProxyPass / http://codeability-austria.uibk.ac.at:10084/ ProxyPassReverse / http://codeability-austria.uibk.ac.at:10084/ Configuration in GitLab ^^^^^^^^^^^^^^^^^^^^^^^ As an administrator choose "Admin Area" -> "Applications" and select "New Application" or edit for an existing Application. In the form enter - any descriptive name - as redirect URI the full URI of the gitsearch endpoint (e.g. https://dev-exchange.codeability-austria.uibk.ac.at/login/oauth2/code/oidc). You can specify serveral urls. - select "Trusted" (to avoid nasty confirmation questions from gitlab) - deselect "Confidential" (not necessary) - select api, openid and read_user After submission gitlab will have generated - an application ID and - an application secret Configuration in gitsearch ^^^^^^^^^^^^^^^^^^^^^^^^^^ Enter the ids from the last section in application{-prod | -dev | -staging}.properties under .. code-block:: YAML security: oauth2: client: provider: oidc: issuer-uri: https://sharing.codeability-austria.uibk.ac.at registration: oidc: client-id: 149276ac11138d9ba72fb3cd12815e3fa2f372866df0eac0f7d1aae5fdffea24 client-secret: 6f480635241f420a361581f4837594ea6f48f5ee6f515c1aa89f325dd922dbb0 The issuer-uri must be changed to the base url of your gitlab installation (e.g. https://sharing.codeability.uibk.ac.at) For the "Entwicklungsserver" an additional staging profile was defined, in order to override certain production configuration settings. To activate it, edit the docker-compose file src/main/docker/gitsearch.yml .. code-block:: YAML gitsearch-app: image: docker.uibk.ac.at:443/csar9407/gitsearch:development-8ad4fb188a8358c34355cebc0400ae731cdf966e # image: docker.uibk.ac.at:443/csar9407/gitsearch:master-1bf1f21ceee62f335eeb4b070d93443cdbf4a14b container_name: sharing_search environment: - _JAVA_OPTIONS=-Xmx512m -Xms256m - SPRING_PROFILES_ACTIVE=prod,swagger,staging .. note:: Remark, to register several oauth2 service providers, add them under different "oidc" tags.