.. _ref_git_search: GitSearch Application ===================== This document presents the basic idea of the GitLab search service and the setup of the development environment. Concept ------- Ideas ~~~~~ The main idea of the frontend of the CodeAbility Sharing Platform is providing a search engine which allows searching the projects provided in any repository that lies within the group ``sharing``. Additionally other repositories are indexed, if they contain a metadata-file in the root directory of the repository. The search engine should also allow users to search/filter for specific metadata (e.g., license(s), programming language(s)). Search results should be presented visually appealing so that users can easily find and use resources without any prior knowledge (especially Git). Moreover, the Sharing Platform should provide a starting and an imprint page, as well as data privacy pages. Technology ~~~~~~~~~~ The application was generated with `JHipster `_ v7.9.3. `JWT `_-tokens are used for authentication. In production `MySQL `_ is utilized as database management system. H2 with disk-based persistence is employed in development. Maven is used for building the backend. Production/Development environment ---------------------------------- Before starting the search, please ensure that the backend is up and running! Afterward, the following steps can be executed to start the search. The ci/cd pipeline is configured to automatically build and deploy the application to the development (CI task 'deploy') or the production environment (CI task 'deploy-prod'). The required environment variables are defined in the GitLab project settings (CI/CD -> Variables) and .env file. Local development environment ----------------------------- A local development environment may help develop, test, and debug the application. It is however quite resource intensive and requires a lot of setup. Therefore the recommended way to develop the GitLab search is to use docker infrastructure on the development environment. I.e. use the gitlab instance at https://sharing.codeability-austria.uibk.ac.at/dashboard. Additionally to use the elasticsearch index, open an ssh-tunnel to the elasticsearch docker container. .. code-block:: bash ssh -L 9200:sharing_elasticsearch:9200 @codeability-austria.uibk.ac.at or in windows (e.g. with putty) .. image:: puttyConfig.png :width: 300pt :alt: Putty tunnel config Frontend ~~~~~~~~ 1. Clone the GitLab search project 2. Start the spring backend of GitLab search in your IDE or by running ``./mvnw``. If this does not work, running ``./mvnw clean && ./mvnw`` might fix it. 3. To enable automatic recompilation upon changes in the frontend code (Angular), run ``npm start`` in the root directory of the repository. After the frontend and backend were started successfully, the services should be available on the following default ports. +--------------------------------+-------+ | Service | Port | +================================+=======+ | GitLab search (no auto-update) | 8080 | +--------------------------------+-------+ | GitLab search (auto-update) | 9000 | +--------------------------------+-------+ | Elasticsearch (if enabled) | 9200 | +--------------------------------+-------+ | GitLab (optional) | 10082 | +--------------------------------+-------+ If port ``9200`` is accessible, it can be useful to check ``http://localhost:9200/metadata/_search`` for debugging purposes. When data has been indexed by Elasticsearch it should show up there in JSON format. Note that GitLab will only tell Elasticsearch to index repositories in the group ``sharing``. Only valid metadata is indexed. Authentication and Integration with GitLab ------------------------------------------ It is important that the user of the search platform does not need to register / login twice for the CodeAbility repository To this end a keycloak server is installed. Basic Concepts ~~~~~~~~~~~~~~ GitLab offers an OAuth2 Service to share authentication and further to allow access to other resources. In the following we describe the configuration issues for GitLab. Establish Preconditions ^^^^^^^^^^^^^^^^^^^^^^^ gitsearch must run behind an (apache) reverse proxy that forwards the original URL. In order to do so, add .. code-block:: SSLProxyEngine On ProxyPreserveHost On ProxyRequests Off RequestHeader set X-Forwarded-Proto "https" ProxyPass / http://codeability-austria.uibk.ac.at:10084/ ProxyPassReverse / http://codeability-austria.uibk.ac.at:10084/ Configuration in GitLab ^^^^^^^^^^^^^^^^^^^^^^^ As an administrator choose "Admin Area" -> "Applications" and select "New Application" or edit for an existing Application. In the form enter - any descriptive name - as redirect URI the full URI of the gitsearch endpoint (e.g. https://dev-exchange.codeability-austria.uibk.ac.at/login/oauth2/code/oidc). You can specify serveral urls. - select "Trusted" (to avoid nasty confirmation questions from gitlab) - deselect "Confidential" (not necessary) - select api, openid and read_user After submission gitlab will have generated - an application ID and - an application secret Configuration in gitsearch ^^^^^^^^^^^^^^^^^^^^^^^^^^ Enter the ids from the last section in application{-prod | -dev | -staging}.properties under .. code-block:: YAML security: oauth2: client: provider: oidc: issuer-uri: https://sharing.codeability-austria.uibk.ac.at registration: oidc: client-id: 149276ac11138d9ba72fb3cd12815e3fa2f372866df0eac0f7d1aae5fdffea24 client-secret: 6f480635241f420a361581f4837594ea6f48f5ee6f515c1aa89f325dd922dbb0 The issuer-uri must be changed to the base url of your gitlab installation (e.g. https://sharing.codeability.uibk.ac.at) For the "Entwicklungsserver" an additional staging profile was defined, in order to override certain production configuration settings. To activate it, edit the docker-compose file src/main/docker/gitsearch.yml .. code-block:: YAML gitsearch-app: image: docker.uibk.ac.at:443/csar9407/gitsearch:development-8ad4fb188a8358c34355cebc0400ae731cdf966e # image: docker.uibk.ac.at:443/csar9407/gitsearch:master-1bf1f21ceee62f335eeb4b070d93443cdbf4a14b container_name: sharing_search environment: - _JAVA_OPTIONS=-Xmx512m -Xms256m - SPRING_PROFILES_ACTIVE=prod,swagger,staging .. note:: Remark, to register several oauth2 service providers, add them under different "oidc" tags.