trigger_project_update¶

This script is called as a file-hook by GitLab upon events. It handles the event is is given. If appropriate for the event, the repository’s metadata is validated, the user who caused the event gets notified by email about errors in case there are any, and the metadata index is updated.

Note

Users cannot disable the health check email.

Metadata validation¶

Metadata files must adhere to the following schema. There are additional restrictions for collections. Should a metadata file indicate a collection, it must declare the collectionContent, which must be valid metadata files in subdirectories of the metadata file declaring the collection. Collections may be nested.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Metadata",
  "description": "A metadata description of a repository",
  "type": "object",
  "properties": {
    "assesses": {
      "description": "The item being described is intended to assess the competency or learning outcome defined by the referenced term. (LRMI)",
      "type": "array",
      "items": {
        "type": "string"
      },
      "uniqueItems": true
    },
    "audience": {
      "description": "description of the people for whom the content is intended",
      "type": "string"
    },
    "collectionContent": {
      "description": "If the material's type is collection, this attribute allows specifying where other metadata files are located.",
      "type": "array",
      "items": {
        "type": "string"
      },
      "minItems": 1,
      "uniqueItems": true
    },
    "contributor": {
      "description": "people who contributed to the project",
      "type": "array",
      "items": {
        "$ref": "#/definitions/person"
      },
      "uniqueItems": true
    },
    "creator": {
      "description": "the authors",
      "type": "array",
      "items": {
        "$ref": "#/definitions/person"
      },
      "minItems": 0,
      "_comment": "only mandatory on top level",
      "uniqueItems": true
    },
    "description": {
      "description": "a brief description of the provided material.",
      "type": "string"
    },
    "difficulty": {
      "description": "the material covers either simple, medium, or advanced topics",
      "enum": ["easy", "medium", "difficult", "simple", "advanced"]
    },
    "educationalAlignment": {
      "description": "An alignment to an established educational framework. (LRMI) (Work in Progress)",
      "type": "string"
    },
    "educationalFramework": {
      "description": "The framework to which the resource being described is aligned. (LRMI) (Work in Progress)",
      "type": "string"
    },
    "educationalLevel": {
      "description": "The level in terms of progression through an educational or training context. Examples of educational levels include the formal sets of level indicators, described in the competence vocabulary. (LRMI - CA specific)",
      "type": "array",
      "items": {
        "type": "string"
      },
      "uniqueItems": true
    },
    "educationalUse": {
      "description": "The purpose of a work in the context of education; for example, 'assignment', 'group work'. (LRMI)",
      "type": "array",
      "items": {
        "type": "string"
      },
      "uniqueItems": true
    },
    "format": {
      "description": "indicates the file type, e.g. latex, ms word, pdf. might be used for plugins in the future",
      "type": "array",
      "items": {
        "type": "string"
      },
      "minItems": 1,
      "uniqueItems": true
    },
    "identifier": {
      "description": "an identifier for the exercise. format is not established yet",
      "type": "string"
    },
    "image": {
      "description": "path to an image location (either in the repository or a general URL)",
      "type": "string",
      "format": "uri-reference"
    },
    "interactivityType": {
      "description": "The predominant mode of learning supported by the learning resource. Acceptable values are 'active', 'expositive', or 'mixed'. (LRMI)",
      "enum": ["active", "expositive", "mixed"]
    },
    "isBasedOn": {
      "description": "A resource from which this work is derived or from which it is a modification or adaption. (LRMI)",
      "type": "array",
      "items": {
        "type": "string"
      },
      "uniqueItems": true
    },
    "keyword": {
      "description": "a collection of keywords describing the provided material",
      "type": "array",
      "items": {
        "type": "string"
      },
      "minItems": 1,
      "uniqueItems": true
    },
    "language": {
      "description": "a collection of ISO 639-1 language codes specifying the natural language of the provided material",
      "type": "array",
      "items": {
        "type": "string",
        "minLength": 2,
        "maxLength": 2
      },
      "minItems": 0,
      "_comment": "only mandatory on top level",
      "uniqueItems": true
    },
    "learningResourceType": {
      "description": "The predominant learningResourceType or kind characterizing the learning resource. For example, 'presentation', 'handout'. (LRMI)",
      "type": "string"
    },
    "license": {
      "description": "the license of the material provided in the repository",
      "type": "string"
    },
    "metadataVersion": {
      "description": "the metadata version",
      "type": "string",
      "enum": ["0.4"]
    },
    "programmingLanguage": {
      "description": "a collection of programming languages which may be covered/used in the material",
      "type": "array",
      "items": {
        "type": "string"
      },
      "uniqueItems": true
    },
    "publicVisibility": {
      "description": "Controls the public visibility of an exercise",
      "type": "object",
      "properties": {
        "except": {
          "description": "exempts folders or files from public visibility",
          "type": "array",
          "items": {
            "type": "string"
          },
          "uniqueItems": true
        }
      },
      "additionalProperties": false
    },
    "publisher": {
      "description": "the maintainers",
      "type": "array",
      "items": {
        "$ref": "#/definitions/person"
      },
      "minItems": 0,
      "_comment": "only mandatory on top level",
      "uniqueItems": true
    },
    "requires": {
      "description": "collection of required skills for solving the exercise",
      "type": "array",
      "items": {
        "type": "string"
      },
      "uniqueItems": true
    },
    "status": {
      "description": "status of the content",
      "type": "string"
    },
    "structure": {
      "description": "Describes how exercises are related. Default is atomic. Other values can be used when the type attribute is collection. Functionality is not implemented yet.",
      "enum": ["atomic", "networked", "hierarchical", "linear"]
    },
    "subject": {
      "description": "University module/school subject, for which the resource is intended",
      "type": "array",
      "items": {
        "type": "string"
      },
      "uniqueItems": true
    },
    "teaches": {
      "description": "The item being described is intended to help a person learn the competency or learning outcome defined by the referenced term. (LRMI)",
      "type": "array",
      "items": {
        "type": "string"
      },
      "uniqueItems": true
    },
    "timeRequired": {
      "description": "the time it typically takes to solve the exercise / work on the content",
      "pattern": "^[0-9]+(:[0-5][0-9]){1,2}$"
    },
    "title": {
      "description": "a title for the provided material.",
      "type": "string"
    },
    "typicalAgeRange": {
      "description": "The typical expected age range, e.g. \"7-9\" (from age 7 to 9), \"11-\" (up to age 11), \"8+\" (over the age of 8). (LRMI)",
      "pattern": "^[0-9]+((-[0-9]*)|\\+)$"
    },
    "valid": {
      "description": "the dates between which the exercise is valid",
      "type": "object",
      "properties": {
        "from": {
          "$ref": "#/definitions/date"
        },
        "to": {
          "$ref": "#/definitions/date"
        }
      },
      "additionalProperties": false,
      "required": ["from", "to"]
    },
    "version": {
      "description": "the version of the content. up to the content creators",
      "type": "string"
    }
  },
  "required": [
    "description",
    "format",
    "keyword",
    "learningResourceType",
    "metadataVersion",
    "title"
  ],
  "_comment": "license, creator, publisher, and language only mandatory on top level",
  "additionalProperties": false,
  "definitions": {
    "person": {
      "description": "a person",
      "type": "object",
      "properties": {
        "name": {
          "description": "the full name of a person",
          "type": "string"
        },
        "affiliation": {
          "description": "the affiliation of a person",
          "type": "string"
        },
        "email": {
          "description": "the email of a person",
          "type": "string",
          "format": "email"
        }
      },
      "required": ["name", "affiliation", "email"],
      "additionalProperties": false
    },
    "date": {
      "description": "a date in the format YYYY-MM-DD",
      "type": "string",
      "pattern": "^[0-9]{4}(-(0[1-9]|1[0-2])(-(0[1-9]|[12][0-9]|3[01]))?)?$"
    }
  }
}

Indexing¶

For the indexing, the alias metadata is used. The metadata files are indexed in the metadata alias using the schema:

{
  "settings": {
    "number_of_shards": 1,
    "index.mapping.total_fields.limit": 2000
  },
  "mappings": {
    "properties": {
      "project": {
        "properties": {
          "project_id": {
            "type": "long"
          },
          "project_name": {
            "type": "keyword"
          },
          "url": {
            "type": "keyword"
          },
          "namespace": {
            "type": "keyword"
          },
          "main_group": {
            "type": "keyword"
          },
          "sub_group": {
            "type": "keyword"
          },
          "visibility": {
            "type": "keyword"
          },
          "users": {
            "type": "keyword"
          },
          "groups": {
            "type": "keyword"
          },
          "archived": {
            "type": "boolean"
          },
          "star_count": {
            "type": "integer"
          },
          "open_issues_count": {
            "type": "integer"
          },
          "forks_count": {
            "type": "integer"
          },
          "last_activity_at": {
            "type": "date",
            "format": "date_optional_time"
          },
          "description": {
            "type": "text"
          }
        }
      },
      "file": {
        "properties": {
          "filename": {
            "type": "keyword"
          },
          "path": {
            "type": "keyword"
          },
          "commit_id": {
            "type": "keyword"
          },
          "indexing_date": {
            "type": "date",
            "format": "date_optional_time"
          },
          "last_activity_at": {
            "type": "date",
            "format": "date_optional_time"
          },
          "parentId": {
            "type": "keyword"
          },
          "children": {
            "type": "keyword"
          }
        }
      },
      "metadata": {
        "properties": {
          "assesses": {
            "type": "keyword"
          },
          "audience": {
            "type": "text"
          },
          "contributor": {
            "properties": {
              "name": {
                "type": "text"
              },
              "affiliation": {
                "type": "text"
              },
              "email": {
                "type": "text"
              }
            }
          },
          "creator": {
            "properties": {
              "name": {
                "type": "text"
              },
              "affiliation": {
                "type": "text"
              },
              "email": {
                "type": "text"
              }
            }
          },
          "description": {
            "type": "text"
          },
          "difficulty": {
            "type": "keyword"
          },
          "educationalAlignment": {
            "type": "keyword"
          },
          "educationalFramework": {
            "type": "keyword"
          },
          "educationalLevel": {
            "type": "keyword"
          },
          "educationalUse": {
            "type": "text"
          },
          "format": {
            "type": "keyword"
          },
          "identifier": {
            "type": "keyword"
          },
          "image": {
            "type": "keyword"
          },
          "interactivityType": {
            "type": "keyword"
          },
          "isBasedOn": {
            "type": "text"
          },
          "keyword": {
            "type": "text"
          },
          "language": {
            "type": "keyword"
          },
          "learningResourceType": {
            "type": "text"
          },
          "license": {
            "type": "keyword"
          },
          "publicVisibility": {
            "properties": {
              "except": {
                "type": "keyword"
              }
            }
          },
          "metadataVersion": {
            "type": "keyword"
          },
          "programmingLanguage": {
            "type": "keyword"
          },
          "publisher": {
            "properties": {
              "name": {
                "type": "text"
              },
              "affiliation": {
                "type": "text"
              },
              "email": {
                "type": "text"
              }
            }
          },
          "requires": {
            "type": "text"
          },
          "status": {
            "type": "keyword"
          },
          "structure": {
            "type": "keyword"
          },
          "subject": {
            "type": "text"
          },
          "teaches": {
            "type": "text"
          },
          "timeRequired": {
            "type": "keyword"
          },
          "title": {
            "type": "text"
          },
          "typicalAgeRange": {
            "type": "keyword"
          },
          "valid": {
            "properties": {
              "from": {
                "type": "keyword"
              },
              "to": {
                "type": "keyword"
              }
            }
          },
          "version": {
            "type": "keyword"
          }
        }
      },
    "statistics": {
            "properties": {
              "views": {
                "type": "long"
              },
              "downloads": {
                "type": "long"
              },
              "batchRewarded": {
                "type": "boolean"
              }
            }
          }
    }
  }
}

In the GitLab docker container, one can check the current content of the alias metadata by issuing the following command.

curl -GET http://sharing_elasticsearch:9200/metadata/_search

Logging¶

Logging information can be found at:

General logging information is provided at /var/log/gitlab/gitlab-rails/trigger_project_update.log
If an error which does not get handled occurs during execution of the file hook, the error message is logged to /var/log/gitlab/gitlab-rails/plugin.log. Such errors indicate a problem which should be investigated.

trigger_project_update module¶

When this module is installed as a file hook in GitLab, it forwards the event to a web service for further handling

class scripts.trigger_project_update.ConfigType(value)[source]¶

Bases: enum.Enum

Enum for choosing the desired configuration

DEBUG = 2¶

LOCAL = 4¶

PRODUCTION = 0¶

STAGING = 3¶

TEST = 1¶

scripts.trigger_project_update.load_config(config_type=<ConfigType.PRODUCTION: 0>)[source]¶

scripts.trigger_project_update.logger_setup(filepath: str) → Dict[str, Any][source]¶

Returns a dictionary which can be used to configure a logger.

Parameters:: filepath – path of the log file
Returns:: a dictionary to configure a logger

scripts.trigger_project_update.read_gitlab_event() → str[source]¶

Reads the GitLab system hook event from stdin.

Returns:: The event