# Hog-CI with GitLab CI/CD ```{warning} Outdated Documentation! This documentation version is out of date. Please check the [latest version 2026.1](https://hog.readthedocs.io/en/latest/). ``` :::{toctree} :glob: :hidden: * ::: ## Setting up Hog CI with GitLab CI/CD On GitLab repository, Hog Continuous Integration makes use of the [GitLab CI/CD tool](https://docs.gitLab.com/ee/ci/). Both the GitLab repository and your local area must be set-up to work properly with Hog CI. In this paragraph, we assume that we are working with a GitLab Project called `MyProject` under the GitLab group `MyGroup`. Please, replace these with the actual names of your project and group. ### Preliminary requirements To run the Hog-CI you must create a GitLab access token or create a GitLab service account. Follow only one of them based on your convenience. #### GitLab access token GitLab access tokens can be used to grant your runner full access to your repository and GitLab API for running your CI/CD jobs. You can create two different kind of tokens, project tokens or group tokens, based on the desired scope. To create a project or group token please follow the steps indicated below, but be reminded that at least you would required a maintainer role: - To create a project token go to https://gitlab.com/MyGroup/MyProject/-/settings/access_tokens or to create a group token go to https://gitlab.com/MyGroup/-/settings/access_tokens - Give a name to your token, e.g. hog-ci - Set or remove *expiration date* - Select a *maintainer* role for the token - Select all the scopes or only the necessary access permissions based on your application - Click on *Create project access token*. After this, your new token will be provided, copy it and be aware that it will not be shown again. If you forget it, you can always delete it and create a new one. - Go to the settings *CI/CD* tab of your project or group and open the *Variables* section (e.g. for your project: https://gitlab.com/MyGroup/MyProject/-/settings/ci_cd) - Create HOG_USER variable and enter your token name - Create HOG_PUSH_TOKEN variable and enter your token In case that you have a project with references to other GitLab projects (submodules), you would need to enable the token access in the project added as a submodule by following the steps below: - Expand the settings tab of your project: https://gitlab.com/MyGroup/MyProject/settings - Expand *CI/CD* - Expand *General pipeline configuration* - Expand *Token Access* - Disable *Allow access to this project with a CI_JOB_TOKEN* option or to avoid any security risk (recommended) you can let this option enabled and add projects to the allow-list only when cross-project access is needed #### GitLab service account At CERN, you can easily request it [at this link](https://account.cern.ch/account/Management/NewAccount.aspx). The service account will run part of the Hog CI. For that, it needs to have access to your local repository. - Go to [https://gitlab.cern.ch/MyGroup/MyProject/-/project_members](https://gitlab.cern.ch/MyGroup/MyProject/-/project_members) and give *Maintainer* rights to your service account - Log in to GitLab with your service account and create a private access token with API rights [here](https://gitlab.cern.ch/profile/personal_access_tokens) Once you have your service account, you should also get 1 TB of space on EOS, that can be used to store the results of Hog CI. If, for some reasons, your service account doesn't have space on EOS, you could request it [here](https://resources.web.cern.ch/resources/Manage/EOS/Default.aspx). On GitLab.com, we suggest to create an other account, different from your main account. ### Set up your personal GitLab CI YAML GitLab CI uses a [YAML configuration file](https://docs.gitlab.com/ee/ci/yaml/) to define which commands it must run. By default this file is called `.gitlab-ci.yml` and must be stored in the root folder of your repository. Hog cannot provide a full YAML file for your project, but a template file can be found under [`Hog` -> `Templates` -> `gitlab-ci.yml` ](https://gitlab.com/hog-cern/Hog/-/blob/master/Templates/gitlab-ci.yml) as a reference. For example, suppose we want to write the `.gitlab-ci.yml` configuration file to run the Hog project `my_project` on the CI. This file will actually include the Hog `hog.yml` configuration file, where the CI stages are defined. To include the reference to the Hog parent file, add at the beginning of your `.gitlab-ci.yml` ```yaml include: - project: 'hog-cern/Hog' file: '/hog.yml' ref: 'vX.Y.Z' ``` Here you must substitute 'vX.Y.Z' with the version of Hog you want to use. The version of Hog **MUST** be specified. If you fail to do so , the CI will pick up the parent configuration file from the latest Hog master branch. This is discouraged, since Hog development could lead to not back-compatible changes that could break your CI. Moreover the pre synthesis script will check that the reference in your `.gitlab-ci.yml` file is consistent with your local Hog submodule, giving a Critical Warning if the two don't match. :::{admonition} Working :class: tip If your project is not stored on gitlab.com, but on another GitLab server (e.g. `gitlab.cern.ch`), you need to check the correct path to be included. First, you need first to mirror the Hog repository (see [here](https://docs.gitlab.com/ee/user/project/repository/repository_mirroring.html)), and then update the yaml file accordingly. We already created mirrors in some GitLab server, for example, at CERN one shall just include in the `gitlab-ci.yml`, ```yaml include: - project: 'hog/Hog' file: '/hog.yml' ref: 'vX.Y.Z' ``` ::: Now, you need to define the stages you want to run in the CI for our project. Hog CI runs always the stages that are not project-specific (e.g. *Merge*), therefore there is no need to declare them in your file. To add a stage `stage_1` for your `my_project`, use the following syntax: ```YAML stage_1:my_project: extends: .stage_1 variables: extends: .vars VARIABLE: ``` In this snippet, the first line is the stage name, i.e. you are defining a stage named `stage_1:my_project`. The second line tells the script that the stage is an extension of `.stage_1` defined in the parent `hog.yml` file. The third line starts the variable declaration section of the script. Since your script extends `.stage_1`, then it must define the variable used by this script. The line `extends: .vars` informs the variables section that it is an extension of the `.vars` object defined in `hog.yml`. The last line shows how to set the value for one named `VARIABLE` defined in the `.vars` object. So, for example, if you want to add a *Generate* stage for your `my_project`, you should add to the `.gitlab-ci.yml`, the following lines: ```yaml generate_project:my_project: extends: .generate_project variables: extends: .vars PROJECT_NAME: my_project ``` A more detailed description of the CI stages and their YAML configuration can be found [here](03-HOG-YAML.md) ### Remove merge commit - Go to https://gitlab.com/MyGroup/MyProject/edit - Expand __Merge Request settings__ - Select *Fast-forward merge* ### Pipeline configuration - Go to https://gitlab.com/MyGroup/MyProject/-/settings/ci_cd - Expand _General pipelines_ - Select *git clone* - Set *Git shallow clone* to 0 - Set *Timeout* to a long threshold, for example 1d ### Set-up Runners Unfortunately, we cannot use shared runners for every CI job, as the necessary software (Xilinx Vivado, Mentor Graphics Questasim, etc.) are not available. The download, installation and licensing processes would have to be done at each time that the CI is started, slowing down the entire process. As a consequence, you need to set-up your own physical or virtual machines. Please refer to [Setting up a Virtual Machines](07-Virtual-Machines.md) section for more information. Once this is done, we can install our gitlab-runner. First, we need to find the gitlab-runner registering token. Open your Gitlab repository on a web browser, and go to `Settings->CI/CD->Runners`. Copy the registration token. ![](../../07-Hog-Tutorial/figures/gitlab-token.png) Go back to your virtual machine, and launch the `install_runner.sh` script, without options to see the Usage description. ```bash Hog - Install gitlab runner --------------------------- Install the Gitlab Runner on a linux machine Usage: ./install_runner.sh -u -g -t [OPTIONS] Options: --tags: , default: hog,vivado --url: . Default: https://gitlab.cern.ch ``` Required arguments for the script are: - the user that will run the Hog-CI. If you work at CERN, we suggest to use a service account. More info [here](https://account.cern.ch/account/Management/NewAccount.aspx)); - The user group. At CERN, this corresponds to your experiment group (e.g. for ATLAS is `zp`); - The Gitlab registering token just obtained. As you can see, it is possible to specify the tags for the runner and the gitlab_url, to create a runner for other Gitlab's locations (e.g. gitlab.com). ```{note} By default, Hog requires the `vivado` and `hog` tags to run its CI. However, it is possible to change the required tags, individually for each CI job. This allows the users to direct the jobs to the desired VMs. ``` Launch now the script with the right options. This will create the home folder for your service account on the VM, and register the gitlab runner. The gitlab runner configuration will be saved on `/etc/gitlab-runner/config.toml`, and will look like this, ```toml concurrent = 1 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "Hog runner on tutorial-hog" output_limit = 20000 url = "https://gitlab.cern.ch" token = "" executor = "shell" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] ``` Here you can change the concurrent value, to set the maximum numbers of gitlab CI jobs this machine will run in parallel. If you go now back to your gitlab repository webpage, under `Settings->CI/CD->Runners`, you should see your new runner under the `Other available runners` list. The Virtual Machine is now ready to run the Hog-CI with GitLab CI/CD. Now take the following actions: - Go to `Settings` -> `CI/CD` - Expand `Runners` - Make sure the shared runners and eventual group runners are enabled for this project - On the left enable the private runners that you have installed on your machines. ### Environment variables - Go to `Settings` -> `CI/CD` - Expand `Variables` - Click on `Add Variable` for each variable you want to add. In the following table, all Hog-CI variables are listed. Please note, that some of them are **required** for the Hog-CI to work, so if any of them is not defined, or defined to a wrong value, Hog-CI will fail. | Name | Required | Value | |--------------------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `HOG_USER` | **YES** | GitLab access token name or service account name depending on your Hog-CI configuration (e.g. `my_access_token` or `my_service_account`). | | `HOG_EMAIL` | **YES** | Your GitLab account or service account's email address (e.g. service_account_mail@cern.ch). | | `HOG_PATH` | **YES** | The `PATH` variable for your VM, should include your ISE bin directory. | | `HOG_PUSH_TOKEN` | **YES** | The push token you generated for your GitLab access token or your service account (should be masked). | | `HOG_TCLLIB_PATH` | NO | **MANDATORY for Libero SoC**. Path of installed `tcllib` libraries on your VMs | | `HOG_OFFICIAL_BIN_PATH` | NO | The path for archiving the official binary files of your project. If it starts with `/eos`, it will copy the binary files into the CERN EOS storage, otherwise on the local machine. | | `EOS_MGM_URL` | NO | Set the EOS instance. If your EOS storage is a user storage use `root://eosuser.cern.ch`. For EOS projects, have a look [here](http://cernbox-manual.web.cern.ch/cernbox-manual/en/project_space/access-to-project-space.html). | | `EOS_USER` | NO | The name of your CERN service account . Only required if you are using the EOS storage cloud space. | | `EOS_PASSWORD` | NO | The password of your CERN service account (should be masked). Only required if you are using the EOS storage cloud space. | | `HOG_XIL_LICENSE` | NO | Should contain the Xilinx license servers, separated by a comma. You can also use the standard Xilinx variables to set up your licenses.[^license] | | `HOG_USE_DOXYGEN` | NO | Should be set to `1`, if you want the Hog CI to create the Doxygen documentation of your project. | | `HOG_CHECK_PROJVER` | NO | Should be set to `1`, if you want that the Hog CI runs only the projects that have been modified with respect to the target branch | | `HOG_CHECK_SYNTAX` | NO | Should be set to `1`, if you want the Hog CI to check the syntax of your code. | | `HOG_CHECK_YAMLREF` | NO | If this variable is set to 1, Hog CI will check that "ref" in `.gitlab-ci.yml` actually matches the gitlab-ci file in the Hog submodule | | `HOG_IP_PATH` | NO | The path where to store the IP generated results. If not set, the CI will synthesise the IPs each time. If the path starts with `/eos/`, Hog will store the products in the specified EOS storage, otherwise on the running machine. | | `HOG_ONLY_SYNTH` | NO | If this variable is set to `1`, Hog-CI will run only the synthesis for all projects in the CI. | | `HOG_MR_MSG` | NO | This variable can be set to 0, 1 (default), 2, 3. It specifies the amount of messages that Hog will write into the pull request page. If set to 0 Hog will not write any message. If set to 1, a single message with a version and resource summary for all the built project is written. If set to 2, a short message declaring that the pipeline is passed is written. If set to 2 Hog will write one message per successful project. | | `HOG_NO_BITSTREAM` | NO | If this variable is set to `1`, Hog-CI runs the implementation but does NOT run the write_bitstream stage | | `HOG_RESET_FILES` | NO | All the files (or wildcards) contained in this variable (separated by spaces or new lines) be reset at the pre-synthesis time by Hog-CI. | | `HOG_NO_RESET_BD` | NO | If this variable is set to `1`, BD files will NOT be reset at the pre-synthesis time by Hog-CI. | | `HOG_CREATE_OFFICIAL_RELEASE` | NO | If this variable is set to `1`, Hog-CI creates an official release note using the version and timing summaries taken from the artefact of the projects. | | `HOG_SIMULATION_LIB_PATH` | NO | The PATH in your VM, where the Simulation Lib files are stored (Vivado only). | | `HOG_TARGET_BRANCH` | NO | Project target branch. Pull request should start from this branch. Default: master. | | `HOG_DEVELOP_BRANCH` | NO | Project develop branch. If defined, pull request should start from this branch. For more information see the [Hog versioning chapter](../02-Hog-Versioning). | | `HOG_INTERMEDIATE_BRANCH` | NO | Intermediate build branch. Merge request targeting this branch activates an intermediate pipeline. More info [here](./03-gitlab-workflow.md#intermediate-build-branches). | | `HOG_NJOBS` | NO | Number of CPU jobs for the synthesis and implementation. Default: 4. | | `HOG_MAX_ZIP_SIZE` | NO | Max size of zip files uploaded to GitLab release. Use zip syntax e.g. 2m for 2 MB. Default: 10m | | `HOG_EXTERNAL_PATH` | NO | If specified, entries in the `.ext` list file would be relative this path | | `HOG_BADGE_PROJECTS` | NO | List of projects, for which Hog will create a GitLab badge in the home page of your GitLab repository, showing the resource utilisation and the timing closure. | | `HOG_APPTAINER_IMAGE` | NO | Path to your `.sif` Apptainer container file, to be used to run the CI. Please, refer to [this chapter](../07a-Containers.md) for more details. | | `HOG_APPTAINER_EXTRA_PATH` | NO | External path to be passed to your Apptainer execution. | :::{note} These variables can be also written inside the `.gitlab-ci.yml` file. GitLab uses the following priority scheme for the environmental variables. `GitLab CI/CD Variables -> Jobs Specific Variables -> Global gitlab-ci.yml variables`. So if, for example, you want to specify a variable that it is same for all jobs expect one, you need to write the common value inside the `.gitlab-ci.yml` in the main variable section and in the job variable sections declare the specific value. ```yaml variables: TEST_VAR: "All jobs can use this variable's value" job1: variables: TEST_VAR_JOB: "Only job1 can use this variable's value" script: - echo "$TEST_VAR" and "$TEST_VAR_JOB" ``` More info [here](https://docs.gitlab.com/ee/ci/variables/). ::: [^license]: Please use the `LM_LICENSE_FILE` variable for Intel, MentorGraphics, etc. ### GitLab badges with resource utilisation and timing (Optional) Hog offers the possibility to automatically create GitLab badges containing the resource utilisation and the timing results for any of the projects in the repository. Two badges will be created for each selected project, one containing the utilisation summary and one containing the timing results: OK if timing was met and NO if it wasn't. To choose which projects are enabled, you must add the names of the projects to the `HOG_BADGE_PROJECTS` environment variable as explained in the table above, separated by a space. ### Archive binary files on EOS space (Optional, CERN-only) The GitLab CI will produce some artefacts. These include the resulting binary files of your firmware projects and, optionally, the Doxygen documentation html files. Hog has also the ability to copy these files into a desired folder, either on the local build machine or, if working at CERN, on the EOS cloud storage. To enable this feature, we have to specify the CI/CD `HOG_OFFICIAL_BIN_PATH` variable. If this variable starts with `/eos`, Hog will understand that it is a EOS folder and will copy the binary there using the [`eos` client software](https://eos-docs.web.cern.ch/diopside/manual/hardware-installation.html#installation), which must be installed on your local machine. To work, you need also to setup the `EOS_MGM_URL` CI/CD variable accordingly. If you wish to have your files to be accessible in a web browser, you should create a web page in EOS, following [these instructions](http://cernbox-manual.web.cern.ch/cernbox-manual/en/web/). For a personal project, by default, the website will be stored in `/eos/user///www`. The Hog EOS paths must be then sub-folders of the website root path. To expose the files in the website, follow these [instructions](http://cernbox-manual.web.cern.ch/cernbox-manual/en/web/expose_files_in_website.html).