## Setting up Hog Continuous Integration ```{warning} Outdated Documentation! This documentation version is out of date. Please check the [latest version 2026.1](https://hog.readthedocs.io/en/latest/). ``` Hog Continuous Integration makes use of the [GitLab CI/CD tool](https://docs.gitLab.com/ee/ci/). Both the GitLab repository and your local area must be set-up to work properly with Hog CI. In this paragraph, we assume that we are working with a GitLab Project called `MyProject` under the GitLab group `MyGroup`. Please, replace these with the actual names of your project and group. ### Preliminary requirements To run the Hog-CI, you need a GitLab service account. At CERN, you can easily request it [at this link](https://account.cern.ch/account/Management/NewAccount.aspx). The service account will run part of the Hog CI. For that, it needs to have access to your local repository. - Go to [https://gitlab.cern.ch/MyGroup/MyProject/-/project_members](https://gitlab.cern.ch/MyGroup/MyProject/-/project_members) and give *Maintainer* rights to your service account - Log in to GitLab with your service account and create a private access token with API rights [here](https://gitlab.cern.ch/profile/personal_access_tokens) Once you have your service account, you should also get 1 TB of space on EOS, that can be used to store the results of Hog CI. If, for some reasons, your service account doesn't have space on EOS, you could request it [here](https://resources.web.cern.ch/resources/Manage/EOS/Default.aspx). ### Set up your personal GitLab CI YAML GitLab CI uses a [YAML configuration file](https://docs.gitlab.com/ee/ci/yaml/) to define which commands it must run. By default this file is called `.gitlab-ci.yml` and must be stored in the root folder of your repository. Hog cannot provide a full YAML file for your project, but a template file can be found under [`Hog` -> `Templates` -> `gitlab-ci.yml` ](https://gitlab.cern.ch/hog/Hog/-/blob/master/Templates/gitlab-ci.yml) as a reference. For example, suppose we want to write the `.gitlab-ci.yml` configuration file to run the Hog project `my_project` on the CI. This file will actually include the Hog `hog.yml` configuration file, where the CI stages are defined. To include the reference to the Hog parent file, add at the beginning of your `.gitlab-ci.yml` ```yaml include: - project: 'hog/Hog' file: '/hog.yml' ref: 'vX.Y.Z' ``` Here you must substitute 'vX.Y.Z' with the version of Hog you want to use. The version of Hog **MUST** be specified. If you fail to do so , the CI will pick up the parent configuration file from the latest Hog master branch. This is discouraged, since Hog development could lead to not back-compatible changes that could break your CI. Moreover the pre synthesis script will check that the reference in your `.gitlab-ci.yml` file is consistent with your local Hog submodule, giving a Critical Warning if the two don't match. :::{admonition} Working outside CERN :class: tip If your project is not stored in the CERN GitLab, you need to check to correct path to be included. For example, if you are working on gitlab.com the top of the `gitlab-ci.yml` looks like ```yaml include: - project: 'hog-cern/Hog' file: '/hog.yml' ref: 'vX.Y.Z' ``` For other GitLab providers, you need first to mirror the Hog repository (see [here](https://docs.gitlab.com/ee/user/project/repository/repository_mirroring.html)), and then update the yaml file accordingly. ::: Now, you need to define the stages you want to run in the CI for our project. Hog CI runs always the stages that are not project-specific (e.g. *Merge*), therefore there is no need to declare them in your file. To add a stage `stage_1` for your `my_project`, use the following syntax: ```YAML stage_1:my_project: extends: .stage_1 variables: extends: .vars VARIABLE: ``` In this snippet, the first line is the stage name, i.e. you are defining a stage named `stage_1:my_project`. The second line tells the script that the stage is an extension of `.stage_1` defined in the parent `hog.yml` file. The third line starts the variable declaration section of the script. Since your script extends `.stage_1`, then it must define the variable used by this script. The line `extends: .vars` informs the variables section that it is an extension of the `.vars` object defined in `hog.yml`. The last line shows how to set the value for one named `VARIABLE` defined in the `.vars` object. So, for example, if you want to add a *Generate* stage for your `my_project`, you should add to the `.gitlab-ci.yml`, the following lines: ```yaml generate_project:my_project: extends: .generate_project variables: extends: .vars PROJECT_NAME: my_project ``` A more detailed description of the CI stages and their YAML configuration can be found [here](04-HOG-YAML.md) ### Remove merge commit - Go to https://gitlab.cern.ch/MyGroup/MyProject/edit - Expand __Merge Request settings__ - Select *Fast-forward merge* ### Pipeline configuration - Go to https://gitlab.cern.ch/MyGroup/MyProject/-/settings/ci_cd - Expand _General pipelines_ - Select *git clone* - Set *Git shallow clone* to 0 - Set *Timeout* to a long threshold, for example 1d ### Set-up Runners Unfortunately, we cannot use shared runners for every CI job, as the necessary software (Xilinx Vivado, Mentor Graphics Questasim, etc.) are not available. The download, installation and licensing processes would have to be done at each time that the CI is started, slowing down the entire process. As a consequence, you need to set-up your own physical or virtual machines. Please refer to [Setting up a Virtual Machines](07-Virtual-Machines.md) section for more information. Now take the following actions: - Go to `Settings` -> `CI/CD` - Expand `Runners` - Make sure the shared runners and eventual group runners are enabled for this project - On the left enable the private runners that you have installed on your machines. ### Environment variables - Go to `Settings` -> `CI/CD` - Expand `Variables` The following variables are **needed** for Hog-CI to work, so if any of them is not defined, or defined to a wrong value, Hog-CI will fail. | Name | Value | |--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | __HOG_USER__ | Your service account name (e.g. my_service_account) | | __HOG_EMAIL__ | Your service account's email address (e.g. service_account_mail@cern.ch) | | __HOG_PATH__ | The PATH variable for your VM, should include Vivado bin directory | | __HOG_PUSH_TOKEN__ | The push token you generated for your service account (should be masked) | | __HOG_COMPILER__ | **Mandatory for Quartus and planAhead.** The IDE to be used for project compilation. If multiple IDEs are needed you can separate them with `;`. Supported options are vivado (default), quartus, planAhead. | With the following **optional** variables you can configure the behaviour of Hog-CI: | Name | Description | |---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | __HOG_PASSWORD__ | The password of your service account (should be masked). Only required if you are using the EOS storage cloud space | | __HOG_XIL_LICENSE__[^license] | Should contain the Xilinx license servers, separated by a comma. You can also use the standard Xilinx variables to set up your licenses. | | __EOS_MGM_URL__ | Set the EOS instance. If your EOS storage is a user storage use `root://eosuser.cern.ch`. For EOS projects, have a look [here](http://cernbox-manual.web.cern.ch/cernbox-manual/en/project_space/access-to-project-space.html) | | __HOG_OFFICIAL_BIN_EOS_PATH__ | The EOS path for archiving the official binary files of your project | | __HOG_USE_DOXYGEN__ | Should be set to 1, if you want the Hog CI to create the doxygen documentation of your project | | __HOG_CHECK_PROJVER__ | Should be set to 1, if you want that the Hog CI runs only the projects that have been modified with respect to the target branch | | __HOG_CHECK_SYNTAX__ | Should be set to 1, if you want the Hog CI to run check syntax | | __HOG_CHECK_YAMLREF__ | If this variable is set to 1, Hog CI will check that "ref" in `.gitlab-ci.yml` actually matches the gitlab-ci file in the Hog submodule | | __HOG_IP_PATH__ | The path where to store the IP generated results. If not set, the CI will synthesise the IPs each time. If the path starts with `/eos/`, Hog will store the products in the specified EOS storage, otherwise on the running machine. | | __HOG_ONLY_SYNTH__ | If this variable is set to 1, Hog-CI will run only the synthesis for all projects in the CI | | __HOG_MR_MSG__ | This variable can be set to 0, 1 (default), 2, 3. It specifies the amount of messages that Hog will write into the merge request page. If set to 0 Hog will not write any message. If set to 1, a single message with a version and resource summary for all the built project is written. If set to 2, a short message declaring that the pipeline is passed is written. If set to 2 Hog will write one message per successful project. | | __HOG_NO_BITSTREAM__ | If this variable is set to 1, Hog-CI runs the implementation but does NOT run the write_bitstream stage | | __HOG_RESET_FILES__ | All the files (or wildcards) contained in this variable (separated by spaces or new lines) be reset at the pre-synthesis time by Hog-CI | | __HOG_NO_RESET_BD__ | If this variable is set to 1, BD files will NOT be reset at the pre-synthesis time by Hog-CI | | __HOG_CREATE_OFFICIAL_RELEASE__ | If this variable is set to 1, Hog-CI creates an official release note using the version and timing summaries taken from the artifact of the projects. | | __HOG_SIMULATION_LIB_PATH__ | The PATH in your VM, where the Simulation Lib files are stored (Vivado only) | | __HOG_TARGET_BRANCH__ | Project target branch. Merge request should start from this branch. Default: master | | __HOG_INTERMEDIATE_BRANCH__ | Intermediate build branch. Merge request targeting this branch activates an intermediate pipeline. More info [here](./03-gitlab-workflow.md#intermediate-build-branches). | | __HOG_NJOBS__ | Number of CPU jobs for the synthesis and implementation. Default: 4 | | __HOG_MAX_ZIP_SIZE__ | Max size of zip files uploaded to gitlab release. Use zip syntax e.g. 2m for 2 MB. Default: 10m | | __HOG_EXTERNAL_PATH__ | If specified, entries in the `.ext` list file would be relative to this path | :::{note} These variables can be also written inside the `.gitlab-ci.yml` file. Gitlab uses the following priority scheme for the environmental variables. `Gitlab CI/CD Variables -> Jobs Specific Variables -> Global gitlab-ci.yml variables`. So if, for example, you want to specify a variable that it is same for all jobs expect one, you need to write the common value inside the `.gitlab-ci.yml` in the main variable section and in the job variable sections declare the specific value. ```yaml variables: TEST_VAR: "All jobs can use this variable's value" job1: variables: TEST_VAR_JOB: "Only job1 can use this variable's value" script: - echo "$TEST_VAR" and "$TEST_VAR_JOB" ``` More info [here](https://docs.gitlab.com/ee/ci/variables/). ::: [^license]: Please use the `LM_LICENSE_FILE` variable for Intel, MentorGraphics, etc. ### Archive binary files on EOS space (Optional) The Gitlab CI will produce some artefacts. These include the resulting binary files of your firmware projects and, optionally, the Doxygen documentation html files. Hog has also the ability to copy these files into a desired EOS repository. To enable this feature, we have to specify the following environmental variables: __EOS_MGM_URL__, __HOG_OFFICIAL_BIN_EOS_PATH__. If you wish to have your files to be accessible in a web browser, you should create a web page in EOS, following [these instructions](http://cernbox-manual.web.cern.ch/cernbox-manual/en/web/). For a personal project, by default, the website will be stored in `/eos/user///www`. The Hog EOS paths must be then sub-folders of the website root path. To expose the files in the website, follow these [instructions](http://cernbox-manual.web.cern.ch/cernbox-manual/en/web/expose_files_in_website.html). ### Speed up the IP synthesis (Optional) In order to save resources and possibly speed up the workflow, the products of the IP synthesis can be copied into an external directory, and retried before starting the firmware synthesis. The external path is specified via the __HOG_IP_PATH__ variable. The IPs are archived in the IP path using the md5 hash of the `.xci` file, so if an IP is modified, it will be copied again. It is recommended to clean the IP path regularly as Hog will not delete old IP products. :::{admonition} Storing IPs on EOS Until Hog2021.2, the IP could be only stored on EOS using the now deprecated __HOG_IP_EOS_PATH__. Staring from the version 2022.1, Hog will automatically detects if the IP storage path is on EOS or on the running machine, verifying that the path starts with `/eos/` or not. :::