Hog-CI with GitLab CI/CD#

Setting up Hog CI with GitLab CI/CD#

On GitLab repository, Hog Continuous Integration makes use of the GitLab CI/CD tool. Both the GitLab repository and your local area must be set-up to work properly with Hog CI. In this paragraph, we assume that we are working with a GitLab Project called MyProject under the GitLab group MyGroup. Please, replace these with the actual names of your project and group.

Preliminary requirements#

To run the Hog-CI you must create a GitLab access token or create a GitLab service account. Follow only one of them based on your convenience.

GitLab access token#

GitLab access tokens can be used to grant your runner full access to your repository and GitLab API for running your CI/CD jobs. You can create two different kind of tokens, project tokens or group tokens, based on the desired scope.

To create a project or group token please follow the steps indicated below, but be reminded that at least you would required a maintainer role:

To create a project token go to https://gitlab.com/MyGroup/MyProject/-/settings/access_tokens or to create a group token go to https://gitlab.com/MyGroup/-/settings/access_tokens
Give a name to your token, e.g. hog-ci
Set or remove expiration date
Select a maintainer role for the token
Select all the scopes or only the necessary access permissions based on your application
Click on Create project access token. After this, your new token will be provided, copy it and be aware that it will not be shown again. If you forget it, you can always delete it and create a new one.
Go to the settings CI/CD tab of your project or group and open the Variables section (e.g. for your project: https://gitlab.com/MyGroup/MyProject/-/settings/ci_cd)
Create HOG_USER variable and enter your token name
Create HOG_PUSH_TOKEN variable and enter your token

In case that you have a project with references to other GitLab projects (submodules), you would need to enable the token access in the project added as a submodule by following the steps below:

Expand the settings tab of your project: https://gitlab.com/MyGroup/MyProject/settings
Expand CI/CD
Expand General pipeline configuration
Expand Token Access
Disable Allow access to this project with a CI_JOB_TOKEN option or to avoid any security risk (recommended) you can let this option enabled and add projects to the allow-list only when cross-project access is needed

GitLab service account#

At CERN, you can easily request it at this link. The service account will run part of the Hog CI. For that, it needs to have access to your local repository.

Go to https://gitlab.cern.ch/MyGroup/MyProject/-/project_members and give Maintainer rights to your service account
Log in to GitLab with your service account and create a private access token with API rights here

Once you have your service account, you should also get 1 TB of space on EOS, that can be used to store the results of Hog CI. If, for some reasons, your service account doesn’t have space on EOS, you could request it here.

On GitLab.com, we suggest to create an other account, different from your main account.

Set up your personal GitLab CI YAML#

GitLab CI uses a YAML configuration file to define which commands it must run. By default this file is called .gitlab-ci.yml and must be stored in the root folder of your repository. Hog cannot provide a full YAML file for your project, but a template file can be found under Hog -> Templates -> gitlab-ci.yml as a reference. For example, suppose we want to write the .gitlab-ci.yml configuration file to run the Hog project my_project on the CI. This file will actually include the Hog hog.yml configuration file, where the CI stages are defined. To include the reference to the Hog parent file, add at the beginning of your .gitlab-ci.yml

include:
    - project: 'hog-cern/Hog'
      file: '/hog.yml'
      ref: 'vX.Y.Z'

Here you must substitute ‘vX.Y.Z’ with the version of Hog you want to use. The version of Hog MUST be specified. If you fail to do so , the CI will pick up the parent configuration file from the latest Hog master branch. This is discouraged, since Hog development could lead to not back-compatible changes that could break your CI. Moreover the pre synthesis script will check that the reference in your .gitlab-ci.yml file is consistent with your local Hog submodule, giving a Critical Warning if the two don’t match.

Working

If your project is not stored on gitlab.com, but on another GitLab server (e.g. gitlab.cern.ch), you need to check the correct path to be included. First, you need first to mirror the Hog repository (see here), and then update the yaml file accordingly.

We already created mirrors in some GitLab server, for example, at CERN one shall just include in the gitlab-ci.yml,

include:
    - project: 'hog/Hog'
      file: '/hog.yml'
      ref: 'vX.Y.Z'

Now, you need to define the stages you want to run in the CI for our project. Hog CI runs always the stages that are not project-specific (e.g. Merge), therefore there is no need to declare them in your file. To add a stage stage_1 for your my_project, use the following syntax:

  stage_1:my_project:
    extends: .stage_1
    variables:
      extends: .vars
      VARIABLE: <variable_value>

In this snippet, the first line is the stage name, i.e. you are defining a stage named stage_1:my_project. The second line tells the script that the stage is an extension of .stage_1 defined in the parent hog.yml file. The third line starts the variable declaration section of the script. Since your script extends .stage_1, then it must define the variable used by this script. The line extends: .vars informs the variables section that it is an extension of the .vars object defined in hog.yml. The last line shows how to set the value for one named VARIABLE defined in the .vars object.

So, for example, if you want to add a Generate stage for your my_project, you should add to the .gitlab-ci.yml, the following lines:

  generate_project:my_project:
    extends: .generate_project
    variables:
      extends: .vars
      PROJECT_NAME: my_project

A more detailed description of the CI stages and their YAML configuration can be found here

Remove merge commit#

Go to https://gitlab.com/MyGroup/MyProject/edit
Expand Merge Request settings
Select Fast-forward merge

Pipeline configuration#

Go to https://gitlab.com/MyGroup/MyProject/-/settings/ci_cd
Expand General pipelines
Select git clone
Set Git shallow clone to 0
Set Timeout to a long threshold, for example 1d

Set-up Runners#

Unfortunately, we cannot use shared runners for every CI job, as the necessary software (Xilinx Vivado, Mentor Graphics Questasim, etc.) are not available. The download, installation and licensing processes would have to be done at each time that the CI is started, slowing down the entire process. As a consequence, you need to set-up your own physical or virtual machines. Please refer to Setting up a Virtual Machines section for more information.

Once this is done, we can install our gitlab-runner. First, we need to find the gitlab-runner registering token.

Open your Gitlab repository on a web browser, and go to Settings->CI/CD->Runners. Copy the registration token.

Go back to your virtual machine, and launch the install_runner.sh script, without options to see the Usage description.

 Hog - Install gitlab runner
 ---------------------------
 Install the Gitlab Runner on a linux machine

 Usage: ./install_runner.sh -u <user_name> -g <user_group> -t <gitlab_token> [OPTIONS]
 Options: --tags: <list of tags for this VM>, default: hog,vivado
          --url: <Gitlab url>. Default: https://gitlab.cern.ch

Required arguments for the script are:

the user that will run the Hog-CI. If you work at CERN, we suggest to use a service account. More info here);
The user group. At CERN, this corresponds to your experiment group (e.g. for ATLAS is zp);
The Gitlab registering token just obtained.

As you can see, it is possible to specify the tags for the runner and the gitlab_url, to create a runner for other Gitlab’s locations (e.g. gitlab.com).

Note

By default, Hog requires the vivado and hog tags to run its CI. However, it is possible to change the required tags, individually for each CI job. This allows the users to direct the jobs to the desired VMs.

Launch now the script with the right options. This will create the home folder for your service account on the VM, and register the gitlab runner. The gitlab runner configuration will be saved on /etc/gitlab-runner/config.toml, and will look like this,

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Hog runner on tutorial-hog"
  output_limit = 20000
  url = "https://gitlab.cern.ch"
  token = "<my_token>"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Here you can change the concurrent value, to set the maximum numbers of gitlab CI jobs this machine will run in parallel.

If you go now back to your gitlab repository webpage, under Settings->CI/CD->Runners, you should see your new runner under the Other available runners list.

The Virtual Machine is now ready to run the Hog-CI with GitLab CI/CD.

Now take the following actions:

Go to Settings -> CI/CD
Expand Runners
Make sure the shared runners and eventual group runners are enabled for this project
On the left enable the private runners that you have installed on your machines.

Environment variables#

Go to Settings -> CI/CD
Expand Variables
Click on Add Variable for each variable you want to add.

In the following table, all Hog-CI variables are listed. Please note, that some of them are required for the Hog-CI to work, so if any of them is not defined, or defined to a wrong value, Hog-CI will fail.

Name	Required	Value
`HOG_USER`	YES	GitLab access token name or service account name depending on your Hog-CI configuration (e.g. `my_access_token` or `my_service_account`).
`HOG_EMAIL`	YES	Your GitLab account or service account’s email address (e.g. service_account_mail@cern.ch).
`HOG_PATH`	YES	The `PATH` variable for your VM, should include your ISE bin directory.
`HOG_PUSH_TOKEN`	YES	The push token you generated for your GitLab access token or your service account (should be masked).
`HOG_TCLLIB_PATH`	NO	MANDATORY for Libero SoC. Path of installed `tcllib` libraries on your VMs
`HOG_OFFICIAL_BIN_PATH`	NO	The path for archiving the official binary files of your project. If it starts with `/eos`, it will copy the binary files into the CERN EOS storage, otherwise on the local machine.
`EOS_MGM_URL`	NO	Set the EOS instance. If your EOS storage is a user storage use `root://eosuser.cern.ch`. For EOS projects, have a look here.
`EOS_USER`	NO	The name of your CERN service account . Only required if you are using the EOS storage cloud space.
`EOS_PASSWORD`	NO	The password of your CERN service account (should be masked). Only required if you are using the EOS storage cloud space.
`HOG_XIL_LICENSE`	NO	Should contain the Xilinx license servers, separated by a comma. You can also use the standard Xilinx variables to set up your licenses.[1]
`HOG_USE_DOXYGEN`	NO	Should be set to `1`, if you want the Hog CI to create the Doxygen documentation of your project.
`HOG_CHECK_PROJVER`	NO	Should be set to `1`, if you want that the Hog CI runs only the projects that have been modified with respect to the target branch
`HOG_CHECK_SYNTAX`	NO	Should be set to `1`, if you want the Hog CI to check the syntax of your code.
`HOG_CHECK_YAMLREF`	NO	If this variable is set to 1, Hog CI will check that “ref” in `.gitlab-ci.yml` actually matches the gitlab-ci file in the Hog submodule
`HOG_IP_PATH`	NO	The path where to store the IP generated results. If not set, the CI will synthesise the IPs each time. If the path starts with `/eos/`, Hog will store the products in the specified EOS storage, otherwise on the running machine.
`HOG_ONLY_SYNTH`	NO	If this variable is set to `1`, Hog-CI will run only the synthesis for all projects in the CI.
`HOG_MR_MSG`	NO	This variable can be set to 0, 1 (default), 2, 3. It specifies the amount of messages that Hog will write into the pull request page. If set to 0 Hog will not write any message. If set to 1, a single message with a version and resource summary for all the built project is written. If set to 2, a short message declaring that the pipeline is passed is written. If set to 2 Hog will write one message per successful project.
`HOG_NO_BITSTREAM`	NO	If this variable is set to `1`, Hog-CI runs the implementation but does NOT run the write_bitstream stage
`HOG_RESET_FILES`	NO	All the files (or wildcards) contained in this variable (separated by spaces or new lines) be reset at the pre-synthesis time by Hog-CI.
`HOG_NO_RESET_BD`	NO	If this variable is set to `1`, BD files will NOT be reset at the pre-synthesis time by Hog-CI.
`HOG_CREATE_OFFICIAL_RELEASE`	NO	If this variable is set to `1`, Hog-CI creates an official release note using the version and timing summaries taken from the artefact of the projects.
`HOG_SIMULATION_LIB_PATH`	NO	The PATH in your VM, where the Simulation Lib files are stored (Vivado only).
`HOG_TARGET_BRANCH`	NO	Project target branch. Pull request should start from this branch. Default: master.
`HOG_DEVELOP_BRANCH`	NO	Project develop branch. If defined, pull request should start from this branch. For more information see the Hog versioning chapter.
`HOG_INTERMEDIATE_BRANCH`	NO	Intermediate build branch. Merge request targeting this branch activates an intermediate pipeline. More info here.
`HOG_NJOBS`	NO	Number of CPU jobs for the synthesis and implementation. Default: 4.
`HOG_MAX_ZIP_SIZE`	NO	Max size of zip files uploaded to GitLab release. Use zip syntax e.g. 2m for 2 MB. Default: 10m
`HOG_EXTERNAL_PATH`	NO	If specified, entries in the `.ext` list file would be relative this path
`HOG_BADGE_PROJECTS`	NO	List of projects, for which Hog will create a GitLab badge in the home page of your GitLab repository, showing the resource utilisation and the timing closure.
`HOG_APPTAINER_IMAGE`	NO	Path to your `.sif` Apptainer container file, to be used to run the CI. Please, refer to this chapter for more details.
`HOG_APPTAINER_EXTRA_PATH`	NO	External path to be passed to your Apptainer execution.

Note

These variables can be also written inside the .gitlab-ci.yml file. GitLab uses the following priority scheme for the environmental variables.

GitLab CI/CD Variables -> Jobs Specific Variables -> Global gitlab-ci.yml variables.

So if, for example, you want to specify a variable that it is same for all jobs expect one, you need to write the common value inside the .gitlab-ci.yml in the main variable section and in the job variable sections declare the specific value.

variables:
  TEST_VAR: "All jobs can use this variable's value"

job1:
  variables:
    TEST_VAR_JOB: "Only job1 can use this variable's value"
  script:
    - echo "$TEST_VAR" and "$TEST_VAR_JOB"  

More info here.

GitLab badges with resource utilisation and timing (Optional)#

Hog offers the possibility to automatically create GitLab badges containing the resource utilisation and the timing results for any of the projects in the repository.

Two badges will be created for each selected project, one containing the utilisation summary and one containing the timing results: OK if timing was met, FAILED if it was not, or UNKNOWN if the implementation has not run. To choose which projects are enabled, you must add the names of the projects to the HOG_BADGE_PROJECTS environment variable as explained in the table above, separated by a space.

Archive binary files on EOS space (Optional, CERN-only)#

The GitLab CI will produce some artefacts. These include the resulting binary files of your firmware projects and, optionally, the Doxygen documentation html files. Hog has also the ability to copy these files into a desired folder, either on the local build machine or, if working at CERN, on the EOS cloud storage.

To enable this feature, we have to specify the CI/CD HOG_OFFICIAL_BIN_PATH variable. If this variable starts with /eos, Hog will understand that it is a EOS folder and will copy the binary there using the eos client software, which must be installed on your local machine. To work, you need also to setup the EOS_MGM_URL CI/CD variable accordingly.

If you wish to have your files to be accessible in a web browser, you should create a web page in EOS, following these instructions. For a personal project, by default, the website will be stored in /eos/user/<initial>/<userID>/www. The Hog EOS paths must be then sub-folders of the website root path. To expose the files in the website, follow these instructions.