Hog-CI with GitLab CI/CD#

Setting up Hog CI with GitLab CI/CD#

On GitLab repository, Hog Continuous Integration makes use of the GitLab CI/CD tool. Both the GitLab repository and your local area must be set-up to work properly with Hog CI. In this paragraph, we assume that we are working with a GitLab Project called MyProject under the GitLab group MyGroup. Please, replace these with the actual names of your project and group.

Preliminary requirements#

To run the Hog-CI you must create a GitLab access token or create a GitLab service account. Follow only one of them based on your convenience.

GitLab access token#

GitLab access tokens can be used to grant your runner full access to your repository and GitLab API for running your CI/CD jobs. You can create two different kind of tokens, project tokens or group tokens, based on the desired scope.

To create a project or group token please follow the steps indicated below, but be reminded that at least you would required a maintainer role:

  • To create a project token go to https://gitlab.com/MyGroup/MyProject/-/settings/access_tokens or to create a group token go to https://gitlab.com/MyGroup/-/settings/access_tokens

  • Give a name to your token, e.g. hog-ci

  • Set or remove expiration date

  • Select a maintainer role for the token

  • Select all the scopes or only the necessary access permissions based on your application

  • Click on Create project access token. After this, your new token will be provided, copy it and be aware that it will not be shown again. If you forget it, you can always delete it and create a new one.

  • Go to the settings CI/CD tab of your project or group and open the Variables section (e.g. for your project: https://gitlab.com/MyGroup/MyProject/-/settings/ci_cd)

  • Create HOG_USER variable and enter your token name

  • Create HOG_PUSH_TOKEN variable and enter your token

In case that you have a project with references to other GitLab projects (submodules), you would need to enable the token access in the project added as a submodule by following the steps below:

  • Expand the settings tab of your project: https://gitlab.com/MyGroup/MyProject/settings

  • Expand CI/CD

  • Expand General pipeline configuration

  • Expand Token Access

  • Disable Allow access to this project with a CI_JOB_TOKEN option or to avoid any security risk (recommended) you can let this option enabled and add projects to the allow-list only when cross-project access is needed

GitLab service account#

At CERN, you can easily request it at this link. The service account will run part of the Hog CI. For that, it needs to have access to your local repository.

Once you have your service account, you should also get 1 TB of space on EOS, that can be used to store the results of Hog CI. If, for some reasons, your service account doesn’t have space on EOS, you could request it here.

On GitLab.com, we suggest to create an other account, different from your main account.

Set up your personal GitLab CI YAML#

GitLab CI uses a YAML configuration file to define which commands it must run. By default this file is called .gitlab-ci.yml and must be stored in the root folder of your repository. Hog cannot provide a full YAML file for your project, but a template file can be found under Hog -> Templates -> gitlab-ci.yml as a reference. For example, suppose we want to write the .gitlab-ci.yml configuration file to run the Hog project my_project on the CI. This file will actually include the Hog hog.yml configuration file, where the CI stages are defined. To include the reference to the Hog parent file, add at the beginning of your .gitlab-ci.yml

include:
    - project: 'hog-cern/Hog'
      file: '/hog.yml'
      ref: 'vX.Y.Z'

Here you must substitute ‘vX.Y.Z’ with the version of Hog you want to use. The version of Hog MUST be specified. If you fail to do so , the CI will pick up the parent configuration file from the latest Hog master branch. This is discouraged, since Hog development could lead to not back-compatible changes that could break your CI. Moreover the pre synthesis script will check that the reference in your .gitlab-ci.yml file is consistent with your local Hog submodule, giving a Critical Warning if the two don’t match.

Working

If your project is not stored on gitlab.com, but on another GitLab server (e.g. gitlab.cern.ch), you need to check the correct path to be included. First, you need first to mirror the Hog repository (see here), and then update the yaml file accordingly.

We already created mirrors in some GitLab server, for example, at CERN one shall just include in the gitlab-ci.yml,

include:
    - project: 'hog/Hog'
      file: '/hog.yml'
      ref: 'vX.Y.Z'

Now, you need to define the stages you want to run in the CI for our project. Hog CI runs always the stages that are not project-specific (e.g. Merge), therefore there is no need to declare them in your file. To add a stage stage_1 for your my_project, use the following syntax:

  stage_1:my_project:
    extends: .stage_1
    variables:
      extends: .vars
      VARIABLE: <variable_value>

In this snippet, the first line is the stage name, i.e. you are defining a stage named stage_1:my_project. The second line tells the script that the stage is an extension of .stage_1 defined in the parent hog.yml file. The third line starts the variable declaration section of the script. Since your script extends .stage_1, then it must define the variable used by this script. The line extends: .vars informs the variables section that it is an extension of the .vars object defined in hog.yml. The last line shows how to set the value for one named VARIABLE defined in the .vars object.

So, for example, if you want to add a Generate stage for your my_project, you should add to the .gitlab-ci.yml, the following lines:

  generate_project:my_project:
    extends: .generate_project
    variables:
      extends: .vars
      PROJECT_NAME: my_project

A more detailed description of the CI stages and their YAML configuration can be found here

Remove merge commit#

  • Go to https://gitlab.com/MyGroup/MyProject/edit

  • Expand Merge Request settings

  • Select Fast-forward merge

../../../_images/fast-forward.png

Pipeline configuration#

  • Go to https://gitlab.com/MyGroup/MyProject/-/settings/ci_cd

  • Expand General pipelines

  • Select git clone

  • Set Git shallow clone to 0

  • Set Timeout to a long threshold, for example 1d

../../../_images/pipeline.png

Set-up Runners#

Unfortunately, we cannot use shared runners for every CI job, as the necessary software (Xilinx Vivado, Mentor Graphics Questasim, etc.) are not available. The download, installation and licensing processes would have to be done at each time that the CI is started, slowing down the entire process. As a consequence, you need to set-up your own physical or virtual machines. Please refer to Setting up a Virtual Machines section for more information.

Once this is done, we can install our gitlab-runner. First, we need to find the gitlab-runner registering token.

Open your Gitlab repository on a web browser, and go to Settings->CI/CD->Runners. Copy the registration token.

Go back to your virtual machine, and launch the install_runner.sh script, without options to see the Usage description.

 Hog - Install gitlab runner
 ---------------------------
 Install the Gitlab Runner on a linux machine

 Usage: ./install_runner.sh -u <user_name> -g <user_group> -t <gitlab_token> [OPTIONS]
 Options: --tags: <list of tags for this VM>, default: hog,vivado
          --url: <Gitlab url>. Default: https://gitlab.cern.ch

Required arguments for the script are:

  • the user that will run the Hog-CI. If you work at CERN, we suggest to use a service account. More info here);

  • The user group. At CERN, this corresponds to your experiment group (e.g. for ATLAS is zp);

  • The Gitlab registering token just obtained.

As you can see, it is possible to specify the tags for the runner and the gitlab_url, to create a runner for other Gitlab’s locations (e.g. gitlab.com).

Note

By default, Hog requires the vivado and hog tags to run its CI. However, it is possible to change the required tags, individually for each CI job. This allows the users to direct the jobs to the desired VMs.

Launch now the script with the right options. This will create the home folder for your service account on the VM, and register the gitlab runner. The gitlab runner configuration will be saved on /etc/gitlab-runner/config.toml, and will look like this,

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Hog runner on tutorial-hog"
  output_limit = 20000
  url = "https://gitlab.cern.ch"
  token = "<my_token>"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Here you can change the concurrent value, to set the maximum numbers of gitlab CI jobs this machine will run in parallel.

If you go now back to your gitlab repository webpage, under Settings->CI/CD->Runners, you should see your new runner under the Other available runners list.

The Virtual Machine is now ready to run the Hog-CI with GitLab CI/CD.

Now take the following actions:

  • Go to Settings -> CI/CD

  • Expand Runners

  • Make sure the shared runners and eventual group runners are enabled for this project

  • On the left enable the private runners that you have installed on your machines.

Environment variables#

  • Go to Settings -> CI/CD

  • Expand Variables

  • Click on Add Variable for each variable you want to add.

In the following table, all Hog-CI variables are listed. Please note, that some of them are required for the Hog-CI to work, so if any of them is not defined, or defined to a wrong value, Hog-CI will fail.

Name

Required

Value

HOG_USER

YES

GitLab access token name or service account name depending on your Hog-CI configuration (e.g. my_access_token or my_service_account).

HOG_EMAIL

YES

Your GitLab account or service account’s email address (e.g. service_account_mail@cern.ch).

HOG_PATH

YES

The PATH variable for your VM, should include your ISE bin directory.

HOG_PUSH_TOKEN

YES

The push token you generated for your GitLab access token or your service account (should be masked).

HOG_TCLLIB_PATH

NO

MANDATORY for Libero SoC. Path of installed tcllib libraries on your VMs

HOG_OFFICIAL_BIN_PATH

NO

The path for archiving the official binary files of your project. If it starts with /eos, it will copy the binary files into the CERN EOS storage, otherwise on the local machine.

EOS_MGM_URL

NO

Set the EOS instance. If your EOS storage is a user storage use root://eosuser.cern.ch. For EOS projects, have a look here.

EOS_USER

NO

The name of your CERN service account . Only required if you are using the EOS storage cloud space.

EOS_PASSWORD

NO

The password of your CERN service account (should be masked). Only required if you are using the EOS storage cloud space.

HOG_XIL_LICENSE

NO

Should contain the Xilinx license servers, separated by a comma. You can also use the standard Xilinx variables to set up your licenses.1

HOG_USE_DOXYGEN

NO

Should be set to 1, if you want the Hog CI to create the Doxygen documentation of your project.

HOG_CHECK_PROJVER

NO

Should be set to 1, if you want that the Hog CI runs only the projects that have been modified with respect to the target branch

HOG_CHECK_SYNTAX

NO

Should be set to 1, if you want the Hog CI to check the syntax of your code.

HOG_CHECK_YAMLREF

NO

If this variable is set to 1, Hog CI will check that “ref” in .gitlab-ci.yml actually matches the gitlab-ci file in the Hog submodule

HOG_IP_PATH

NO

The path where to store the IP generated results. If not set, the CI will synthesise the IPs each time. If the path starts with /eos/, Hog will store the products in the specified EOS storage, otherwise on the running machine.

HOG_ONLY_SYNTH

NO

If this variable is set to 1, Hog-CI will run only the synthesis for all projects in the CI.

HOG_MR_MSG

NO

This variable can be set to 0, 1 (default), 2, 3. It specifies the amount of messages that Hog will write into the pull request page. If set to 0 Hog will not write any message. If set to 1, a single message with a version and resource summary for all the built project is written. If set to 2, a short message declaring that the pipeline is passed is written. If set to 2 Hog will write one message per successful project.

HOG_NO_BITSTREAM

NO

If this variable is set to 1, Hog-CI runs the implementation but does NOT run the write_bitstream stage

HOG_RESET_FILES

NO

All the files (or wildcards) contained in this variable (separated by spaces or new lines) be reset at the pre-synthesis time by Hog-CI.

HOG_NO_RESET_BD

NO

If this variable is set to 1, BD files will NOT be reset at the pre-synthesis time by Hog-CI.

HOG_CREATE_OFFICIAL_RELEASE

NO

If this variable is set to 1, Hog-CI creates an official release note using the version and timing summaries taken from the artefact of the projects.

HOG_SIMULATION_LIB_PATH

NO

The PATH in your VM, where the Simulation Lib files are stored (Vivado only).

HOG_TARGET_BRANCH

NO

Project target branch. Pull request should start from this branch. Default: master.

HOG_DEVELOP_BRANCH

NO

Project develop branch. If defined, pull request should start from this branch. For more information see the Hog versioning chapter.

HOG_INTERMEDIATE_BRANCH

NO

Intermediate build branch. Merge request targeting this branch activates an intermediate pipeline. More info here.

HOG_NJOBS

NO

Number of CPU jobs for the synthesis and implementation. Default: 4.

HOG_MAX_ZIP_SIZE

NO

Max size of zip files uploaded to GitLab release. Use zip syntax e.g. 2m for 2 MB. Default: 10m

HOG_EXTERNAL_PATH

NO

If specified, entries in the .ext list file would be relative this path

HOG_BADGE_PROJECTS

NO

List of projects, for which Hog will create a GitLab badge in the home page of your GitLab repository, showing the resource utilisation and the timing closure.

HOG_APPTAINER_IMAGE

NO

Path to your .sif Apptainer container file, to be used to run the CI. Please, refer to this chapter for more details.

HOG_APPTAINER_EXTRA_PATH

NO

External path to be passed to your Apptainer execution.

Note

These variables can be also written inside the .gitlab-ci.yml file. GitLab uses the following priority scheme for the environmental variables.

GitLab CI/CD Variables -> Jobs Specific Variables -> Global gitlab-ci.yml variables.

So if, for example, you want to specify a variable that it is same for all jobs expect one, you need to write the common value inside the .gitlab-ci.yml in the main variable section and in the job variable sections declare the specific value.

variables:
  TEST_VAR: "All jobs can use this variable's value"

job1:
  variables:
    TEST_VAR_JOB: "Only job1 can use this variable's value"
  script:
    - echo "$TEST_VAR" and "$TEST_VAR_JOB"  

More info here.

GitLab badges with resource utilisation and timing (Optional)#

Hog offers the possibility to automatically create GitLab badges containing the resource utilisation and the timing results for any of the projects in the repository.

../../../_images/badges.png

Two badges will be created for each selected project, one containing the utilisation summary and one containing the timing results: OK if timing was met, FAILED if it was not, or UNKNOWN if the implementation has not run. To choose which projects are enabled, you must add the names of the projects to the HOG_BADGE_PROJECTS environment variable as explained in the table above, separated by a space.

Archive binary files on EOS space (Optional, CERN-only)#

The GitLab CI will produce some artefacts. These include the resulting binary files of your firmware projects and, optionally, the Doxygen documentation html files. Hog has also the ability to copy these files into a desired folder, either on the local build machine or, if working at CERN, on the EOS cloud storage.

To enable this feature, we have to specify the CI/CD HOG_OFFICIAL_BIN_PATH variable. If this variable starts with /eos, Hog will understand that it is a EOS folder and will copy the binary there using the eos client software, which must be installed on your local machine. To work, you need also to setup the EOS_MGM_URL CI/CD variable accordingly.

If you wish to have your files to be accessible in a web browser, you should create a web page in EOS, following these instructions. For a personal project, by default, the website will be stored in /eos/user/<initial>/<userID>/www. The Hog EOS paths must be then sub-folders of the website root path. To expose the files in the website, follow these instructions.


1

Please use the LM_LICENSE_FILE variable for Intel, MentorGraphics, etc.