Setting up Hog Continuous Integration

Hog Continuous Integration makes use of the Gitlab CI/CD tool. Both the Gitlab repository and your local area must be set-up to work properly with Hog CI. In this paragraph, we assume that we are working with a Gitlab Project called MyProject under the Gitlab group MyGroup. Please, replace these with the actual names of your project and group.

Preliminary requirements

To run the Hog-CI, you need a CERN service account. If you don’t have one, you can easily request it here. The service account will run the Hog CI. For that, it needs to have access to your local repository.

Once you have your service account, you should also get 1 TB of space on EOS, that can be used to store the results of Hog CI. If, for some reasons, your service account doesn’t have space on EOS, you could request it here.

Set up your personal Gitlab CI YAML

Gitlab CI uses a YAML configuration file to define which commands it must run. By default this file is called .gitlab-ci.yml and must be stored in the root folder of your repository. Hog cannot provide a full YAML file for your project, but a template file can be found under Hog -> Templates -> gitlab-ci.yml as a reference. For example, suppose we want to write the .gitlab-ci.yml configuration file to run the Hog project my_project on the CI. This file will actually include the Hog hog.yml configuration file, where the CI stages are defined. To include the reference to the Hog parent file, add at the beginning of your .gitlab-ci.yml

    include:
        - project: 'hog/Hog'
        file: '/hog.yml'
        ref: 'vX.Y.Z'

Here you must substitute ‘vX.Y.Z’ with the version of Hog you want to use. The version of Hog MUST be specified. If you fail to do so , the CI will pick up the parent configuration file from the latest Hog master branch. This is discouraged, since Hog development could lead to not back-compatible changes that could break your CI. Moreover the pre synthesis script will check that the reference in your .gitlab-ci.yml file is consistent with your local Hog submodule, giving a Critical Warning if the two don’t match.

Working outside CERN

If your project is not stored in the CERN Gitlab, you need to check to correct path to be included. For example, if you are working on gitlab.com the top of the gitlab-ci.yml looks like

    include:
        - project: 'hog-cern/Hog'
        file: '/hog.yml'
        ref: 'vX.Y.Z'

For other Gitlab providers, you need first to mirror the Hog repository (see here), and then update the yaml file accordingly.

Now, you need to define the stages you want to run in the CI for our project. Hog CI runs always the stages that are not project-specific (e.g. Merge), therefore there is no need to declare them in your file. To add a stage stage_1 for your my_project, use the following syntax:

  stage_1:my_project:
    extends: .stage_1
    variables:
      extends: .vars
      VARIABLE: <variable_value>

In this snippet the first line is the stage name, i.e. you are defining a stage named ‘stage_1:my_project’. The second line tells the script that the stage is an extension of ‘.stage_1’ defined in the parent hog.yml file. The third line starts the variable declaration section of the script. Since your script extends .stage_1, then it must define the variable used by this script. The line extends: .vars informs the variables section that it is an extension of the .vars object defined in hog.yml. The last line shows how to set the value for one named VARIABLE defined in the .vars object.

So, for example, if you want to add a Generate stage for your my_project, you should add to the .gitlab-ci.yml, the following lines:

  generate_project:my_project:
    extends: .generate_project
    variables:
      extends: .vars
      PROJECT_NAME: my_project

A more detailed description of the CI stages and their YAML configuration can be found here

Remove merge commit

  • Go to https://gitlab.cern.ch/MyGroup/MyProject/edit

  • Expand Merge Request settings

  • Select Fast-forward merge

../../_images/fast-forward.png

Pipeline configuration

  • Go to https://gitlab.cern.ch/MyGroup/MyProject/-/settings/ci_cd

  • Expand General pipelines

  • Select git clone

  • Set Git shallow clone to 0

  • Set Timeout to a long threshold, for example 1d

../../_images/pipeline.png

Set-up Runners

Unfortunately, we cannot use shared runners for every CI job, as the necessary software (Xilinx Vivado, Mentor Graphics Questasim, etc.) are not available. The download, installation and licensing processes would have to be done at each time that the CI is started, slowing down the entire process. As a consequence, you need to set-up your own physical or virtual machines. Please refer to Setting up a Virtual Machines section for more information.

Now take the following actions:

  • Go to Settings -> CI/CD

  • Expand Runners

  • Make sure the shared runners and eventual group runners are enabled for this project

  • On the left enable the private runners that you have installed on your machines.

Environment variables

  • Go to Settings -> CI/CD

  • Expand Variables

The following variables are needed for Hog-CI to work, so if any of them is not defined, or defined to a wrong value, Hog-CI will fail.

Name

Value

HOG_USER

Your service account name (e.g. my_service_account)

HOG_EMAIL

Your service account’s email address (e.g. service_account_mail@cern.ch)

HOG_PASSWORD

The password of your service account (should be masked). To be set to NONE if not used.

HOG_PATH

The PATH variable for your VM, should include Vivado bin directory

HOG_PUSH_TOKEN

The push token you generated for your service account (should be masked)

HOG_XIL_LICENSE1

Should contain the Xilinx license servers, separated by a comma. To be set to NONE if not used.

HOG_COMPILER

Mandatory for Quartus end planAhead. The IDE to be used for project compilation. If multiple IDEs are needed you can separate them with ;. Supported options are vivado (default), quartus, planAhead.

With the following optional variables you can configure the behaviour of Hog-CI:

Name

Description

EOS_MGM_URL

Set the EOS instance. If your EOS storage is a user storage use root://eosuser.cern.ch. For EOS projects, have a look here

HOG_OFFICIAL_BIN_EOS_PATH

The EOS path for archiving the official binary files of your project

HOG_USE_DOXYGEN

Should be set to 1, if you want the Hog CI to create the doxygen documentation of your project

HOG_CHECK_PROJVER

Should be set to 1, if you want that the Hog CI runs only the projects that have been modified with respect to the target branch

HOG_CHECK_SYNTAX

Should be set to 1, if you want the Hog CI to run check syntax

HOG_CHECK_YAMLREF

If this variable is set to 1, Hog CI will check that “ref” in .gitlab-ci.yml actually matches the gitlab-ci file in the Hog submodule

HOG_IP_EOS_PATH

The EOS path where to store the IP generated results. If not set, the CI will synthesise the IPs each time

HOG_ONLY_SYNTH

If this variable is set to 1, Hog-CI will run only the synthesis for all projects in the CI

HOG_MR_MSG

This variable can be set to 0, 1 (default), 2. It specifies the amount of messages that Hog will write into the merge request page. If set to 0 Hog will not write any massage, if set to 1 Hog will write just one message after all the project have been successful, if set to 2 Hog will write one message per successful project.

HOG_NO_BITSTREAM

If this variable is set to 1, Hog-CI runs the implementation but does NOT run the write_bitstream stage

HOG_RESET_FILES

All the files (or wildcards) contained in this variable (separated by spaces or new lines) be reset at the pre-synthesis time by Hog-CI

HOG_NO_RESET_BD

If this variable is set to 1, BD files will NOT be reset at the pre-synthesis time by Hog-CI

HOG_CREATE_OFFICIAL_RELEASE

If this variable is set to 1, Hog-CI creates an official release note using the version and timing summaries taken from the artifact of the projects.

HOG_SIMULATION_LIB_PATH

The PATH in your VM, where the Simulation Lib files are stored (Vivado only)

HOG_TARGET_BRANCH

Project target branch. Merge request should start from this branch. Default: master

HOG_INTERMEDIATE_BRANCH

Intermediate build branch. Merge request targeting this branch activates an intermediate pipeline. More info here.

HOG_NJOBS

Number of CPU jobs for the synthesis and implementation. Default: 4

HOG_MAX_ZIP_SIZE

Max size of zip files uploaded to gitlab release. Use zip syntax e.g. 2m for 2 MB. Default: 10m

HOG_EXTERNAL_PATH

If specified, entries in the .ext list file would be relative to this path

Note

These variables can be also written inside the .gitlab-ci.yml file. Gitlab uses the following priority scheme for the environmental variables.

Gitlab CI/CD Variables -> Jobs Specific Variables -> Global gitlab-ci.yml variables.

So if, for example, you want to specify a variable that it is same for all jobs expect one, you need to write the common value inside the .gitlab-ci.yml in the main variable section and in the job variable sections declare the specific value.

variables:
  TEST_VAR: "All jobs can use this variable's value"

job1:
  variables:
    TEST_VAR_JOB: "Only job1 can use this variable's value"
  script:
    - echo "$TEST_VAR" and "$TEST_VAR_JOB"  

More info here.

Archive binary files on EOS space (Optional)

The Gitlab CI will produce some artefacts. These include the resulting binary files of your firmware projects and, optionally, the Doxygen documentation html files. Hog has also the ability to copy these files into a desired EOS repository. To enable this feature, we have to specify the following environmental variables: EOS_MGM_URL, HOG_OFFICIAL_BIN_EOS_PATH.

If you wish to have your files to be accessible in a web browser, you should create a web page in EOS, following these instructions. For a personal project, by default, the website will be stored in /eos/user/<initial>/<userID>/www. The Hog EOS paths must be then sub-folders of the website root path. To expose the files in the website, follow these instructions.

Using EOS space for IP synthesis (Optional)

In order to save resources and possibly speed up the workflow, the products of the IP synthesis arr copied to a directory on EOS, specified via HOG_IP_EOS_PATH. Hog-CI will retrieve the already synthesised IPs rather than waste time and resources to produce them again.

The IPs are archived in the IP path on EOS using the md5 hash of the .xci file, so if an IP is modified, it will be copied again on EOS.

It is recommended to clean the IP EOS space regularly as Hog will not delete old IP products.


1

Please use the LM_LICENSE_FILE variable for Intel, MentorGreaphics, etc.