Hog Dynamic CI#

Warning

Hog Dynamic CI is still under development.

A novel feature of the Gitlab Continuous Integration allows to generate dynamically a YAML file for the Gitlab-CI. Exploiting this feature Hog is now able to create a CI configuration for each project inside the Top directory automatically.

To use the dynamic CI simply include the hog-dynamic.yml in your .gitlab-ci.yml file.

  include:
    - project: 'hog-cern/Hog'
      file: '/hog-dynamic.yml'
      ref: 'v0.2.1'

Remember to add the correct path in the reference, relative to your GitLab server provider.

The Dynamic CI pipeline#

The dynamic CI pipeline is activated for each non-Draft merge request commit and it is divided into a main pipeline and a child pipeline.

The main pipeline is composed by the following stages:

  1. Merge: checks that all the required Hog environmental variables are set up and that the source branch is not outdated with respect to the target branch. If it is, the pipeline fails and asks the user to update the source branch.

  2. Generate: generates dynamically the YAML configuration file for the child pipeline.

  3. Triggers: starts the child pipeline using the dynamically-generated YAML file.

  4. Collect: Collects all the artefacts from the child pipeline and, if activated, creates the Doxygen documentation. If EOS is used, it copies the implementation outputs and the Doxygen documentation to the EOS repository.

../../../_images/main.png

The child pipeline simulates, synthesises and implements the chosen HDL projects. It is composed by the following stages:

  1. Generation_and_simulation: launches the workflow for the chosen Hog projects.

  2. Collect: Collects all the artefacts of the previous stage.

../../../_images/child.png

If the HOG_CHECK_PROJVER variable is set to 1, the child pipeline will contain only jobs relative to the projects that have been modified with respect to the target branch.

Configuring the projects#

By default the dynamic CI will create a YAML configuration for each project inside the Top folder with a full implementation and simulation. To configure the CI, a ci.conf file should be created for each project in the Top/project/ directory.

This file uses the .ini syntax to configure the CI jobs. Hog will read this file to generate dynamically the YAML running the pipeline.

An example ci.conf looks as follow.

#NO_VER_CHECK
[generate_project]
allow_failure=true
[generate_project:variables]
HOG_ONLY_SYNTH=0
MY_VAR=3
[simulate_project]
[my_user_job]

The first optional line #NO_VER_CHECK signals Hog to not check the version of the project. In this way, the CI is always run for the specific project, independently of whether it has been modified with respect to the target branch.

The [generate_project] section creates the actual generate_project job in the YAML. In this section you can specify all the standard GitLab CI settings, to modify your job. For example, here we want that this job is allowed to fail using the allow_failure=true line.

Job variables must be specified in the subsection [generate_project:variables]. There is no need to specify again the PROJECT_NAME, since this is automatically parsed by Hog. In the example, we specified a Hog specific CI variable (HOG_ONLY_SYNTH) and also a user variable MY_VAR, which may be used by the developers.

Similarly, the developers can set-up the simulation using the [simulate_project] section, or even a custom user job in the [my_user_job] section, where my_user_job is the name of the custom user CI job. For more info, please read the next subsection.

To exclude totally a project from the CI, the developer just have to create an empty ci.conf file in the project folder.

User jobs in the dynamic CI.#

Custom user jobs must be specified in the hog-ci-users.yml file, in the main directory of your repository, following the instructions in the user stages section. Jobs that will be project-specific, must be declared using the standard template syntax.

.my_user_job: &my_user_job
    <<: *only-default # Default Hog variables. Mandatory.
    <<: *tag-hog # To run on your private machine, use tag-sr for the shared-machines running the Hog Docker containers.
    stage: user_pre # The stage when the job runs 
    script:
        - my_script # the script to run 
    allow_failure: false # extra options of GitLab

Naturally, you can also define global jobs, following the GitLab-CI syntax.

Known Issues#

Child pipeline status do not appear in the main pipeline page

To access the child pipeline status you have to click on the status button in the first column of the Gitlab pipeline page, under CI/CD > Pipelines. Then you have to click on the Downstream box to show the child pipeline.

../../../_images/pipeline_page.png ../../../_images/downstream.png

If child pipeline passes after a retry, the main pipeline remains in the fail status

If a child pipeline fails, the main pipeline will fail subsequently, and even if a restarted child pipeline succeed, the main pipeline would stay in a failed status. This can lead to cases where a Gitlab 504 error would make your pipeline job fail.