# CI Tutorial ```{warning} Outdated Documentation! This documentation version is out of date. Please check the [latest version 2026.1](https://hog.readthedocs.io/en/latest/). ``` Requirements for this tutorial are a CERN account, a CERN service account ([https://account.cern.ch/account/Management/NewAccount.aspx](https://account.cern.ch/account/Management/NewAccount.aspx)). ## Create Openstack machine The first step is to create an Openstack Virtual Machine, running CENTOS7. CERN user can easily request a virtual machine for production, test and development purposes. For more information, visit [https://clouddocs.web.cern.ch/](https://clouddocs.web.cern.ch/). To create the virtual machine, go to [https://openstack.cern.ch/](https://openstack.cern.ch/), click in the left sidebar on Compute->Instances. Then click on `Launch Instance`. ![](figures/launch-instances.png) Choose an Instance Name (e.g. `tutorial-hog`) and click on `Next`. ![](figures/inst-0.png) Select the `CC7 - x86_64` image and click on `Next`. ![](figures/inst-1.png) Select the largest flavour available (typically `m2.large`) and click on `Next`. ![](figures/inst-2.png) This will take few minutes. Meanwhile create an external volume to be attached to the VM. From the left sidebar, click on `Volumes->Volumes`, and then on `CREATE VOLUME`. ![](figures/volume-0.png) Choose a Volume Name (e.g. `HOG-DISK`), and allocate at least 100 GB to it. Then click on `CREATE VOLUME`. ![](figures/volume-1.png) Once the volume has been created and the Virtual Machine has been setup, click on `Actions->MANAGE ATTACHMENTS`. ![](figures/volume-2.png) Select in the list the VM you just created, e.g. `tutorial-hog`, and click on `ATTACH VOLUME`. ![](figures/volume-3.png) We are now ready to configure the machine to run the Hog CI. ### Configure the Openstack VM Login to `lxplus.cern.ch` ```bash ssh -XY user@lxplus.cern.ch ``` Clone the Hog VM setup repository, in any folder inside your AFS area. ```bash git clone https://gitlab.cern.ch/hog/vm-setup.git ``` ```{note} EOS is not accessible from the Virtual Machine, so don't clone the repository there. ``` Now login into your VM. ```bash ssh -XY tutorial-hog ``` Go to the just cloned `vm-setup` folder, and switch to `root` permission. ```bash sudo su ``` Now install the required dependencies, ```bash ./install_dependecies.sh ``` Then format and mount the volume, ```bash ./volume-setup.sh ``` Finally, install the required software (Vivado, QuestaSim, Quartus) in the attached volume (`/mnt/vd`). Follow the instructions to install them from the correspondent vendor website. ```{warning} You are the one responsible for correctly licensing the software. ``` Once this is done, we can install our gitlab-runner. First, we need to find the gitlab-runner registering token. Open your gitlab repository on a web browser, and go to `Settings->CI/CD->Runners`. Copy the registration token. ![](figures/gitlab-token.png) Go back to your virtual machine, and launch the `install_runner.sh` script, without options to see the Usage description. ```bash Hog - Install gitlab runner --------------------------- Install the Gitlab Runner on a linux machine Usage: ./install_runner.sh -u -g -t [OPTIONS] Options: --tags: , default: hog,vivado --url: . Default: https://gitlab.cern.ch --output_limit: ``` As you can see, it is possible to specify the tags for the runner and the gitlab_url, to create a runner for other Gitlab's locations (e.g. gitlab.com). ```{note} By default, Hog requires the `vivado` and `hog` tags to run its CI. However, it is possible to change the required tags, individually for each CI job, as we'll be shown later in the tutorial. ``` Launch now the script with the right options. This will create the home folder for your service account on the VM, and register the gitlab runner. The gitlab runner configuration will be saved on `/etc/gitlab-runner/config.toml`, and will look like this, ```toml concurrent = 1 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "Hog runner on tutorial-hog" url = "https://gitlab.cern.ch" token = "" executor = "shell" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] ``` Here you can change the concurrent value, to set the maximum numbers of gitlab CI jobs this machine will run in parallel. If you go now back to your gitlab repository webpage, under `Settings->CI/CD->Runners`, you should see your new runner under the `Other available runners` list. We are now ready to configure the repository to run the Hog-CI. ## Setup the Hog-CI on your repository ### Configure Gitlab CI/CD variables To run the Hog-CI, we need to configure some variables in the CI/CD page of our Gitlab repository. Open a web browser and go to [https://gitlab.cern.ch//PiLUP_devel/-/settings/ci_cd#js-runners-settings](https://gitlab.cern.ch//PiLUP_devel/-/settings/ci_cd#js-runners-settings). Expand the `Variables` section and add the following variables. - __HOG_USER__: Your service account; - __HOG_EMAIL__: The email of your service account; - __HOG_PASSWORD__: The password of your service account (Mask it!!); - __HOG_PATH__: The `PATH` environmental variable for your VM, should include the Vivado bin directory; - __HOG_PUSH_TOKEN__: The push token you have generated for your service account. You can obtain one [here](https://gitlab.cern.ch/-/profile/personal_access_tokens), logging in with your service account a select `api` as scope; - __HOG_XIL_LICENSE__: The Xilinx License Server. More information on the CI/CD variables that Hog uses can be found [here](../02-User-Manual/02-Hog-CI/01-GitLab-CI/01-setup-CI.md#environment-variables). ```{note} These variables can be also written inside the `.gitlab-ci.yml` file. Gitlab uses the following priority scheme for the environmental variables. `Gitlab CI/CD Variables -> Jobs Specific Variables -> Global gitlab-ci.yml variables`. More info [here](https://docs.gitlab.com/ee/ci/variables/) ``` Other interesting variables are: - __EOS_MGM_URL__: EOS root url, e.g. `root://eosuser.cern.ch` - __HOG_OFFICIAL_BIN_PATH__: The path for archiving the official binary files of your project. If it starts with `/eos`, it will copy the binary files into the CERN EOS storage, otherwise on the local machine. - __HOG_CHECK_PROJVER__: If set to 1, Hog CI runs only the projects that have been modified with respect to the target branch; - __HOG_CHECK_SYNTAX__: If set to 1, Hog will check the syntax of your source files, before running the synthesis; - __HOG_CHECK_YAMLREF__: If set to 1, Hog will check that the reference Hog version in your `.gitlab-ci.yml` is the same as the cloned submodule; - __HOG_IP_EOS_PATH__: EOS Path where Hog will store the generated IP files, to speed up the CI; - __HOG_USE_DOXYGEN__: If set to 1, Hog will produce the doxygen documentation and copy to EOS; - __HOG_CREATE_OFFICIAL_RELEASE__: If set to 1, Hog creates an official release note using the version and timing summaries taken from the artifact of the projects. ### Remove merge commit Go to [https://gitlab.cern.ch//PiLUP_devel/edit](https://gitlab.cern.ch//PiLUP_devel/edit), expand the __Merge Request settings__ and select *Fast-forward merge*. ### Pipeline configuration Go to [https://gitlab.cern.ch//PiLUP_devel/ci_cd](https://gitlab.cern.ch//PiLUP_devel/ci_cd), expand the __General pipelines settings__, and set the following configurations, - Select *git clone* - Set *Git shallow clone* to 0 - Set *Timeout* to a long threshold, for example 1d Remember to save the changes! ### Add your service account to the members Go to [https://gitlab.cern.ch//PiLUP_devel/-/project_members](https://gitlab.cern.ch//PiLUP_devel/-/project_members) and give *Maintainer* rights to your service account ### The gitlab-ci.yml configuration file Now we need to add our CI configuration file. Open a terminal and go back to the path, where you cloned the `PiLUP_devel` repository. Copy the Hog `gitlab-ci.yml` template to the main folder of your repository. ```bash cp ./Hog/Templates/gitlab-ci.yml .gitlab-ci.yml ``` Let's have a look at the content of this file. Open it with your favourite editor. ```yaml include: - project: 'hog/Hog' file: '/hog.yml' ref: 'v0.2.1' #################### example ########################### ### Change 'example' with your project name GEN:example: extends: .generate_project variables: extends: .vars PROJECT_NAME: example HOG_ONLY_SYNTH: 0 # if 1 runs only the synthesis SIM:example: extends: .simulate_project variables: extends: .vars PROJECT_NAME: example ``` We need to change the reference value to point to correct Hog version. In this tutorial we are using `Hog2021.2`. ```yaml include: - project: 'hog/Hog' file: '/hog.yml' ref: 'Hog2021.2-2' ``` The project that we want to run on the CI is called `simple`, so the yaml would look like, ```yaml GEN:simple: extends: .generate_project variables: extends: .vars PROJECT_NAME: simple HOG_ONLY_SYNTH: 0 # if 1 runs only the synthesis SIM:simple: extends: .simulate_project variables: extends: .vars PROJECT_NAME: simple ``` Save the file and push it to the remote. ```bash git add .gitlab-ci.yml git commit -m "FEATURE: Enabling Hog Continuous Integration" git push ``` We are now ready to open the merge request, by either visiting the link git suggests in the output log, or going to [https://gitlab.cern.ch//PiLUP_devel/-/merge_requests](https://gitlab.cern.ch//PiLUP_devel/-/merge_requests), and click on `New merge request`. As source branch, select `Converting_to_hog` and as target the master branch in your repository. Click then on `Compare branches and continue`. In the next page, write in the description, ```text MAJOR_VERSION ``` In this way, Hog will increase the Major number, when creating the new tag once the merge request is accepted. By default, Hog increases the patch number. More info [here](../02-User-Manual/02-Hog-CI/01-GitLab-CI/02-gitlab-workflow). Leave the remaining fields empty and click on `Create merge request`. ![](figures/create-mr.png) This will trigger the merge request pipeline (info [here](../02-User-Manual/02-Hog-CI/01-CI-Introduction.md#merge-request-pipeline)). While we wait for the pipeline to finish, one can explore the different executed jobs. ![](figures/pipeline.png) Once this done, Hog will write a message in the Merge Request page, with a summary of the build projects with timing and versions. You can also have a look at the produced binary files, by downloading the artifacts of the `collect_artifacts` job. Decompress the download archive, and have a look at the `bin` folder. It contains the `note.md` file that will be used later for the release, and a folder for each build project, in our case only `PiLUP_Kintex_template`. Inside that you'll find the reports and the produced bit files. :::{admonition} Local and CI bit-files :class: tip Try to do a diff between the bit-file you produced locally in the previous chapter, and the one just downloaded from the CI. They should be identical! ::: Now we can finally merge our request. Go back to the merge request page, and click on `Merge`. ![](figures/merge.png) This triggers the `master` and later the `tag` pipelines, which create the tag and gitlab release. Finally, you can have a look at the generated release at [https://gitlab.cern.ch//PiLUP_devel/-/release](https://gitlab.cern.ch//PiLUP_devel/-/release). ![](figures/release.png) You can see how the pipeline artifacts have been attached to the release. If you configured EOS, these have been also copied to your storage. #### The dynamic CI (Optional) An experimental feature of Gitlab allows the developer to dynamically generate the yaml configuration. Hog supports this feature, even if it still recommended to use the standard CI. To enable the dynamic CI, one should include the `hog-dynamic.yml` in the `.gitlab-ci.yml`, ```yaml include: - project: 'hog/Hog' file: '/hog-dynamic.yml' ref: 'Hog2021.2' ``` That's it! For the dynamic CI you don't need to configure individually each job. Hog will look for each project in the `Top` folder and create dynamically the CI jobs, which will be run in a `child` pipeline. By default, both implementation and simulation jobs are created for each project. It is possible to configure the single projects, by adding a `ci.conf` file in the Top project directory. More info [here](../02-User-Manual/02-Hog-CI/01-GitLab-CI/05-Dynamic-CI). Create a new branch, commit the modified `.gitlab-ci.yml` and open a new merge request. ```bash git checkout -b dynamic_ci git add .gitlab-ci.yml git commit -m "FEATURE: Dynamic Continuous Integration" git push ``` Opening the merge request enables the dynamic CI pipeline. ![](figures/dynamic-ci.png) The jobs are created in the `dynamic_generate` stage, and run in the child pipeline, in the Upstream section. The dynamic CI allows the developers to save resources and computing time, when enabling the __HOG_CHECK_PROJVER__ variable. If this is enabled, in the standard CI, Hog runs in any case all the generation and simulation jobs specified in the YAML file, but it stops the execution before starting the synthesis or the simulation, if the project has not been changed with respect to the target branch. The dynamic CI, on the other hand, performs this check before creating the individual project jobs. If you now, enable the variable in the CI/CD settings of your repository, and relaunch the pipeline by doing a fake commit (just change the README for instance), you will see that the `child-pipeline` this time consists only of the `collect_artifact` job. ![](figures/dynamic-skip.png)