Introduce a global share repository mechanism
This is a full refactor of the way jobs fetch git data: with this
approach, only the top level job, the upstream job, would set a share
folder populated with all required repositories, including
trusted-firmware-a, tf-a-tests, tf-a-job-configs and tf-a-ci-scripts,
ultimately consumed by the builder job, tf-a-builder. This would
reduce considerably the CI build times, avoiding multiple clones per
job thus overloading the TF git servers as reported in [1].
The way it works is simple: the top level job sets the SHARE_FOLDER
parameter based on its name and number on top of the share
volume (/srv/shared/<job name>/<job number>) then it calls the clone
script (clone.sh), which in turn it fetches the repositories mentioned
above. Jobs triggered on behalf of the latter, share the same
SHARE_FOLDER value, and these in turn also call the clone script, but
in this case, the script detects that the folder is already populated so
its role is to simply copy the repositories into the job's
workspace. As seen, all jobs work with repositories on their own
workspace, which are just copies of the share folder, so there is no
change of a race condition, i.e every job works with its own copy. The
worst case scenario is where the down-level job, tf-a-builder, uses its
default SHARE_FOLDER value, in this case, it would simply clone its
own repositories without reusing any file however the current approach
prevents the latter unless the job is triggered manually from the
builder job itself.
Besides the git-cloning strategy presented, the change implied moving
all of jobs to 'docker-amd64-tf-a-bionic' container node, instead of
'master', the former having access to the docker share volume.
[1] https://linaro.atlassian.net/browse/TFC-20
Signed-off-by: Leonardo Sandoval <leonardo.sandoval@linaro.org>
Change-Id: I001bc1148cde4b1952161d84aa3d2e089a4281a2
diff --git a/tf-static-checks.yaml b/tf-static-checks.yaml
index 81ab262..4912ef6 100644
--- a/tf-static-checks.yaml
+++ b/tf-static-checks.yaml
@@ -1,44 +1,3 @@
-- scm:
- name: tf-a-ci-scripts
- scm:
- - git:
- url: https://git.trustedfirmware.org/ci/tf-a-ci-scripts.git
- refspec: +refs/heads/master:refs/remotes/origin/master
- name: origin
- branches:
- - refs/heads/master
- basedir: tf-a-ci-scripts
- skip-tag: true
- shallow-clone: true
- wipe-workspace: false
-- scm:
- name: trusted-firmware-a
- scm:
- - git:
- url: https://review.trustedfirmware.org/${TF_GERRIT_PROJECT}
- refspec: ${TF_GERRIT_REFSPEC}
- name: origin
- branches:
- - ${TF_GERRIT_BRANCH}
- basedir: trusted-firmware-a
- choosing-strategy: gerrit
- skip-tag: true
- shallow-clone: false
- wipe-workspace: false
-- scm:
- name: tf-a-tests
- scm:
- - git:
- url: https://review.trustedfirmware.org/${TFTF_GERRIT_PROJECT}
- refspec: ${TFTF_GERRIT_REFSPEC}
- name: origin
- branches:
- - ${TFTF_GERRIT_BRANCH}
- basedir: tf-a-tests
- choosing-strategy: gerrit
- skip-tag: true
- shallow-clone: false
- wipe-workspace: false
- job:
name: tf-static-checks
node: docker-amd64-tf-a-bionic
@@ -79,16 +38,18 @@
default: trusted-firmware-a
description: |-
Repository to run static checks on.
- scm:
- - tf-a-ci-scripts
- - trusted-firmware-a
- - tf-a-tests
+ - string:
+ name: SHARE_FOLDER
+ default: '/srv/shared/${JOB_NAME}/${BUILD_NUMBER}'
+ description: 'Folder containing shared repositories for downstream pipeline jobs'
wrappers:
- timestamps
- timeout:
timeout: 180
fail: true
builders:
+ - shell:
+ !include-raw: scripts/clone.sh
- shell: |
#!/bin/bash
set -e