lava_wait_jobs: Catch exceptions during 2nd pass on fetching job artifacts
Catch exception, log as error, but continue the rest of processing like
resubmission of (other) failed jobs. This is to get realistic overall
result figures for tests. For example, we had a case
(https://ci.trustedfirmware.org/job/tf-m-nightly/1702/) when, due to
test error, gigantic log failes were produced which simply cannot be
reliably downloaded from LAVA. But at the same run, we had 189 other
included LAVA jobs. Some of them were clearly random board errors which
might succeed on resubmission.
Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>
Change-Id: I8356fe070f61ccd3570af8550365f4f3b388b2d7
diff --git a/lava_helper/lava_wait_jobs.py b/lava_helper/lava_wait_jobs.py
index 50cae41..12862c5 100755
--- a/lava_helper/lava_wait_jobs.py
+++ b/lava_helper/lava_wait_jobs.py
@@ -76,7 +76,10 @@
info['job_dir'] = os.path.join(user_args.artifacts_path, "{}_{}".format(str(job), info['description']))
to_fetch = {job_id: info for job_id, info in finished_jobs.items() if job_id not in fetched_artifacts}
_log.info("Fetching artifacts for remaining jobs: %s", to_fetch.keys())
- fetch_artifacts(to_fetch, user_args, lava)
+ try:
+ fetch_artifacts(to_fetch, user_args, lava)
+ except Exception as e:
+ _log.exception("Still failed to fetch artifacts for some jobs; continuing, but overall result is failure")
return finished_jobs
def resubmit_failed_jobs(jobs, user_args):