Hermetic builds inside a container
Adds 'build/docker/Dockerfile' which describes the base container image of
Hafnium compilation environment. This image is built and uploaded to GCP
where users download it from. The feature is always enabled for Kokoro
and can optionally be enabled for local builds too. Once rootless
containers are easier to set up, we might make it the default for local
builds too.
An arbitrary command can be executed inside the container with
'build/run_in_container.sh [-i] <command> ...'. This is done
automatically inside 'Makefile' and 'kokoro/ubuntu/build.sh' which
detect whether they are already running inside the container and respawn
themselves using 'run_in_container.sh' if not.
The feature is guarded with HAFNIUM_HERMETIC_BUILD environment variable,
switched on if the value is "true". All other values switch it off, e.g.
'run_in_container.sh' sets it to 'inside' to avoid recursion.
Bug: 132428451
Test: HAFNIUM_HERMETIC_BUILD=<value> make
Test: HAFNIUM_HERMETIC_BUILD=<value> kokoro/ubuntu/build.sh
Change-Id: I0737a868ab4f67c0fdbf78fa8a97cc91714d2e10
diff --git a/build/docker/Dockerfile b/build/docker/Dockerfile
new file mode 100644
index 0000000..93c8caa
--- /dev/null
+++ b/build/docker/Dockerfile
@@ -0,0 +1,43 @@
+# Copyright 2019 The Hafnium Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Base container image to be uploaded to Google Cloud Platform as
+# "eu.gcr.io/hafnium-build/hafnium_ci". Each user derives their own container
+# with local user permissions from this base image. It should contain everything
+# needed to build and test Hafnium.
+#
+FROM launcher.gcr.io/google/ubuntu1804
+MAINTAINER Hafnium Team <hafnium-team+build@google.com>
+
+# Install dependencies. Clear APT cache at the end to save space.
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt-get update \
+ && apt-get install -y \
+ bc `# for Linux headers` \
+ binutils-aarch64-linux-gnu \
+ bison \
+ build-essential \
+ cpio \
+ device-tree-compiler \
+ flex \
+ git \
+ libpixman-1-0 `# for QEMU` \
+ libsdl2-2.0-0 `# for QEMU` \
+ libglib2.0 `# for QEMU` \
+ libssl-dev `# for Linux headers` \
+ python \
+ python-git `# for Linux checkpatch` \
+ python-ply `# for Linux checkpatch` \
+ && rm -rf /var/lib/apt/lists/*
diff --git a/build/docker/Dockerfile.local b/build/docker/Dockerfile.local
new file mode 100644
index 0000000..67eb92f
--- /dev/null
+++ b/build/docker/Dockerfile.local
@@ -0,0 +1,35 @@
+# Copyright 2019 The Hafnium Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Container derived from the base image hosted on Google Cloud Platform.
+# It sets up a user with the same UID/GID as the local user, so that generated
+# files can be accessed by the host.
+# Please keep the diff between base and local images as small as possible.
+#
+FROM eu.gcr.io/hafnium-build/hafnium_ci
+ARG LOCAL_UID=1000
+ARG LOCAL_GID=1000
+
+RUN addgroup \
+ --gid "${LOCAL_GID}" \
+ hafnium \
+ && adduser \
+ -disabled-password \
+ -gecos "" \
+ --uid "${LOCAL_UID}" \
+ --shell "/bin/bash" \
+ --ingroup hafnium \
+ hafnium
+USER hafnium
\ No newline at end of file
diff --git a/build/docker/build.sh b/build/docker/build.sh
new file mode 100755
index 0000000..5972038
--- /dev/null
+++ b/build/docker/build.sh
@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# Copyright 2019 The Hafnium Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+set -euo pipefail
+
+SCRIPT_DIR="$(realpath "$(dirname "${BASH_SOURCE[0]}")")"
+source "${SCRIPT_DIR}/common.inc"
+
+${DOCKER} build \
+ --pull \
+ -f "${SCRIPT_DIR}/Dockerfile" \
+ -t "${CONTAINER_TAG}" \
+ "${SCRIPT_DIR}"
diff --git a/build/docker/common.inc b/build/docker/common.inc
new file mode 100644
index 0000000..0d1e1db
--- /dev/null
+++ b/build/docker/common.inc
@@ -0,0 +1,21 @@
+# Copyright 2019 The Hafnium Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+CONTAINER_TAG="eu.gcr.io/hafnium-build/hafnium_ci"
+
+if [[ ! -v DOCKER ]]
+then
+ DOCKER="$(which docker)" \
+ || (echo "ERROR: Could not find Docker binary" 1>&2; exit 1)
+fi
\ No newline at end of file
diff --git a/build/docker/publish.sh b/build/docker/publish.sh
new file mode 100755
index 0000000..96ec2f1
--- /dev/null
+++ b/build/docker/publish.sh
@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Copyright 2019 The Hafnium Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+set -euo pipefail
+
+SCRIPT_DIR="$(realpath "$(dirname "${BASH_SOURCE[0]}")")"
+source "${SCRIPT_DIR}/common.inc"
+
+# Requires for the user to be an owner of the GCP 'hafnium-build' project and
+# have gcloud SDK installed and authenticated.
+
+${DOCKER} push "${CONTAINER_TAG}"
diff --git a/build/run_in_container.sh b/build/run_in_container.sh
new file mode 100755
index 0000000..e3c6bd0
--- /dev/null
+++ b/build/run_in_container.sh
@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+# Copyright 2019 The Hafnium Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+set -euo pipefail
+
+SCRIPT_DIR="$(realpath "$(dirname "${BASH_SOURCE[0]}")")"
+ROOT_DIR="$(realpath ${SCRIPT_DIR}/..)"
+
+source "${SCRIPT_DIR}/docker/common.inc"
+
+if [ "${HAFNIUM_HERMETIC_BUILD:-}" == "inside" ]
+then
+ echo "ERROR: Invoked $0 recursively" 1>&2
+ exit 1
+fi
+
+# Set up a temp directory and register a cleanup function on exit.
+TMP_DIR="$(mktemp -d)"
+function cleanup() {
+ rm -rf "${TMP_DIR}"
+}
+trap cleanup EXIT
+
+# Build local image and write its hash to a temporary file.
+IID_FILE="${TMP_DIR}/imgid.txt"
+"${DOCKER}" build \
+ --build-arg LOCAL_UID="$(id -u)" \
+ --build-arg LOCAL_GID="$(id -g)" \
+ --iidfile="${IID_FILE}" \
+ -f "${SCRIPT_DIR}/docker/Dockerfile.local" \
+ "${SCRIPT_DIR}/docker"
+IMAGE_ID="$(cat ${IID_FILE})"
+
+# Check if script was invoked with '-i' as first argument. If so, run
+# container in interactive mode.
+INTERACTIVE=false
+if [ "${1:-}" == "-i" ]
+then
+ INTERACTIVE=true
+ shift
+fi
+
+ARGS=()
+# Run with a pseduo-TTY for nicer logging.
+ARGS+=(-t)
+# Run interactive if this script was invoked with '-i'.
+if [ "${INTERACTIVE}" == "true" ]
+then
+ ARGS+=(-i)
+fi
+# Set environment variable informing the build that we are running inside
+# a container.
+ARGS+=(-e HAFNIUM_HERMETIC_BUILD=inside)
+# Bind-mount the Hafnium root directory. We mount it at the same absolute
+# location so that all paths match across the host and guest.
+ARGS+=(-v "${ROOT_DIR}":"${ROOT_DIR}")
+# Make all files outside of the Hafnium directory read-only to ensure that all
+# generated files are written there.
+ARGS+=(--read-only)
+# Mount a writable /tmp folder. Required by LLVM/Clang for intermediate files.
+ARGS+=(--tmpfs /tmp)
+# Set working directory.
+ARGS+=(-w "${ROOT_DIR}")
+
+echo "Running in container: $*" 1>&2
+${DOCKER} run \
+ ${ARGS[@]} \
+ "${IMAGE_ID}" \
+ /bin/bash -c "$*"
\ No newline at end of file