I have been trying to speed up my CI/CD pipeline in GitLab to save both time and money. The reason for this is that I have a docker image which requires a lot of packages to be installed. This, as we all know, docker handles this pretty well by caching intermediate images which speeds up subsequent builds from the same Dockerfile. However, this is not the case with GitLab CI.

The problem is that my CI/CD pipeline uses docker in docker, therefore I get a completely new docker environment each time the pipeline is executed. Thus no caching and slower builds.

The solution to this problem relates to one of my earlier post: Multiple Docker Images from a Single Dockerfile. The general idea is to separate the Dockerfile into different targets and then manually cache each stage for later builds. For convenience I will use the same Dockerfile as in the previous mentioned post.

FROM swift:latest as builder

RUN apt-get -y update && \
    apt-get -y upgrade
RUN apt-get -y install openssl libssl-dev && \
    apt-get -y install libcurl4-openssl-dev 

WORKDIR /build
COPY Package.* .
RUN swift package resolve


#
#   Build for unittest
#
FROM builder as unittest
WORKDIR /build

COPY Sources Sources
COPY Tests Tests

RUN swift test

#
#   Build for staging
#
FROM builder as staging
WORKDIR /build

COPY Sources Sources
COPY Tests Tests

WORKDIR /staging
RUN cp "$(swift build --package-path /build -c release --show-bin-path)/Application" ./


#
#   Build for production
#
FROM swift:5.2-focal-slim as production

WORKDIR /app

COPY --from=staging /staging .

ENTRYPOINT ["./Application"]


Caching the images

With GitLab there are two ways to solve the problem of caching docker images between stages when using docker in docker. The first option is to use the container registry as a cache and the second option is to use GitLab artifacts.

The main difference between these two approaches is that the container registry will persist your cached images between pipeline executions while artifacts will only live within as single pipeline.

In this example I will use both approaches to improve performance. The setup is pretty straight forward and the complete CI-file can be found at the end of this post.

Initial setup

Before anything else, I sign in to the container registry for this repository. I then define my three stages, build, test and production. These stages will run sequentially, meaning that all jobs in build will finish before any job in test can begin.

image: docker:latest

services:
  - docker:dind
  
before_script:
  - > 
    docker login 
    -u "$CI_REGISTRY_USER" 
    -p "$CI_REGISTRY_PASSWORD" 
    $CI_REGISTRY

stages:
  - build
  - test
  - production

Build Stage

The first thing I do, is to pull a cached base image from the container registry. Note that this command must return true even if docker pull fails (e.g. if there is no image in the registry), otherwise the job would fail and the pipeline execution stop.

docker build will build the base image, using the pulled base as a cached image. A new image will be created, either from scratch (if docker pull failed) or with the pulled image as cache.

Then push the new base image to the container registry and create a tar-file from  the image with docker save. Finally  save the tar-file as an artifact, to be used in subsequent jobs.


build:
  stage: build
  script:
    - docker pull $CI_REGISTRY_IMAGE:base || true
    - >
      docker build 
      --build-arg BUILDKIT_INLINE_CACHE=1
      --cache-from $CI_REGISTRY_IMAGE:base
      --target builder
      --tag $CI_REGISTRY_IMAGE:base .
    - docker push $CI_REGISTRY_IMAGE:base
    - mkdir image
    - docker save $CI_REGISTRY_IMAGE:base > image/image.tar
  artifacts:
    paths:
      - image

Test & Production

The next two stages are test and production. The two jobs are very similar. Both starts by  loading the base image artifact from the previous stage using docker load. When the image is loaded it will be used as a cached image in docker build.

And that's it! The only thing left to do is to push the production image to the container registry.

Note that both the test and production stages are not independent of each other and could potentially run in parallel. However, I do not want to overwrite the latest image in the container registry if unittest fails, and therefore they are run sequentially.

unittest:
  stage: test
  script:
    - docker load -i image/image.tar
    - >
      docker build 
      --build-arg BUILDKIT_INLINE_CACHE=1
      --cache-from $CI_REGISTRY_IMAGE:base
      --target unittest  .

production:
  stage: production
  script:
    - docker load -i image/image.tar
    - >
      docker build 
      --build-arg BUILDKIT_INLINE_CACHE=1
      --cache-from $CI_REGISTRY_IMAGE:base
      --target production
      --tag $CI_REGISTRY_IMAGE:latest  .
    - docker push $CI_REGISTRY_IMAGE:latest

Conclusion

Caching docker images between and within pipeline executions can potentially speed up the build process. However, both pulling images from the container registry and archiving images can be costly both in time and computation. This is especially true if you use shared runners on gitlab.com.

In conclusion, if your jobs already run fast, there is no need for this kind of optimization. But, if you, like me, have a time consuming build stage in your Dockerfile that can be reused in later stages, caching docker images may save you a lot of time (and money)

Complete CI-file

image: docker:latest

services:
  - docker:dind
  
before_script:
  - > 
    docker login 
    -u "$CI_REGISTRY_USER" 
    -p "$CI_REGISTRY_PASSWORD" 
    $CI_REGISTRY

stages:
  - build
  - test
  - production


build:
  stage: build
  script:
    - docker pull $CI_REGISTRY_IMAGE:base || true
    - >
      docker build 
      --build-arg BUILDKIT_INLINE_CACHE=1
      --cache-from $CI_REGISTRY_IMAGE:base
      --target builder
      --tag $CI_REGISTRY_IMAGE:base .
    - docker push $CI_REGISTRY_IMAGE:base
    - mkdir image
    - docker save $CI_REGISTRY_IMAGE:base > image/image.tar
  artifacts:
    paths:
      - image


unittest:
  stage: test
  script:
    - docker load -i image/image.tar
    - >
      docker build 
      --build-arg BUILDKIT_INLINE_CACHE=1
      --cache-from $CI_REGISTRY_IMAGE:base
      --target unittest  .

production:
  stage: production
  script:
    - docker load -i image/image.tar
    - >
      docker build 
      --build-arg BUILDKIT_INLINE_CACHE=1
      --cache-from $CI_REGISTRY_IMAGE:base
      --target production
      --tag $CI_REGISTRY_IMAGE:latest  .
    - docker push $CI_REGISTRY_IMAGE:latest