Implementing Image Signing for ECR Images via Cosign in a GitLab Pipeline

In this post, we’ll see how we can sign ECR images using Cosign in a GitLab pipeline. We’ll implement keyless as well as creating and using your own key for image signing.

Prerequisites:

1-OIDC connectivity between GitLab and AWS is required . You may refer to https://docs.gitlab.com/ee/ci/cloud_services/aws/ to set this up.

2-Make sure your GitLab IAM role has necessary permission. For this demo, I gave Administrator permission but in real world scenarios, access should be restricted.

Option 1: Keyless Image Signing

Reference: https://about.gitlab.com/blog/2023/09/13/keyless-signing-with-cosign/

Step 1: Fork https://gitlab.com/bwill/container-signing inn your GitLab account.

Step 2: In your project, go to Settings->CI/CD->Variables and create a variable named AWS_ACCOUNT with value set to your AWS account.

Step 3: Update the content of .gitlab-ci.yml with this. Change the variable values accordingly.

stages:
  - build
  
.protected:
  rules:
    - if: '$CI_COMMIT_REF_PROTECTED == "true"'

.setup-script:
  before_script:
    - mkdir -p ~/.aws
    - echo "${MY_OIDC_TOKEN}" > /tmp/web_identity_token
    - echo -e
      "[default]\nrole_arn=${ROLE_ARN}\nweb_identity_token_file=/tmp/web_identity_token"
      > ~/.aws/config

docker-build:
  image: docker:latest
  stage: build
  extends:
    - .protected
  variables:
    COSIGN_YES: "true"
    ROLE_ARN: "arn:aws:iam::$AWS_ACCOUNT:role/gitlab"
    DOCKER_REGISTRY: $AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com
    AWS_DEFAULT_REGION: us-east-1
    REPO_NAME: app-api

  id_tokens:
    SIGSTORE_ID_TOKEN:
      aud: sigstore
    MY_OIDC_TOKEN:
      aud: https://gitlab.com
  services:
    - docker:dind
  before_script:
    - apk add --no-cache curl jq python3 py3-pip
    - pip3 install awscli --break-system-packages
    - !reference [.setup-script, before_script]
    - aws ecr get-login-password | docker login --username AWS --password-stdin $DOCKER_REGISTRY
  
  script:
    - docker build -t $DOCKER_REGISTRY/$REPO_NAME:v$CI_PIPELINE_IID .
    - docker push "$DOCKER_REGISTRY/$REPO_NAME:v$CI_PIPELINE_IID"
    - apk add --update cosign
    - IMAGE_DIGEST="$(docker inspect --format='{{index .RepoDigests 0}}' "$DOCKER_REGISTRY/$REPO_NAME:v$CI_PIPELINE_IID")"
    - echo "$IMAGE_DIGEST"
    - cosign sign "$IMAGE_DIGEST"
    - cosign verify "$DOCKER_REGISTRY/$REPO_NAME:v$CI_PIPELINE_IID" --certificate-identity "$CI_PROJECT_URL//.gitlab-ci.yml@refs/heads/$CI_COMMIT_BRANCH" --certificate-oidc-issuer "$CI_SERVER_URL"

Step 4: Trigger the pipeline and once it’s completed, you’ll see the image along with SHA digest pushed to your ECR repo.

Option 2: Generate your own key for image signing

Step 1: Launch an EC2 instance using Amazon Linux 2 Image and SSH in to it and execute following commands to install Cosign and generate a key pair

sudo su -
curl -O -L "https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64" cosign-linux-amd64
mv cosign-linux-amd64 /usr/local/bin/cosign
chmod +x /usr/local/bin/cosign
cosign generate-key-pair #You may choose not to specify any password for private key

Step 2: Store private and public generated by cosign generate-key-pair command, in parameter store.

Step 3: Update the content of .gitlab-ci.yml with this. Change the variable values accordingly

stages:
  - build

.setup-script:
  before_script:
    - mkdir -p ~/.aws
    - echo "${MY_OIDC_TOKEN}" > /tmp/web_identity_token
    - echo -e
      "[default]\nrole_arn=${ROLE_ARN}\nweb_identity_token_file=/tmp/web_identity_token"
      > ~/.aws/config

docker-build:
  image: docker:latest
  stage: build
  variables:
    ROLE_ARN: "arn:aws:iam::$AWS_ACCOUNT:role/gitlab"
    DOCKER_REGISTRY: $AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com
    AWS_DEFAULT_REGION: us-east-1
    REPO_NAME: app-api
    COSIGN_PRIVATE_KEY_PARAMETER: cosign-private-key
    COSIGN_PUBLIC_KEY_PARAMETER: cosign-public-key

  id_tokens:
    MY_OIDC_TOKEN:
      aud: https://gitlab.com
  services:
    - docker:dind
  before_script:
    - apk add --no-cache curl jq python3 py3-pip
    - pip3 install awscli --break-system-packages
    - !reference [.setup-script, before_script]
    - aws ecr get-login-password | docker login --username AWS --password-stdin $DOCKER_REGISTRY
  
  script:
    - docker build -t $DOCKER_REGISTRY/$REPO_NAME:v$CI_PIPELINE_IID .
    - docker push "$DOCKER_REGISTRY/$REPO_NAME:v$CI_PIPELINE_IID"
    - apk add --update cosign
    - IMAGE_DIGEST="$(docker inspect --format='{{index .RepoDigests 0}}' "$DOCKER_REGISTRY/$REPO_NAME:v$CI_PIPELINE_IID")"
    - echo "$IMAGE_DIGEST"
    - aws ssm get-parameter --name $COSIGN_PRIVATE_KEY_PARAMETER --region $AWS_DEFAULT_REGION --with-decryption --query "Parameter.Value" --output text > cosign.key
    - cosign sign --key cosign.key "$IMAGE_DIGEST" -y
    - rm -rf cosign.key
    - aws ssm get-parameter --name $COSIGN_PUBLIC_KEY_PARAMETER --region $AWS_DEFAULT_REGION --with-decryption --query "Parameter.Value" --output text > cosign.pub
    - cosign verify --key cosign.pub "$DOCKER_REGISTRY/$REPO_NAME:v$CI_PIPELINE_IID" | jq -r .
    - rm -rf cosign.pub

Step 4: Trigger the pipeline and once it’s completed, you’ll see the image along with SHA digest pushed to your ECR repo.

Note: You may also use KMS key for image signing.