Deploying Grafana On EKS Fargate With EFS Persistent Storage

In this post, we’ll see how we can deploy Grafana on EKS Fargate with EFS persistent storage. This ensures that we don’t lose Grafana data if the pod restarts.

Reference: https://github.com/aws-samples/eks-efs-share-within-fargate/tree/master

Step 1: Launch an EC2 instance with Amazon Linux 2 AMI.This instance will act as a command center :).

Connect to it and Install kubectl,eksctl,helm,docker and git using following command:

sudo yum install -y git docker jq
sudo service docker start
sudo curl --silent --location -o /usr/local/bin/kubectl \
   https://s3.us-west-2.amazonaws.com/amazon-eks/1.28.5/2024-01-04/bin/linux/amd64/kubectl
sudo chmod +x /usr/local/bin/kubectl
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv -v /tmp/eksctl /usr/local/bin
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Step 2: Follow https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#getting-started-install-instructions to upgrade AWS CLI to v2.

Step 3: Now create a file named eks-cluster-config.yaml with following content:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-east-1
  version: '1.29'
iam:
  withOIDC: true
availabilityZones: ['us-east-1a','us-east-1b','us-east-1c']
fargateProfiles:
  - name: defaultfp
    selectors:
      - namespace: default
      - namespace: kube-system
cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

Now execute following command to create our EKS cluster:

eksctl create cluster -f eks-cluster-config.yaml

It will take around 20 minutes before cluster is ready.

Step 4: Verify you can connect to your cluster using following commands:

aws eks update-kubeconfig --region us-east-1 --name my-cluster
kubectl get nodes

Step 5: Create a file named grafana.yaml with following content and deploy it using kubectl apply -f grafana.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "16"
    meta.helm.sh/release-name: grafana
    meta.helm.sh/release-namespace: kube-system
  labels:
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 8.3.1
    helm.sh/chart: grafana-6.19.0
  name: grafana
  namespace: kube-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: grafana
      app.kubernetes.io/name: grafana
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/config: e273d11a0e7bfb234f70c87d5790a463762f078cb093bf61e26d1a8f470c983e
        checksum/dashboards-json-config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
        checksum/sc-dashboard-provider-config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
        checksum/secret: da2f12621be635c6bb8140e206def2c309b99d31b987355d440068d83fc126ba
        kubectl.kubernetes.io/restartedAt: "2024-03-22T17:01:26+08:00"
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: grafana
        app.kubernetes.io/name: grafana
    spec:
      automountServiceAccountToken: true
      containers:
      - env:
        - name: GF_SECURITY_ADMIN_USER
          valueFrom:
            secretKeyRef:
              key: admin-user
              name: grafana
        - name: GF_SECURITY_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: admin-password
              name: grafana
        - name: GF_PATHS_DATA
          value: /var/lib/grafana/
        - name: GF_PATHS_LOGS
          value: /var/log/grafana
        - name: GF_PATHS_PLUGINS
          value: /var/lib/grafana/plugins
        - name: GF_PATHS_PROVISIONING
          value: /etc/grafana/provisioning
        - name: AWS_STS_REGIONAL_ENDPOINTS
          value: regional
        image: grafana/grafana
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 10
          httpGet:
            path: /api/health
            port: 3000
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 30
        name: grafana
        ports:
        - containerPort: 80
          name: service
          protocol: TCP
        - containerPort: 3000
          name: grafana
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /api/health
            port: 3000
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 1Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/grafana/grafana.ini
          name: config
          subPath: grafana.ini
        - mountPath: /var/lib/grafana
          name: storage
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 472
        runAsGroup: 472
        runAsUser: 472
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: grafana
        name: config
      - emptyDir: {}
        name: storage
---
apiVersion: v1
data:
  grafana.ini: |
    [analytics]
    check_for_updates = true
    [auth]
    sigv4_auth_enabled = true
    [grafana_net]
    url = https://grafana.net
    [log]
    mode = console
    [paths]
    data = /var/lib/grafana/
    logs = /var/log/grafana
    plugins = /var/lib/grafana/plugins
    provisioning = /etc/grafana/provisioning
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: grafana
    meta.helm.sh/release-namespace: kube-system
  labels:
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 8.3.1
    helm.sh/chart: grafana-6.19.0
  name: grafana
  namespace: kube-system

---
apiVersion: v1
data:
  admin-password: emdCNHNuOGs0YU01ODRzRGYweHU1TzhYRW9IYUh6bEd4ckk1U1JOOQ==
  admin-user: YWRtaW4=
  ldap-toml: ""
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: grafana
    meta.helm.sh/release-namespace: kube-system
  labels:
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 8.3.1
    helm.sh/chart: grafana-6.19.0
  name: grafana
  namespace: kube-system
type: Opaque

This will create a Grafana deployment.

Step 6: Create EFS file system with following command:

git clone https://github.com/aws-samples/eks-efs-share-within-fargate.git
export EFS_CREATION_TOKEN=`uuidgen`
export CLUSTER_NAME=my-cluster
cd eks-efs-share-within-fargate
./scripts/epic02/create-efs.sh -c "$CLUSTER_NAME" -t "$EFS_CREATION_TOKEN"
./scripts/epic03/create-k8s-efs-csi-sc.sh

Step 7: Now go to EFS file system and create an access point with following details. Since Grafana pod is running as user 472, we have specified 472 as User and Group ID.

Step 8: Create a file named pv.yaml with following content and apply it.

Change volumeHandle: fs-02c5741aea873e03e::fsap-0823e1b8744e67ee1 as per your file system and access point id.

apiVersion: v1
kind: PersistentVolume
metadata:
   name: grafana-pv
   namespace: kube-system
spec:
   capacity:
      # Required field by Kubernetes.
      # Can be any value since EFS is an elastic file system
      # it doesn't really enforce any file system capacity.
      storage: 1Mi
   volumeMode: Filesystem
   accessModes:
      - ReadWriteMany
   persistentVolumeReclaimPolicy: Retain
   storageClassName: efs-sc
   csi:
      driver: efs.csi.aws.com
      volumeHandle: fs-02c5741aea873e03e::fsap-0823e1b8744e67ee1
      volumeAttributes:
        encryptInTransit: "true"


---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
   name: grafana-pvc
   namespace: kube-system
spec:
   accessModes:
      - ReadWriteMany
   storageClassName: efs-sc
   resources:
      requests:
         # Required field by Kubernetes.
         # Can be any value since EFS is an elastic file system 
         # it doesn't really enforce any file system capacity.
         storage: 1Mi

Step 9: Now we have our PV and PVC ready, let’s edit the grafana deployment using kubectl edit deployment grafana -n kube-system command and change

- name: storage
  emptyDir: {}

to

- name: storage
  persistentVolumeClaim:
    claimName: grafana-pvc

Check pod status using kubectl get pods -A and make sure pod is in running state.

At this point of time, our Grafana deployment is backed by persistent storage using EFS.