Using CloudWatch Custom Data Identifiers To Mask Sensitive Data In Logs

Reference: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL-custom-data-identifiers.html

View unmasked data

In this post, we’ll explore how to use CloudWatch Custom Data Identifiers to mask sensitive data in CloudWatch Logs.

Step 1: Create a CloudWatch Log group and a log stream named test. Based on your requirements, create a Data Protection policy.

I have created a policy with following regex which is used to match anything that comes immediately after /v1/ in a string, stopping when it encounters a comma (,), closing brace (}), double quote ("), or space ( ).

\/v1\/[^,}" ]+

Step 2: Create a script to send JSON style logs to your CloudWatch log group

#!/bin/bash
EVENT_TIME=$(date +%s000)
LOG_LEVEL="/v1/vinayak"
EVENT_SOURCE=myapp
MESSAGE=/v1/vinayak

OUTPUT=$(jq -n \
  --arg EventTime "$EVENT_TIME" \
  --arg LogLevel "$LOG_LEVEL" \
  --arg EventSource "$EVENT_SOURCE" \
  --arg Message "$MESSAGE" \
      '{EventTime:$EventTime,LogLevel:$LogLevel,EventSource:$EventSource,Message:$Message}')


LOG_MESSAGE=$(echo $OUTPUT | sed 's/"/\\"/g') 
   
aws logs put-log-events --log-group-name test --log-stream-name test --log-events timestamp=$(date +%s000),message=\""$LOG_MESSAGE"\"

Step 3: Execute the script and check CloudWatch logs. Anything after /v1 should be masked like this

{
    "EventTime": "1737615105000",
    "LogLevel": "***********",
    "EventSource": "myapp",
    "Message": "***********"
}

Step 4: If you have the required permission, you can temporarily unmask protected data.