Securely Accessing Opensearch Serverless Collection From A Lambda Function

In this post, we’ll see how to access Opensearch serverless collection from a Lambda using VPC endpoint.

Prerequisite: Need to have an OpenSearch serverless collection with an index and some data.

Step 1: Create a network policy for Opensearch serverless collection which allows access from a VPC endpoint only. If you don’t have an existing VPC endpoint for Opensearch service, you can create one from here. For demo purpose, I have allowed all traffic in the security group.

Step 2: Create a Lambda function with Python 3.11 runtime and 30 seconds timeout. Follow https://docs.aws.amazon.com/lambda/latest/dg/python-package.html to create a zip with lambda code and required packages.

Packages required:

pip install --target ./package boto3 requests requests_aws4auth

Lambda code:

import json
import requests
from requests_aws4auth import AWS4Auth
import boto3
import logging
import os

# Configure logging
logger = logging.getLogger()
logger.setLevel("INFO")

host = os.environ['OPENSEARCH_ENDPOINT']
region = "us-east-1"
service = 'aoss'
headers = {'Content-Type': 'application/json'}

credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

def get_request(index_name,path, http_body):
    if index_name =='':
        get_url=f'{host}{path}'
    else:
        get_url = f'{host}{index_name}{path}'
    logger.info(f'GET Request URL: {get_url}')
    logger.info(f'http_body: {json.dumps(http_body)}')
    logger.info(f'Headers: {json.dumps(headers)}')
    response = requests.get(get_url, auth=awsauth, data=http_body.encode('utf-8'),headers=headers)
    if index_name =='':
        output = response.text  
    else:
        output = response.json()
    if response.status_code == 200:
        return {
            'statusCode': 200,
            'body': output
        }
    else:
        return {
            'statusCode': response.status_code,
            'body': response.text
        }
'''
def post_request(path, http_body):
    post_url = f'{host}{index_name}{path}'
    response = requests.post(post_url, data=http_body.encode('utf-8'), auth=awsauth,headers=headers)
    if response.status_code == 200:
        return {
            'statusCode': 200,
            'body': response.json()
        }
    else:
        return {
            'statusCode': response.status_code,
            'body': response.text
        }
'''
def lambda_handler(event, context):
    logger.info(event)
    http_method = event['httpMethod']
    http_path = event['httpPath']
    if 'indexName' in event:
        index_name = event['indexName']
    else:
        index_name=''
    http_body = json.dumps(event['httpBody'])

    if not http_method or not http_path or not http_body :
        return {
            'statusCode': 400,
            'headers': {'Content-Type': 'application/json'},
            'body': json.dumps({'error': 'Missing required parameters'})
        }
    if http_method == 'GET':
        return get_request(index_name,http_path, http_body)
    #elif http_method == 'POST':
    #    return post_request(http_path, http_body)
    else:
        return {
            'statusCode': 400,
            'body': json.dumps('Invalid method specified')
        }

    

Associate this Lambda with the VPC where we have created VPC endpoint for Opensearch. For this you need to add following IAM policy to the Lambda IAM role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DeleteNetworkInterface"
            ],
            "Resource": "*"
        }
    ]
}

You may use the same subnets and security group which we used for VPC endpoint. Also add an environment variable for Opensearch endpoint.

OPENSEARCH_ENDPOINT https://<COLLECTION_ID>.us-east-1.aoss.amazonaws.com/

Step 3: Add following policy to the Lambda IAM role to grant access to Opensearch collection.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "aoss:DashboardsAccessAll",
                "aoss:APIAccessAll"
            ],
            "Resource": "*"
        }
    ]
}

Step 4: Edit the data access policy of the Opensearch collection and grant following access to the IAM role associated with the Lambda function.

Step 5: Now you can trigger your lambda with payloads like this:

To get index details

{
    "httpMethod": "GET",
    "httpPath": "_cat/indices",
    "httpBody" :{}
}

To search a document in an index

{
    "httpMethod": "GET",
    "httpPath": "/_search",
    "httpBody" :{
        "query":{
            "ids": {
            "values": [1]
        }
        }
    },
     "indexName": "my_index"
}

To find the number of documents in an index

{
    "httpMethod": "GET",
    "httpPath": "/_count",
    "httpBody" :{},
    "indexName": "my_index"
}