How to use EFS to store cx_Oracle, Pandas, and other python packages?

This post focuses on how to use the EFS storage to store large packages and libraries like cx_Oracle, pandas, and pymssql and import the packages in AWS Lambda. Considering the Lambda package size limitation that is inclusive of layers, larger functions packages and libraries must be stored outside the Lambda package.

There are some steps that you do not need to follow as it has been done, and you can mount the EFS to your lambda and import the package. However, I will be logging the steps to ensure we all can reference the steps in the future – technical debt.

In short:

1. launched an EC2
2. created an EFS Storage
3. SSH to the EC2
4. Mount the EFS to the EC2
5. created a sample python venv project
6. installed all the package requirements I had in the virtual environment
7. moved the site_packages contents to the EFS/packages directory
8. created a directory in EFS and called it sl (shared libraries)
9. moved the libraries including the cx_Oracle the EFS/sl/oracle
10. created a test function in AWS Lambda using the code below
11. added an environment variable entry in the AWS Lambda Configuration
12. and Test

In Long:

I will start the details from step 4 onwards:

mkdir efs mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-6212b722.efs.ap-southeast-1.amazonaws.com:/ efs

Then I downloaded the latest version of instant client from Oracle website here (Basic Package (ZIP)):

wget https://download.oracle.com/otn_software/linux/instantclient/211000/instantclient-basic-linux.x64-21.1.0.0.0.zip
unzip instantclient-basic-linux.x64-21.1.0.0.0.zip

Renaming the directory to oracle before copying it to the EFS:mv instantclient_21_1 oracle

creating the necessary directories in EFS

mkdir -p efs/sl/

Then moving the oracle instant client to the efs

mv oracle efs/sl/

Now I will be creating a python virtual environment in EC2 to download the necessary packages and copy them to the EFS

python3.8 -m venv venv 
source venv/bin/activate 
pip install pandas cx_Oracle pymysql mymssql

Let’s check the packages in the venv site packages and then copy them to the EFS

ls venv/lib64/python3.8/site-packages/ 
mkdir -p efs/packages 
mv venv/lib64/python3.8/site-packages/* efs/packages/

At this point, we have python requirements and shared objects/libraries copied to the EFS. Let’s mount the EFS in Lambda and try using the libraries and objects.

To mount the EFS in AWS Lambda go to Configuration > File systems and click on Add file system.

Once you select the EFS file system and Access point you will need to enter the Local mount path in the AWS Lambda which must be an absolute path under /mnt. Save the file system and go to the next step.

You must add the environment variable before moving to the function test.

To add an environment variable go to Configuration > Environment variables

Click on the Edit and then Add environment variable and enter the key and value as per below:

LD_LIBRARY_PATH=/mnt/lib/packages:/mnt/lib/sl/oracle/lib:$LD_LIBRARY_PATH

^^^^^^^^^^ Pay attention to the path and joins

Sample Python code to test the libraries:

import json 
import sys 
import os 

sys.path.append("/mnt/lib/packages") 

import cx_Oracle 


conn = 'username/password@123.123.123.123/ORCL' 
curs = cx_Oracle.Connection(conn) 

You must append the system path with the packages directory from the mount point sys.path.append(“/mnt/lib/packages”).

Cheers