Adding Custom Python Packages for AWS Lambda Functions

Python is a popular language along with Javascript (NodeJS) for writing AWS lambda functions. Lambda function written in Python support the core modules, so one may choose to use the http.client instead of much simpler requests. However, if the function is to use some custom or non-native packages such as request and response we have few methods available to us.

In this article I will be discussing one such method of uploading a zip file containing all such custom packages and adding an AWS Lambda Layer to use this zip file for the particular function. We will be making use of Docker containers for this process. To be honest we actually do not need to go through the process of using a docker container. We can use only a simple pip install -t, zip the directory and upload it. However, certain python modules need to compile extensions written in C or C++. For such modules, the pip install -t approach will not work as the AWS Lambda functions use AWS Linux environment and you may have OSX, Windows or any other linux distribution of your choice. If you are sure that your modules do not have compiled extensions, please follow steps 2 and 3 below in this post.

Step 1 – Build and Run the Docker Container

The pre-requisite for this step is to have Docker installed. If you are on OSX, you can use Docker for Desktop. In this step we will use Amazon Linux base image and install desired version of python and few modules and OS packages. Amazon Linux 2 is a long term support release available at this moment. Amazon Linux 2 provides amazon-linux-extras which allows availability of newest application software on a stable base. At the time of writing this, Python 2.7 has been depricated by Amazon and the recommended version is Python 3.8. We would be needing to use amazon-linux-extras to install Python 3.8. Following Dockerfile is a very simple and self-explanatory file which we will be using to build our container –

FROM amazonlinux:2

RUN amazon-linux-extras enable python3.8 && \
          yum install -y python38 && \
          yum install -y python3-pip && \
          yum install -y zip && \
          yum clean all

RUN python3.8 -m pip install --upgrade pip && \
          python3.8 -m pip install virtualenv

Build the container using the following command –

$ docker build -f Dockerfile.awslambda -t aws_lambda_layer:latest

Once the container is built, it can be run as –

user1@macbook-air $ docker run -it --name aws_lambda_layer aws_lambda_layer:latest bash

This will give the bash shell inside the container. Next step will install the required modules in a python 3.8 virtual-environment and package as a zip file

Step 2 – Install Non-Native Packages and Package These As A Zip File

We will install the required packages inside a virtual environment, this will allow to reuse the same container for other future packaging also.

# python3.8 -m venv venv-telethon

Next, activate the virtual environment and install the packages in it under a specific folder, so that the same can be packaged. After packaging the folder, the zip file needs to be copied outside the container so that the same can be uploaded –

# source venv-telethon/bin/activate
(venv-telethon) # pip install telethon -t ./python
(venv-telethon) # deactivate

# zip -r python.zip ./python/

user1@macbook-air $ docker cp aws_lambda_layer:python.zip ./Desktop/

Step 3 – Upload the Package to the AWS Lambda Layer or S3

If the zip file is more than 50 MB, it has to be uploaded to Amazon S3 store. In case you decide to upload it S3 store, ensure that the path of the file is recorded carefully.

To upload the file, under the Lambda->Layer, click on Create Layer and fill up the form. The form will allow to upload the zip file or specify the path of the AWS S3 location where the file is uploaded.

Now write your lambda function and use the modules which were uploaded as a part of the zip file.