Jun 22, 2021

AWS Python Lambda: Not Quite So Simple?

Python + AWS Lambda…simple, quick, effective code delivered at the click of an upload button? Not always.

Python lets you write complex solutions quickly using English-like syntax and AWS Lambda lets you edit in a browser-based editor within the AWS Console (or upload a .zip file of your code with the click of a button); so when is it not that quick and easy and how can you avoid the headaches?

Where the Python/Lambda relationship starts to degrade a bit is around dependency packages. Packages outside of the Python Standard Library (and a few AWS-specific libraries provided in the Boto3 AWS SDK for Python) cannot be added in the online Lambda editor but need to be zipped up and uploaded. Although this may feel a bit gauche by today’s modern development standards, it functionally solves the problem…usually.

Now we get to the crux of our issue here. What if your beautifully simple Python code either directly or indirectly relies on one or more of those operating system specific c libraries? And what if you are not, in fact, developing on a Linux system? Turns out, your non-Linux (usually MacOS or Windows) system libraries might not work on the Linux runtime that Lambdas use. So what to do (switching to a Linux-based development environment is not always a viable, or desirable, option)?

One solution is to utilize a Linux-based docker container to bundle all your perfect Python code and its dependencies into a .zip file for you. This solution assumes a few things:

  1. You are using Python 3.8 (other versions can utilize this solution but the specific example is for 3.8).
  2. You have pip installed and are using it for dependent packages.
  3. You have docker installed in your development environment (who doesn’t these days, it is terribly handy!)

Here’s an example Dockerfile for just such a container:

FROM centos/python-38-centos7
COPY ./docker-entrypoint.sh /
WORKDIR /working
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["bash"]

Breaking this down, this is for Python 3.8 code (hence the python-38-centos7 base image, other versions would use their own base image… this could be generalized by installing Python on a centos base image, but I took the shortcut this time). And we’re going to utilize a “magic” docker-entrypoint.sh script to do our bidding within the container…

#!/bin/bash
set -e
rm -rf temp
rm -rf bundle.zip
pip install -r requirements.txt -t temp
cd temp; zip -r ../bundle.zip *; cd ..
zip -g bundle.zip *.py
rm -rf temp
exec "$@"

Here we have a bit more to delve into! First we clean up any previous runs in lines 3 and 4. We then use pip to install all our dependent packages, as listed in our requirements.txt, into a temp directory and we zip all those packages and our code into a tidy little bundle.zip. Despite our initial cleanup on line 3, we attempt to cleanup after ourselves in line 8 as well.

Once you docker build -t your-clever-tag name . from the directory your Dockerfile is in to create your local image (and push it up to your shared repo if you want the rest of your team to have such an all powerful image), how does it all work? Well we leverage a little docker cleverness to link the magic with our actual code via another relatively simple script:

while test $# -gt 0; do
  case "$1" in
    -h|--help)
      echo "options:"
      echo "-f, --freeze execute pip freeze to update requirements.txt"
      ;;
    -f| --freeze)
      shift
      pip freeze > requirements.txt
      ;;
  esac
done
docker run --rm --name lambda_python_bundler -v ${PWD}:/working lambda_python_bundler
echo "built"

Execute this script from your Python code directory (I’ve called my docker image lambda_python_bundler, and am just running the latest version). There’s a simple -f/--freeze option to auto-replace your requirements.txt file with a current snapshot via pip, and it simply mounts your Python code directory to the container (-v ${PWD}:/working). Once running, the aforementioned “magical” docker-entypoint.sh script will do its thing et voila; you will have a Linux-kernel-compatible `bundle.zip` to upload and run in AWS Lambda.

Source code can be found at https://github.com/rosspenna/lambda-python-bundler

About the Author

Ross Penna profile.

Ross Penna

Sr. Consultant

Ross is experienced in providing solutions in software; with a background in Java, Spring, APIs, SQL, various AWS services, and a track record of picking up new technologies and techniques quickly and efficiently while delivering enterprise scale solutions. A veteran of multiple SDLCs including Agile, agile-fall, Kanban, Scrum, and Waterfall, he is able to integrate quickly into teams and situations to start providing solutions.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
Planning an Apache Airflow Deployment
This is part one of a five-part series addressing Airflow at an enterprise scale. I will update these with links as they are published. Airflow: Planning a Deployment Apache Airflow is a platform for authoring, […]
Rockstar Development
This originally appeared on Marty Henderson’s personal blog Or, how to use Gitpod and GitLab so that no one else has to care about your questionable coding language choices. A true rockstar has a good […]
Testing a Quarkus Kafka Application
Quarkus, a “Kubernetes Native Java stack,” enables lighter Java applications with faster startup times. In a recent post, I talked about scaling Kafka consumers in Kubernetes. Quarkus applications fit right into this picture because they […]
Gitpod and Hringvegurinn
Iceland Ever seen an advertisement for visiting Iceland? Have you noticed that they all mention Hringvegurinn or the Ring Road, as a good tour? (If you haven’t seen a tour ad for Iceland, Steindi Jr […]