Jun 22, 2021

AWS Python Lambda: Not Quite So Simple?

Python + AWS Lambda…simple, quick, effective code delivered at the click of an upload button? Not always.

Python lets you write complex solutions quickly using English-like syntax and AWS Lambda lets you edit in a browser-based editor within the AWS Console (or upload a .zip file of your code with the click of a button); so when is it not that quick and easy and how can you avoid the headaches?

Where the Python/Lambda relationship starts to degrade a bit is around dependency packages. Packages outside of the Python Standard Library (and a few AWS-specific libraries provided in the Boto3 AWS SDK for Python) cannot be added in the online Lambda editor but need to be zipped up and uploaded. Although this may feel a bit gauche by today’s modern development standards, it functionally solves the problem…usually.

Now we get to the crux of our issue here. What if your beautifully simple Python code either directly or indirectly relies on one or more of those operating system specific c libraries? And what if you are not, in fact, developing on a Linux system? Turns out, your non-Linux (usually MacOS or Windows) system libraries might not work on the Linux runtime that Lambdas use. So what to do (switching to a Linux-based development environment is not always a viable, or desirable, option)?

One solution is to utilize a Linux-based docker container to bundle all your perfect Python code and its dependencies into a .zip file for you. This solution assumes a few things:

  1. You are using Python 3.8 (other versions can utilize this solution but the specific example is for 3.8).
  2. You have pip installed and are using it for dependent packages.
  3. You have docker installed in your development environment (who doesn’t these days, it is terribly handy!)

Here’s an example Dockerfile for just such a container:

FROM centos/python-38-centos7
COPY ./docker-entrypoint.sh /
WORKDIR /working
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["bash"]

Breaking this down, this is for Python 3.8 code (hence the python-38-centos7 base image, other versions would use their own base image… this could be generalized by installing Python on a centos base image, but I took the shortcut this time). And we’re going to utilize a “magic” docker-entrypoint.sh script to do our bidding within the container…

#!/bin/bash
set -e
rm -rf temp
rm -rf bundle.zip
pip install -r requirements.txt -t temp
cd temp; zip -r ../bundle.zip *; cd ..
zip -g bundle.zip *.py
rm -rf temp
exec "$@"

Here we have a bit more to delve into! First we clean up any previous runs in lines 3 and 4. We then use pip to install all our dependent packages, as listed in our requirements.txt, into a temp directory and we zip all those packages and our code into a tidy little bundle.zip. Despite our initial cleanup on line 3, we attempt to cleanup after ourselves in line 8 as well.

Once you docker build -t your-clever-tag name . from the directory your Dockerfile is in to create your local image (and push it up to your shared repo if you want the rest of your team to have such an all powerful image), how does it all work? Well we leverage a little docker cleverness to link the magic with our actual code via another relatively simple script:

while test $# -gt 0; do
  case "$1" in
    -h|--help)
      echo "options:"
      echo "-f, --freeze execute pip freeze to update requirements.txt"
      ;;
    -f| --freeze)
      shift
      pip freeze > requirements.txt
      ;;
  esac
done
docker run --rm --name lambda_python_bundler -v ${PWD}:/working lambda_python_bundler
echo "built"

Execute this script from your Python code directory (I’ve called my docker image lambda_python_bundler, and am just running the latest version). There’s a simple -f/--freeze option to auto-replace your requirements.txt file with a current snapshot via pip, and it simply mounts your Python code directory to the container (-v ${PWD}:/working). Once running, the aforementioned “magical” docker-entypoint.sh script will do its thing et voila; you will have a Linux-kernel-compatible `bundle.zip` to upload and run in AWS Lambda.

Source code can be found at https://github.com/rosspenna/lambda-python-bundler

About the Author

Ross Penna profile.

Ross Penna

Sr. Consultant

Ross is experienced in providing solutions in software; with a background in Java, Spring, APIs, SQL, various AWS services, and a track record of picking up new technologies and techniques quickly and efficiently while delivering enterprise scale solutions. A veteran of multiple SDLCs including Agile, agile-fall, Kanban, Scrum, and Waterfall, he is able to integrate quickly into teams and situations to start providing solutions.

Leave a Reply

Your email address will not be published.

Related Blog Posts
A security model for developers
Software security is more important than ever, but developing secure applications is more confusing than ever. TLS, mTLS, RBAC, SAML, OAUTH, OWASP, GDPR, SASL, RSA, JWT, cookie, attack vector, DDoS, firewall, VPN, security groups, exploit, […]
Building Better Data Visualization Experiences: Part 1 of 2
Through direct experience with data scientists, business analysts, lab technicians, as well as other UX professionals, I have found that we need a better understanding of the people who will be using our data visualization products in order to build them. Creating a product utilizing data with the goal of providing insight is fundamentally different from a typical user-centric web experience, although traditional UX process methods can help.
Kafka Schema Evolution With Java Spring Boot and Protobuf
In this blog I will be demonstrating Kafka schema evolution with Java, Spring Boot and Protobuf.  This app is for tutorial purposes, so there will be instances where a refactor could happen. I tried to […]
Redis Bitmaps: Storing state in small places
Redis is a popular open source in-memory data store that supports all kinds of abstract data structures. In this post and in an accompanying example Java project, I am going to explore two great use […]