Jun 22, 2021

AWS Python Lambda: Not Quite So Simple?

Python + AWS Lambda…simple, quick, effective code delivered at the click of an upload button? Not always.

Python lets you write complex solutions quickly using English-like syntax and AWS Lambda lets you edit in a browser-based editor within the AWS Console (or upload a .zip file of your code with the click of a button); so when is it not that quick and easy and how can you avoid the headaches?

Where the Python/Lambda relationship starts to degrade a bit is around dependency packages. Packages outside of the Python Standard Library (and a few AWS-specific libraries provided in the Boto3 AWS SDK for Python) cannot be added in the online Lambda editor but need to be zipped up and uploaded. Although this may feel a bit gauche by today’s modern development standards, it functionally solves the problem…usually.

Now we get to the crux of our issue here. What if your beautifully simple Python code either directly or indirectly relies on one or more of those operating system specific c libraries? And what if you are not, in fact, developing on a Linux system? Turns out, your non-Linux (usually MacOS or Windows) system libraries might not work on the Linux runtime that Lambdas use. So what to do (switching to a Linux-based development environment is not always a viable, or desirable, option)?

One solution is to utilize a Linux-based docker container to bundle all your perfect Python code and its dependencies into a .zip file for you. This solution assumes a few things:

  1. You are using Python 3.8 (other versions can utilize this solution but the specific example is for 3.8).
  2. You have pip installed and are using it for dependent packages.
  3. You have docker installed in your development environment (who doesn’t these days, it is terribly handy!)

Here’s an example Dockerfile for just such a container:

FROM centos/python-38-centos7
COPY ./docker-entrypoint.sh /
WORKDIR /working
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["bash"]

Breaking this down, this is for Python 3.8 code (hence the python-38-centos7 base image, other versions would use their own base image… this could be generalized by installing Python on a centos base image, but I took the shortcut this time). And we’re going to utilize a “magic” docker-entrypoint.sh script to do our bidding within the container…

set -e
rm -rf temp
rm -rf bundle.zip
pip install -r requirements.txt -t temp
cd temp; zip -r ../bundle.zip *; cd ..
zip -g bundle.zip *.py
rm -rf temp
exec "$@"

Here we have a bit more to delve into! First we clean up any previous runs in lines 3 and 4. We then use pip to install all our dependent packages, as listed in our requirements.txt, into a temp directory and we zip all those packages and our code into a tidy little bundle.zip. Despite our initial cleanup on line 3, we attempt to cleanup after ourselves in line 8 as well.

Once you docker build -t your-clever-tag name . from the directory your Dockerfile is in to create your local image (and push it up to your shared repo if you want the rest of your team to have such an all powerful image), how does it all work? Well we leverage a little docker cleverness to link the magic with our actual code via another relatively simple script:

while test $# -gt 0; do
  case "$1" in
      echo "options:"
      echo "-f, --freeze execute pip freeze to update requirements.txt"
    -f| --freeze)
      pip freeze > requirements.txt
docker run --rm --name lambda_python_bundler -v ${PWD}:/working lambda_python_bundler
echo "built"

Execute this script from your Python code directory (I’ve called my docker image lambda_python_bundler, and am just running the latest version). There’s a simple -f/--freeze option to auto-replace your requirements.txt file with a current snapshot via pip, and it simply mounts your Python code directory to the container (-v ${PWD}:/working). Once running, the aforementioned “magical” docker-entypoint.sh script will do its thing et voila; you will have a Linux-kernel-compatible `bundle.zip` to upload and run in AWS Lambda.

Source code can be found at https://github.com/rosspenna/lambda-python-bundler

About the Author

Ross Penna profile.

Ross Penna

Sr. Consultant

Ross is experienced in providing solutions in software; with a background in Java, Spring, APIs, SQL, various AWS services, and a track record of picking up new technologies and techniques quickly and efficiently while delivering enterprise scale solutions. A veteran of multiple SDLCs including Agile, agile-fall, Kanban, Scrum, and Waterfall, he is able to integrate quickly into teams and situations to start providing solutions.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
An Exploration in Rust: Musings From a Java/C++ Developer
Why Rust? It’s fast (runtime performance) It’s small (binary size) It’s safe (no memory leaks) It’s modern (build system, language features, etc) When Is It Worth It? Embedded systems (where it is implied that interpreted […]
Getting Started with CSS Container Queries
For as long as I’ve been working full-time on the front-end, I’ve heard about the promise of container queries and their potential to solve the majority of our responsive web design needs. And, for as […]
Simple improvements to making decisions in teams
Software development teams need to make a lot of decisions. Functional requirements, non-functional requirements, user experience, API contracts, tech stack, architecture, database schemas, cloud providers, deployment strategy, test strategy, security, and the list goes on. […]
JavaScript Bundle Optimization – Polyfills
If you are lucky enough to only support a small subset of browsers (for example, you are targeting a controlled set of users), feel free to move along. However, if your website is open to […]