AWS Python Lambda: Not Quite So Simple?
Python + AWS Lambda…simple, quick, effective code delivered at the click of an upload button? Not always.
Python lets you write complex solutions quickly using English-like syntax and AWS Lambda lets you edit in a browser-based editor within the AWS Console (or upload a .zip
file of your code with the click of a button); so when is it not that quick and easy and how can you avoid the headaches?
Where the Python/Lambda relationship starts to degrade a bit is around dependency packages. Packages outside of the Python Standard Library (and a few AWS-specific libraries provided in the Boto3 AWS SDK for Python) cannot be added in the online Lambda editor but need to be zipped up and uploaded. Although this may feel a bit gauche by today’s modern development standards, it functionally solves the problem…usually.
Now we get to the crux of our issue here. What if your beautifully simple Python code either directly or indirectly relies on one or more of those operating system specific c libraries? And what if you are not, in fact, developing on a Linux system? Turns out, your non-Linux (usually MacOS or Windows) system libraries might not work on the Linux runtime that Lambdas use. So what to do (switching to a Linux-based development environment is not always a viable, or desirable, option)?
One solution is to utilize a Linux-based docker container to bundle all your perfect Python code and its dependencies into a .zip
file for you. This solution assumes a few things:
- You are using Python 3.8 (other versions can utilize this solution but the specific example is for 3.8).
- You have
pip
installed and are using it for dependent packages. - You have
docker
installed in your development environment (who doesn’t these days, it is terribly handy!)
Here’s an example Dockerfile
for just such a container:
FROM centos/python-38-centos7
COPY ./docker-entrypoint.sh /
WORKDIR /working
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["bash"]
Breaking this down, this is for Python 3.8 code (hence the python-38-centos7
base image, other versions would use their own base image… this could be generalized by installing Python on a centos
base image, but I took the shortcut this time). And we’re going to utilize a “magic” docker-entrypoint.sh
script to do our bidding within the container…
#!/bin/bash
set -e
rm -rf temp
rm -rf bundle.zip
pip install -r requirements.txt -t temp
cd temp; zip -r ../bundle.zip *; cd ..
zip -g bundle.zip *.py
rm -rf temp
exec "$@"
Here we have a bit more to delve into! First we clean up any previous runs in lines 3 and 4. We then use pip
to install all our dependent packages, as listed in our requirements.txt
, into a temp
directory and we zip all those packages and our code into a tidy little bundle.zip
. Despite our initial cleanup on line 3, we attempt to cleanup after ourselves in line 8 as well.
Once you docker build -t your-clever-tag name .
from the directory your Dockerfile
is in to create your local image (and push it up to your shared repo if you want the rest of your team to have such an all powerful image), how does it all work? Well we leverage a little docker
cleverness to link the magic with our actual code via another relatively simple script:
while test $# -gt 0; do
case "$1" in
-h|--help)
echo "options:"
echo "-f, --freeze execute pip freeze to update requirements.txt"
;;
-f| --freeze)
shift
pip freeze > requirements.txt
;;
esac
done
docker run --rm --name lambda_python_bundler -v ${PWD}:/working lambda_python_bundler
echo "built"
Execute this script from your Python code directory (I’ve called my docker image lambda_python_bundler
, and am just running the latest
version). There’s a simple -f/--freeze
option to auto-replace your requirements.txt
file with a current snapshot via pip
, and it simply mounts your Python code directory to the container (-v ${PWD}:/working
). Once running, the aforementioned “magical” docker-entypoint.sh
script will do its thing et voila; you will have a Linux-kernel-compatible `bundle.zip` to upload and run in AWS Lambda.
Source code can be found at https://github.com/rosspenna/lambda-python-bundler