AWS Lambda in Python with SAM

The AWS SAM and CloudFormation mix works well for my projects.
I have been working mainly with Python for building Lambda functions on AWS.
However, managing code from project to production has been less than trivial.
AWS Lambda Deployment Package in Python

This article and the sample project on GitHub shows how to

• structure a Python project for Serverless functions,
• deploy app using SAM and CloudFormation.

A sample project demonstrating this approach is at  GitHub:designfultech/python-aws

Building and deploying JavaScript functions using SAM was super simple – see Watcher project.
SAM’s original project structure for Python is less than ideal.

• 3rd party libraries, dependencies, are expected to be in the project root.
• All project assets in the root will be deployed.
• I want to maintain a virtualenv for development and define different dependencies (requirements.txt) for the runtime.

Project structure

I use the Pycharm IDE for development. My projects have their own virtual environments, and I use the following project structure.

Where

• source has all the deployable source code
  • ext 3rd party libraries, not a Python package
    • requirements.txt – use it to install dependencies
  • lib keeps my own shared libraries
    • init.py it is a Python package
    • vendor.py library for vendoring, more on this later
  • functions all function(s) code
    • __init.py it is a Python package
  • I may add other packages like models for ORM
  • runtime_context.py more on this later
  • requirements.txt this one is kept empty for SAM build
• dist is created by SAM build for the deployment
• requirements.txt development-time dependencies for the project
• template.yaml AWS serverless application template

Vendoring

Vendoring is a technique to incorporate 3rd party packages, dependencies into the project. It is a neat trick used in other languages, and this specific one is adopted from Google’s App Engine library.

1. Create a directory in the root of the project for the 3rd party packages ext. Add ext to your Python path so dependencies resolve during development.
2. Create and maintain the requirements.txt inside ext for the deployed runtime.
3. Install the packages in the ext directory.
   pip install -t . -r requirements.txt
4. How to make the code in the ext directory available to the runtime?
   This is where Google’s helper – google.appengine.ext.vendor – comes in. It adjusts the path for the Python runtime.
5. Add the code to a project file, for example: /lib/vendor.py
6. My approach is then to create the runtime_context.py in the root of the project
   import os from lib import vendor vendor.add(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'ext'))
7. Any Python application that needs access to the packages in the extdirectory needs a single line of import.
   import runtime_context

Using the runtime_context.py

I use the runtime_context.py to setup vendoring for all functions.
I also place shared configurations and variables here, for example:

• logging
• environment variables

import logging LOGGER = logging.getLogger()
LOGGER.setLevel(logging.DEBUG)

import os
GREETING = os.environ.get('GREETING', 'Hello World!')

SAM

The approach and project structure described here works for development, local test with SAM, and AWS runtime.

When deploying to AWS, run the SAM CLI from the root of the project, where the template file is. Use a distribution directory dist for building and packaging.

• Install the 3rd party modules

cd source/ext pip install -r requirements.txt

• Build the function

sam build --template template.yaml --build-dir ./dist

• Package the function
  [BUCKET_NAME] is the name for the bucket where the deployment artefacts are uploaded.

sam package --s3-bucket [BUCKET_NAME] --template-file dist/template.yaml\
  --output-template-file dist/packaged.yaml

• Deploy the function
  [STACK_NAME] is the name of the stack for the deployment.

aws cloudformation deploy --template-file dist/packaged.yaml\
  --stack-name [STACK_NAME] --s3-bucket [BUCKET_NAME]

Remove deployed app

When done with the application, un-dpeloy it by removing the stack.

aws cloudformation delete-stack --stack-name PythonAppStack

Final notes

This is just one approach that works for me. There are probably numerous other ways to make coding Python functions easy and comfortable.