Starting a Python-oriented Serverless-dot-com Project

In the past few months, I've created about a half-dozen projects using "serverless" infrastructure on Amazon Web Services (AWS). (And I'm about to start another one.) Over the course of these projects, I've refined my development environment into something that I think is useful to share, so read on for how to make Python, Node, and Serverless.com work together and work independently from your other projects.

About "Serverless"

"Serverless" is both a term for a kind of computing environment and the name of a framework that helps manage such environments. As a computing environment, "serverless" abstracts away the needs to manage the servers and underlying operating systems from the task of writing and running code. If you assume that a fully-patched server at the required capacity is ready and waiting to run your code, then a serverless environment allows the developer to focus on just running the code. Someone else will deal with the other parts. AWS' Lambda is probably the best known, but other major cloud computing environments (Microsoft Azure, Google Cloud Services, Cloudflare Workers) and datacenter tools (Apache OpenWhisk, Kubeless) have the same thing.

"serverless.com " is also the name of a specific framework that helps developers manage serverless environments. It takes care of the tasks of bundling up code, setting up the appropriate triggers (web APIs, message queues, etc.), managing versions, and similar tasks. To make matters even more confusing, "Serverless.com" is also a service for managing workloads in serverless environments...so hopefully you can see that talking about "serverless" quickly gets one to "what 'serverless' are you talking about?" As far as understanding serverless-the-framework, I recommend skipping the homepage and going right to the framework documentation.

Building Up the Environment

There is one globally-installed prerequisite that I use: pipenv. Pipenv creates isolated Python environments...the python executable and installed modules for the project are separated from those of the underlying operating system. There are many isolated Python environment tools—pipenv, virtualenv, poetry—but I've used pipenv for a long time and it has the advantage of working with Eugene Kalinin's nodeenv project: a Node isolation tool that integrates with pipenv. In other words, in one directory I'm getting both Python isolation and Node isolation.

The numbered steps below are the sequence of commands to set this up. If you want to see what an empty shell looks like—along with some strong opinions about how I like to set up Serverless for myself—check out this GitHub repository: dltj/serverless-template.

1. mkdir serverless_project && cd serverless_project — create an empty directory and change into it
2. PIPENV_VENV_IN_PROJECT=1 pipenv install — create an isolated installation of Python in this environment [note 1]
3. pipenv install --dev nodeenv — install the 'nodeenv' package in the isolated Python environment
4. pipenv shell — enter the isolated Python environment
5. nodeenv -p — install node in the isolated Python environment and hook into the pipenv script
6. npm install serverless — install Serverless into the node environment and pin version in package.json [note 2]
7. exit — For serverless to install correctly in the environment...
8. pipenv shell — ...we need to exit out and re-enter the environment
9. npm install -g serverless — add the Serverless commands into the Python environment (.venv/bin) [note 3]
10. npm install --include=dev serverless-python-requirements — we're probably going to want to use the serverless-python-requirements plug-in since we are developing the AWS Lambdas in Python

You are then ready to start your serverless framework development using the serverless command to build an empty shell.

Note 1: I also have a stong preference for putting the Python virtual environment in the directory where the code lives, hence the PIPENV_VENV_IN_PROJECT=1 environment variable fed into the pipenv command.

Note 2: At this point, you can also specify a version of the Serverless framework. As I'm writing this, Serverless jumped to a 3.x.x version but I'm waiting for that major release to settle down. So I use npm install serverless@2.72.2

Note 3: Although the -g global flag is being used, we are inside the Python environment at this point so the Serverless install is in the Python/Node environment. Note, too, that this command "globally" installs the version pinned in package.json from the earlier npm install command.

How Others Use Your Environment

With the versions of commands and dependencies pinned in their respective Python and Node package files, someone can reproduce the same isolated Python/Node environment with these commands. Add these instructions to your README file:

1. git clone serverless_project && cd serverless_project
2. PIPENV_VENV_IN_PROJECT=1 pipenv install # create an isolated installation of Python in this environment
3. pipenv shell
4. nodeenv -p # Installs Node environment inside Python environment
5. npm install --include=dev # Installs Node packages inside combined Python/Node environment
6. exit # For serverless to install correctly in the environment...
7. pipenv shell # ...we need to exit out and re-enter the environment
8. npm install -g serverless # Although the '-g' global flag is being used, Serverless install is in the Python/Node environment