Package your Python project into a wheel Link to heading

Today, let’s go over how to package your Python codebase into a wheel file so that anyone can easily install with pip install command. In particular, we will also cover the following:

  • resolve dependencies of the codebase
  • include data files
  • exclude tests and other files
  • dev environment setup
  • explicitly specify top-level packages to include

Image

Let’s start with a minimal project. Suppose we have two files in the project. Link to heading 

project/
├── mean.py
└── sum.py

# sum.py
import numpy as np
from typing import Iterable

def add_all(xs: Iterable):
    return np.sum(xs).item()


# mean.py
from sum import add_all
from typing import Iterable

def take_average(xs: Iterable):
    n = len(xs)
    return add_all(xs) / n

Notice that our codebase requires numpy package. Let’s first test if our project works as expected.

# install dependency
pip install numpy

# should print "3.0"
python -c 'import mean; print(mean.take_average([1,2,3,4,5]))'

Looks good. Now, we want to package this as a wheel file so that people can easily install and use it like below

# install the project from a wheel file
pip install project.whl

# should print "3.0"
python -c 'from awesome_project import mean; print(mean.take_average([1,2,3,4,5])'

Step 1. Reorganize Link to heading

The first step is to reorganize the project directory following the convention. We will create a new folder as the package name and move the files to the folder. We will name our package awesome-project. Note that we must replace — with _ when naming the folder; that is, we create a new folder awesome_project, not awesome-project.

project/
└── awesome_project
    ├── mean.py
    └── sum.py

Step 2. Use relative import Link to heading

Change imports within the codebase as relative import. For example, mean.py imports sum.py with the following import statement

from sum import add_all

When we package the project, this will no longer work. Hence, we need to add . in the import statement

from .sum import add_all

Step 3. Add pyproject.toml Link to heading

In the past, setup.py used to be the way to go. These days, pyproject.toml file is the way to go. If you are interested in the details, check out PEP518. Let’s create a pyproject.toml file in the project root.

project/
├── awesome_project
│   ├── mean.py
│   └── sum.py
└── pyproject.toml

Let’s start with the minimal

# pyproject.toml
[build-system]
requires = ["setuptools >= 61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "awesome-project"
version = "0.1.0"
dependencies = ["numpy>=1.26"]
requires-python = ">= 3.9"

build-system table determine how to build the package. There are multiple build systems for packaging the project. For this example, we will use setuptools, and build-system table is pretty much fixed by this.

Now, a more important properties are under project. First, we must have a name of the project, which we will call awesome-project. We also need to define the version of the package.

Other attributes are optional. For our example, however, we need to declare dependencies as we call numpy within the package. We also declare that our package requires Python version 3.9 or later.

Other attributes you can declare are

  • authors
  • maintainers
  • description
  • readme
  • license
  • keywords
  • classifiers
  • urls

Check out Python Packaging User Guide for more details.

Step 4. Build Link to heading

Now, we are ready to build and test our package. Run the following from the project root directory

# install `build` package
pip install build

# build
python -m build

If everything runs successful, you should see .tar.gz file and .whl file in the dist/ directory.

Step 5. Test Link to heading

Let’s verify that our package works.

# change directory to some other folder to avoid reading locally
cd SOME/DIRECTORY/OTHER/THAN/PROJECT/ROOT

# create a new fresh python virtual env
python -m venv venv
source venv/bin/activate

# install the package
# we force-reinstall in case there is a change in the package but the version is not updated
pip install --force-reinstall /PROJECT/ROOT/dist/awesome_project-0.1.0-py3-none-any.whl

# test
python -c 'from awesome_package import mean; print(mean.take_average([1,2,3,4,5]))'

If everything works successful, you should see the output 3.0 printed.

We have the codebase that we can package. However, this is just a bare minimum project. In practice, the project can be a a bit more complex than this. Let’s have a look at a few more scenarios. Link to heading

Adding data files Link to heading

The next thing we want to do is to include any data files, such as plain text or json files that are required for the project. Let’s first create a new Python file that reads some default config json file

# config.py
import json
import os

def load_config(file: str = None):
    if file is None:
        pwd = os.path.dirname(os.path.realpath(__file__))
        file = os.path.join(pwd, 'default.json')
    with open(file) as f:
        return json.load(f)

Here, we expect that there is default.json file in the same directory. For that, save the following as default.json

{
    "color": "blue"
}

If you re-build the package, you will notice that the package omits default.json file, as this is not a Python source file. To tell your build system to include this file, we need to create MANIFEST.in file in the project root directory

include awesome_project/*.json

This file tells which files to include or exclude. Here, we are asking it to include any json files in the awesome_project directory. Other available options are

  • exclude
  • recursive-include
  • recursive-exclude
  • global-include
  • global-exclude
  • graft
  • prune

For more details, check out this page. Now, our final project directory should like below.

project/
├── MANIFEST.in
├── awesome_project
│   ├── config.py
│   ├── default.json
│   ├── mean.py
│   └── sum.py
└── pyproject.toml

Let’s re-build our package and test it. Make sure to execute it inside some directory other than the project root.

pip install --force-reinstall /PROJECT/ROOT/dist/awesome_project-0.1.0-py3-none-any.whl

# should print `{'color': 'blue'}`
python -c 'from awesome_project import config; print(config.load_config())'

Setting up Dev Environment Link to heading

So far, we have installed the package from a user’s perspective. What about the developer who maintains the package? To set up the developer environment, the command is a bit different

pip install -e .

That’s it. This will install the package in editable mode so that you can make changes to the code and don’t have to reinstall it every time.

Excluding Tests Link to heading

In practice, a codebase comes with unit tests. Let’s create a simple unit test. For that, let’s create tests directory and test_mean.py file inside.

# tests/test_mean.py
import unittest
from awesome_project import mean

class TestMean(unittest.TestCase):
    def test_mean(self):
        self.assertEqual(mean.take_average([1,2,3,4,5]), 3.0)

For unittest module to discover this, let’s also create a blank __init__.py file in the tests folder. Now, the codebase structure is

project/
├── MANIFEST.in
├── README.md
├── awesome_project
│   ├── config.py
│   ├── default.json
│   ├── mean.py
│   └── sum.py
├── pyproject.toml
└── tests
    ├── __init__.py
    └── test_mean.py

To build, we do python -m build again. The tests folder is automatically excluded from the package.

Excluding a Folder Link to heading

Let’s say we want to add some Python files, intended only for the development and not for the client. We will create dev folder and add main.py file

from awesome_project.mean import take_average

if __name__ == '__main__':
    with open('/dev/stdin') as f:
        xs = [int(x) for x in f.read().split()]
    print(take_average(xs))

We notice that we can no longer build the package

python -m build
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 61.0)
* Getting build dependencies for sdist...
error: Multiple top-level packages discovered in a flat-layout: ['dev', 'awesome_project'].

To avoid accidental inclusion of unwanted files or directories,
setuptools will not proceed with this build.

If you are trying to create a single distribution with multiple packages
on purpose, you should not rely on automatic discovery.

Instead, consider the following options:

1. set up custom discovery (`find` directive with `include` or `exclude`)
2. use a `src-layout`
3. explicitly set `py_modules` or `packages` with a list of names

To find more information, look for "package discovery" on setuptools docs.

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist

This is because we created dev folder with a Python file in it, so the build tool thinks there are multiple packages.

project/
├── MANIFEST.in
├── README.md
├── awesome_project
│   ├── config.py
│   ├── default.json
│   ├── mean.py
│   └── sum.py
├── dev
│   └── main.py
├── pyproject.toml
└── tests
    ├── __init__.py
    └── test_mean.py

To resolve this, we need to explicitly specify which packages to include in the build. In the pyproject.toml file, append the following

[tool.setuptools]
packages = ["awesome_project"]

Now, the build should work again and won’t include dev folder.

Check out this repo for full source code.