Packaging
Questions
How to organize Python projects larger than one script?
What is a good file and folder structure for Python projects?
How can you make your Python functions most usable by your collaborators?
How to prepare your code to make a Python package?
How to publish your Python package?
Objectives
Learn to identify the components of a Python package
Learn to create a Python package
Learn to publish a Python package
Organizing Python projects
Python projects often start as a single script or Jupyter notebook but they can grow out of a single file.
In the Scripts episode we have also learned how to import functions and objects from other Python files (modules). Now we will take it a step further.
Recommendations:
Collect related functions into modules (files).
Collect related modules into packages (we will show how).
Add a
LICENSEfile to your code (see Software Licensing and Open source explained with cakes).Write a
README.mdfile describing what the code does and how to use it.It is also recommended to document your package.
When the project grows, you might need automated testing.
To have a concrete but still simple example, we will create a project
consisting of 3 functions, each in its own file. We can then imagine that each
file would contain many more functions. To make it more interesting,
one of these functions will depend on an external library: scipy.
These are the 3 files:
def add(x, y):
return x + y
def subtract(x, y):
return x - y
from scipy import integrate
def integral(function, lower_limit, upper_limit):
return integrate.quad(function, lower_limit, upper_limit)
We will add a fourth file:
"""
Example calculator package.
"""
from .adding import add
from .subtracting import subtract
from .integrating import integral
__version__ = "0.1.0"
This __init__.py file will be the interface of our package/library.
It also holds the package docstring and the version string.
Note how it imports functions from the various modules using relative imports
(with the dot).
This is how we will arrange the files in the project folder/repository:
project-folder
├── calculator
│ ├── adding.py
│ ├── __init__.py
│ ├── integrating.py
│ └── subtracting.py
├── LICENSE
└── README.md
Now we are ready to test the package. For this we need to be in the “root”
folder, what we have called the project-folder. We also need to have
scipy available in our environment:
from calculator import add, subtract, integral
print(add(2, 3))
print(subtract(2, 3))
print(integral(lambda x: x * x, 0.0, 1.0))
The package is not yet pip-installable, though. We will make this possible in the next section.
Testing a local pip install
To make our example package pip-installable we need to add one more file:
project-folder
├── calculator
│ ├── adding.py
│ ├── __init__.py
│ ├── integrating.py
│ └── subtracting.py
├── LICENSE
├── README.md
└── pyproject.toml
This is how pyproject.toml looks:
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[project]
name = "calculator-myname"
description = "A small example package"
version = "0.1.0"
readme = "README.md"
authors = [
{ name = "Firstname Lastname", email = "firstname.lastname@example.org" }
]
dependencies = [
"scipy"
]
Note how our package requires scipy and we decided to not pin the version
here (see Version pinning for package creators).
Now we have all the building blocks to test a local pip install. This is a good test before trying to upload a package to PyPI or test-PyPI (see PyPI (The Python Package Index) and (Ana)conda)
Exercises 1
Packaging-1
To test a local pip install:
Create a new folder outside of our example project
Create a new virtual environment (Dependency management)
Install the example package from the project folder into the new environment:
$ pip install /path/to/project-folder/Test the local installation:
from calculator import add, subtract, integral
print(add(2, 3))
print(subtract(2, 3))
print(integral(lambda x: x * x, 0.0, 1.0))
Tools that simplify sharing via PyPI
The solution that we have used to create the example package (using
setuptools and twine) is not the only approach. There are many ways to
achieve this and we avoided going into too many details and comparisons to not
confuse too much. If you web-search this, you will also see that recently the
trend goes towards using pyproject.toml as more general
alternative to the previous setup.py.
There are at least two tools which try to make the packaging and PyPI interaction easier: