Create and publish your first Python Package

Overview

If you have never created a Python package are interested in creating one, this post would be the right place to start with!

Tools

1. pyenv: Python version management
2. conda: Python environment management
3. poetry: Dependency package management
4. Python Packages (py-pkgs.org): How to create a Python Package
5. cookiecutter: Tool to create a Python project from a template
6. PyPI (Python Package Index): Python package hub where you can publish your package and you can use packages from.
7. Test Python Package Index: Python package hub for testing
8. python-semantic-release: Tool to automate Python package release based on semantic commit
9. flake8, black, isort: Formater、Linter

Prepare environment with Conda

Installation is here:

1. Download the miniconda3 installer for your platform. e.g. Miniconda3 macOS Apple M1 ARM 64-bit bash.
2. Execute the installer

bash Miniconda3-latest-MacOSX-x86_64.sh

Now, you can create a conda environment with the following command:

conda create --name <environment name> python=3.9 -y

Then, you need to activate it with the following command:

conda activate <environment name>

Manage your package with Poetry

First, you need to install poetry:

curl -sSL https://install.python-poetry.org | python3 -

If you encounter the following warning, just follow the instruction and move the directory.

Configuration file exists at xxx/Library/Application Support/pypoetry, reusing this directory.

Consider moving configuration to xxx/Library/Preferences/pypoetry, as support for the legacy directory will be removed in an upcoming release.

mkdir -p ~/Library/Preferences/pypoetry
mv ~"/Library/Application Support/pypoetry/config.toml" ~/Library/Preferences/pypoetry/config.toml

Create a package with cookiecutter

Cookiecutter is a tool to create a python project from a template.

You can install cookiecutter by the following command:

pip install cookiecutter

or you can use brew if you’re using MacOS

brew install cookiecutter

Then, create Python project from this cookiecutter template: https://github.com/py-pkgs/py-pkgs-cookiecutter

cookiecutter https://github.com/py-pkgs/py-pkgs-cookiecutter.git

Implement your package

You can implement your own package in the generated Python project above. The structure would be like the following (from How to package a Python):

pycounts
├── CHANGELOG.md               ┐
├── CONDUCT.md                 │
├── CONTRIBUTING.md            │
├── docs                       │ Package documentation
│   └── ...                    │
├── LICENSE                    │
├── README.md                  ┘
├── pyproject.toml             ┐ 
├── src                        │
│   └── pycounts               │ Package source code, metadata,
│       ├── __init__.py        │ and build instructions 
│       ├── moduleA.py         │
│       └── moduleB.py         ┘
└── tests                      ┐
    └── ...                    ┘ Package tests

pycounts is the name of the example package.

Build your package

You can build your Python package with the following command:

poetry build

Everything is ready because you created your project from the template.

This command will generate wheel and src under dist directory, which can be configured by dist_pathin pyproject.toml.

Update package version

When you add some feature or create the first version of your package. You need to upgrade your package version.

You can do it by the poetry versioncommand:

poetry version <rule>

rule can be one of the followings:

1. major: major version upgrade e.g. 1.2.0 -> 2.0.0
2. minor: minor version upgrade e.g. 1.2.0 -> 1.3.0
3. patch: patch version upgrade e.g. 1.2.0 -> 1.2.1
4. premajor: premajor version upgrade e.g. 1.2.0 -> 2.0.0a0
5. preminor: preminor version upgrade e.g. 1.2.0 -> 1.3.0a0
6. prepatch: prepatch version upgrade e.g. 1.2.0 -> 1.2.1a0
7. etc

For more details, you can read https://python-poetry.org/docs/cli/#version.

Publish your Package to PyPi

You can publish your Python package so others can use it as any other Python libraries that you usually install with pip install <package-name>.

Settings of PyPi

You need to sign up https://test.pypi.org/account/register/ and https://pypi.org/account/register for testing and prod, respectively. You can skip this step if you already have your account.

For testing, you can add the source test-pypi to poetry configuration:

poetry source add test-pypi https://test.pypi.org/simple/

And generate API key from https://test.pypi.org/manage/account/ for test account and set it to pypi-token.test-pypi

poetry config pypi-token.test-pypi pypi-xxxxxx

Now you’re ready to publish your Package to test-pypi.

You need to do the same thing for pypi https://pypi.org/ by setting your API key:

poetry config pypi-token.pypi pypi-xxxxx

(Be careful, this command sets pypi-token.pypi!)

Publish your package to test pypi

Build your package by the following command:

poetry build

Publish by the poetry publish command by specifying test-pypi:

poetry publish -r test-pypi

You’ll be able to check your package with the url: https://test.pypi.org/project/<yourpackage>/

Check your package published to test pypi

You can install your package from test pypi

poetry add --source test-pypi <your package>

And you can check the behavior as a user.

After confirming everything’s fine, now it’s time to publish to Pypi.

Publish your package to Pypi

You can publish your package almost same way as you did it for test pypi:

poetry build
poetry publish

Release by Python Semantic Release (PSR)

Python Semantic Release (PSR) is a tool to automate releasing based on semantic commit message.

To configure PSR, you need to install it by the command:

poetry add --dev python-semantic-release

And update pyproject.toml:

[tool.semantic_release]
version_variable = "pyproject.toml:version"
version_source = "tag"

There are several command options:

1. Check the next version: semantic-release print-version
2. Check the current version: semantic-release print-version --current
3. Create a new version, commit it, and create a tag (only in local): semantic-release version
4. Publish your package (including updating changelog, push to git, build and upload dist to github release): semantic-release publish

Lint your Python codes

You can use isort, black, and flake8

poetry run isort --check --diff .
poetry run black --check --diff .
poetry run flake8 .

If you need extra configuration for flake8 such as excluding some directories or files, you can configure it in .flake8 file.

[flake8]
exclude =
    # No need to traverse our git directory
    .git,
    # There's no value in checking cache directories
    __pycache__,
    # The conf file is mostly autogenerated, ignore it
    docs/source/conf.py,
    # The old directory contains Flake8 2.0
    old,
    # This contains our built documentation
    build,
    # This contains builds of flake8 that we don't want to check
    dist
    .venv

GitHub Actions for your CI

I splitted the ci-cd from the cookiecutter template as it publishes a new release every time a new PR is merged to the main branch.

GitHub Actions — CI

This workflow is executed when creating/updating a PR and pushing a new commit to the main branch. This workflow include test, lint, codecov, docs, etc.

name: dev

on:
  pull_request:
  push:
    branches:
      - main

jobs:
  ci:
    # Set up operating system
    runs-on: ubuntu-latest

    # Define job steps
    steps:
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.9"

      - name: Check-out repository
        uses: actions/checkout@v3

      - name: Load cached Poetry installation
        id: cached-poetry
        uses: actions/cache@v3
        with:
          path: ~/.local  # the path depends on the OS
          key: poetry  # increment to reset cache

      - name: Install poetry
        if: steps.cached-poetry.outputs.cache-hit != 'true'
        uses: snok/install-poetry@v1

      - name: Restore cached dependencies
        uses: actions/cache@v3
        with:
          path: ~/.cache/pypoetry
          key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
          restore-keys: |
            ${{ runner.os }}-poetry-
      - name: Install package
        run: poetry install

      - name: Lint
        run: |
          poetry run isort --check --diff .
          poetry run black --check --diff .
          poetry run flake8 .

      - name: Test with pytest
        run: poetry run pytest tests/ --cov=<package_name> --cov-report=xml

      - name: Use Codecov to track coverage
        uses: codecov/codecov-action@v3
        with:
          files: ./coverage.xml   # coverage report

      - name: Build documentation
        run: poetry run make html --directory docs/

Please replace the package name with your own.

GitHub Actions — semantic-pull-request

As semantic-release decides version tag based on the commits in each release, it’s better to check if the title of a pull request matches semantic commit message (if you use pull request title for each commit to the main branch.)

name: semantic-pull-request

on:
  pull_request:
    types:
      - opened
      - edited
      - synchronize

jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: amannn/action-semantic-pull-request@v5
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

GitHub Actions — release

Lastly, I’m too lazy to manually create a new release on GitHub so I created a GitHub Actions workflow with dispatch_workflow type, which execute semantic-release publish.

To use trusted publishers, you need to follow the configuration here: https://docs.pypi.org/trusted-publishers/adding-a-publisher/. With this configuration, you don’t need to set any password in your GitHub repository.

name: release

on:
  workflow_dispatch:

jobs:
  release:
    if: github.ref == 'refs/heads/main'
    environment:
      name: pypi
      url: https://pypi.org/p/autonote
    permissions:
      id-token: write  # IMPORTANT: this permission is mandatory for trusted publishing
    runs-on: ubuntu-latest
    steps:
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.9"

      - name: Check-out repository
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Load cached Poetry installation
        id: cached-poetry
        uses: actions/cache@v3
        with:
          path: ~/.local  # the path depends on the OS
          key: poetry  # increment to reset cache

      - name: Install poetry
        if: steps.cached-poetry.outputs.cache-hit != 'true'
        uses: snok/install-poetry@v1

      - name: "Restore cached dependencies"
        uses: actions/cache@v3
        with:
          path: ~/.cache/pypoetry
          key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
          restore-keys: |
            ${{ runner.os }}-poetry-
      - name: Install package
        run: poetry install

      - name: Use Python Semantic Release to prepare release
        env:
          # This token is created automatically by GH Actions
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
            git config user.name github-actions
            git config user.email github-actions@github.com
            poetry run semantic-release publish
      - name: Publish to TestPyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          repository-url: https://test.pypi.org/legacy/

      - name: Test install from TestPyPI
        run: |
            pip install \
            --index-url https://test.pypi.org/simple/ \
            --extra-index-url https://pypi.org/simple \
            <package_name>
      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1

Summary

This post covered basic steps to create your own package, publish to PyPi with CI/CD pipelines using GitHub Actions.

For future work, I’d try to find a better release flow and write about documentation.