Packaging & project structure

A single script is fine for small jobs. But once your code grows — multiple modules, tests, dependencies, a command-line tool, teammates — you need a project structure that every Python developer recognises, and a way to package it so it can be installed with pip. This lesson walks through a modern, professional layout built around pyproject.toml, then builds and installs a real package.

A recommended layout#

We'll build a small tool called textstats that reports word statistics. Here's the structure:

textstats/
├── pyproject.toml          # metadata, dependencies, tool configuration
├── README.md
├── LICENSE
├── .gitignore
├── src/
│   └── textstats/
│       ├── __init__.py     # marks the package; exposes the public API
│       ├── core.py         # the logic
│       └── cli.py          # the command-line interface
└── tests/
    └── test_core.py

Why this shape?

• src/ layout: your package lives in src/textstats/, not at the project root. That means Python can only import it once it's installed, so your tests exercise the package exactly as users will get it — missing files and broken imports show up immediately. It's the layout recommended by the Python Packaging User Guide.
• Tests outside the package, in tests/, so they aren't shipped to users.
• One pyproject.toml replaces the old setup.py, setup.cfg and many tool-specific config files.

The code#

"""Core text statistics."""
import re
from collections import Counter
from dataclasses import dataclass

WORD = re.compile(r"[a-zA-Z']+")


@dataclass(frozen=True)
class Stats:
    words: int
    unique: int
    top: list[tuple[str, int]]


def analyse(text: str, top_n: int = 3) -> Stats:
    """Count words, unique words and the most common words in text."""
    words = [w.lower() for w in WORD.findall(text)]
    counts = Counter(words)
    return Stats(words=len(words), unique=len(counts), top=counts.most_common(top_n))

"""textstats: quick word statistics."""
from .core import Stats, analyse

__all__ = ["Stats", "analyse"]
__version__ = "0.1.0"

import argparse
import sys
from pathlib import Path

from . import __version__
from .core import analyse


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(prog="textstats", description="Word statistics for text files.")
    parser.add_argument("file", nargs="?", help="file to analyse (default: stdin)")
    parser.add_argument("-n", "--top", type=int, default=3, help="number of top words")
    parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
    args = parser.parse_args(argv)

    text = Path(args.file).read_text(encoding="utf-8") if args.file else sys.stdin.read()
    stats = analyse(text, top_n=args.top)
    print(f"words: {stats.words}, unique: {stats.unique}")
    for word, count in stats.top:
        print(f"  {word:<10}{count}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

from textstats import analyse


def test_counts_words_case_insensitively():
    stats = analyse("The cat and the hat. THE end!")
    assert stats.words == 7
    assert stats.unique == 5
    assert stats.top[0] == ("the", 3)


def test_empty_text():
    assert analyse("").words == 0

pyproject.toml#

This one file describes the project:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "textstats"
version = "0.1.0"
description = "Quick word statistics for text files"
readme = "README.md"
requires-python = ">=3.12"
license = "MIT"
authors = [{ name = "Your Name", email = "you@example.com" }]
dependencies = []                      # e.g. ["requests>=2.32"]

[project.optional-dependencies]
dev = ["pytest>=8", "ruff", "mypy"]

[project.scripts]
textstats = "textstats.cli:main"       # creates a `textstats` command

[tool.pytest.ini_options]
testpaths = ["tests"]

[tool.ruff]
line-length = 100

[tool.mypy]
strict = true

The sections:

• [build-system] says which build backend turns your source into installable files. Hatchling, setuptools, Flit, PDM-backend and uv's backend are all fine; we use Hatchling.
• [project] is the standard metadata (PEP 621): name, version, Python requirement and runtime dependencies. Use loose lower bounds here (requests>=2.32), not exact pins — exact pins belong in an application's lock file.
• [project.optional-dependencies] defines extras, installed with pip install -e ".[dev]".
• [project.scripts] creates console commands: after installing, typing textstats runs textstats.cli:main.
• [tool.*] sections configure pytest, ruff, mypy and other tools in the same file.

Developing: editable installs#

From the project root, inside a virtual environment:

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

-e (editable) installs a link to your source folder, so edits take effect immediately without reinstalling. Now the package imports from anywhere, the tests pass, and the command exists:

pytest -q
echo "the cat and the hat and the bat" | textstats -n 2
textstats --version

..                                                                       [100%]
2 passed in 0.01s
words: 8, unique: 5
  the       3
  and       2
textstats 0.1.0

You can also run the CLI as a module: python -m textstats.cli README.md.

Building distributions#

To share your package, build it into two standard files:

pip install build
python -m build

...
Successfully built textstats-0.1.0.tar.gz and textstats-0.1.0-py3-none-any.whl

• The sdist (.tar.gz) is a source archive.
• The wheel (.whl) is a ready-to-install archive — pip prefers wheels because installing them is fast and needs no build step. py3-none-any means "pure Python, any OS".

Anyone can now pip install dist/textstats-0.1.0-py3-none-any.whl. To publish on PyPI so the world can pip install textstats, create an account at pypi.org, generate an API token, and upload with twine upload dist/* (try TestPyPI first). Many projects publish automatically from CI using PyPI's trusted publishing.

Applications vs libraries#

The layout above suits libraries and CLI tools. For applications (a web service, a data pipeline), the same ideas apply, with a few differences:

• Pin exact versions for reproducible deployments — with a lock file (uv lock, pip-tools' requirements.txt, Poetry's poetry.lock).
• Keep configuration in environment variables, not in code.
• A typical web app adds folders like src/app/api/, src/app/models/, src/app/services/, plus a Dockerfile.

my-service/
├── pyproject.toml
├── uv.lock
├── Dockerfile
├── .env.example          # documents required env vars (no real secrets!)
├── src/my_service/
│   ├── __init__.py
│   ├── main.py           # creates the FastAPI/Flask app
│   ├── config.py         # reads settings from the environment
│   ├── api/              # routes
│   ├── services/         # business logic
│   └── db/               # database access
└── tests/

The guiding principle: separate what the app does (pure business logic, easy to test) from how it talks to the world (HTTP, databases, files).

Tooling every project should have#

Run them all in CI (e.g. GitHub Actions) on every pull request, so problems are caught before they reach the main branch.

A good .gitignore for Python starts with:

.venv/
__pycache__/
*.pyc
dist/
build/
*.egg-info/
.pytest_cache/
.mypy_cache/
.ruff_cache/
.env

Versioning#

Use semantic versioning: MAJOR.MINOR.PATCH. Bump PATCH for bug fixes, MINOR for backwards-compatible features, MAJOR for breaking changes. Keep a CHANGELOG.md, and tag releases in git (git tag v0.1.0).

Common mistakes#

• Putting the package at the project root and running tests from there — tests pass locally but the installed package is broken. Use the src/ layout.
• Committing .venv, dist/ or __pycache__.
• Pinning exact versions in a library's dependencies, which causes conflicts for users. Pin in applications, not libraries.
• Forgetting __init__.py re-exports, forcing users to import from deep internal modules.
• Mismatched names: the distribution name (pip install textstats) and import name (import textstats) should match whenever possible.
• Secrets in the repository — use environment variables and .env.example.

What's next#

Your code is well structured. The final lesson pulls everything together: Pythonic style, performance and where to go next.