Part 1: Setting up a simple CI/CD pipeline for Python and PyPI
Published on 17 July 2025
Sommaire
Introduction
Objectif : Accompany the reader in setting up a minimal but functional CI/CD pipeline for a Python application, with automatic deployment to PyPI.
Automation of development processes has become indispensable in modern projects. A well-designed CI/CD pipeline enables not only early detection of regressions in the development cycle but also full automation of the deployment process.
In this article, we will explore how to set up a complete pipeline with GitHub Actions for a Python application, from continuous integration (CI) to continuous deployment (CD) on PyPI.
Pipeline Architecture
Our pipeline consists of two distinct workflows:
-
CI PipelineExecuted on every push and pull request
-
CD Pipeline: Triggered only on GitHub releases
@startuml
!theme plain
package "GitHub Repository" {
[Source Code] as source
[GitHub Actions] as actions
}
package "CI Pipeline" {
[Checkout] as checkout_ci
[Setup Python] as python_ci
[Install Dependencies] as deps_ci
[Linting (Ruff)] as lint
[Unit Tests] as tests
}
package "CD pipeline" {
[Checkout] as checkout_cd
[Setup Python] as python_cd
[Build Package] as build
[Publish to PyPI] as pypi
}
source --> actions : Push/PR
actions --> checkout_ci
checkout_ci --> python_ci
python_ci --> deps_ci
deps_ci --> lint
lint --> tests
source --> actions : Release
actions --> checkout_cd : On Release
checkout_cd --> python_cd
python_cd --> build
build --> pypi
@enduml
Configuration of the Continuous Integration (CI) Pipeline
CI Workflow Structure
The CI workflow is designed to validate every contribution to the code. Here is its full configuration:
name: CI/CD Pipeline
on:
push:
branches:
- main
pull_request:
branches:
- main
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.x'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install ruff pytest pytest-mock
- name: Run Linting (Ruff)
run: ruff check .
- name: Run Tests (Pytest)
run: pytestAnalysis of CI Steps
Triggers (on)
----
----
on:
push:
branches:
- main
pull_request:
branches:
- main
----
The pipeline triggers on: - Each push to the branch`main` - Each pull request towards`main`
This approach ensures that the main code remains stable and that any contribution is validated before integration.
==== 2. Execution Environment
----
runs-on: ubuntu-latest
Ubuntu Latest offers a good compromise between performance, cost, and compatibility for most Python projects.
==== 3. Checkout of the Code
```yaml
----
----
- name: Checkout code
uses: actions/checkout@v4
----
The action`checkout@v4`Retrieves the source code from the repository. Version v4 brings performance and security improvements.
==== 4. Python Configuration
----
- name: Set up Python uses: actions/setup-python@v5 with: python-version: '3.x'
The use of`'3.x'`allows automatically using the latest stable version of Python 3, simplifying maintenance.
==== 5. Installation of Dependencies
```yaml
----
----
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install ruff pytest pytest-mock
----
This step: - Update pip to the latest version - Install the project dependencies - Add the development tools (linting and tests)
==== 6. Linting with Ruff
----
- name: Run Linting (Ruff) run: ruff check .
**Ruff**is an ultra-fast Python linter written in Rust. It combines the features of several tools (Flake8, Black, isort) into a single high-performance tool.
==== 7. Execution of Tests
```yaml
----
----
- name: Run Tests (Pytest)
run: pytest
----
Pytest runs the entire test suite, ensuring that changes do not introduce regressions.
== Configuration of the Deployment Pipeline (CD)
=== CD Workflow Structure
The CD workflow triggers only on GitHub releases and automates publishing to PyPI:
[source,yaml]
----
name: Publish to PyPI
on:
release:
types:
- published
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.x'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools wheel twine
- name: Build and publish
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: |
python setup.py sdist bdist_wheel
twine upload dist/*
----
=== Analysis of CD Steps
==== 1. Trigger Release
----
on: release: types: - published
The CD pipeline is triggered only when a GitHub release is published. This approach ensures precise control over deployments.
==== 2. Installation of Build Tools
```yaml
----
----
pip install setuptools wheel twine
----
- **setuptools**: Python packaging tools - **wheel**: Modern Python distribution format - **twine**: Secure tool for uploading to PyPI
==== 3. Configuration of Authentication
----
env: TWINE_USERNAME: __token__ TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
Authentication uses a PyPI API token stored as a GitHub secret, which is more secure than classic credentials.
==== 4. Build and Publication
```yaml
----
----
run: |
python setup.py sdist bdist_wheel
twine upload dist/*
----
- `sdist`Create a source distribution - `bdist_wheel`: Create a wheel (binary distribution) - `twine upload`: Publish the distributions on PyPI
== Python Package Configuration
=== Structure of setup.py
For the pipeline to work, your project must include a file`setup.py`:
[source,python]
----
from setuptools import setup, find_packages
with open("README.adoc", "r", encoding="utf-8") as fh:
long_description = fh.read()
setup(
name="playlist-downloader",
version="1.0.0",
author="Votre Nom",
author_email="[email protected]",
description="CLI tool for managing YouTube playlists",
long_description=long_description,
long_description_content_type="text/plain",
url="https://github.com/cheroliv/playlist-downloader",
packages=find_packages(),
classifiers=[
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
],
python_requires=">=3.8",
install_requires=[
"typer>=0.9.0",
"yt-dlp>=2023.1.6",
"google-api-python-client>=2.70.0",
"google-auth-oauthlib>=0.7.1",
"pymonad>=2.4.0",
"pyyaml>=6.0",
],
entry_points={
"console_scripts": [
"playlist-downloader=cli:app",
],
},
)
----
=== Key Points of setup.py
1. **Metadata**: Name, version, author, description
2. **Dependencies**: List of required packages
3. **Entry Points**: Exposed CLI commands
4. **Classifiers**: Metadata for PyPI
== Securing with GitHub Secrets
=== PyPI Token Configuration
1. **Create an API token on PyPI**(empty)
- Log in to PyPI - Go to Account Settings > API tokens - Create a new token with appropriate permissions
1. **Add the secret in GitHub**(empty)
- Repository Settings > Secrets and variables > Actions - Create a new secret named`PYPI_API_TOKEN` - Paste your token PyPI
[plantuml, secrets-flow, svg]
----
@startuml
!theme plain
actor Developer as dev
participant "GitHub Repository" as repo
participant "GitHub Actions" as actions
participant PyPI
dev -> repo : Configure PYPI_API_TOKEN secret
repo -> actions : Trigger CD pipeline on release
actions -> actions : Access secret securely
actions -> PyPI : Authenticate with token
PyPI -> PyPI : Validate and publish package
note over actions, PyPI
Token never exposed in logs
Automatic rotation possible
end note
@enduml
----
== Full Deployment Workflow
=== Deployment Sequence
[plantuml, deployment-sequence, svg]
----
@startuml
!theme plain
actor Developer as dev
participant "Local Git" as git
participant "GitHub" as github
participant "GitHub Actions" as actions
participant PyPI
participant "End Users" as users
dev -> git : git tag v1.0.0
dev -> git : git push origin v1.0.0
git -> github : Push tag
dev -> github : Create release from tag
github -> actions : Trigger CD pipeline
actions -> actions : Checkout code
actions -> actions : Setup Python environment
actions -> actions : Install build tools
actions -> actions : Build distributions (sdist + wheel)
actions -> PyPI : Upload to PyPI with token
PyPI -> PyPI : Validate and publish
users -> PyPI : pip install playlist-downloader
note over dev, github
Release creation can be automated
or done manually through GitHub UI
end note
@enduml
----
=== Pipeline States
[plantuml, pipeline-states, svg]
----
@startuml
!theme plain
[*] --> Idle
Idle --> CI_Running : Push/PR created
CI_Running --> CI_Success : All checks pass
CI_Running --> CI_Failed : Linting/Tests fail
CI_Success --> Idle : Merge completed
CI_Failed --> Idle : Fix and retry
Idle --> CD_Running : Release published
CD_Running --> CD_Success : Package published
CD_Running --> CD_Failed : Build/Upload error
CD_Success --> Idle : Package available on PyPI
CD_Failed --> Idle : Fix and retry release
note on link #red : Blocks merge
note on link #green : Allows deployment
@enduml
----
== Best Practices and Optimizations
=== 1. Version Management
Use semantic Git tags:
----
git tag -a v1.2.3 -m "Release version 1.2.3" git push origin v1.2.3
=== 2. Matrix Tests
To test on multiple Python versions:
```yaml
----
----
strategy:
matrix:
python-version: [3.8, 3.9, "3.10", "3.11"]
----
=== 3. Dependencies cache
Speed up builds with the cache:
----
- name: Cache pip dependencies uses: actions/cache@v3 with: path: ~/.cache/pip key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
=== 4. Deployment Environments
Use GitHub environments for controlled deployments:
```yaml
----
----
jobs:
deploy:
environment: production
runs-on: ubuntu-latest
----
== Use Cases and Architecture
=== Use Case Diagram
[plantuml, use-cases, svg]
----
@startuml
!theme plain
left to right direction
actor "Developer" as dev
actor "GitHub Actions" as ga
actor "End User" as user
package "CI/CD System" {
usecase "Run Linting" as lint
usecase "Execute Tests" as test
usecase "Build Package" as build
usecase "Publish to PyPI" as publish
usecase "Create Release" as release
}
dev --> lint : Pushes code
dev --> test : Pushes code
dev --> release : Creates release
ga --> build : On release trigger
ga --> publish : After successful build
user --> publish : Downloads package
lint .> test : triggers
test .> build : on success
build .> publish : on success
@enduml
----
=== Deployment Architecture
[plantuml, deployment-architecture, svg]
----
@startuml
!theme plain
cloud "GitHub" {
[Source Repository]
[GitHub Actions]
[Secrets Store]
}
cloud "PyPI" {
[Package Registry]
[Distribution Files]
}
node "CI/CD Pipeline" {
[Linting]
[Testing]
[Building]
[Publishing]
}
[Source Repository] --> [GitHub Actions] : Triggers
[GitHub Actions] --> [Linting]
[Linting] --> [Testing]
[Testing] --> [Building]
[Building] --> [Publishing]
[Publishing] --> [Package Registry] : Uploads
[Secrets Store] --> [Publishing] : Provides token
note as N1
Secure token-based
authentication
end note
[Secrets Store] .. N1
@enduml
----
== Monitoring and Debugging
=== Logs and Monitoring
GitHub Actions provides detailed logs for each step. To debug:
1. **Examine the logs**of each step
2. **Enable debug**with`ACTIONS_STEP_DEBUG`
3. **Use some artifacts**to save the build files
----
- name: Upload build artifacts uses: actions/upload-artifact@v3 if: failure() with: name: build-logs path: build/
=== Notifications
Add Slack or email notifications:
```yaml
----
----
- name: Notify on failure
if: failure()
uses: 8398a7/action-slack@v3
with:
status: ${{ job.status }}
webhook_url: ${{ secrets.SLACK_WEBHOOK }}
----
== Conclusion
Setting up a robust CI/CD pipeline with GitHub Actions radically transforms the development experience. By automating linting, tests, and deployment, you:
- **Reduce the errors**in production - **Accelerate the cycles**of development - **Improve the trust**in your releases - **Facilitate collaboration**as a team
This pipeline can be adapted to different types of Python projects by adjusting linting tools, testing frameworks, or deployment targets.
The initial investment in configuring these workflows is quickly recouped through time savings and the reduction of manual errors during deployments.
== Additional Resources
- https://docs.github.com/en/actions[Documentation GitHub Actions] - https://packaging.python.org/[Python Packaging Guide] - https://docs.pytest.org/[Documentation Pytest] - https://docs.astral.sh/ruff/[Documentation Ruff] - https://twine.readthedocs.io/[Twine documentation]
✅ Functional pipeline reached ! You now have a simple CI/CD pipeline that lets you automate your tests and publish your Python package to PyPI directly from GitHub Actions.
However, this pipeline remains deliberately minimalist. It does not yet cover some essential aspects in a professional context:
Multi-version tests of Python,
Automatic security analysis,
Gradual deployment via Test PyPI,
Pipeline monitoring and metrics,
Automation of versioning and integration of modern best practices (pyproject.toml).
In the next part, we will move to the next level. You will learn to transform this basic pipeline into a true industrial deployment chain, robust and secure, ready for production Python projects.
----