Parte 2: Industrializar y asegurar un pipeline CI/CD de Python avanzado

Introducción

Objetivo: Convertir un pipeline simple en un pipeline industrial robusto, respetando los estándares modernos de empaquetado e ingeniería de software.

En la primera parte de esta serie, hemos construido un pipeline CI/CD sencillo pero eficaz para automatizar las pruebas y la publicación de un paquete Python en PyPI.

Ahora es momento de profesionalizar este pipeline. En esta segunda parte, vamos a:

Migrar al estándar moderno pyproject.toml,
Agregar herramientas de calidad y seguridad (Black, Mypy, Bandit, Safety),
Implementar pruebas multi-versiones,
Integrar un despliegue progresivo a través de Test PyPI,
Automatizar el versioning y mejorar la vigilancia del pipeline.

Prepárate para pasar de un pipeline funcional a una infraestructura CI/CD profesional.

Desplegar una aplicación Python en PyPI requiere un pipeline CI/CD robusto que automatice las pruebas, la construcción y la publicación de los paquetes. Este artículo detalla la configuración de un pipeline completo utilizando GitHub Actions para una aplicación CLI de Python, apoyándose en las buenas prácticas del ecosistema Python moderno.

Arquitectura del Pipeline CI/CD

El pipeline que vamos a construir sigue un enfoque en varias etapas:

@startuml
!theme plain
title Pipeline CI/CD Python a PyPI
skinparam backgroundColor #f8f9fa
skinparam componentStyle rectangle

rectangle "Desarrollador" as dev
rectangle "Repositorio de GitHub" as repo {
  rectangle ".github/workflows/" as workflows
  rectangle "pruebas/" as tests
  rectangle "setup.py / pyproject.toml" as setup
  rectangle "requirements.txt" as req
}

rectangle "GitHub Actions" as actions {
  rectangle "Test Runner" as test_runner
  rectangle "Construir" as build
  rectangle "Escaneo de seguridad" as security
  rectangle "Comprobación de calidad" as quality
}

rectangle "PyPI" as pypi {
  rectangle "Test PyPI" as test_pypi
  rectangle "Producción PyPI" as prod_pypi
}

rectangle "Usuarios" as users

dev --> repo : push/PR
repo --> actions : trigger workflow
actions --> test_runner : run tests
actions --> security : security checks
actions --> quality : code quality
actions --> build : build package
build --> test_pypi : deploy (pre-release)
build --> prod_pypi : deploy (release)
prod_pypi --> users : install package

@enduml

Estructura del Proyecto

Una aplicación CLI de Python preparada para la distribución debe respetar una estructura estandarizada:

playlist-downloader/
├── .github/
│   └── workflows/
│       ├── ci.yml
│       ├── release.yml
│       └── security.yml
├── src/
│   └── playlist_downloader/
│       ├── __init__.py
│       ├── cli.py
│       ├── core/
│       └── adapters/
├── tests/
│   ├── unit/
│   ├── integration/
│   └── conftest.py
├── docs/
├── pyproject.toml
├── requirements.txt
├── requirements-dev.txt
├── MANIFEST.in
├── README.md
├── LICENSE
└── CHANGELOG.md

Configuración del paquete con pyproject.toml

El archivo`pyproject.toml`es el estándar moderno para configurar los paquetes de Python:

[build-system]
requires = ["setuptools>=45", "wheel", "setuptools_scm>=6.2"]
build-backend = "setuptools.build_meta"

[project]
name = "playlist-downloader"
authors = [
    {name = "Christophe Hérolivier", email = "[email protected]"},
]
description = "CLI tool for YouTube playlist management"
readme = "README.md"
requires-python = ">=3.8"
keywords = ["youtube", "playlist", "cli", "downloader"]
license = {text = "MIT"}
classifiers = [
    "Development Status :: 4 - Beta",
    "Environment :: Console",
    "Intended Audience :: End Users/Desktop",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Topic :: Multimedia :: Sound/Audio",
    "Topic :: Utilities",
]
dependencies = [
    "typer>=0.9.0",
    "yt-dlp>=2023.7.6",
    "google-api-python-client>=2.0.0",
    "google-auth-oauthlib>=1.0.0",
    "pyyaml>=6.0",
    "rich>=13.0.0",
]
dynamic = ["version"]

[project.optional-dependencies]
dev = [
    "pytest>=7.0.0",
    "pytest-cov>=4.0.0",
    "pytest-mock>=3.10.0",
    "black>=23.0.0",
    "flake8>=6.0.0",
    "mypy>=1.0.0",
    "pre-commit>=3.0.0",
    "tox>=4.0.0",
]
test = [
    "pytest>=7.0.0",
    "pytest-cov>=4.0.0",
    "pytest-mock>=3.10.0",
]

[project.urls]
Homepage = "https://github.com/cheroliv/playlist-downloader"
Documentation = "https://github.com/cheroliv/playlist-downloader#readme"
Repository = "https://github.com/cheroliv/playlist-downloader.git"
"Bug Tracker" = "https://github.com/cheroliv/playlist-downloader/issues"

[project.scripts]
playlist-downloader = "playlist_downloader.cli:main"

[tool.setuptools_scm]
write_to = "src/playlist_downloader/_version.py"

[tool.setuptools.packages.find]
where = ["src"]

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = [
    "--cov=src/playlist_downloader",
    "--cov-report=html",
    "--cov-report=term-missing",
    "--cov-fail-under=85",
]

[tool.black]
line-length = 88
target-version = ['py38']
include = '\.pyi?$'
extend-exclude = '''
/(
  \.eggs
  | \.git
  | \.hg
  | \.mypy_cache
  | \.tox
  | \.venv
  | _build
  | buck-out
  | build
  | dist
)/
'''

[tool.mypy]
python_version = "3.8"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
disallow_untyped_decorators = true
no_implicit_optional = true
warn_redundant_casts = true
warn_unused_ignores = true
warn_no_return = true
warn_unreachable = true
strict_equality = true

[[tool.mypy.overrides]]
module = [
    "yt_dlp.*",
    "googleapiclient.*",
    "google_auth_oauthlib.*",
]
ignore_missing_imports = true

Flujo de trabajo de CI/CD - Pruebas y Calidad

El flujo de trabajo principal (ci.yml) ejecuta las pruebas en varias versiones de Python:

name: CI

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11"]

    steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}

    - name: Cache dependencies
      uses: actions/cache@v3
      with:
        path: |
          ~/.cache/pip
          ~/.cache/pre-commit
        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }}
        restore-keys: |
          ${{ runner.os }}-pip-

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -e ".[dev]"

    - name: Lint with flake8
      run: |
        flake8 src tests --count --select=E9,F63,F7,F82 --show-source --statistics
        flake8 src tests --count --exit-zero --max-complexity=10 --max-line-length=88 --statistics

    - name: Check code formatting with Black
      run: black --check src tests

    - name: Type checking with mypy
      run: mypy src

    - name: Run tests with pytest
      run: |
        pytest tests/ -v --cov=src/playlist_downloader \
          --cov-report=xml --cov-report=term-missing

    - name: Upload coverage to Codecov
      uses: codecov/codecov-action@v3
      if: matrix.python-version == '3.11'
      with:
        file: ./coverage.xml
        flags: unittests
        name: codecov-umbrella

  security:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: "3.11"

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install bandit[toml] safety

    - name: Run security checks with bandit
      run: bandit -r src/ -f json -o bandit-report.json

    - name: Check dependencies with safety
      run: safety check --json --output safety-report.json

    - name: Upload security reports
      uses: actions/upload-artifact@v3
      if: always()
      with:
        name: security-reports
        path: |
          bandit-report.json
          safety-report.json

  build:
    needs: [test, security]
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: "3.11"

    - name: Install build dependencies
      run: |
        python -m pip install --upgrade pip
        pip install build twine

    - name: Build package
      run: python -m build

    - name: Check package with twine
      run: twine check dist/*

    - name: Upload build artifacts
      uses: actions/upload-artifact@v3
      with:
        name: dist
        path: dist/

Workflow de Lanzamiento y Despliegue

El flujo de trabajo de release (release.yml) gestiona la publicación automática en PyPI :

name: Release

on:
  push:
    tags:
      - 'v*.*.*'
  workflow_dispatch:
    inputs:
      environment:
        description: 'Deployment environment'
        required: true
        default: 'test'
        type: choice
        options:
        - test
        - production

env:
  PYTHON_VERSION: "3.11"

jobs:
  release:
    runs-on: ubuntu-latest
    environment:
      name: ${{ github.event.inputs.environment || (startsWith(github.ref, 'refs/tags/') && 'production' || 'test') }}

    steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install build twine

    - name: Build package
      run: python -m build

    - name: Check package
      run: twine check dist/*

    - name: Publish to Test PyPI
      if: github.event.inputs.environment == 'test' || (startsWith(github.ref, 'refs/tags/') && contains(github.ref, 'rc'))
      env:
        TWINE_USERNAME: __token__
        TWINE_PASSWORD: ${{ secrets.TEST_PYPI_API_TOKEN }}
      run: |
        twine upload --repository testpypi dist/*

    - name: Publish to PyPI
      if: github.event.inputs.environment == 'production' || (startsWith(github.ref, 'refs/tags/') && !contains(github.ref, 'rc'))
      env:
        TWINE_USERNAME: __token__
        TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
      run: |
        twine upload dist/*

    - name: Create GitHub Release
      if: startsWith(github.ref, 'refs/tags/')
      uses: actions/create-release@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        tag_name: ${{ github.ref }}
        release_name: Release ${{ github.ref }}
        draft: false
        prerelease: ${{ contains(github.ref, 'rc') }}

  post-release:
    needs: release
    runs-on: ubuntu-latest
    if: startsWith(github.ref, 'refs/tags/')

    steps:
    - uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}

    - name: Test installation from PyPI
      run: |
        sleep 60  # Attendre la propagation sur PyPI
        pip install playlist-downloader
        playlist-downloader --version

    - name: Update documentation
      run: |
        # Script pour mettre à jour la documentation
        echo "Documentation updated for version ${GITHUB_REF#refs/tags/}"

Diagramas de Arquitectura

Diagrama de Secuencia - Proceso de Release

@startuml
!theme plain
title Secuencia de lanzamiento hacia PyPI
actor Developer as dev
participant "GitHub" as gh
participant "GitHub Actions" as ga
participant "Prueba PyPI" as tpypi
participant "PyPI" as pypi
participant "Usuarios" as users

dev -> gh : git push --tags v1.2.3
gh -> ga : trigger release workflow

ga -> ga : checkout code
ga -> ga : setup Python environment
ga -> ga : install dependencies
ga -> ga : run tests
ga -> ga : build package (wheel + sdist)
ga -> ga : check package with twine

alt Pre-release (rc tag)
    ga -> tpypi : upload to Test PyPI
    tpypi -> ga : confirm upload
else Stable release
    ga -> pypi : upload to PyPI
    pypi -> ga : confirm upload
end

ga -> gh : create GitHub release
ga -> ga : test installation from PyPI

users -> pypi : pip install playlist-downloader
pypi -> users : download package

@enduml

Diagrama de Estados - Ciclo de Vida del Paquete

@startuml
!theme plain
title Estados del paquete Python
[*] --> Development
Development --> Testing : commit/PR
Testing --> Development : tests fail
Testing --> Built : tests pass
Built --> TestPyPI : pre-release tag
Built --> PyPI : stable tag
TestPyPI --> PyPI : validation OK
PyPI --> Published
Published --> [*]

state Development {
  [*] --> Coding
  Coding --> LocalTesting
  LocalTesting --> Coding : fix issues
  LocalTesting --> ReadyForCI : all tests pass
}

state Testing {
  [*] --> CITests
  CITests --> SecurityScan
  SecurityScan --> QualityCheck
  QualityCheck --> BuildValidation
}

@enduml

Diagrama de despliegue - infraestructura CI/CD

@startuml
!theme plain
title Infraestructura de despliegue
node "GitHub" {
  component "Repositorio" as repo
  component "Ejecutor de acciones" as runner
  component "Almacén de secretos" as secrets
}

node "PyPI Infraestructura" {
  component "PyPI" as pypi
  component "Prueba PyPI" as testpypi
  database "Índice de paquetes" as index
}

node "Máquina del desarrollador" {
  component "Cliente Git" as git
  component "Entorno Python" as python
  component "IDE" as ide
}

node "Entorno de usuario" {
  component "pip" as pip_client
  component "Entorno de ejecución de Python" as py_runtime
}

git --> repo : push code/tags
repo --> runner : activar flujos de trabajo
runner --> secrets : lee los tokens API
runner --> testpypi : cargar pre‑lanzamiento
runner --> pypi : cargar versión
pypi --> index : paquete de tienda
pip_client --> pypi : descargar paquete
py_runtime <-- pip_client : instalar paquete

@enduml

Objetos y Modelos del Pipeline

Diagrama de Clases - Modelos CI/CD

@startuml
!theme plain
title Modelos del Pipeline CI/CD
class PipelineConfig {
  +python_versions: List[str]
  +test_environments: List[str]
  +security_checks: bool
  +coverage_threshold: float
  +validate()
}

class BuildArtifact {
  +name: str
  +version: str
  +wheel_path: str
  +sdist_path: str
  +checksums: Dict[str, str]
  +validate_integrity()
}

class TestResult {
  +test_suite: str
  +python_version: str
  +passed: int
  +failed: int
  +coverage: float
  +duration: float
  +is_success(): bool
}

class SecurityReport {
  +bandit_issues: List[Issue]
  +safety_vulnerabilities: List[Vulnerability]
  +severity_level: str
  +is_secure(): bool
}

class DeploymentTarget {
  +name: str
  +url: str
  +api_token: str
  +environment: str
  +deploy(artifact: BuildArtifact)
}

class ReleaseManager {
  +version: str
  +changelog: str
  +artifacts: List[BuildArtifact]
  +test_results: List[TestResult]
  +security_report: SecurityReport
  +deploy_to_test()
  +deploy_to_production()
  +create_github_release()
}

PipelineConfig ||--o{ TestResult
ReleaseManager *-- BuildArtifact
ReleaseManager *-- SecurityReport
ReleaseManager o-- DeploymentTarget
DeploymentTarget ..> BuildArtifact : uses

@enduml

Configuración de secretos

Para que el pipeline funcione, debe configurar los siguientes secretos en GitHub :

Secretos GitHub Actions

# Dans Settings > Secrets and variables > Actions

# Token PyPI pour la production
PYPI_API_TOKEN=pypi-...

# Token Test PyPI pour les pré-releases
TEST_PYPI_API_TOKEN=pypi-...

# Token GitHub pour créer les releases
GITHUB_TOKEN=(automatiquement fourni)

# Token Codecov (optionnel)
CODECOV_TOKEN=...

Generación de los Tokens PyPI

# 1. Créer un compte sur PyPI et Test PyPI
# 2. Aller dans Account Settings > API tokens
# 3. Créer un token avec scope "Entire account" ou spécifique au projet
# 4. Format du token : pypi-AgEIcHlwaS5vcmc...

Scripts de Desarrollo Local

Para facilitar el desarrollo, cree scripts de utilidad:

Makefile

.PHONY: install test lint format security build clean release-test release-prod

install:
	pip install -e ".[dev]"

test:
	pytest tests/ -v --cov=src/playlist_downloader

lint:
	flake8 src tests
	mypy src

format:
	black src tests

security:
	bandit -r src/
	safety check

build:
	python -m build
	twine check dist/*

clean:
	rm -rf build/ dist/ *.egg-info/
	find . -type d -name __pycache__ -delete
	find . -name "*.pyc" -delete

release-test: clean build
	twine upload --repository testpypi dist/*

release-prod: clean build
	twine upload dist/*

pre-commit: format lint test security
	@echo "✅ Prêt pour commit"

Guion de Versión

#!/usr/bin/env python3
"""Script pour gérer les versions du projet."""

import sys
import subprocess
from pathlib import Path

def get_current_version():
    """Récupère la version actuelle depuis git."""
    try:
        result = subprocess.run(
            ["git", "describe", "--tags", "--abbrev=0"],
            capture_output=True,
            text=True,
            check=True
        )
        return result.stdout.strip()
    except subprocess.CalledProcessError:
        return "0.0.0"

def create_version_tag(version, message=None):
    """Crée un tag de version."""
    if not version.startswith('v'):
        version = f'v{version}'

    tag_message = message or f"Release {version}"

    subprocess.run(["git", "tag", "-a", version, "-m", tag_message], check=True)
    print(f"✅ Tag {version} créé")

    # Push le tag
    subprocess.run(["git", "push", "origin", version], check=True)
    print(f"✅ Tag {version} poussé vers origin")

if __name__ == "__main__":
    if len(sys.argv) < 2:
        current = get_current_version()
        print(f"Version actuelle: {current}")
        print("Usage: python version.py <new_version> [message]")
        sys.exit(1)

    new_version = sys.argv[1]
    message = sys.argv[2] if len(sys.argv) > 2 else None

    create_version_tag(new_version, message)

Buenas Prácticas y Recomendaciones

Versionamiento semántico

Utilice el versionado semántico (SemVer):

MAJOR.MINOR.PATCH (ex: 1.2.3)
MAJOR: cambios incompatibles
`MINOR`nuevas funcionalidades compatibles
PATCH: correcciones de errores compatibles

Estrategia de Ramificación

main          ──●──●──●──●──●────●── (releases stables)
               /       /          /
develop    ──●──●──●──●──●──●──●──●── (développement)
            /     /        /
feature/xxx  ●──●──●──●──●──/ (fonctionnalités)

Pruebas y cobertura

Cobertura mínima de código: 85%
Pruebas unitarias para la lógica de negocio
Pruebas de integración para los adaptadores
Pruebas de extremo a extremo para las CLI

Seguridad

Escaneo automático de dependencias (Safety)
Análisis estático del código (Bandit)
Secretos nunca en el código
Utilización de tokens específicos de PyPI

Monitoreo y Observabilidad

Métricas de pipeline

# .github/workflows/metrics.yml
name: Pipeline Metrics

on:
  workflow_run:
    workflows: ["CI", "Release"]
    types: [completed]

jobs:
  metrics:
    runs-on: ubuntu-latest
    steps:
    - name: Collect metrics
      run: |
        echo "Pipeline: ${{ github.event.workflow_run.name }}"
        echo "Status: ${{ github.event.workflow_run.conclusion }}"
        echo "Duration: ${{ github.event.workflow_run.updated_at - github.event.workflow_run.created_at }}"
        # Envoyer vers système de monitoring

Notificaciones

# Ajout dans les workflows pour notifications
- name: Notify on failure
  if: failure()
  uses: 8398a7/action-slack@v3
  with:
    status: failure
    text: "❌ Pipeline failed for ${{ github.repository }}"
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

Conclusión

Este pipeline CI/CD completo para Python ofrece :

Automatización completa: de la validación del código a la publicación
Seguridad: exploraciones automáticas y gestión segura de secretos
Calidad: pruebas multi-versiones, linting y cobertura de código
fiabilidad: despliegue progresivo via Test PyPI
Trazabilidad: artefactos, informes y lanzamientos de GitHub

La adopción de estas prácticas garantiza un proceso de entrega robusto y profesional para sus aplicaciones Python CLI, facilitando el mantenimiento y la evolución de sus proyectos a largo plazo.