Parte 2: Industrializar e proteger um pipeline avançado de CI/CD em Python

Introdução

Objetivo : Transformar um pipeline simples em um pipeline industrial robusto, respeitando os padrões modernos de empacotamento e engenharia de software.

Na primeira parte desta série, construímos um pipeline CI/CD simples mas eficaz para automatizar os testes e a publicação de um pacote Python no PyPI.

Agora é hora de profissionalizar este pipeline. Nesta segunda parte, vamos:

Migrar para o padrão moderno pyproject.toml,
Adicionar ferramentas de qualidade e segurança (Black, Mypy, Bandit, Safety),
Implementar testes multi-versão,
Integrar um lançamento gradual via Test PyPI,
Automatizar o versionamento e melhorar o monitoramento do pipeline.

Prepare-se para passar de um pipeline funcional para uma infraestrutura CI/CD profissional.

Implantar uma aplicação Python no PyPI requer um pipeline CI/CD robusto que automatiza os testes, a construção e a publicação dos pacotes. Este artigo detalha a implementação de um pipeline completo usando GitHub Actions para uma aplicação CLI Python, baseando-se nas boas práticas do ecossistema Python moderno.

Arquitetura do Pipeline CI/CD

O pipeline que vamos construir segue uma abordagem em várias etapas :

@startuml
!theme plain
title Pipeline CI/CD Python para o PyPI
skinparam backgroundColor #f8f9fa
skinparam componentStyle rectangle

rectangle "Desenvolvedor" as dev
rectangle "repositório GitHub" as repo {
  rectangle ".github/workflows/" as workflows
  rectangle "testes/" as tests
  rectangle "setup.py / pyproject.toml" as setup
  rectangle "requirements.txt" as req
}

rectangle "GitHub Actions" as actions {
  rectangle "Executor de testes" as test_runner
  rectangle "Build" as build
  rectangle "Varredura de Segurança" as security
  rectangle "Verificação de Qualidade" as quality
}

rectangle "PyPI" as pypi {
  rectangle "Test PyPI" as test_pypi
  rectangle "Produção PyPI" as prod_pypi
}

rectangle "Usuários" as users

dev --> repo : push/PR
repo --> actions : trigger workflow
actions --> test_runner : run tests
actions --> security : security checks
actions --> quality : code quality
actions --> build : build package
build --> test_pypi : deploy (pre-release)
build --> prod_pypi : deploy (release)
prod_pypi --> users : install package

@enduml

Estrutura do Projeto

Uma aplicação CLI Python pronta para distribuição deve seguir uma estrutura padronizada :

playlist-downloader/
├── .github/
│   └── workflows/
│       ├── ci.yml
│       ├── release.yml
│       └── security.yml
├── src/
│   └── playlist_downloader/
│       ├── __init__.py
│       ├── cli.py
│       ├── core/
│       └── adapters/
├── tests/
│   ├── unit/
│   ├── integration/
│   └── conftest.py
├── docs/
├── pyproject.toml
├── requirements.txt
├── requirements-dev.txt
├── MANIFEST.in
├── README.md
├── LICENSE
└── CHANGELOG.md

Configuração do Pacote com pyproject.toml

O arquivo`pyproject.toml`é o padrão moderno para configurar pacotes Python :

[build-system]
requires = ["setuptools>=45", "wheel", "setuptools_scm>=6.2"]
build-backend = "setuptools.build_meta"

[project]
name = "playlist-downloader"
authors = [
    {name = "Christophe Hérolivier", email = "[email protected]"},
]
description = "CLI tool for YouTube playlist management"
readme = "README.md"
requires-python = ">=3.8"
keywords = ["youtube", "playlist", "cli", "downloader"]
license = {text = "MIT"}
classifiers = [
    "Development Status :: 4 - Beta",
    "Environment :: Console",
    "Intended Audience :: End Users/Desktop",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Topic :: Multimedia :: Sound/Audio",
    "Topic :: Utilities",
]
dependencies = [
    "typer>=0.9.0",
    "yt-dlp>=2023.7.6",
    "google-api-python-client>=2.0.0",
    "google-auth-oauthlib>=1.0.0",
    "pyyaml>=6.0",
    "rich>=13.0.0",
]
dynamic = ["version"]

[project.optional-dependencies]
dev = [
    "pytest>=7.0.0",
    "pytest-cov>=4.0.0",
    "pytest-mock>=3.10.0",
    "black>=23.0.0",
    "flake8>=6.0.0",
    "mypy>=1.0.0",
    "pre-commit>=3.0.0",
    "tox>=4.0.0",
]
test = [
    "pytest>=7.0.0",
    "pytest-cov>=4.0.0",
    "pytest-mock>=3.10.0",
]

[project.urls]
Homepage = "https://github.com/cheroliv/playlist-downloader"
Documentation = "https://github.com/cheroliv/playlist-downloader#readme"
Repository = "https://github.com/cheroliv/playlist-downloader.git"
"Bug Tracker" = "https://github.com/cheroliv/playlist-downloader/issues"

[project.scripts]
playlist-downloader = "playlist_downloader.cli:main"

[tool.setuptools_scm]
write_to = "src/playlist_downloader/_version.py"

[tool.setuptools.packages.find]
where = ["src"]

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = [
    "--cov=src/playlist_downloader",
    "--cov-report=html",
    "--cov-report=term-missing",
    "--cov-fail-under=85",
]

[tool.black]
line-length = 88
target-version = ['py38']
include = '\.pyi?$'
extend-exclude = '''
/(
  \.eggs
  | \.git
  | \.hg
  | \.mypy_cache
  | \.tox
  | \.venv
  | _build
  | buck-out
  | build
  | dist
)/
'''

[tool.mypy]
python_version = "3.8"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
disallow_untyped_decorators = true
no_implicit_optional = true
warn_redundant_casts = true
warn_unused_ignores = true
warn_no_return = true
warn_unreachable = true
strict_equality = true

[[tool.mypy.overrides]]
module = [
    "yt_dlp.*",
    "googleapiclient.*",
    "google_auth_oauthlib.*",
]
ignore_missing_imports = true

Workflow de CI/CD - Testes e Qualidade

O fluxo de trabalho principal`ci.yml`) executa os testes em várias versões do Python :

name: CI

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11"]

    steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}

    - name: Cache dependencies
      uses: actions/cache@v3
      with:
        path: |
          ~/.cache/pip
          ~/.cache/pre-commit
        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }}
        restore-keys: |
          ${{ runner.os }}-pip-

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -e ".[dev]"

    - name: Lint with flake8
      run: |
        flake8 src tests --count --select=E9,F63,F7,F82 --show-source --statistics
        flake8 src tests --count --exit-zero --max-complexity=10 --max-line-length=88 --statistics

    - name: Check code formatting with Black
      run: black --check src tests

    - name: Type checking with mypy
      run: mypy src

    - name: Run tests with pytest
      run: |
        pytest tests/ -v --cov=src/playlist_downloader \
          --cov-report=xml --cov-report=term-missing

    - name: Upload coverage to Codecov
      uses: codecov/codecov-action@v3
      if: matrix.python-version == '3.11'
      with:
        file: ./coverage.xml
        flags: unittests
        name: codecov-umbrella

  security:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: "3.11"

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install bandit[toml] safety

    - name: Run security checks with bandit
      run: bandit -r src/ -f json -o bandit-report.json

    - name: Check dependencies with safety
      run: safety check --json --output safety-report.json

    - name: Upload security reports
      uses: actions/upload-artifact@v3
      if: always()
      with:
        name: security-reports
        path: |
          bandit-report.json
          safety-report.json

  build:
    needs: [test, security]
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: "3.11"

    - name: Install build dependencies
      run: |
        python -m pip install --upgrade pip
        pip install build twine

    - name: Build package
      run: python -m build

    - name: Check package with twine
      run: twine check dist/*

    - name: Upload build artifacts
      uses: actions/upload-artifact@v3
      with:
        name: dist
        path: dist/

Workflow de Lançamento e Implantação

O workflow de release (release.yml) gerencia a publicação automática no PyPI :

name: Release

on:
  push:
    tags:
      - 'v*.*.*'
  workflow_dispatch:
    inputs:
      environment:
        description: 'Deployment environment'
        required: true
        default: 'test'
        type: choice
        options:
        - test
        - production

env:
  PYTHON_VERSION: "3.11"

jobs:
  release:
    runs-on: ubuntu-latest
    environment:
      name: ${{ github.event.inputs.environment || (startsWith(github.ref, 'refs/tags/') && 'production' || 'test') }}

    steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install build twine

    - name: Build package
      run: python -m build

    - name: Check package
      run: twine check dist/*

    - name: Publish to Test PyPI
      if: github.event.inputs.environment == 'test' || (startsWith(github.ref, 'refs/tags/') && contains(github.ref, 'rc'))
      env:
        TWINE_USERNAME: __token__
        TWINE_PASSWORD: ${{ secrets.TEST_PYPI_API_TOKEN }}
      run: |
        twine upload --repository testpypi dist/*

    - name: Publish to PyPI
      if: github.event.inputs.environment == 'production' || (startsWith(github.ref, 'refs/tags/') && !contains(github.ref, 'rc'))
      env:
        TWINE_USERNAME: __token__
        TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
      run: |
        twine upload dist/*

    - name: Create GitHub Release
      if: startsWith(github.ref, 'refs/tags/')
      uses: actions/create-release@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        tag_name: ${{ github.ref }}
        release_name: Release ${{ github.ref }}
        draft: false
        prerelease: ${{ contains(github.ref, 'rc') }}

  post-release:
    needs: release
    runs-on: ubuntu-latest
    if: startsWith(github.ref, 'refs/tags/')

    steps:
    - uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}

    - name: Test installation from PyPI
      run: |
        sleep 60  # Attendre la propagation sur PyPI
        pip install playlist-downloader
        playlist-downloader --version

    - name: Update documentation
      run: |
        # Script pour mettre à jour la documentation
        echo "Documentation updated for version ${GITHUB_REF#refs/tags/}"

Diagramas de Arquitetura

Diagrama de Sequência - Processo de Lançamento

@startuml
!theme plain
title Sequência de Release para o PyPI
actor Developer as dev
participant "GitHub" as gh
participant "Ações do GitHub" as ga
participant "Teste PyPI" as tpypi
participant "PyPI" as pypi
participant "Utilizadores" as users

dev -> gh : git push --tags v1.2.3
gh -> ga : trigger release workflow

ga -> ga : checkout code
ga -> ga : setup Python environment
ga -> ga : install dependencies
ga -> ga : run tests
ga -> ga : build package (wheel + sdist)
ga -> ga : check package with twine

alt Pre-release (rc tag)
    ga -> tpypi : upload to Test PyPI
    tpypi -> ga : confirm upload
else Stable release
    ga -> pypi : upload to PyPI
    pypi -> ga : confirm upload
end

ga -> gh : create GitHub release
ga -> ga : test installation from PyPI

users -> pypi : pip install playlist-downloader
pypi -> users : download package

@enduml

Diagrama de estados - ciclo de vida do pacote

@startuml
!theme plain
title Estados do Pacote Python
[*] --> Development
Development --> Testing : commit/PR
Testing --> Development : tests fail
Testing --> Built : tests pass
Built --> TestPyPI : pre-release tag
Built --> PyPI : stable tag
TestPyPI --> PyPI : validation OK
PyPI --> Published
Published --> [*]

state Development {
  [*] --> Coding
  Coding --> LocalTesting
  LocalTesting --> Coding : fix issues
  LocalTesting --> ReadyForCI : all tests pass
}

state Testing {
  [*] --> CITests
  CITests --> SecurityScan
  SecurityScan --> QualityCheck
  QualityCheck --> BuildValidation
}

@enduml

Diagrama de Implantação - Infraestrutura CI/CD

@startuml
!theme plain
title Infraestrutura de Implantação
node "GitHub" {
  component "Repositório" as repo
  component "Actions Runner" as runner
  component "Armazenamento de Segredos" as secrets
}

node "Infraestrutura do PyPI" {
  component "PyPI" as pypi
  component "Teste PyPI" as testpypi
  database "Índice de Pacotes" as index
}

node "Máquina do desenvolvedor" {
  component "Cliente Git" as git
  component "Ambiente Python" as python
  component "IDE" as ide
}

node "Ambiente do Usuário" {
  component "pip" as pip_client
  component "ambiente de execução do Python" as py_runtime
}

git --> repo : push code/tags
repo --> runner : trigger workflows
runner --> secrets : read API tokens
runner --> testpypi : upload pre-release
runner --> pypi : upload release
pypi --> index : store package
pip_client --> pypi : download package
py_runtime <-- pip_client : install package

@enduml

Objetos e Modelos do Pipeline

Diagrama de Classes - Modelos CI/CD

@startuml
!theme plain
title Modelos do Pipeline CI/CD
class PipelineConfig {
  +python_versions: List[str]
  +test_environments: List[str]
  +security_checks: bool
  +coverage_threshold: float
  +validate()
}

class BuildArtifact {
  +name: str
  +version: str
  +wheel_path: str
  +sdist_path: str
  +checksums: Dict[str, str]
  +validate_integrity()
}

class TestResult {
  +test_suite: str
  +python_version: str
  +passed: int
  +failed: int
  +coverage: float
  +duration: float
  +is_success(): bool
}

class SecurityReport {
  +bandit_issues: List[Issue]
  +safety_vulnerabilities: List[Vulnerability]
  +severity_level: str
  +is_secure(): bool
}

class DeploymentTarget {
  +name: str
  +url: str
  +api_token: str
  +environment: str
  +deploy(artifact: BuildArtifact)
}

class ReleaseManager {
  +version: str
  +changelog: str
  +artifacts: List[BuildArtifact]
  +test_results: List[TestResult]
  +security_report: SecurityReport
  +deploy_to_test()
  +deploy_to_production()
  +create_github_release()
}

PipelineConfig ||--o{ TestResult
ReleaseManager *-- BuildArtifact
ReleaseManager *-- SecurityReport
ReleaseManager o-- DeploymentTarget
DeploymentTarget ..> BuildArtifact : uses

@enduml

Configuração dos Segredos

Para que o pipeline funcione, você deve configurar os seguintes segredos no GitHub

Segredos GitHub Actions

# Dans Settings > Secrets and variables > Actions

# Token PyPI pour la production
PYPI_API_TOKEN=pypi-...

# Token Test PyPI pour les pré-releases
TEST_PYPI_API_TOKEN=pypi-...

# Token GitHub pour créer les releases
GITHUB_TOKEN=(automatiquement fourni)

# Token Codecov (optionnel)
CODECOV_TOKEN=...

Geração de Tokens PyPI

# 1. Créer un compte sur PyPI et Test PyPI
# 2. Aller dans Account Settings > API tokens
# 3. Créer un token avec scope "Entire account" ou spécifique au projet
# 4. Format du token : pypi-AgEIcHlwaS5vcmc...

Scripts de Desenvolvimento Local

Para facilitar o desenvolvimento, crie scripts utilitários:

Makefile

.PHONY: install test lint format security build clean release-test release-prod

install:
	pip install -e ".[dev]"

test:
	pytest tests/ -v --cov=src/playlist_downloader

lint:
	flake8 src tests
	mypy src

format:
	black src tests

security:
	bandit -r src/
	safety check

build:
	python -m build
	twine check dist/*

clean:
	rm -rf build/ dist/ *.egg-info/
	find . -type d -name __pycache__ -delete
	find . -name "*.pyc" -delete

release-test: clean build
	twine upload --repository testpypi dist/*

release-prod: clean build
	twine upload dist/*

pre-commit: format lint test security
	@echo "✅ Prêt pour commit"

Script de Versão

#!/usr/bin/env python3
"""Script pour gérer les versions du projet."""

import sys
import subprocess
from pathlib import Path

def get_current_version():
    """Récupère la version actuelle depuis git."""
    try:
        result = subprocess.run(
            ["git", "describe", "--tags", "--abbrev=0"],
            capture_output=True,
            text=True,
            check=True
        )
        return result.stdout.strip()
    except subprocess.CalledProcessError:
        return "0.0.0"

def create_version_tag(version, message=None):
    """Crée un tag de version."""
    if not version.startswith('v'):
        version = f'v{version}'

    tag_message = message or f"Release {version}"

    subprocess.run(["git", "tag", "-a", version, "-m", tag_message], check=True)
    print(f"✅ Tag {version} créé")

    # Push le tag
    subprocess.run(["git", "push", "origin", version], check=True)
    print(f"✅ Tag {version} poussé vers origin")

if __name__ == "__main__":
    if len(sys.argv) < 2:
        current = get_current_version()
        print(f"Version actuelle: {current}")
        print("Usage: python version.py <new_version> [message]")
        sys.exit(1)

    new_version = sys.argv[1]
    message = sys.argv[2] if len(sys.argv) > 2 else None

    create_version_tag(new_version, message)

Boas práticas e recomendações

Versionamento Semântico

Utilize o versionamento semântico (SemVer):

MAJOR.MINOR.PATCH (ex: 1.2.3)
MAJOR: alterações incompatíveis
`MINOR`novas funcionalidades compatíveis
PATCH: correções de bugs compatíveis

Estratégia de ramificação

main          ──●──●──●──●──●────●── (releases stables)
               /       /          /
develop    ──●──●──●──●──●──●──●──●── (développement)
            /     /        /
feature/xxx  ●──●──●──●──●──/ (fonctionnalités)

Testes e Cobertura

Cobertura mínima de código: 85%
Testes unitários para a lógica de negócio
Testes de integração para os adaptadores
Testes de ponta a ponta para as CLI

Segurança

Escaneamento automático de dependências (Safety)
Análise estática do código (Bandit)
Segredos nunca no código
Utilização de tokens PyPI específicos

Monitoramento e Observabilidade

Métricas do Pipeline

# .github/workflows/metrics.yml
name: Pipeline Metrics

on:
  workflow_run:
    workflows: ["CI", "Release"]
    types: [completed]

jobs:
  metrics:
    runs-on: ubuntu-latest
    steps:
    - name: Collect metrics
      run: |
        echo "Pipeline: ${{ github.event.workflow_run.name }}"
        echo "Status: ${{ github.event.workflow_run.conclusion }}"
        echo "Duration: ${{ github.event.workflow_run.updated_at - github.event.workflow_run.created_at }}"
        # Envoyer vers système de monitoring

Notificações

# Ajout dans les workflows pour notifications
- name: Notify on failure
  if: failure()
  uses: 8398a7/action-slack@v3
  with:
    status: failure
    text: "❌ Pipeline failed for ${{ github.repository }}"
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

Conclusão

Este pipeline CI/CD completo para Python oferece :

Automação completa: da validação do código à publicação
Segurança: digitalizações automáticas e gestão segura dos segredos
Qualidadetestes multi-versões, linting e cobertura de código
Fiabilidadeimplementação gradual via Test PyPI
Traçabilidade: artefatos, relatórios e lançamentos GitHub

A adoção dessas práticas garante um processo de entrega robusto e profissional para suas aplicações Python CLI, facilitando a manutenção e a evolução de seus projetos a longo prazo.