Bagian 2 : Mengindustrialisasikan dan mengamankan pipeline CI/CD Python tingkat lanjut

Pengantar

Tujuan : Mengubah pipeline sederhana menjadi pipeline industri yang kuat, mematuhi standar modern dari packaging dan rekayasa perangkat lunak.

Dalam bagian pertama dari seri ini, kita telah membangun pipeline CI/CD sederhana namun efektif untuk mengotomatiskan pengujian dan publikasi paket Python ke PyPI.

Saatnya untuk profesionalisasi pipeline ini. Dalam bagian kedua ini, kita akan :

Migrasikan ke standar modern pyproject.toml,
Menambahkan alat kualitas dan keamanan (Black, Mypy, Bandit, Safety),
Menyiapkan tes multi-versi,
Mengintegrasikan peluncuran bertahap melalui Test PyPI,
Otomatiskan versioning dan perbaiki pemantauan pipeline.

Bersiaplah untuk beralih dari pipeline fungsional ke infrastruktur CI/CD profesional.

Menerapkan aplikasi Python ke PyPI membutuhkan pipeline CI/CD yang kuat yang mengotomatiskan pengujian, pembangunan, dan publikasi paket. Artikel ini merinci penyiapan pipeline lengkap yang menggunakan GitHub Actions untuk aplikasi CLI Python, dengan mengandalkan praktik terbaik ekosistem Python modern.

Arsitektur Pipeline CI/CD

Pipeline yang akan kita bangun mengikuti pendekatan bertahap:

@startuml
!theme plain
title Alur CI/CD Python ke PyPI
skinparam backgroundColor #f8f9fa
skinparam componentStyle rectangle

rectangle "Pengembang" as dev
rectangle "Repositori GitHub" as repo {
  rectangle ".github/workflows/" as workflows
  rectangle "ujian/" as tests
  rectangle "setup.py / pyproject.toml" as setup
  rectangle "requirements.txt" as req
}

rectangle "GitHub Tindakan" as actions {
  rectangle "Pelaksana Pengujian" as test_runner
  rectangle "Bangun" as build
  rectangle "Pindai Keamanan" as security
  rectangle "Pemeriksaan Kualitas" as quality
}

rectangle "PyPI" as pypi {
  rectangle "Uji PyPI" as test_pypi
  rectangle "Produksi PyPI" as prod_pypi
}

rectangle "Pengguna" as users

dev --> repo : push/PR
repo --> actions : trigger workflow
actions --> test_runner : run tests
actions --> security : security checks
actions --> quality : code quality
actions --> build : build package
build --> test_pypi : deploy (pre-release)
build --> prod_pypi : deploy (release)
prod_pypi --> users : install package

@enduml

Struktur Proyek

Sebuah aplikasi CLI Python yang siap untuk didistribusikan harus mematuhi struktur yang terstandar :

playlist-downloader/
├── .github/
│   └── workflows/
│       ├── ci.yml
│       ├── release.yml
│       └── security.yml
├── src/
│   └── playlist_downloader/
│       ├── __init__.py
│       ├── cli.py
│       ├── core/
│       └── adapters/
├── tests/
│   ├── unit/
│   ├── integration/
│   └── conftest.py
├── docs/
├── pyproject.toml
├── requirements.txt
├── requirements-dev.txt
├── MANIFEST.in
├── README.md
├── LICENSE
└── CHANGELOG.md

Konfigurasi Paket dengan pyproject.toml

File`pyproject.toml`adalah standar modern untuk mengonfigurasi paket Python:

[build-system]
requires = ["setuptools>=45", "wheel", "setuptools_scm>=6.2"]
build-backend = "setuptools.build_meta"

[project]
name = "playlist-downloader"
authors = [
    {name = "Christophe Hérolivier", email = "[email protected]"},
]
description = "CLI tool for YouTube playlist management"
readme = "README.md"
requires-python = ">=3.8"
keywords = ["youtube", "playlist", "cli", "downloader"]
license = {text = "MIT"}
classifiers = [
    "Development Status :: 4 - Beta",
    "Environment :: Console",
    "Intended Audience :: End Users/Desktop",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Topic :: Multimedia :: Sound/Audio",
    "Topic :: Utilities",
]
dependencies = [
    "typer>=0.9.0",
    "yt-dlp>=2023.7.6",
    "google-api-python-client>=2.0.0",
    "google-auth-oauthlib>=1.0.0",
    "pyyaml>=6.0",
    "rich>=13.0.0",
]
dynamic = ["version"]

[project.optional-dependencies]
dev = [
    "pytest>=7.0.0",
    "pytest-cov>=4.0.0",
    "pytest-mock>=3.10.0",
    "black>=23.0.0",
    "flake8>=6.0.0",
    "mypy>=1.0.0",
    "pre-commit>=3.0.0",
    "tox>=4.0.0",
]
test = [
    "pytest>=7.0.0",
    "pytest-cov>=4.0.0",
    "pytest-mock>=3.10.0",
]

[project.urls]
Homepage = "https://github.com/cheroliv/playlist-downloader"
Documentation = "https://github.com/cheroliv/playlist-downloader#readme"
Repository = "https://github.com/cheroliv/playlist-downloader.git"
"Bug Tracker" = "https://github.com/cheroliv/playlist-downloader/issues"

[project.scripts]
playlist-downloader = "playlist_downloader.cli:main"

[tool.setuptools_scm]
write_to = "src/playlist_downloader/_version.py"

[tool.setuptools.packages.find]
where = ["src"]

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = [
    "--cov=src/playlist_downloader",
    "--cov-report=html",
    "--cov-report=term-missing",
    "--cov-fail-under=85",
]

[tool.black]
line-length = 88
target-version = ['py38']
include = '\.pyi?$'
extend-exclude = '''
/(
  \.eggs
  | \.git
  | \.hg
  | \.mypy_cache
  | \.tox
  | \.venv
  | _build
  | buck-out
  | build
  | dist
)/
'''

[tool.mypy]
python_version = "3.8"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
disallow_untyped_decorators = true
no_implicit_optional = true
warn_redundant_casts = true
warn_unused_ignores = true
warn_no_return = true
warn_unreachable = true
strict_equality = true

[[tool.mypy.overrides]]
module = [
    "yt_dlp.*",
    "googleapiclient.*",
    "google_auth_oauthlib.*",
]
ignore_missing_imports = true

Alur kerja CI/CD - Pengujian dan Kualitas

Alur kerja utama (ci.yml) menjalankan tes pada beberapa versi Python :

name: CI

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11"]

    steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}

    - name: Cache dependencies
      uses: actions/cache@v3
      with:
        path: |
          ~/.cache/pip
          ~/.cache/pre-commit
        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }}
        restore-keys: |
          ${{ runner.os }}-pip-

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -e ".[dev]"

    - name: Lint with flake8
      run: |
        flake8 src tests --count --select=E9,F63,F7,F82 --show-source --statistics
        flake8 src tests --count --exit-zero --max-complexity=10 --max-line-length=88 --statistics

    - name: Check code formatting with Black
      run: black --check src tests

    - name: Type checking with mypy
      run: mypy src

    - name: Run tests with pytest
      run: |
        pytest tests/ -v --cov=src/playlist_downloader \
          --cov-report=xml --cov-report=term-missing

    - name: Upload coverage to Codecov
      uses: codecov/codecov-action@v3
      if: matrix.python-version == '3.11'
      with:
        file: ./coverage.xml
        flags: unittests
        name: codecov-umbrella

  security:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: "3.11"

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install bandit[toml] safety

    - name: Run security checks with bandit
      run: bandit -r src/ -f json -o bandit-report.json

    - name: Check dependencies with safety
      run: safety check --json --output safety-report.json

    - name: Upload security reports
      uses: actions/upload-artifact@v3
      if: always()
      with:
        name: security-reports
        path: |
          bandit-report.json
          safety-report.json

  build:
    needs: [test, security]
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: "3.11"

    - name: Install build dependencies
      run: |
        python -m pip install --upgrade pip
        pip install build twine

    - name: Build package
      run: python -m build

    - name: Check package with twine
      run: twine check dist/*

    - name: Upload build artifacts
      uses: actions/upload-artifact@v3
      with:
        name: dist
        path: dist/

Workflow Rilis dan Penerapan

Alur kerja rilis (release.yml) mengelola publikasi otomatis ke PyPI:

name: Release

on:
  push:
    tags:
      - 'v*.*.*'
  workflow_dispatch:
    inputs:
      environment:
        description: 'Deployment environment'
        required: true
        default: 'test'
        type: choice
        options:
        - test
        - production

env:
  PYTHON_VERSION: "3.11"

jobs:
  release:
    runs-on: ubuntu-latest
    environment:
      name: ${{ github.event.inputs.environment || (startsWith(github.ref, 'refs/tags/') && 'production' || 'test') }}

    steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install build twine

    - name: Build package
      run: python -m build

    - name: Check package
      run: twine check dist/*

    - name: Publish to Test PyPI
      if: github.event.inputs.environment == 'test' || (startsWith(github.ref, 'refs/tags/') && contains(github.ref, 'rc'))
      env:
        TWINE_USERNAME: __token__
        TWINE_PASSWORD: ${{ secrets.TEST_PYPI_API_TOKEN }}
      run: |
        twine upload --repository testpypi dist/*

    - name: Publish to PyPI
      if: github.event.inputs.environment == 'production' || (startsWith(github.ref, 'refs/tags/') && !contains(github.ref, 'rc'))
      env:
        TWINE_USERNAME: __token__
        TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
      run: |
        twine upload dist/*

    - name: Create GitHub Release
      if: startsWith(github.ref, 'refs/tags/')
      uses: actions/create-release@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        tag_name: ${{ github.ref }}
        release_name: Release ${{ github.ref }}
        draft: false
        prerelease: ${{ contains(github.ref, 'rc') }}

  post-release:
    needs: release
    runs-on: ubuntu-latest
    if: startsWith(github.ref, 'refs/tags/')

    steps:
    - uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}

    - name: Test installation from PyPI
      run: |
        sleep 60  # Attendre la propagation sur PyPI
        pip install playlist-downloader
        playlist-downloader --version

    - name: Update documentation
      run: |
        # Script pour mettre à jour la documentation
        echo "Documentation updated for version ${GITHUB_REF#refs/tags/}"

diagram arsitektur

Diagram Urutan - Proses Rilis

@startuml
!theme plain
title Urutan rilis ke PyPI
actor Developer as dev
participant "GitHub" as gh
participant "GitHub Actions" as ga
participant "Test PyPI" as tpypi
participant "PyPI" as pypi
participant "Pengguna" as users

dev -> gh : git push --tags v1.2.3
gh -> ga : trigger release workflow

ga -> ga : checkout code
ga -> ga : setup Python environment
ga -> ga : install dependencies
ga -> ga : run tests
ga -> ga : build package (wheel + sdist)
ga -> ga : check package with twine

alt Pre-release (rc tag)
    ga -> tpypi : upload to Test PyPI
    tpypi -> ga : confirm upload
else Stable release
    ga -> pypi : upload to PyPI
    pypi -> ga : confirm upload
end

ga -> gh : create GitHub release
ga -> ga : test installation from PyPI

users -> pypi : pip install playlist-downloader
pypi -> users : download package

@enduml

Diagram Kondisi - Siklus Hidup Paket

@startuml
!theme plain
title Status Paket Python
[*] --> Development
Development --> Testing : commit/PR
Testing --> Development : tests fail
Testing --> Built : tests pass
Built --> TestPyPI : pre-release tag
Built --> PyPI : stable tag
TestPyPI --> PyPI : validation OK
PyPI --> Published
Published --> [*]

state Development {
  [*] --> Coding
  Coding --> LocalTesting
  LocalTesting --> Coding : fix issues
  LocalTesting --> ReadyForCI : all tests pass
}

state Testing {
  [*] --> CITests
  CITests --> SecurityScan
  SecurityScan --> QualityCheck
  QualityCheck --> BuildValidation
}

@enduml

Diagram Penyebaran - Infrastruktur CI/CD

@startuml
!theme plain
title Infrastruktur penyebaran
node "GitHub" {
  component "Repositori" as repo
  component "Penjalani Tindakan" as runner
  component "Toko Rahasia" as secrets
}

node "PyPI infrastruktur" {
  component "PyPI" as pypi
  component "Test PyPI" as testpypi
  database "Indeks Paket" as index
}

node "mesin pengembang" {
  component "Klien Git" as git
  component "lingkungan Python" as python
  component "IDE" as ide
}

node "Lingkungan Pengguna" {
  component "pip" as pip_client
  component "Runtime Python" as py_runtime
}

git --> repo : push code/tags
repo --> runner : trigger workflows
runner --> secrets : read API tokens
runner --> testpypi : upload pre-release
runner --> pypi : upload release
pypi --> index : store package
pip_client --> pypi : download package
py_runtime <-- pip_client : install package

@enduml

Objek dan Model Pipeline

Diagram kelas - Model-model CI/CD

@startuml
!theme plain
title Model Pipeline CI/CD
class PipelineConfig {
  +python_versions: List[str]
  +test_environments: List[str]
  +security_checks: bool
  +coverage_threshold: float
  +validate()
}

class BuildArtifact {
  +name: str
  +version: str
  +wheel_path: str
  +sdist_path: str
  +checksums: Dict[str, str]
  +validate_integrity()
}

class TestResult {
  +test_suite: str
  +python_version: str
  +passed: int
  +failed: int
  +coverage: float
  +duration: float
  +is_success(): bool
}

class SecurityReport {
  +bandit_issues: List[Issue]
  +safety_vulnerabilities: List[Vulnerability]
  +severity_level: str
  +is_secure(): bool
}

class DeploymentTarget {
  +name: str
  +url: str
  +api_token: str
  +environment: str
  +deploy(artifact: BuildArtifact)
}

class ReleaseManager {
  +version: str
  +changelog: str
  +artifacts: List[BuildArtifact]
  +test_results: List[TestResult]
  +security_report: SecurityReport
  +deploy_to_test()
  +deploy_to_production()
  +create_github_release()
}

PipelineConfig ||--o{ TestResult
ReleaseManager *-- BuildArtifact
ReleaseManager *-- SecurityReport
ReleaseManager o-- DeploymentTarget
DeploymentTarget ..> BuildArtifact : uses

@enduml

Konfigurasi Rahasia

Agar pipeline berfungsi, Anda harus mengonfigurasi rahasia berikut di GitHub :

Rahasia GitHub Actions

# Dans Settings > Secrets and variables > Actions

# Token PyPI pour la production
PYPI_API_TOKEN=pypi-...

# Token Test PyPI pour les pré-releases
TEST_PYPI_API_TOKEN=pypi-...

# Token GitHub pour créer les releases
GITHUB_TOKEN=(automatiquement fourni)

# Token Codecov (optionnel)
CODECOV_TOKEN=...

Pembuatan Token PyPI

# 1. Créer un compte sur PyPI et Test PyPI
# 2. Aller dans Account Settings > API tokens
# 3. Créer un token avec scope "Entire account" ou spécifique au projet
# 4. Format du token : pypi-AgEIcHlwaS5vcmc...

Skrip Pengembangan Lokal

Untuk memudahkan pengembangan, buat skrip utilitas:

Makefile

.PHONY: install test lint format security build clean release-test release-prod

install:
	pip install -e ".[dev]"

test:
	pytest tests/ -v --cov=src/playlist_downloader

lint:
	flake8 src tests
	mypy src

format:
	black src tests

security:
	bandit -r src/
	safety check

build:
	python -m build
	twine check dist/*

clean:
	rm -rf build/ dist/ *.egg-info/
	find . -type d -name __pycache__ -delete
	find . -name "*.pyc" -delete

release-test: clean build
	twine upload --repository testpypi dist/*

release-prod: clean build
	twine upload dist/*

pre-commit: format lint test security
	@echo "✅ Prêt pour commit"

Skrip Versi

#!/usr/bin/env python3
"""Script pour gérer les versions du projet."""

import sys
import subprocess
from pathlib import Path

def get_current_version():
    """Récupère la version actuelle depuis git."""
    try:
        result = subprocess.run(
            ["git", "describe", "--tags", "--abbrev=0"],
            capture_output=True,
            text=True,
            check=True
        )
        return result.stdout.strip()
    except subprocess.CalledProcessError:
        return "0.0.0"

def create_version_tag(version, message=None):
    """Crée un tag de version."""
    if not version.startswith('v'):
        version = f'v{version}'

    tag_message = message or f"Release {version}"

    subprocess.run(["git", "tag", "-a", version, "-m", tag_message], check=True)
    print(f"✅ Tag {version} créé")

    # Push le tag
    subprocess.run(["git", "push", "origin", version], check=True)
    print(f"✅ Tag {version} poussé vers origin")

if __name__ == "__main__":
    if len(sys.argv) < 2:
        current = get_current_version()
        print(f"Version actuelle: {current}")
        print("Usage: python version.py <new_version> [message]")
        sys.exit(1)

    new_version = sys.argv[1]
    message = sys.argv[2] if len(sys.argv) > 2 else None

    create_version_tag(new_version, message)

Praktik Baik dan Rekomendasi

Versioning Semantik

Gunakan versioning semantik (SemVer):

MAJOR.MINOR.PATCH (ex: 1.2.3)
MAJOR: perubahan tidak kompatibel
MINOR: fitur baru yang kompatibel
PATCH: perbaikan bug yang kompatibel

Strategi Pembranching

main          ──●──●──●──●──●────●── (releases stables)
               /       /          /
develop    ──●──●──●──●──●──●──●──●── (développement)
            /     /        /
feature/xxx  ●──●──●──●──●──/ (fonctionnalités)

Pengujian dan Cakupan

Cakupan kode minimum : 85%
Uji unit untuk logika bisnis
Uji integrasi untuk adapter
Uji ujung ke ujung untuk CLI

Keamanan

Pemindaian otomatis dependensi (Safety)
Analisis statis kode (Bandit)
Rahasia tidak pernah ada dalam kode
Penggunaan token PyPI khusus

monitoring dan observability

Metrik Pipeline

# .github/workflows/metrics.yml
name: Pipeline Metrics

on:
  workflow_run:
    workflows: ["CI", "Release"]
    types: [completed]

jobs:
  metrics:
    runs-on: ubuntu-latest
    steps:
    - name: Collect metrics
      run: |
        echo "Pipeline: ${{ github.event.workflow_run.name }}"
        echo "Status: ${{ github.event.workflow_run.conclusion }}"
        echo "Duration: ${{ github.event.workflow_run.updated_at - github.event.workflow_run.created_at }}"
        # Envoyer vers système de monitoring

Notifikasi

# Ajout dans les workflows pour notifications
- name: Notify on failure
  if: failure()
  uses: 8398a7/action-slack@v3
  with:
    status: failure
    text: "❌ Pipeline failed for ${{ github.repository }}"
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

Kesimpulan

Pipeline CI/CD lengkap untuk Python menawarkan:

Otomatisasi lengkapdari validasi kode hingga publikasi
Keamanan: pindai otomatis dan pengelolaan aman rahasia
Kualitastes multi-versi, linting dan cakupan kode
keandalan: penerapan progresif melalui Test PyPI
pelacakan: artefak, laporan dan rilis GitHub

Adoptasi praktik ini menjamin proses pengiriman yang kuat dan profesional untuk aplikasi CLI Python Anda, memudahkan pemeliharaan dan perkembangan proyek Anda dalam jangka panjang.