Git and GitHub Workflows for Data Projects — The Cheat Code You Actually Need

"Pushing to master should not be an extreme sport." — Your future calm self

You just finished training a sweet CNN or a transformer (Deep Learning Foundations — nice). You joined tables with pandas + SQLAlchemy (solid), ran SQL queries that made your analyst friend weep with joy (SQL Fundamentals — epic). Now someone — maybe production, maybe you in three months — will want to re-run, debug, or deploy that pipeline. Enter: Git and GitHub workflows — the scaffolding that keeps your experiments from becoming chaotic archaeological sites of Jupyter notebooks and mystery CSVs.

Why Git & GitHub workflows matter for data science and deployment

• Reproducibility: Track code, SQL queries, and scripts that produced a dataset or a model. When your model's accuracy changes, you can trace it to the exact commit.
• Collaboration: Branches + pull requests (PRs) = non-destructive teamwork. No one overwrites your pretrained weights accidentally (hopefully).
• Automated checks & deployment: Run tests, data validations, and deploy models via CI/CD so your pipeline doesn’t collapse when someone renames a column.
• Separation of concerns: Keep large data out of Git; keep SQL migrations, scripts, and model-serving code versioned.

Core concepts (quick micro explanations)

Repos, commits, branches

• Repo: project folder with history.
• Commit: snapshot + message. Write good messages.
• Branch: parallel timelines. Feature branches are your playground.

Pull Requests (PRs)

• Ask for code review. Run CI on PRs. Use templates for reproducibility steps and data notes.

Tags & releases

• Tag releases (v1.2.0) to mark deployable states — useful for model versioning.

Git LFS, DVC, and artifacts

• Git LFS: stores large objects (models) outside Git but tracked.
• DVC: data-versioning tool that works with remote storage; great for datasets and model artifacts.
• Artifact registries: GitHub Packages, S3, or MLflow model stores — production-grade storage for model binaries.

A practical GitHub workflow for data science projects

Think of this as the checklist you wish your team already used.

1. Main branches
   • main or master: always deployable; protected by branch rules.
   • develop (optional): integration branch for ongoing work.
2. Feature branches
   • feature/clean-sqlalchemy-models, feature/experiment-resnet50
3. PR + automated checks
   • Lint (flake8/black), unit tests (pytest), small data tests, reproducibility smoke test, model eval (if fast).
4. Review & merge
   • Require 1–2 approvals, passing CI, and a PR checklist (see below).
5. Release
   • Tag and publish artifacts; update model registry and deployment triggers.

Example git commands (feature branch flow)

# create and switch
git checkout -b feature/clean-sql-stmt
# make changes
git add .
git commit -m "refactor: use SQLAlchemy core for monthly agg"
# push and open PR
git push -u origin feature/clean-sql-stmt

PR Checklist (copy into your repo as template)

• Code passes linting and tests
• SQL queries are parameterized / checked for injection
• No large data files or model weights committed
• Data schema changes recorded (migration scripts or docs)
• Repro steps in PR description (how to run the pipeline locally)
• Performance/accuracy checks included if model changes

CI/CD in practice: a minimal GitHub Actions example

This snippet shows a CI job to run tests and build a Docker image for deployment. (Trim before using; replace secrets and registry values.)

name: CI
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: 3.10
      - name: Install deps
        run: pip install -r requirements.txt
      - name: Run tests
        run: pytest -q
  build-and-push:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v3
      - name: Build Docker image
        run: docker build -t ghcr.io/${{ github.repository }}:v${{ github.run_number }} .
      - name: Push to GitHub Container Registry
        uses: docker/login-action@v2
        with: registry: ghcr.io, username: ${{ github.actor }}, password: ${{ secrets.GITHUB_TOKEN }}
      - name: Push image
        run: docker push ghcr.io/${{ github.repository }}:v${{ github.run_number }}

Tip: Extend CI to run a fast model evaluation (e.g., smoke test on a small validation set) so you don't deploy a broken model.

Notebooks, diffs, and secrets — the practical nasties

• Use nbdime for notebook diffs or convert notebooks into scripts with nbconvert.
• Add .gitattributes to strip outputs (or use nbstripout) so diffs stay readable.
• Never commit credentials or DB connection strings. Use GitHub Secrets and environment variables in workflows to connect to SQL servers or artifact stores.
• Use .gitignore for data: data/, *.h5, *.pt, .env

Example .gitignore snippet:

# data and models
data/
models/
*.h5
*.pt
# python env
.env
venv/
__pycache__/

Large files and data: what to store where

• Source code, SQL migrations, helper scripts: Git.
• Large datasets and model binaries: DVC + remote (S3, GCS) or Git LFS for simpler cases.
• Database snapshots or raw exports: store a pointer (SQL query, DVC file) — not the file itself.

This is where your earlier work with SQL and pandas shines: store the query and the ETL script (SQLAlchemy code) in Git so anyone can recreate the dataset from the canonical source.

From experiment to deployment — high-level flow

1. Create feature branch, code model changes, add reproducible training script.
2. Push branch -> PR triggers CI: lint, tests, small dataset training or evaluation.
3. Approve PR -> merge to main -> GitHub Actions builds image or package -> pushes artifact.
4. Release workflow tags version -> deploys container to staging -> run smoke-tests -> promote to prod.

If your model training is heavy, use CI to run only fast checks and trigger external training pipelines (Kubernetes job, cloud training service) as part of release automation. Use model registry (MLflow, DVC + artifacts) for tracking metrics and versions.

Key takeaways — what to do tomorrow morning

• Protect the main branch and use PRs with automated checks.
• Never store raw data or credentials in Git; use DVC/GCS/S3 and secrets.
• Use Git LFS or artifact registries for model binaries — DVC if you want dataset reproducibility.
• Add a PR template that requires reproducibility steps and a quick evaluation summary.
• Automate builds and deployments with GitHub Actions; treat CI failures as sacred.

"Treat your Git history like a museum: future you will be a grateful visitor."

Go push that tidy, testable, and deployable pipeline. Your future self (and your teammates) will send you a thank-you emoji and maybe even buy you coffee.

Further reading / next steps

• Integrate DVC with your SQL + pandas workflows to reproduce datasets from DB exports or queries.
• Add model evaluation stages to your CI for every PR that touches model code.
• Explore GitHub Environments & Secrets for safe deployments.