Linux, Git, CLI, and Reproducible Dev Environments

Why Terminal Fluency Compounds

Every ML engineer works in terminals: connecting to remote training machines, navigating experiment directories, inspecting logs, launching jobs, debugging package conflicts. Engineers who navigate this environment confidently move faster, make fewer environment-related mistakes, and ship more reproducible work.

This module is not about memorizing commands. It is about internalizing the patterns that make your ML work reproducible and your debugging efficient.

Shell Navigation and File Operations

The commands every ML engineer uses daily:

bashCopy
# Navigation
cd ~/projects/my_model       # change directory
pwd                          # print working directory
ls -la                       # list with permissions and hidden files
ls -lh data/                 # human-readable file sizes

# Files
cp data/raw/events.csv data/backup/events_backup.csv
mv features_v1.py features_v2.py
rm -f tmp/*.json             # remove all .json files in tmp/
mkdir -p experiments/run_001/checkpoints

# Finding files
find . -name "*.py" -newer requirements.txt   # Python files changed recently
find . -name "*.ipynb" -size +10M             # Notebooks larger than 10 MB
find . -name "*.pkl" -exec ls -lh {} \;       # Find and inspect model files

Pipes and Redirection

The shell's power comes from composing simple commands:

bashCopy
# Count training examples in a CSV (minus header)
echo "$(wc -l < data/train.csv) - 1" | bc

# Show class distribution (assuming label is the last field)
cut -d',' -f-1 data/train.csv | sort | uniq -c | sort -rn

# Find all Python files that import torch
grep -rl "import torch" src/

# Count errors per hour in training logs
grep "ERROR" logs/training.log | awk '{print $1}' | uniq -c

# Save command output to a file (overwrite)
python scripts/evaluate.py > results/eval_run_42.txt 2>&1

# Append to a file
python scripts/train.py >> logs/training.log 2>&1

Environment Variables

Used for secrets, paths, and runtime configuration:

bashCopy
# Set temporarily for one command
CUDA_VISIBLE_DEVICES=0,1 python train.py

# Export for the current session
export MLFLOW_TRACKING_URI="http://localhost:5000"
export HF_HOME="/mnt/fast-drive/.cache/huggingface"

# Load from a .env file
set -a; source .env; set +a

Never hardcode API keys or database URIs in source code. Store them in .env files and add .env to .gitignore.

Git for ML Projects

The Core Workflow

bashCopy
git init                          # Initialize a repo
git clone <url>                   # Clone an existing repo
git status                        # See what changed
git diff                          # See unstaged changes

# Stage specific files - not git add . which can include secrets
git add src/features/pipeline.py configs/train_v3.yaml
git commit -m "Add median imputation for missing income feature"

git push origin main

Good commit messages describe why - the diff shows the what.

Branching for Experiments

bashCopy
git checkout -b experiment/bge-reranker     # Create + switch
# ... iterate ...
git commit -m "bge-reranker-v2: +4.2 Precision@5 on dev set"
git checkout main
git merge experiment/bge-reranker           # Merge if it worked
git branch -d experiment/bge-reranker       # Clean up

Branch per experiment. Delete failed branches. Merge succeeded ones with a commit that records the result.

.gitignore for ML Projects

gitignoreCopy
# Python
__pycache__/
*.pyc
.venv/
*.egg-info/

# Data and models - use DVC or object storage instead
data/raw/
data/processed/
models/
checkpoints/
*.parquet
*.pkl
*.h5
*.safetensors

# Secrets
.env
*.key
secrets.yaml

# Experiment outputs
mlruns/
wandb/
logs/
outputs/

# Jupyter checkpoints
.ipynb_checkpoints/

Never commit data files or model weights. They make the repository slow and the history useless. Use DVC, Git LFS, or object storage (S3, GCS) for versioned data and artifacts.

Navigating History

bashCopy
git log --oneline -20                 # Compact recent history
git log --grep "baseline"             # Find commits mentioning baseline
git show abc1234                      # Show changes in a specific commit
git diff main..experiment/v2          # Diff between branches
git blame src/features/pipeline.py    # See who changed each line and when

git bisect is invaluable when a regression was introduced somewhere in a sequence of commits:

bashCopy
git bisect start
git bisect bad                   # Current commit is broken
git bisect good v1.2.0           # Last known good commit
# Git checks out the midpoint commit; you test it and mark good/bad
# Repeats until the first breaking commit is identified
git bisect reset

Environment Management

The Problem With Global Python

If you install packages globally, every project shares the same package set. One project upgrades NumPy; another project breaks. This is not hypothetical.

Every ML project needs an isolated environment.

venv (built-in, simplest):

bashCopy
python -m venv .venv
source .venv/bin/activate        # macOS / Linux
.venv\Scripts\activate           # Windows
pip install -r requirements.txt

conda (better for scientific packages; manages non-Python dependencies like CUDA):

bashCopy
conda create -n myproject python=3.11
conda activate myproject
conda install pytorch torchvision -c pytorch

pyenv (when you need multiple Python versions):

bashCopy
pyenv install 3.11.7
pyenv local 3.11.7    # Sets the Python version for this directory

For most ML projects: venv with pinned requirements. For projects requiring specific CUDA, BLAS, or HDF5 versions: conda.

Pinning Dependencies

bashCopy
pip freeze > requirements.txt   # Records exact installed versions

numpy==1.26.4
pandas==2.2.1
scikit-learn==1.4.0
xgboost==2.0.3
torch==2.2.1+cu121

Anyone who runs pip install -r requirements.txt gets an identical environment. This is the minimum reproducibility bar.

For a lockfile that includes all transitive dependencies:

bashCopy
# uv (fast, modern alternative to pip-compile)
uv pip compile requirements.in -o requirements.txt

Makefiles for Project Automation

A Makefile records the commands needed to work with a project, making the workflow self-documenting:

makefileCopy
.PHONY: setup data train evaluate test clean

setup:
	python -m venv .venv
	.venv/bin/pip install --upgrade pip
	.venv/bin/pip install -r requirements.txt
	.venv/bin/pre-commit install

data:
	python scripts/download_data.py --version v2024-03
	python scripts/validate_data.py

train:
	python src/models/train.py --config configs/train_v3.yaml

evaluate:
	python src/models/evaluate.py --run-id $(RUN_ID)

test:
	pytest tests/ -v

clean:
	find . -name "*.pyc" -delete
	find . -name "__pycache__" -type d -exec rm -rf {} +
	rm -rf .venv/

make setup sets up the environment. make train runs the training job. A new teammate can be productive on day one.

SSH and Remote Machines

ML training often runs on remote servers. SSH is how you connect:

bashCopy
# Basic connection
ssh username@server-ip

# With a private key
ssh -i ~/.ssh/my_key.pem [email protected]

# Port forwarding (to access Jupyter or MLflow running on the remote machine)
ssh -L 8888:localhost:8888 username@server-ip
# Now open localhost:8888 in your local browser

# Copy files
scp local_script.py username@server:/home/username/project/
rsync -avz --progress local_data/ username@server:/data/raw/

Add frequently-used servers to ~/.ssh/config:

Host training-server
    HostName ec2-xx.compute.amazonaws.com
    User ec2-user
    IdentityFile ~/.ssh/my_key.pem
    ServerAliveInterval 60

Now ssh training-server connects with one command.

Pre-commit Hooks

Pre-commit hooks run checks before each Git commit, catching issues before they land in the repository:

yamlCopy
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.4.1
    hooks:
      - id: ruff           # Linting
      - id: ruff-format    # Formatting

  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.9.0
    hooks:
      - id: mypy
        args: [--ignore-missing-imports]

  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: check-added-large-files   # Blocks accidental data commits
        args: [--maxkb=500]
      - id: detect-private-key        # Blocks accidental secret commits

bashCopy
pre-commit install           # Register hooks once after cloning
pre-commit run --all-files   # Run manually on all files

After pre-commit install, every git commit automatically runs these checks. If any fail, the commit is blocked until you fix the issues. This keeps the codebase clean without relying on code-review discipline.

Common Mistakes and Bad Instincts

Committing data files. Data belongs in object storage, versioned with DVC or referenced by path. Git is for code. A 200 MB parquet file in the repo makes every git clone painful and the history almost useless.

Global Python for project work. The first time a package conflict causes two hours of debugging, this habit changes. Set up a virtual environment before writing the first line of project code.

Loose .gitignore. *.pkl, *.parquet, models/, and .env must be in .gitignore from day one. Accidentally committing a model or API key is hard to undo.

No Makefile or equivalent. If the steps to set up, run, and test the project are not in a file, they exist only in one person's head. A Makefile (or justfile, or Taskfile) makes the project self-documenting.

Where to Go Next

The deliverable for this module: a clean project repository with Git initialized, .gitignore configured for ML artifacts and secrets, a virtual environment set up, a Makefile with setup, train, and clean targets, and pre-commit hooks installed.

Module 3 begins the mathematical foundation: linear algebra as the language ML models use internally.