Skip to main content

GitHub Actions and S3

Published

2026-11-06

GitHub Actions can automatically build and publish our vetiver models to our S3 whenever we push changes to a specific branch (we covered these in Lab 5 and GitHub Actions as CI/CD). I’ve stored the Quarto files for building, pushing, and pulling the model data in the _labs/lab07/ folder.

In this section I’ll cover setting up AWS credentials as repository secrets and creating a workflow to automate the model publishing process.

%%{init: {'theme': 'base', 'look': 'classic', 'themeVariables': {'fontFamily': 'monospace'}}}%%

graph TD
    Push(["<strong>Developer</strong><br/><em>Pushes to branch</em>"]) 
    GHA["<strong>GitHub Actions</strong>"]
    Push --Triggers--> GHA
    GHA --"Build Models"--> Build("<strong>Vetiver Models</strong><br/><em>(py_model.qmd, <br>r_model.qmd)</em>")
    Build --> Auth("<strong>Configure</strong><br><em>AWS Credentials</em>")
    Auth --"Upload to"--> S3[("<strong>S3</strong><br/><em>(using pins)</em>")]
    S3 --> Success(["Models<br/>Published"])

    style Push fill:#5B8C5A,stroke:#000000,stroke-width:1px,color:#ffffff
    style GHA fill:#E8A33D,stroke:#000000,stroke-width:1px,color:#ffffff
    style Build fill:#1B2A41,stroke:#000000,stroke-width:1px,color:#ffffff
    style Auth fill:#2A6F77,stroke:#000000,stroke-width:1px,color:#ffffff
    style S3 fill:#D2562B,stroke:#000000,stroke-width:1px,color:#ffffff
    style Success fill:#5B8C5A,stroke:#000000,stroke-width:1px,color:#ffffff

GitHub Actions Automated Deployment

Automating Model Deployment with GitHub Actions

We need to add AWS configuration to the GitHub repository (Settings > Secrets and variables > Actions). Sensitive credentials go in Secrets, while non-sensitive configuration goes in Variables:

Click to enlarge Repo Action secrets and variables

Click to enlarge Repo Action secrets and variables

Secrets:

  1. AWS_ACCESS_KEY_ID: AWS access key ID from the mjfrigaard-cli IAM user
  2. AWS_SECRET_ACCESS_KEY: AWS secret access key

Variables:

  1. AWS_REGION: AWS region (e.g., us-east-1)

These credentials belong to the mjfrigaard-cli IAM user with AmazonS3FullAccess permissions (same user for local AWS CLI configuration).

Setting up GitHub Secrets and Variables

The diagram below shows how secrets and variables reach the workflow job.

%%{init: {'theme': 'base', 'look': 'classic', 'themeVariables': {'fontFamily': 'monospace'}}}%%

graph TD
    Secrets(["GitHub Secrets<br/>(Sensitive)"])
    Vars(["GitHub Variables<br/>(Config)"])

    Secrets --> AccessKey("<strong>AWS_ACCESS_KEY_ID</strong>")
    Secrets --> SecretKey("<strong>AWS_SECRET_ACCESS_KEY</strong>")
    Vars --> Region("<strong>AWS_REGION</strong>")

    AccessKey --> Workflow("Workflow Job<br/>(aws-actions/<br>configure-aws-<br>credentials)")
    SecretKey --> Workflow
    Region --> Workflow

    Workflow --> S3("Temporary AWS<br/>Credentials")
    S3 --> Access[("Access to S3")]

    style Secrets fill:#D2562B,stroke:#000000,stroke-width:1px,color:#ffffff
    style Vars fill:#2A6F77,stroke:#000000,stroke-width:1px,color:#ffffff
    style AccessKey fill:#1B2A41,stroke:#000000,stroke-width:1px,font-size:14px,text-align:center,font-family:monospace,color:#ffffff
    style SecretKey fill:#1B2A41,stroke:#000000,stroke-width:1px,font-size:14px,text-align:center,font-family:monospace,color:#ffffff
    style Region fill:#5B8C5A,stroke:#000000,stroke-width:1px,font-size:14px,text-align:center,font-family:monospace,color:#ffffff
    style Workflow fill:#E8A33D,stroke:#000000,stroke-width:1px,color:#ffffff
    style S3 fill:#2A6F77,stroke:#000000,stroke-width:1px,color:#ffffff
    style Access fill:#5B8C5A,stroke:#000000,stroke-width:1px,color:#ffffff

GitHub Secrets and Variables Flow

GitHub Actions Workflow File

Next, we create .github/workflows/publish.yml1 to build and publish the models to our S3 bucket. The models are automatically versioned by vetiver (e.g., 20260710T065220Z-91fdd), so we’re creating a new version in S3 with each workflow run.

name: Publish Models to S3

on:
  push:
    branches: [lab07-s3]
    paths:
      - '_labs/lab07/py_model.qmd'
      - '_labs/lab07/r_model.qmd'
      - '.github/workflows/publish.yml'

jobs:
  publish-models:
    runs-on: ubuntu-latest

    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ vars.AWS_REGION || 'us-east-1' }}

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Set up R
        uses: r-lib/actions/setup-r@v2
        with:
          r-version: '4.4'
          use-public-rspm: true

      - name: Set up Quarto
        uses: quarto-dev/quarto-actions/setup@v2

      - name: Install Python dependencies
        working-directory: ./_labs/lab07
        run: |
          python -m pip install --upgrade pip
          pip install -r Python/requirements.txt pins vetiver python-dotenv s3fs jupyter "setuptools<81"

      - name: Install R dependencies
        working-directory: ./_labs/lab07
        run: |
          R -e "install.packages(c('vetiver', 'pins', 'plumber2', 'paws.storage', 'palmerpenguins', 'duckdb', 'DBI', 'dplyr', 'rmarkdown', 'knitr'))"

      - name: Build and publish Python model
        working-directory: ./_labs/lab07
        env:
          USE_S3: 'true'
          AWS_REGION: ${{ vars.AWS_REGION || 'us-east-1' }}
          PINS_ALLOW_PICKLE_READ: '1'
        run: |
          quarto render py_model.qmd

      - name: Build and publish R model
        working-directory: ./_labs/lab07
        env:
          USE_S3: 'true'
          AWS_REGION: ${{ vars.AWS_REGION || 'us-east-1' }}
        run: |
          quarto render r_model.qmd
1
Trigger: Runs when py_model.qmd, r_model.qmd, or this workflow file changes on the lab07-s3 branch
2
Checkout: Downloads the latest code from the repository
3
AWS Auth: Uses aws-actions/configure-aws-credentials to load the IAM user’s access keys from encrypted GitHub secrets
4
Setup: Installs Python 3.12, R 4.4, Quarto, and the required dependencies
5
Build & Publish: Renders py_model.qmd and r_model.qmd with the USE_S3=true environment variable, which causes the model code to write directly to S3

The workflow renders the same .qmd files we render locally, so there are no separate scripts to keep in sync.

The use-public-rspm: true option in the Set up R step points R at the Posit Public Package Manager, which serves prebuilt Linux binaries. Without it, install.packages() compiles every R package from source, and the first successful run of this workflow took about 25 minutes (most of it in Install R dependencies).

Publishing with GitHub Actions automates the model deployment while keeping the AWS credentials in encrypted repository secrets rather than in the code.

Developing on the lab07-s3 Branch

Because I want the main branch reserved for the Quarto book rendering, I’ll create and check out a lab07-s3 branch from main for model changes (pushing to this branch triggers the model publishing workflow):

git checkout -b lab07-s3

And push the new branch to GitHub:

git push -u origin lab07-s3

Once on the lab07-s3 branch, I’ll need to follow this process:

Make changes

Make changes to the model files (i.e., _labs/lab07/py_model.qmd or _labs/lab07/r_model.qmd)

Test changes (locally)

To test changes locally, I’ll need to set the environment variables for S3 (or use local for testing):

export USE_S3=false  # or true if testing S3 locally

Then render/run the files with the changes

quarto render py_model.qmd

Or

quarto render r_model.qmd

Commit changes

Commit the changes (be specific about the files):

git add _labs/lab07/py_model.qmd _labs/lab07/r_model.qmd
git commit -m "Update model configuration or training logic"

Push changes

Push to the lab07-s3 branch:

git push origin lab07-s3

GitHub Actions will automatically detects the push to lab07-s3, build the models using the latest code, and publish the versioned models to S3 with AWS credentials from secrets.

Verify

The GitHub CLI (gh) lets us follow the workflow from the Terminal. First, list the most recent runs of publish.yml. Each row shows the status, the commit message, the branch, the run ID, and how long the run took:

gh run list --workflow publish.yml --limit 2

Pass the run ID from that list to gh run watch to follow the run until it finishes. The --exit-status flag makes the command fail when the run fails, so we don’t have to read the output to know the result:

gh run watch <run-id> --exit-status

To see where the time goes, gh api requests the run’s jobs from the GitHub REST API ({owner} and {repo} are filled in from the current repository). The --jq filter prints each step’s name and how many seconds it took. The Install R dependencies step should take about a minute with use-public-rspm: true (compiling from source took most of the 25 minute first run):

gh api repos/{owner}/{repo}/actions/runs/<run-id>/jobs \
  --jq '.jobs[0].steps[] | "\(.name): \(((.completed_at|fromdate)-(.started_at|fromdate)))s"'

We can also confirm R downloaded its packages from Package Manager by searching the run’s log for its address (packagemanager.posit.co). The __linux__ part of each URL means Package Manager sends a prebuilt binary for the runner’s Linux version:

gh run view <run-id> --log | grep -m3 "packagemanager.posit.co"

Finally, we verify the models were published by listing the files in the bucket. Each run adds a new version folder (named with the date and time, e.g., 20261106T183259Z-6a79d) under Python/models/penguin_lab07_model/ and R/models/penguin_lab07_model/:

# List files in S3
aws s3 ls s3://penguin-vetiver-model-data/ --recursive --profile dev-deploy

Merge to main

Then switch back to the main branch:

git checkout main

This will merge the changes in lab07-s3 to main (if the changes also belong in the book).

git merge lab07-s3

Push changes to main.

git push origin main

Then publish the book:

quarto publish gh-pages

Recap

  1. The AWS access key ID and secret access key go in the repository’s Secrets, and AWS_REGION goes in Variables. The workflow loads them with aws-actions/configure-aws-credentials, so the credentials never appear in the code.

  2. publish.yml runs on pushes to the lab07-s3 branch that change py_model.qmd, r_model.qmd, or the workflow file itself. It installs Python and R, then renders both model documents with USE_S3=true, so each run writes a new vetiver version to S3.

  3. Model changes happen on lab07-s3 (make changes, test locally, commit, push, and verify with aws s3 ls), which keeps main reserved for rendering the book. Changes that also belong in the book get merged back into main.

Lab 8 moves to the command line and how SSH keys work.

  1. Note that file is stored in the root directory (.github/workflows/)–where all workflow files belong.↩︎