From c9b8da9ec55207b880f55b19513a8a422f507d99 Mon Sep 17 00:00:00 2001 From: Alan Wizemann Date: Thu, 23 Apr 2026 00:12:29 +0200 Subject: [PATCH] feat(ci): validate template submissions on PR + tailored checklist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the CI gate that runs on every PR touching templates/, the catalog validator, or its tests. The Action: - runs tools/test_build_catalog.py (catches drift between validator + its own test suite on the same PR that introduces the drift) - runs tools/build-catalog.py --check (validates every shipped .scarftemplate against the same invariants ProjectTemplateService.verifyClaims enforces at install time) - posts a PR comment with the last 3 KB of the validator log on failure, so contributors see the specific mismatch without hunting through the Actions UI .github/PULL_REQUEST_TEMPLATE/template-submission.md is the author-facing checklist that mirrors templates/CONTRIBUTING.md. Opt-in via the ?template=template-submission.md compare URL (documented in the contribution guide). CONTRIBUTING.md now links both the PR template and the workflow file so authors know what to expect. Phase 4 closes the community loop — from this commit on, a stranger can fork the repo, follow templates/CONTRIBUTING.md, push a PR, and get deterministic green/red feedback before a maintainer ever looks at it. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../template-submission.md | 42 +++++++++++ .github/workflows/validate-template-pr.yml | 73 +++++++++++++++++++ templates/CONTRIBUTING.md | 4 +- 3 files changed, 118 insertions(+), 1 deletion(-) create mode 100644 .github/PULL_REQUEST_TEMPLATE/template-submission.md create mode 100644 .github/workflows/validate-template-pr.yml diff --git a/.github/PULL_REQUEST_TEMPLATE/template-submission.md b/.github/PULL_REQUEST_TEMPLATE/template-submission.md new file mode 100644 index 0000000..03ffa3e --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE/template-submission.md @@ -0,0 +1,42 @@ + + +## What's in this PR + +- [ ] New template: `templates///` +- [ ] Update to existing template: `templates///` (which one and why) + +## One-line pitch + +_What does this template do for its installers? Two sentences max._ + +## Checklist + +- [ ] I wrote this template, or have the author's explicit permission to submit it. +- [ ] `AGENTS.md` is present and tells any cross-agent what the project does and how to run it. +- [ ] `README.md` includes install, customize, and uninstall instructions. +- [ ] The bundle's `template.json` `contents` claim matches what's actually in the zip. +- [ ] Cron jobs (if any) ship paused and use self-contained prompts. +- [ ] No secrets in any file (API keys, tokens, hostnames, IPs, credentials). +- [ ] No writes to `config.yaml`, `auth.json`, or credential paths — v1 installer will refuse. +- [ ] `python3 tools/build-catalog.py --check` passes locally. +- [ ] I installed + uninstalled this template on my machine and verified the `AGENTS.md` contract works end-to-end. +- [ ] I did **not** edit `templates/catalog.json` — the maintainer regenerates it post-merge. + +## Testing notes + +_What did you run, what did you see? Paste the log output of the cron job +firing once, or the chat transcript of asking the agent to do the main +thing. Reviewers don't have your machine — show, don't tell._ + +## Screenshots (optional) + +_Drop screenshots of the installed dashboard, or the catalog detail page +rendered locally (`./scripts/catalog.sh preview && open /tmp/scarf-catalog-preview/templates//index.html`)._ diff --git a/.github/workflows/validate-template-pr.yml b/.github/workflows/validate-template-pr.yml new file mode 100644 index 0000000..33bf840 --- /dev/null +++ b/.github/workflows/validate-template-pr.yml @@ -0,0 +1,73 @@ +# Validates `.scarftemplate` bundles on PRs that touch templates/. +# +# Mirrors the invariants `ProjectTemplateService.verifyClaims` enforces at +# install time. Runs the same Python script the maintainer uses locally +# (tools/build-catalog.py --check) so a bundle can't reach main unless the +# validator is happy. +# +# Also runs tools/test_build_catalog.py so drift between the validator and +# its own test suite is caught on the same PR. + +name: Validate template submissions + +on: + pull_request: + paths: + - 'templates/**' + - 'tools/build-catalog.py' + - 'tools/test_build_catalog.py' + - '.github/workflows/validate-template-pr.yml' + +permissions: + contents: read + pull-requests: write + +jobs: + validate: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + # Full clone so we can diff against the PR base and scope + # --only to just the changed templates if we want to later. + fetch-depth: 0 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + # Stdlib-only script; any modern Python works. 3.9 matches the + # system Python on current macOS so we exercise the same code + # path maintainers hit locally. + python-version: '3.11' + + - name: Run validator unit tests + run: python3 tools/test_build_catalog.py -v + + - name: Validate every template + id: validate + run: | + set -o pipefail + python3 tools/build-catalog.py --check 2>&1 | tee /tmp/validator.log + + - name: Post failure comment + if: failure() && steps.validate.outcome == 'failure' + uses: actions/github-script@v7 + with: + script: | + const fs = require('fs'); + let body = '## Template validation failed\n\n'; + try { + const log = fs.readFileSync('/tmp/validator.log', 'utf8'); + body += '```\n' + log.slice(-3000) + '\n```\n'; + } catch (e) { + body += 'See the failed job log for details.\n'; + } + body += '\nFix the issues above and push again — the check reruns automatically.\n'; + body += '\nLocal reproduction: `python3 tools/build-catalog.py --check`\n'; + await github.rest.issues.createComment({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + body, + }); diff --git a/templates/CONTRIBUTING.md b/templates/CONTRIBUTING.md index 637652e..49011bf 100644 --- a/templates/CONTRIBUTING.md +++ b/templates/CONTRIBUTING.md @@ -121,7 +121,9 @@ gh pr create **Do not modify `templates/catalog.json`** — the maintainer regenerates it after merge to keep PR diffs small. -GitHub Actions runs the validator on your PR. A green check means the bundle structure is sound; it doesn't mean the content is approved. Expect a maintainer pass for content quality (is the `AGENTS.md` clear, does the prompt do what you describe, is the scope reasonable). +The scarf repo ships a tailored submission checklist at [.github/PULL_REQUEST_TEMPLATE/template-submission.md](../.github/PULL_REQUEST_TEMPLATE/template-submission.md). To apply it to your PR, append `?template=template-submission.md` to the compare URL when opening the PR in the browser, or copy the checkbox list into the body manually. + +GitHub Actions runs the validator on your PR (see [.github/workflows/validate-template-pr.yml](../.github/workflows/validate-template-pr.yml)). A green check means the bundle structure is sound; it doesn't mean the content is approved. Expect a maintainer pass for content quality (is the `AGENTS.md` clear, does the prompt do what you describe, is the scope reasonable). ### 8. Iterate + ship