feat(ci): validate template submissions on PR + tailored checklist

Adds the CI gate that runs on every PR touching templates/, the catalog
validator, or its tests. The Action:
- runs tools/test_build_catalog.py (catches drift between validator +
  its own test suite on the same PR that introduces the drift)
- runs tools/build-catalog.py --check (validates every shipped .scarftemplate
  against the same invariants ProjectTemplateService.verifyClaims enforces
  at install time)
- posts a PR comment with the last 3 KB of the validator log on failure,
  so contributors see the specific mismatch without hunting through the
  Actions UI

.github/PULL_REQUEST_TEMPLATE/template-submission.md is the author-facing
checklist that mirrors templates/CONTRIBUTING.md. Opt-in via the
?template=template-submission.md compare URL (documented in the
contribution guide). CONTRIBUTING.md now links both the PR template and
the workflow file so authors know what to expect.

Phase 4 closes the community loop — from this commit on, a stranger can
fork the repo, follow templates/CONTRIBUTING.md, push a PR, and get
deterministic green/red feedback before a maintainer ever looks at it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alan Wizemann
2026-04-23 00:12:29 +02:00
parent 6175bee27d
commit c9b8da9ec5
3 changed files with 118 additions and 1 deletions
@@ -0,0 +1,42 @@
<!--
Use this template when submitting a new Scarf project template or updating
an existing one. For regular code/docs PRs, delete this template and write
your own summary.
Switch to this template by adding `?template=template-submission.md` to the
compare URL, or let GitHub pick it up automatically when you touch files
under templates/.
-->
## What's in this PR
- [ ] New template: `templates/<your-handle>/<your-template-name>/`
- [ ] Update to existing template: `templates/<author>/<name>/` (which one and why)
## One-line pitch
_What does this template do for its installers? Two sentences max._
## Checklist
- [ ] I wrote this template, or have the author's explicit permission to submit it.
- [ ] `AGENTS.md` is present and tells any cross-agent what the project does and how to run it.
- [ ] `README.md` includes install, customize, and uninstall instructions.
- [ ] The bundle's `template.json` `contents` claim matches what's actually in the zip.
- [ ] Cron jobs (if any) ship paused and use self-contained prompts.
- [ ] No secrets in any file (API keys, tokens, hostnames, IPs, credentials).
- [ ] No writes to `config.yaml`, `auth.json`, or credential paths — v1 installer will refuse.
- [ ] `python3 tools/build-catalog.py --check` passes locally.
- [ ] I installed + uninstalled this template on my machine and verified the `AGENTS.md` contract works end-to-end.
- [ ] I did **not** edit `templates/catalog.json` — the maintainer regenerates it post-merge.
## Testing notes
_What did you run, what did you see? Paste the log output of the cron job
firing once, or the chat transcript of asking the agent to do the main
thing. Reviewers don't have your machine — show, don't tell._
## Screenshots (optional)
_Drop screenshots of the installed dashboard, or the catalog detail page
rendered locally (`./scripts/catalog.sh preview && open /tmp/scarf-catalog-preview/templates/<slug>/index.html`)._
@@ -0,0 +1,73 @@
# Validates `.scarftemplate` bundles on PRs that touch templates/.
#
# Mirrors the invariants `ProjectTemplateService.verifyClaims` enforces at
# install time. Runs the same Python script the maintainer uses locally
# (tools/build-catalog.py --check) so a bundle can't reach main unless the
# validator is happy.
#
# Also runs tools/test_build_catalog.py so drift between the validator and
# its own test suite is caught on the same PR.
name: Validate template submissions
on:
pull_request:
paths:
- 'templates/**'
- 'tools/build-catalog.py'
- 'tools/test_build_catalog.py'
- '.github/workflows/validate-template-pr.yml'
permissions:
contents: read
pull-requests: write
jobs:
validate:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
# Full clone so we can diff against the PR base and scope
# --only to just the changed templates if we want to later.
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v5
with:
# Stdlib-only script; any modern Python works. 3.9 matches the
# system Python on current macOS so we exercise the same code
# path maintainers hit locally.
python-version: '3.11'
- name: Run validator unit tests
run: python3 tools/test_build_catalog.py -v
- name: Validate every template
id: validate
run: |
set -o pipefail
python3 tools/build-catalog.py --check 2>&1 | tee /tmp/validator.log
- name: Post failure comment
if: failure() && steps.validate.outcome == 'failure'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
let body = '## Template validation failed\n\n';
try {
const log = fs.readFileSync('/tmp/validator.log', 'utf8');
body += '```\n' + log.slice(-3000) + '\n```\n';
} catch (e) {
body += 'See the failed job log for details.\n';
}
body += '\nFix the issues above and push again — the check reruns automatically.\n';
body += '\nLocal reproduction: `python3 tools/build-catalog.py --check`\n';
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body,
});
+3 -1
View File
@@ -121,7 +121,9 @@ gh pr create
**Do not modify `templates/catalog.json`** — the maintainer regenerates it after merge to keep PR diffs small.
GitHub Actions runs the validator on your PR. A green check means the bundle structure is sound; it doesn't mean the content is approved. Expect a maintainer pass for content quality (is the `AGENTS.md` clear, does the prompt do what you describe, is the scope reasonable).
The scarf repo ships a tailored submission checklist at [.github/PULL_REQUEST_TEMPLATE/template-submission.md](../.github/PULL_REQUEST_TEMPLATE/template-submission.md). To apply it to your PR, append `?template=template-submission.md` to the compare URL when opening the PR in the browser, or copy the checkbox list into the body manually.
GitHub Actions runs the validator on your PR (see [.github/workflows/validate-template-pr.yml](../.github/workflows/validate-template-pr.yml)). A green check means the bundle structure is sound; it doesn't mean the content is approved. Expect a maintainer pass for content quality (is the `AGENTS.md` clear, does the prompt do what you describe, is the scope reasonable).
### 8. Iterate + ship