mirror of
https://github.com/awizemann/scarf.git
synced 2026-05-08 02:14:37 +00:00
feat(ci): validate template submissions on PR + tailored checklist
Adds the CI gate that runs on every PR touching templates/, the catalog validator, or its tests. The Action: - runs tools/test_build_catalog.py (catches drift between validator + its own test suite on the same PR that introduces the drift) - runs tools/build-catalog.py --check (validates every shipped .scarftemplate against the same invariants ProjectTemplateService.verifyClaims enforces at install time) - posts a PR comment with the last 3 KB of the validator log on failure, so contributors see the specific mismatch without hunting through the Actions UI .github/PULL_REQUEST_TEMPLATE/template-submission.md is the author-facing checklist that mirrors templates/CONTRIBUTING.md. Opt-in via the ?template=template-submission.md compare URL (documented in the contribution guide). CONTRIBUTING.md now links both the PR template and the workflow file so authors know what to expect. Phase 4 closes the community loop — from this commit on, a stranger can fork the repo, follow templates/CONTRIBUTING.md, push a PR, and get deterministic green/red feedback before a maintainer ever looks at it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,42 @@
|
||||
<!--
|
||||
Use this template when submitting a new Scarf project template or updating
|
||||
an existing one. For regular code/docs PRs, delete this template and write
|
||||
your own summary.
|
||||
|
||||
Switch to this template by adding `?template=template-submission.md` to the
|
||||
compare URL, or let GitHub pick it up automatically when you touch files
|
||||
under templates/.
|
||||
-->
|
||||
|
||||
## What's in this PR
|
||||
|
||||
- [ ] New template: `templates/<your-handle>/<your-template-name>/`
|
||||
- [ ] Update to existing template: `templates/<author>/<name>/` (which one and why)
|
||||
|
||||
## One-line pitch
|
||||
|
||||
_What does this template do for its installers? Two sentences max._
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] I wrote this template, or have the author's explicit permission to submit it.
|
||||
- [ ] `AGENTS.md` is present and tells any cross-agent what the project does and how to run it.
|
||||
- [ ] `README.md` includes install, customize, and uninstall instructions.
|
||||
- [ ] The bundle's `template.json` `contents` claim matches what's actually in the zip.
|
||||
- [ ] Cron jobs (if any) ship paused and use self-contained prompts.
|
||||
- [ ] No secrets in any file (API keys, tokens, hostnames, IPs, credentials).
|
||||
- [ ] No writes to `config.yaml`, `auth.json`, or credential paths — v1 installer will refuse.
|
||||
- [ ] `python3 tools/build-catalog.py --check` passes locally.
|
||||
- [ ] I installed + uninstalled this template on my machine and verified the `AGENTS.md` contract works end-to-end.
|
||||
- [ ] I did **not** edit `templates/catalog.json` — the maintainer regenerates it post-merge.
|
||||
|
||||
## Testing notes
|
||||
|
||||
_What did you run, what did you see? Paste the log output of the cron job
|
||||
firing once, or the chat transcript of asking the agent to do the main
|
||||
thing. Reviewers don't have your machine — show, don't tell._
|
||||
|
||||
## Screenshots (optional)
|
||||
|
||||
_Drop screenshots of the installed dashboard, or the catalog detail page
|
||||
rendered locally (`./scripts/catalog.sh preview && open /tmp/scarf-catalog-preview/templates/<slug>/index.html`)._
|
||||
@@ -0,0 +1,73 @@
|
||||
# Validates `.scarftemplate` bundles on PRs that touch templates/.
|
||||
#
|
||||
# Mirrors the invariants `ProjectTemplateService.verifyClaims` enforces at
|
||||
# install time. Runs the same Python script the maintainer uses locally
|
||||
# (tools/build-catalog.py --check) so a bundle can't reach main unless the
|
||||
# validator is happy.
|
||||
#
|
||||
# Also runs tools/test_build_catalog.py so drift between the validator and
|
||||
# its own test suite is caught on the same PR.
|
||||
|
||||
name: Validate template submissions
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'templates/**'
|
||||
- 'tools/build-catalog.py'
|
||||
- 'tools/test_build_catalog.py'
|
||||
- '.github/workflows/validate-template-pr.yml'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
validate:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
# Full clone so we can diff against the PR base and scope
|
||||
# --only to just the changed templates if we want to later.
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
# Stdlib-only script; any modern Python works. 3.9 matches the
|
||||
# system Python on current macOS so we exercise the same code
|
||||
# path maintainers hit locally.
|
||||
python-version: '3.11'
|
||||
|
||||
- name: Run validator unit tests
|
||||
run: python3 tools/test_build_catalog.py -v
|
||||
|
||||
- name: Validate every template
|
||||
id: validate
|
||||
run: |
|
||||
set -o pipefail
|
||||
python3 tools/build-catalog.py --check 2>&1 | tee /tmp/validator.log
|
||||
|
||||
- name: Post failure comment
|
||||
if: failure() && steps.validate.outcome == 'failure'
|
||||
uses: actions/github-script@v7
|
||||
with:
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
let body = '## Template validation failed\n\n';
|
||||
try {
|
||||
const log = fs.readFileSync('/tmp/validator.log', 'utf8');
|
||||
body += '```\n' + log.slice(-3000) + '\n```\n';
|
||||
} catch (e) {
|
||||
body += 'See the failed job log for details.\n';
|
||||
}
|
||||
body += '\nFix the issues above and push again — the check reruns automatically.\n';
|
||||
body += '\nLocal reproduction: `python3 tools/build-catalog.py --check`\n';
|
||||
await github.rest.issues.createComment({
|
||||
issue_number: context.issue.number,
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
body,
|
||||
});
|
||||
@@ -121,7 +121,9 @@ gh pr create
|
||||
|
||||
**Do not modify `templates/catalog.json`** — the maintainer regenerates it after merge to keep PR diffs small.
|
||||
|
||||
GitHub Actions runs the validator on your PR. A green check means the bundle structure is sound; it doesn't mean the content is approved. Expect a maintainer pass for content quality (is the `AGENTS.md` clear, does the prompt do what you describe, is the scope reasonable).
|
||||
The scarf repo ships a tailored submission checklist at [.github/PULL_REQUEST_TEMPLATE/template-submission.md](../.github/PULL_REQUEST_TEMPLATE/template-submission.md). To apply it to your PR, append `?template=template-submission.md` to the compare URL when opening the PR in the browser, or copy the checkbox list into the body manually.
|
||||
|
||||
GitHub Actions runs the validator on your PR (see [.github/workflows/validate-template-pr.yml](../.github/workflows/validate-template-pr.yml)). A green check means the bundle structure is sound; it doesn't mean the content is approved. Expect a maintainer pass for content quality (is the `AGENTS.md` clear, does the prompt do what you describe, is the scope reasonable).
|
||||
|
||||
### 8. Iterate + ship
|
||||
|
||||
|
||||
Reference in New Issue
Block a user