mirror of
https://github.com/awizemann/scarf.git
synced 2026-05-10 10:36:35 +00:00
feat(ci): validate template submissions on PR + tailored checklist
Adds the CI gate that runs on every PR touching templates/, the catalog validator, or its tests. The Action: - runs tools/test_build_catalog.py (catches drift between validator + its own test suite on the same PR that introduces the drift) - runs tools/build-catalog.py --check (validates every shipped .scarftemplate against the same invariants ProjectTemplateService.verifyClaims enforces at install time) - posts a PR comment with the last 3 KB of the validator log on failure, so contributors see the specific mismatch without hunting through the Actions UI .github/PULL_REQUEST_TEMPLATE/template-submission.md is the author-facing checklist that mirrors templates/CONTRIBUTING.md. Opt-in via the ?template=template-submission.md compare URL (documented in the contribution guide). CONTRIBUTING.md now links both the PR template and the workflow file so authors know what to expect. Phase 4 closes the community loop — from this commit on, a stranger can fork the repo, follow templates/CONTRIBUTING.md, push a PR, and get deterministic green/red feedback before a maintainer ever looks at it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,42 @@
|
|||||||
|
<!--
|
||||||
|
Use this template when submitting a new Scarf project template or updating
|
||||||
|
an existing one. For regular code/docs PRs, delete this template and write
|
||||||
|
your own summary.
|
||||||
|
|
||||||
|
Switch to this template by adding `?template=template-submission.md` to the
|
||||||
|
compare URL, or let GitHub pick it up automatically when you touch files
|
||||||
|
under templates/.
|
||||||
|
-->
|
||||||
|
|
||||||
|
## What's in this PR
|
||||||
|
|
||||||
|
- [ ] New template: `templates/<your-handle>/<your-template-name>/`
|
||||||
|
- [ ] Update to existing template: `templates/<author>/<name>/` (which one and why)
|
||||||
|
|
||||||
|
## One-line pitch
|
||||||
|
|
||||||
|
_What does this template do for its installers? Two sentences max._
|
||||||
|
|
||||||
|
## Checklist
|
||||||
|
|
||||||
|
- [ ] I wrote this template, or have the author's explicit permission to submit it.
|
||||||
|
- [ ] `AGENTS.md` is present and tells any cross-agent what the project does and how to run it.
|
||||||
|
- [ ] `README.md` includes install, customize, and uninstall instructions.
|
||||||
|
- [ ] The bundle's `template.json` `contents` claim matches what's actually in the zip.
|
||||||
|
- [ ] Cron jobs (if any) ship paused and use self-contained prompts.
|
||||||
|
- [ ] No secrets in any file (API keys, tokens, hostnames, IPs, credentials).
|
||||||
|
- [ ] No writes to `config.yaml`, `auth.json`, or credential paths — v1 installer will refuse.
|
||||||
|
- [ ] `python3 tools/build-catalog.py --check` passes locally.
|
||||||
|
- [ ] I installed + uninstalled this template on my machine and verified the `AGENTS.md` contract works end-to-end.
|
||||||
|
- [ ] I did **not** edit `templates/catalog.json` — the maintainer regenerates it post-merge.
|
||||||
|
|
||||||
|
## Testing notes
|
||||||
|
|
||||||
|
_What did you run, what did you see? Paste the log output of the cron job
|
||||||
|
firing once, or the chat transcript of asking the agent to do the main
|
||||||
|
thing. Reviewers don't have your machine — show, don't tell._
|
||||||
|
|
||||||
|
## Screenshots (optional)
|
||||||
|
|
||||||
|
_Drop screenshots of the installed dashboard, or the catalog detail page
|
||||||
|
rendered locally (`./scripts/catalog.sh preview && open /tmp/scarf-catalog-preview/templates/<slug>/index.html`)._
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
# Validates `.scarftemplate` bundles on PRs that touch templates/.
|
||||||
|
#
|
||||||
|
# Mirrors the invariants `ProjectTemplateService.verifyClaims` enforces at
|
||||||
|
# install time. Runs the same Python script the maintainer uses locally
|
||||||
|
# (tools/build-catalog.py --check) so a bundle can't reach main unless the
|
||||||
|
# validator is happy.
|
||||||
|
#
|
||||||
|
# Also runs tools/test_build_catalog.py so drift between the validator and
|
||||||
|
# its own test suite is caught on the same PR.
|
||||||
|
|
||||||
|
name: Validate template submissions
|
||||||
|
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
paths:
|
||||||
|
- 'templates/**'
|
||||||
|
- 'tools/build-catalog.py'
|
||||||
|
- 'tools/test_build_catalog.py'
|
||||||
|
- '.github/workflows/validate-template-pr.yml'
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
pull-requests: write
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
validate:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
# Full clone so we can diff against the PR base and scope
|
||||||
|
# --only to just the changed templates if we want to later.
|
||||||
|
fetch-depth: 0
|
||||||
|
|
||||||
|
- name: Set up Python
|
||||||
|
uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
# Stdlib-only script; any modern Python works. 3.9 matches the
|
||||||
|
# system Python on current macOS so we exercise the same code
|
||||||
|
# path maintainers hit locally.
|
||||||
|
python-version: '3.11'
|
||||||
|
|
||||||
|
- name: Run validator unit tests
|
||||||
|
run: python3 tools/test_build_catalog.py -v
|
||||||
|
|
||||||
|
- name: Validate every template
|
||||||
|
id: validate
|
||||||
|
run: |
|
||||||
|
set -o pipefail
|
||||||
|
python3 tools/build-catalog.py --check 2>&1 | tee /tmp/validator.log
|
||||||
|
|
||||||
|
- name: Post failure comment
|
||||||
|
if: failure() && steps.validate.outcome == 'failure'
|
||||||
|
uses: actions/github-script@v7
|
||||||
|
with:
|
||||||
|
script: |
|
||||||
|
const fs = require('fs');
|
||||||
|
let body = '## Template validation failed\n\n';
|
||||||
|
try {
|
||||||
|
const log = fs.readFileSync('/tmp/validator.log', 'utf8');
|
||||||
|
body += '```\n' + log.slice(-3000) + '\n```\n';
|
||||||
|
} catch (e) {
|
||||||
|
body += 'See the failed job log for details.\n';
|
||||||
|
}
|
||||||
|
body += '\nFix the issues above and push again — the check reruns automatically.\n';
|
||||||
|
body += '\nLocal reproduction: `python3 tools/build-catalog.py --check`\n';
|
||||||
|
await github.rest.issues.createComment({
|
||||||
|
issue_number: context.issue.number,
|
||||||
|
owner: context.repo.owner,
|
||||||
|
repo: context.repo.repo,
|
||||||
|
body,
|
||||||
|
});
|
||||||
@@ -121,7 +121,9 @@ gh pr create
|
|||||||
|
|
||||||
**Do not modify `templates/catalog.json`** — the maintainer regenerates it after merge to keep PR diffs small.
|
**Do not modify `templates/catalog.json`** — the maintainer regenerates it after merge to keep PR diffs small.
|
||||||
|
|
||||||
GitHub Actions runs the validator on your PR. A green check means the bundle structure is sound; it doesn't mean the content is approved. Expect a maintainer pass for content quality (is the `AGENTS.md` clear, does the prompt do what you describe, is the scope reasonable).
|
The scarf repo ships a tailored submission checklist at [.github/PULL_REQUEST_TEMPLATE/template-submission.md](../.github/PULL_REQUEST_TEMPLATE/template-submission.md). To apply it to your PR, append `?template=template-submission.md` to the compare URL when opening the PR in the browser, or copy the checkbox list into the body manually.
|
||||||
|
|
||||||
|
GitHub Actions runs the validator on your PR (see [.github/workflows/validate-template-pr.yml](../.github/workflows/validate-template-pr.yml)). A green check means the bundle structure is sound; it doesn't mean the content is approved. Expect a maintainer pass for content quality (is the `AGENTS.md` clear, does the prompt do what you describe, is the scope reasonable).
|
||||||
|
|
||||||
### 8. Iterate + ship
|
### 8. Iterate + ship
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user