ci: cache GitHub runners from NetBox to github-runners.jq#328
Conversation
|
Warning Review limit reached
More reviews will be available in 25 minutes and 15 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughThis pull request extends a GitHub Actions workflow to generate an additional JSON artifact containing GitHub runner information from NetBox. The change adds a new workflow step that queries the NetBox API for "Userlevel runner" virtual machines, extracts the host name and runner count for each VM using a jq filter, saves the result to Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/generate-servers-jsons.yml:
- Around line 122-129: The curl command output is captured into the variable
response via command substitution which masks curl failures; modify the code
around the response="$(curl ...)" invocation to capture curl's exit status
and/or HTTP status explicitly and fail fast: run curl so you capture both output
and exit code (e.g., redirect curl output to a temp file or use "curl -sS ... -o
/tmp/resp && echo $?"), then if curl failed or returned a non-2xx HTTP status,
log the error and exit nonzero before piping to jq; ensure you still use
RUNNERS_JQ_FILTER on the validated response and write to runners_outfile only
after the curl success check.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 2c98ccc6-c7ce-4c83-813b-e2932369feee
📒 Files selected for processing (1)
.github/workflows/generate-servers-jsons.yml
Add a step to the server-JSON generator that queries VMs with role "Userlevel runner" and writes one entry per runner host to github-runners.jq, committed under data/servers/ alongside the mirror lists. The virtual-machines endpoint ignores the `device_role` filter, so we fetch all active VMs and select on .role.name == "Userlevel runner" in jq. Each entry carries host, runners count, vcpus (rounded), memory (MB), and the apt_proxy / oci_cache / redis_cache custom fields. Both the new runner fetch and the existing mirror loop are hardened with curl retry (--retry/--fail/--max-time, -sS) and an empty-result guard, so a slow or empty NetBox response retries or fails loudly instead of timing out (the old `timeout 10` exit 124) or committing a broken list.
5acc15b to
6b0cd98
Compare
iav
left a comment
There was a problem hiding this comment.
I'm not "in the know", so I can't understand much.
But at first glance to an outsider, everything looks normal, it should work.
I also asked Claude to evaluate this pr in the context of the experience gained in the process of debugging my ccache-network and sccache extensions - he also generally approved, without any comments worth mentioning.
|
✅ This PR has been reviewed and approved — all set for merge! |
|
Its just preparing JSON for the next step - using it in the build process so we don't burn Netbox API calls. |
What
Adds a step to the Cache servers JSONs workflow (
.github/workflows/generate-servers-jsons.yml) that caches self-hosted GitHub runners from NetBox intodata/servers/github-runners.jq, alongside the existing mirror lists.Each entry is one runner host:
{ "host": "ampere-1", "runners": 64, "vcpus": 128, "memory": 512000, "apt_proxy": "http://10.0.40.2:3142", "oci_cache": "10.0.40.2:5000", "redis_cache": "redis://10.0.40.2:6379" }Fields:
host— VM namerunners—runnerscustom field (count)vcpus— VM vCPUs, rounded to integermemory— VM memory in MB (as NetBox stores it)apt_proxy,oci_cache,redis_cache— custom fields, default""when unsetHow runners are selected
NetBox's virtual-machines endpoint ignores
device_role(it's a Device filter), so a role query returns all active VMs. We instead fetch all active VMs and filter in jq on.role.name == "Userlevel runner", which reliably excludes unrelated roles (e.g. "Virtual machine").Reliability hardening (both the new runner fetch and the existing mirror loop)
--retry 5 --retry-delay 2 --retry-all-errors --fail,--connect-timeout 10 --max-time 60,-sS.timeout 10failure (exit 124) when NetBox'slimit=0response took >10s.::error::annotation if NetBox returns no matching VMs, instead of committing an empty/broken list to thedatabranch.Validation
Verified end-to-end via workflow runs on this branch — output committed to
data/servers/github-runners.jqwith correct counts and resource/cache fields (e.g.cats→ 6 runners, 16 vcpus, 32768 MB).