mirror of
https://gitea.com/gitea/act_runner.git
synced 2026-08-07 09:24:22 +02:00
573e811492f83ddcd397319564f732af81140d03
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Lunny Xiao
|
573e811492 |
refactor: move act/model and act/exprparser to actionslib
Gitea needs the workflow model and the expression evaluator to parse workflows and to build the task payload this runner consumes, so it had to depend on gitea.com/gitea/runner for it. Both packages now live in gitea.dev/actionslib, the module both sides already share, and this repository consumes them from there. The GithubContext helpers that need a git checkout on disk (SetRef, SetSha and SetRepositoryAndOwner) are not moved: they are runner only and would drag a git client and the act context logger into the shared module. They become functions of the new act/ghcontext package, together with their tests. act/common.CartesianProduct moved to the shared model package as well, act/model was its only user. act/model/testdata/container-volumes is now act/runner/testdata/container-volumes, its only user is runner_test.go. The x-runner-uuid and x-runner-token header names come from actionslib too, so the runner and Gitea cannot drift apart. The generated runner API code moved along with the repository, from gitea.dev/actions-proto-go to gitea.dev/actionslib, so the imports of ping/v1 and runner/v1 follow the new module path. Both cannot be used at once: the generated types would no longer be the same types. Signed-off-by: Lunny Xiao <xiaolunwen@gmail.com> |
||
bircni
|
c43cbe87ca |
feat: add runner health admission checks (#1090)
Opt-in local task-admission checks under a `health_check` config section (disabled by default): - pause new task fetching when free disk space on the workspace volume is below the configured minimum - optional executable health-check script — a non-zero exit, timeout, or start failure marks the runner unavailable - checks run only while the runner is idle; the last result is reused while a job is active, and polling resumes automatically on recovery - `/readyz` reports task-admission readiness (reusing the poll loop's last check); `/healthz` stays a process-liveness endpoint --------- Co-authored-by: silverwind <me@silverwind.io> Reviewed-on: https://gitea.com/gitea/runner/pulls/1090 Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com> |
||
|
bircni
|
3396021e0f |
test: Enhance Coverage + CI (#1055)
Reviewed-on: https://gitea.com/gitea/runner/pulls/1055 Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: bircni <bircni@icloud.com> |
||
Renovate Bot
|
984b47c716 |
fix(deps): update module code.gitea.io/actions-proto-go to gitea.dev/actions-proto-go v0.5.0 (#1009)
This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | code.gitea.io/actions-proto-go | `v0.4.1` → `v0.5.0` |  |  | --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xOTEuMiIsInVwZGF0ZWRJblZlciI6IjQzLjE5MS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119--> --------- Co-authored-by: Nicolas <bircni@icloud.com> Reviewed-on: https://gitea.com/gitea/runner/pulls/1009 Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Renovate Bot <renovate-bot@gitea.com> Co-committed-by: Renovate Bot <renovate-bot@gitea.com> |
||
|
Lunny Xiao
|
13dc9386fe |
Rename act_runner to runner (#850)
## Consumer-facing breaking changes
- **Go module path**: `gitea.com/gitea/act_runner` → `gitea.com/gitea/runner`. Anything importing `act/...` or `internal/...` packages (notably Gitea itself) must update imports.
- **Binary name**: `act_runner` → `gitea-runner`. Wrapper scripts, systemd units, init scripts, and documentation referencing the binary by `act_runner` will break.
- **Docker image**: `gitea/act_runner` → `gitea/runner` (incl. `*-dind-rootless` variants). Users pulling `gitea/act_runner:nightly` etc. will get stale images. Note: the image name is `gitea/runner`, not `gitea/gitea-runner`.
- **Release artifact paths**: S3 directory `act_runner/{{.Version}}` → `gitea-runner/{{.Version}}`, and artifact filenames change with the new project name. Existing download URLs break.
- **Metrics namespace**: changed from `act_runner` to `gitea_runner` (e.g. `act_runner_jobs_total` → `gitea_runner_jobs_total`); existing monitors/dashboards must be updated.
- **ldflags version path**: `gitea.com/gitea/act_runner/internal/pkg/ver.version` → `gitea.com/gitea/runner/internal/pkg/ver.version`. Affects anyone building with custom ldflags.
- **Kubernetes example resource names**: `act-runner` / `act-runner-vol` → `runner` / `runner-vol`. Users who copied the manifests verbatim will see resource churn on apply.
- **s6 service name**: `scripts/s6/act_runner/` → `scripts/s6/gitea-runner/` (image-internal; only matters for downstream image overrides).
Unchanged: YAML config field names, env vars (`GITEA_*`), CLI flags/subcommands, registration file format.
---------
Co-authored-by: silverwind <me@silverwind.io>
Reviewed-on: https://gitea.com/gitea/runner/pulls/850
Reviewed-by: Zettat123 <39446+zettat123@noreply.gitea.com>
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
Reviewed-by: Nicolas <bircni@icloud.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-committed-by: Lunny Xiao <xiaolunwen@gmail.com>
|
||
Bo-Yi Wu
|
9aafec169b |
perf: use single poller with semaphore-based capacity control (#822)
## Background #819 replaced the shared `rate.Limiter` with per-worker exponential backoff counters to add jitter and adaptive polling. Before #819, the poller used: ```go limiter := rate.NewLimiter(rate.Every(p.cfg.Runner.FetchInterval), 1) ``` This limiter was **shared across all N polling goroutines with burst=1**, effectively serializing their `FetchTask` calls — so even with `capacity=60`, the runner issued roughly one `FetchTask` per `FetchInterval` total. #819 replaced this with independent per-worker `consecutiveEmpty` / `consecutiveErrors` counters. Each goroutine now backs off **independently**, which inadvertently removed the cross-worker serialization. With `capacity=N`, the runner now has N goroutines each polling on their own schedule — a regression from the pre-#819 baseline for any runner with `capacity > 1`. (Thanks to @ChristopherHX for catching this in review.) ## Problem With the post-#819 code: - `capacity=N` maintains **N persistent polling goroutines**, each calling `FetchTask` independently - At idle, N goroutines each wake up and send a `FetchTask` RPC per `FetchInterval` - At full load, N goroutines **continue polling** even though no slot is available to run a new task — every one of those RPCs is wasted - The `Shutdown()` timeout branch has a pre-existing bug: the "non-blocking check" is actually a blocking receive, so `shutdownJobs()` is never reached on timeout ## Real-World Impact: 3 Runners × capacity=60 Current production environment: 3 runners each with `capacity=60`. | Metric | Post-#819 (current) | This PR | Reduction | |--------|---------------------|---------|-----------| | Polling goroutines (total) | 3 × 60 = **180** | 3 × 1 = **3** | **98.3%** (177 fewer) | | FetchTask RPCs per poll cycle (idle) | **180** | **3** | **98.3%** | | FetchTask RPCs per poll cycle (full load) | **180** (all wasted) | **0** (blocked on semaphore) | **100%** | | Concurrent connections to Gitea | **180** | **3** | **98.3%** | | Backoff state objects | 180 (per-worker) | 3 (one per runner) | Simplified | ### Idle scenario All 180 goroutines wake up every `FetchInterval`, each sending a `FetchTask` RPC that returns empty. Server handles 180 RPCs per cycle for zero useful work. After this PR: **3 RPCs per cycle** — one per runner. > Note: pre-#819 idle behavior was already ~3 RPCs/cycle due to the shared `rate.Limiter`. This PR restores that property while also addressing the full-load case below. ### Full-load scenario (all 180 slots occupied) All 180 goroutines **continue polling** even though no slot is available. Every RPC is wasted. After this PR: all 3 pollers are **blocked on the semaphore** — **zero RPCs** until a task completes. > This is a scenario neither the pre-#819 shared limiter nor the post-#819 per-worker backoff handles — both still issue `FetchTask` RPCs when no slot is free. The semaphore is the only approach of the three that ties polling to available capacity. ## Why Not Just Revert to `rate.Limiter`? Reverting would restore the serialized behavior but is not the right long-term fix: - **`rate.Limiter` has no concept of available capacity.** At full load it still hands out tokens and issues `FetchTask` RPCs that can't be acted on. The semaphore blocks polling entirely in that case — zero wasted RPCs. - **It composes poorly with adaptive backoff from #819.** A shared limiter and per-worker backoff pull in different directions. - **N goroutines serializing on a shared limiter means N-1 of them exist only to wait in line.** A single poller expresses the same behavior more directly. The semaphore approach ties polling to capacity explicitly: `acquire slot → fetch → dispatch → release`. That invariant becomes structural rather than emergent from a rate limiter. ## Solution Replace N polling goroutines with a **single polling loop** that uses a buffered channel as a semaphore to control concurrent task execution: ```go // New: poller.go Poll() sem := make(chan struct{}, p.cfg.Runner.Capacity) for { select { case sem <- struct{}{}: // Acquire slot (blocks at capacity) case <-p.pollingCtx.Done(): return } task, ok := p.fetchTask(...) // Single FetchTask RPC if !ok { <-sem // Release slot on empty response // backoff... continue } go func(t *runnerv1.Task) { // Dispatch task defer func() { <-sem }() // Release slot when done p.runTaskWithRecover(p.jobsCtx, t) }(task) } ``` The exponential backoff and jitter from #819 are preserved — just driven by a single `workerState` instead of N per-worker states. ## Shutdown Bug Fix Fixed a pre-existing bug in `Shutdown()` where the timeout branch could never force-cancel running jobs: ```go // Before (BROKEN): blocking receive, shutdownJobs() never reached _, ok := <-p.done // blocks until p.done is closed if !ok { return nil } p.shutdownJobs() // dead code when jobs are still running // After (FIXED): proper non-blocking check select { case <-p.done: return nil default: } p.shutdownJobs() // now correctly reached on timeout ``` ## Code Changes | Area | Detail | |------|--------| | `Poller.runner` | `*run.Runner` → `TaskRunner` interface (enables mock-based testing) | | `Poll()` | N goroutines → single loop with buffered-channel semaphore | | `PollOnce()` | Inlined from removed `pollOnce()` | | `waitBackoff()` | New helper, eliminates duplicated backoff logic | | `resetBackoff()` | New method on `workerState`, also resets stale `lastBackoff` metric | | `Shutdown()` | Fixed blocking receive → proper non-blocking select | | Removed | `poll()`, `pollOnce()` private methods (-2 methods, -42 lines) | ## Test Coverage Added `TestPoller_ConcurrencyLimitedByCapacity` which verifies: - With `capacity=3`, at most 3 tasks execute concurrently (`maxConcurrent <= 3`) - Tasks actually overlap in execution (`maxConcurrent >= 2`) - `FetchTask` is never called concurrently — confirms single poller (`maxFetchConcur == 1`) - All 6 tasks complete successfully (`totalCompleted == 6`) - Mock runner respects context cancellation, enabling shutdown path verification ``` === RUN TestPoller_ConcurrencyLimitedByCapacity --- PASS: TestPoller_ConcurrencyLimitedByCapacity (0.10s) PASS ok gitea.com/gitea/act_runner/internal/app/poll 0.59s ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: https://gitea.com/gitea/act_runner/pulls/822 Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com> Co-authored-by: Bo-Yi Wu <appleboy.tw@gmail.com> Co-committed-by: Bo-Yi Wu <appleboy.tw@gmail.com> |
||
|
Bo-Yi Wu
|
40dcee0991 |
chore(deps): upgrade golangci-lint from v2.10.1 to v2.11.4 (#821)
## Summary - Bump golangci-lint from v2.10.1 to v2.11.4 - Remove unused `//nolint:revive` directive on metrics package declaration (detected by stricter nolintlint in new version) ## Changes between v2.10.1 and v2.11.4 - **v2.11.0** — Multiple linter dependency upgrades, Go 1.26 support - **v2.11.2** — Bug fix for `fmt` with path - **v2.11.3** — gosec update - **v2.11.4** — Dependency updates (sqlclosecheck, noctx, etc.) No breaking changes. Reviewed-on: https://gitea.com/gitea/act_runner/pulls/821 Co-authored-by: Bo-Yi Wu <appleboy.tw@gmail.com> Co-committed-by: Bo-Yi Wu <appleboy.tw@gmail.com> |
||
|
Bo-Yi Wu
|
f33e5a6245 |
feat: add Prometheus metrics endpoint for runner observability (#820)
## What
Add an optional Prometheus `/metrics` HTTP endpoint to `act_runner` so operators can observe runner health, polling behavior, job outcomes, and RPC latency without scraping logs.
New surface:
- `internal/pkg/metrics/metrics.go` — metric definitions, custom `Registry`, static Go/process collectors, label constants, `ResultToStatusLabel` helper.
- `internal/pkg/metrics/server.go` — hardened `http.Server` serving `/metrics` and `/healthz` with Slowloris-safe timeouts (`ReadHeaderTimeout` 5s, `ReadTimeout`/`WriteTimeout` 10s, `IdleTimeout` 60s) and a 5s graceful shutdown.
- `daemon.go` wires it up behind `cfg.Metrics.Enabled` (disabled by default).
- `poller.go` / `reporter.go` / `runner.go` instrument their existing hot paths with counters/histograms/gauges — no behavior change.
Metrics exported (namespace `act_runner_`):
| Subsystem | Metric | Type | Labels |
|---|---|---|---|
| — | `info` | Gauge | `version`, `name` |
| — | `capacity`, `uptime_seconds` | Gauge | — |
| `poll` | `fetch_total`, `client_errors_total` | Counter | `result` / `method` |
| `poll` | `fetch_duration_seconds`, `backoff_seconds` | Histogram / Gauge | — |
| `job` | `total` | Counter | `status` |
| `job` | `duration_seconds`, `running`, `capacity_utilization_ratio` | Histogram / GaugeFunc | — |
| `report` | `log_total`, `state_total` | Counter | `result` |
| `report` | `log_duration_seconds`, `state_duration_seconds` | Histogram | — |
| `report` | `log_buffer_rows` | Gauge | — |
| — | `go_*`, `process_*` | standard collectors | — |
All label values are predefined constants — **no high-cardinality labels** (no task IDs, repo URLs, branches, tokens, or secrets) so scraping is safe and bounded.
## Why
Teams self-hosting Gitea + `act_runner` at scale need to answer basic SRE questions that are currently invisible:
- How often are RPCs failing? Which RPC? (`act_runner_client_errors_total`)
- Are runners saturated? (`act_runner_job_capacity_utilization_ratio`, `act_runner_job_running`)
- How long do jobs take? (`act_runner_job_duration_seconds`)
- Is polling backing off? (`act_runner_poll_backoff_seconds`, `act_runner_poll_fetch_total{result=\"error\"}`)
- Are log/state reports slow? (`act_runner_report_{log,state}_duration_seconds`)
- Is the log buffer draining? (`act_runner_report_log_buffer_rows`)
Today operators have to grep logs. This PR makes all of the above first-class metrics so they can feed dashboards and alerts (`rate(act_runner_client_errors_total[5m]) > 0.1`, capacity saturation alerts, etc.).
The endpoint is **disabled by default** and binds to `127.0.0.1:9101` when enabled, so it's opt-in and safe for existing deployments.
## How
### Config
```yaml
metrics:
enabled: false # opt-in
addr: 127.0.0.1:9101 # change to 0.0.0.0:9101 only behind a reverse proxy
```
`config.example.yaml` documents both fields plus a security note about binding externally without auth.
### Wiring
1. `daemon.go` calls `metrics.Init()` (guarded by `sync.Once`), sets `act_runner_info`, `act_runner_capacity`, registers uptime + running-jobs GaugeFuncs, then starts the server goroutine with the daemon context — it shuts down cleanly on `ctx.Done()`.
2. `poller.fetchTask` observes RPC latency / result / error counters. `DeadlineExceeded` (long-poll idle) is treated as an empty result and **not** observed into the histogram so the 5s timeout doesn't swamp the buckets.
3. `poller.pollOnce` reports `poll_backoff_seconds` using the pre-jitter base interval (the true backoff level), and only when it changes — prevents noisy no-op gauge updates at the `FetchIntervalMax` plateau.
4. `reporter.ReportLog` / `ReportState` record duration histograms and success/error counters; `log_buffer_rows` is updated only when the value changes, guarded by the already-held `clientM`.
5. `runner.Run` observes `job_duration_seconds` and increments `job_total` by outcome via `metrics.ResultToStatusLabel`.
### Safety / security review
- All timeouts set; Slowloris-safe.
- Custom `prometheus.NewRegistry()` — no global registration side-effects.
- No sensitive data in labels (reviewed every instrumentation site).
- Single new dependency: `github.com/prometheus/client_golang v1.23.2`.
- Endpoint is unauthenticated by design and documented as such; default localhost bind mitigates exposure. Operators exposing externally should front it with a reverse proxy.
## Verification
### Unit tests
\`\`\`bash
go build ./...
go vet ./...
go test ./...
\`\`\`
### Manual smoke test
1. Enable metrics in `config.yaml`:
\`\`\`yaml
metrics:
enabled: true
addr: 127.0.0.1:9101
\`\`\`
2. Start the runner against a Gitea instance: \`./act_runner daemon\`.
3. Scrape the endpoint:
\`\`\`bash
curl -s http://127.0.0.1:9101/metrics | grep '^act_runner_'
curl -s http://127.0.0.1:9101/healthz # → ok
\`\`\`
4. Confirm the static series appear immediately: \`act_runner_info\`, \`act_runner_capacity\`, \`act_runner_uptime_seconds\`, \`act_runner_job_running\`, \`act_runner_job_capacity_utilization_ratio\`.
5. Trigger a workflow and confirm counters increment: \`act_runner_poll_fetch_total{result=\"task\"}\`, \`act_runner_job_total{status=\"success\"}\`, \`act_runner_report_log_total{result=\"success\"}\`.
6. Leave the runner idle and confirm \`act_runner_poll_backoff_seconds\` settles (and does **not** churn on every poll).
7. Ctrl-C and confirm a clean \"metrics server shutdown\" log line (no port-in-use error on restart within 5s).
### Prometheus integration
Add to \`prometheus.yml\`:
\`\`\`yaml
scrape_configs:
- job_name: act_runner
static_configs:
- targets: ['127.0.0.1:9101']
\`\`\`
Sample alert to try:
\`\`\`
sum(rate(act_runner_client_errors_total[5m])) by (method) > 0.1
\`\`\`
## Out of scope (follow-ups)
- TLS and auth on the metrics endpoint (mitigated today by localhost default; add when operators need external scraping).
- Per-task labels (intentionally avoided for cardinality safety).
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://gitea.com/gitea/act_runner/pulls/820
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Bo-Yi Wu <appleboy.tw@gmail.com>
Co-committed-by: Bo-Yi Wu <appleboy.tw@gmail.com>
|