`ACTIONS_RESULTS_URL` names one origin serving every `github.actions.results.api.v1` service. Gitea serves the artifact half and this runner the cache half, so announcing `ACTIONS_CACHE_SERVICE_V2` while that URL pointed at Gitea was a promise the environment could not keep, and `docker buildx` posted its cache calls at Gitea and got a 404.
The cache server now forwards the artifact half to the instance each job registers with, so it is the whole results service and jobs are pointed at it. The announcement follows, and the bundle patch follows the cache URL instead.
Also fixes three things no JavaScript client reached: camelCase in the v2 responses where the Go clients read proto names, the missing `x-ms-request-id` on blob uploads that panics buildkit, and `cache.external_server` passed through without the trailing slash the v1 client concatenates onto.
Tests run the real actions against the services they look for: `actions/cache` over both API versions, the artifact actions up and back down through the forwarding, and `setup-node`. The regression itself is covered by asserting that whatever a job is handed as `ACTIONS_RESULTS_URL` answers a cache service call.
Fixes https://gitea.com/gitea/runner/issues/1139
Reviewed-on: https://gitea.com/gitea/runner/pulls/1141
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <2021+silverwind@noreply.gitea.com>
Serves `github.actions.results.api.v1.CacheService` next to the v1 cache API, sharing its store, plus the subset of the Azure blob protocol the toolkit uploads with. On by default via `cache.v2`, and works with `external_server`.
Clients reach it through two edits in the action's own bundle: the GHES check is opened, and the cache service URL is taken from `ACTIONS_CACHE_URL`.
The same GHES check is what makes the stock `actions/upload-artifact` and `download-artifact` abort on Gitea. Opening it makes them work without the `gitea-upload-artifact` fork, from `upload-artifact@v4.4.0` on.
Verified against 118 real bundles, every major version of 16 actions: 92 patched, the rest deliberately left alone, and every patched bundle checked with `node --check`. Also end to end against pinned `actions/cache@v6.1.0` with an unreachable results URL, so only the patch can make the cache work.
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: bircni <me@silverwind.io>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1110
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
Leaked per-job networks each hold a subnet of the daemon's address pool and nothing reclaimed them, so a host eventually fails every job with `all predefined address pools have been fully subnetted`. This is what CI hit in https://gitea.com/gitea/runner/actions/runs/742177.
- teardown no longer loses a container, and its network with it: a failed `ContainerRemove` was reported as success, a container whose id was never learned was skipped, and the daemon's own `AutoRemove` teardown was raced
- the idle cleanup reclaims what teardown cannot: networks carry `com.gitea.runner.uuid`, so a runner only touches its own, and a cutoff keeps a job starting during the pass out of scope
- pull failures reported mid-stream were discarded, surfacing later as a confusing `No such image`; they now propagate, and fall back to a local copy instead of failing the job
- `NetworkCreate` retries pool exhaustion briefly, then says which knobs to turn
- digest-pinned images are not re-pulled, and removal kills first so it never waits out Podman's stop timeout (measured 10.1s → 0.09s per container)
---------
Co-authored-by: bircni <bircni@icloud.com>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1124
Reviewed-by: bircni <bircni@icloud.com>
1. Docker rejects two mounts on one target, so a `container.volumes:` or `--volume` aimed at `/opt/hostedtoolcache` failed the job with `Duplicate mount point`. Job and service volumes now displace the mount on the same path, and `name:/target:ro` no longer mounts read-write at the literal path `/target:ro`.
1. Setting `RUNNER_TOOL_CACHE` only changed what the variable said, the cache stayed where it was, so tools writing to it landed outside the mount and `${{ runner.tool_cache }}` disagreed with the variable. It now relocates the cache. Leaving it unset behaves as before.
1. Unknown `config.yaml` keys now warn instead of being dropped without a trace.
Fixes https://gitea.com/gitea/runner/issues/813
---------
Co-authored-by: bircni <bircni@icloud.com>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1122
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
Adds `runner.hooks.job_started` and `runner.hooks.job_completed`: operator scripts that run inside the job environment, before the job's first step and after its last one.
```yaml
runner:
hooks:
job_started: /hooks/started.sh
job_completed: /hooks/completed.sh
```
Equivalent to GitHub's `ACTIONS_RUNNER_HOOK_JOB_STARTED` / `ACTIONS_RUNNER_HOOK_JOB_COMPLETED`, which are read when unset: output is scanned for workflow commands, `$GITHUB_ENV` and `$GITHUB_PATH` are read back, and a non-zero exit fails the job.
Fixes: https://gitea.com/gitea/runner/issues/779
Co-authored-by: silverwind <me@silverwind.io>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1111
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
This PR adds a new `cache.external_secret_file` config, which points at a file holding the secret. So the secret can come from a mounted Kubernetes/Docker secret while the rest of the config stays plain text.
```yaml
cache:
external_server: "http://cache-host:8088/"
external_secret_file: /path/to/cache_external_secret
```
---------
Co-authored-by: bircni <bircni@icloud.com>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1100
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: Zettat123 <zettat123@gmail.com>
Opt-in local task-admission checks under a `health_check` config section (disabled by default):
- pause new task fetching when free disk space on the workspace volume is below the configured minimum
- optional executable health-check script — a non-zero exit, timeout, or start failure marks the runner unavailable
- checks run only while the runner is idle; the last result is reused while a job is active, and polling resumes automatically on recovery
- `/readyz` reports task-admission readiness (reusing the poll loop's last check); `/healthz` stays a process-liveness endpoint
---------
Co-authored-by: silverwind <me@silverwind.io>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1090
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
A batch of small, self-contained fixes and docs/example additions.
Fixes#625 - align the example config's `force_pull` with the actual default (`false`)
Fixes#804 - return an error instead of discarding `os.UserHomeDir()` when defaulting `cache.dir`/`host.workdir_parent`
Fixes#571 - send a `gitea-runner/<version>` User-Agent on API requests
Fixes#650 - detect an `Unauthenticated` fetch response and exit the daemon with an error instead of retrying forever
Fixes#766 - add `exec --eventpath` to supply a JSON event payload file
Fixes#256 - add a `bug-report` subcommand that prints version/Go/OS-arch/CPU info
Fixes#617 - add `runner.set_act_env` (default `true`) to optionally omit the `ACT=true` env var
Fixes#635 - record a failure result (and guard a nil reusable-workflow caller) when the job `if`-expression fails to evaluate
Fixes#1005 - remove README docs for config env-var overrides that were already removed from the code
Fixes#448 - clarify in `exec --job` help that `--workflows` may be needed to disambiguate
Fixes#209 - note that `host`-labelled runners still need Docker for `docker://` actions and service containers
Fixes#757 - add a systemd service example with automatic restart
Fixes#474 - add a Kubernetes StatefulSet example that persists the `.runner` registration across reschedules
Fixes#776 - build the basic (non-dind) docker image for `linux/riscv64`
Fixes#628 - build the basic (non-dind) docker image for `linux/s390x`Reviewed-on: https://gitea.com/gitea/runner/pulls/1075
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Refs https://gitea.com/gitea/runner/issues/155
The remaining failure mode described in the issue comments is the job container timing out when it tries to reach ACTIONS_CACHE_URL. The generated config now explains that cache.host/cache.port must be reachable from job containers, and calls out dockerized runners with auto-created per-job networks as a case that may need a fixed published cache endpoint or a shared Docker network
Co-authored-by: bircni <bircni@icloud.com>
## Summary
When a workflow references a remote action (e.g. `uses: actions/checkout@v4`) the runner clones that repository during job setup.
Previously this was always a full clone(every branch and the complete history) even though only a single ref is needed.
This PR makes the runner shallow-clone the requested ref by default (`--depth=1`, single branch), falling back to a full clone when a shallow clone fails.
Notes:
- Existing on-disk caches are reused as-is; there is no forced re-clone on upgrade.
## Changes
- A new `runner.action_shallow_clone` option (default `true`) lets operators opt back into full clones.
- `cloneAtDepth`: attempt a shallow clone; fall back to a full clone when shallow clone fails.
- Keep a shallow cache cheap on update: fetch the single requested ref at depth 1 and skip `pull`.
---------
Co-authored-by: bircni <bircni@icloud.com>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1053
Reviewed-by: Nicolas <bircni@icloud.com>
Co-authored-by: Zettat123 <zettat123@gmail.com>
Co-committed-by: Zettat123 <zettat123@gmail.com>
- Adds `runner.post_task_script` and `runner.post_task_script_timeout` (default `5m`) to run a host executable after each task’s built-in cleanup (post-steps, container teardown, bind-workdir removal).
- Stops task heartbeats via `Reporter.StopHeartbeats()` while the script runs so Gitea won’t assign overlapping work; the final task acknowledgement still happens in `reporter.Close()`.
- Script output goes to the runner process log; non-zero exits are warned only and do not change the job result.
- Documents lifecycle, offline behavior, timeouts, and Windows limits (`.ps1` not supported yet) in `docs/post-task-script.md`.
Reviewed-on: https://gitea.com/gitea/runner/pulls/1026
Reviewed-by: Zettat123 <39446+zettat123@noreply.gitea.com>
Fixes#1023.
## Problem
In Windows host mode, a single stalled delete syscall (AV/EDR filter driver, unresponsive mount, dying disk) wedged the job forever at `Cleaning up container`. `HostEnvironment.Remove()` bounds every teardown phase (`terminateRunningProcesses`, both `removePathWithRetry` calls) except the `CleanUp` callback — an unbounded `os.RemoveAll(miscpath)` assigned in `startHostEnvironment`. The runner then held its capacity slot indefinitely, the task was reaped as a zombie, and there were no diagnostics.
## Fix
- **Bound the cleanup (availability):** `Remove()` now runs `CleanUp` under `hostCleanupTimeout` (30s) via `runWithTimeout`; on timeout it logs a warning and continues job completion. The stuck goroutine is left to finish (a delete syscall can't be interrupted). Added debug logs around the phase.
- **Reclaim the leak (disk hygiene):** a timed-out cleanup can leave a scratch dir behind, so the existing idle stale-dir sweep is extended to also remove orphaned host-mode scratch dirs (16-hex names) under `Host.WorkdirParent`, leaving the shared `tool_cache` and operator data untouched. The `bind_workdir` gate is dropped from `shouldRunIdleCleanup` so host-mode runners run the sweep.
Reviewed-on: https://gitea.com/gitea/runner/pulls/1024
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
## Summary
- When Gitea cancels a job, the reporter cancels its own task context; the final Close() flush then aborted on that same cancelled context and Gitea never received the runner's acknowledgement (missing tail logs and final state).
- On Windows the cancelled context also neutralised terminateRunningProcesses, leaving step grandchildren alive in the workspace, holding file handles, so the runner could no longer clean up and pick up new work.
- Reporter.Close() now flushes on a detached, bounded context via a new rpcCtx() helper and configurable Runner.ReportCloseTimeout (default 10s).
- terminateRunningProcesses now PowerShell-enumerates Win32_Process and taskkill /T /F's every process whose ExecutablePath or CommandLine references the job's workspace directories, on a detached context.
- The daemon heartbeat loop still exits on <-r.ctx.Done(): the runner is intentionally seen as offline by Gitea during cleanup so it isn't handed a new task overlapping the in-progress teardown.
## Test plan
- [x] go test ./internal/pkg/report/... ./act/container/ -run 'TestReporter_ServerCancelStillFlushesFinal|TestBuildWindowsWorkspaceKillScript'
- [x] make fmt && make lint-go - 0 issues
- [x] GOOS=windows go build ./... - clean
- [x] Manual on a Windows runner: trigger a long-running workflow, cancel from Gitea UI; verify (a) the job ends with tail logs + cancelled state in Gitea, (b) workspace cleans up, (c) the runner picks up a new job without restart.
Authored-by: bircni
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: silverwind <me@silverwind.io>
Reviewed-on: https://gitea.com/gitea/runner/pulls/996
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
Fixes#956.
Pseudo-TTY allocation is now an explicit, runner-wide opt-in via `runner.allocate_pty`, applied to both host and docker backends. Default is off, matching GitHub `actions/runner`.
```yaml
runner:
allocate_pty: false # default
```
**Before:** the host backend hardcoded `if true /* allocate Terminal */` and the docker backend used `term.IsTerminal(os.Stdout.Fd())`. As a result, `docker build` (and other TTY-aware tools) saw a TTY and emitted cursor-control redraw frames that flooded captured logs with thousands of duplicate-looking progress lines — only on host-mode runners in production, and on docker-mode runners when the daemon happened to be launched from a shell rather than a service.
**After:** both backends consult `Config.AllocatePTY`. The `term.IsTerminal` heuristic is gone, so behavior no longer depends on whether the daemon has a controlling terminal.
**Reproduction:** running `docker build` through `HostEnvironment.Exec` with output captured to a buffer:
| | Before (`if true`) | After (`AllocatePTY=false`) |
|---|---:|---:|
| bytes captured | 18,167 | 1,048 |
| ANSI CSI sequences | 556 | 0 |
| cursor-up `\e[1A` | 181 | 0 |
**Side fix:** `ptyWriter.AutoStop` is now `atomic.Bool`. The field is written from the exec goroutine after `cmd.Wait()` and read from the `copyPtyOutput` goroutine via `ptyWriter.Write`; existing tests never tripped the race detector because their commands produced no output before exit. The new host-mode test does.
---
This PR was written with the help of Claude Opus 4.7
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Nicolas <bircni@icloud.com>
Reviewed-on: https://gitea.com/gitea/runner/pulls/961
Reviewed-by: Nicolas <bircni@icloud.com>
Co-authored-by: silverwind <2021+silverwind@noreply.gitea.com>
Co-committed-by: silverwind <2021+silverwind@noreply.gitea.com>
- Add idle-time cleanup for stale bind-workdir task directories instead of cleaning them on the task execution path.
- Make cleanup behavior configurable with `runner.startup_cleanup_age` as the stale-age threshold (default: `24h`) and `runner.idle_cleanup_interval` as the idle cleanup cadence (default: `10m`).
- Restrict cleanup scope to numeric task directory names only, to avoid touching operator-managed folders.
- Document the cleanup settings in `config.example.yaml` and `README.md`.
- Add tests for stale-directory cleanup, idle cleanup throttling, and config default/override parsing.
## Why
When a runner or host crashes, normal per-task cleanup may not run, leaving stale task directories under the bind-workdir root. Running this cleanup only while the runner is idle recovers that disk space without adding overhead to active job execution.
If you want, I can also tighten the wording around `startup_cleanup_age`, since the key name now reads a bit misleadingly relative to the actual behavior.
---------
Co-authored-by: silverwind <me@silverwind.io>
Reviewed-on: https://gitea.com/gitea/runner/pulls/870
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
## Summary
- Lower the default `fetch_interval_max` from 60s to 5s to reduce job pickup latency for common single-runner deployments
- PR #819 optimized defaults for 200-runner fleets, regressing single-runner pickup time from ~2s to ~65s
- Most deployments use few runners; large fleets can still tune this value higher in their config
## Impact
| `fetch_interval_max` | 1 runner pickup | 200 runners idle |
| -------------------- | --------------- | ---------------- |
| 60s (previous) | up to **65s** | 3.3 req/s |
| **5s (new default)** | up to **5s** | 40 req/s |
Closes https://gitea.com/gitea/act_runner/issues/869
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reviewed-on: https://gitea.com/gitea/act_runner/pulls/875
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
Co-authored-by: Bo-Yi Wu <appleboy.tw@gmail.com>
Co-committed-by: Bo-Yi Wu <appleboy.tw@gmail.com>
Closes#848. Addresses [GHSA-82g9-637c-2fx2](https://github.com/go-gitea/gitea/security/advisories/GHSA-82g9-637c-2fx2) and the follow-up points raised by @ChristopherHX and @haroutp in that thread.
The change is breaking only for `cache.external_server` which uses auth via a pre-shared secret.
## How auth works now
1. **Runner starts** → opens the embedded cache server on `:port`. Loads / creates a 32-byte HMAC signing key in `<cache-dir>/.secret`.
2. **Runner receives a task** → calls `handler.RegisterJob(ACTIONS_RUNTIME_TOKEN, repository)` before the job runs, defers a revoker that removes the credential on completion. Registrations are reference-counted so a stray re-register cannot revoke a live job.
3. **Job container runs `actions/cache`** → the toolkit sends `Authorization: Bearer $ACTIONS_RUNTIME_TOKEN` on every management call (`reserve`, `upload`, `commit`, `find`, `clean`). The cache server's middleware looks the token up in the registered-jobs map: miss → 401; hit → the job's repository is injected into the request context.
4. **Repository scoping** — every cache entry is stamped with `Repo` on reserve; `find`, `upload`, `commit` all verify the caller's repo matches. A job in repo A cannot see or poison a cache entry owned by repo B, even when both reach the server over the same docker bridge. GC dedup also groups by `(Repo, Key, Version)` so one repo can't age out another.
5. **Archive downloads** — `@actions/cache` does not attach Authorization when downloading `archiveLocation`, so the `find` response is a short-lived HMAC-signed URL: `…/artifacts/:id?exp=<unix>&sig=<hmac>`, 10-minute TTL, signature binds `cacheID:exp`. Tampered, expired, or foreign-secret URLs get 401.
6. **Defence-in-depth** — `ACTIONS_RUNTIME_TOKEN` is added to `task.Secrets` so the runner's log masker scrubs it from step output.
## `cache.external_server` (standalone `act_runner cache-server`)
Operators set `cache.external_secret` to the same value on the runner config and the `act_runner cache-server` config. The `cache-server` then runs with bearer auth on the cache API and exposes a control-plane at `POST /_internal/{register,revoke}` (gated by the shared secret). The runner pre-registers each task's `ACTIONS_RUNTIME_TOKEN` with the remote server before the job runs and revokes it on completion. Same per-job auth + repo scoping as the embedded handler, just over the network.
`cache-server` refuses to start without `cache.external_secret`; runner config load also fails when `cache.external_server` is set without `cache.external_secret`.
## User-facing changes
- **One-time cache miss after upgrade.** Pre-existing entries in `bolt.db` have no `Repo` stamp and won't match any job — they'll be evicted by the normal GC. First job per cache key rebuilds its cache.
- **`cache.external_server` deployments must add `cache.external_secret`.** Breaking change for anyone running a standalone `act_runner cache-server`: set the same `cache.external_secret` in both the runner config and the cache-server config. Without it neither side starts.
- **No config changes required for the default setup.** Runners using the embedded cache server (the common case) keep working without any yaml edits; the auth mechanism is invisible to workflows.
---
This PR was written with the help of Claude Opus 4.7
---------
Co-authored-by: Nicolas <bircni@icloud.com>
Co-authored-by: Christopher Homberger <christopher.homberger@web.de>
Reviewed-on: https://gitea.com/gitea/act_runner/pulls/849
Reviewed-by: ChristopherHX <38043+christopherhx@noreply.gitea.com>
## What
Add an optional Prometheus `/metrics` HTTP endpoint to `act_runner` so operators can observe runner health, polling behavior, job outcomes, and RPC latency without scraping logs.
New surface:
- `internal/pkg/metrics/metrics.go` — metric definitions, custom `Registry`, static Go/process collectors, label constants, `ResultToStatusLabel` helper.
- `internal/pkg/metrics/server.go` — hardened `http.Server` serving `/metrics` and `/healthz` with Slowloris-safe timeouts (`ReadHeaderTimeout` 5s, `ReadTimeout`/`WriteTimeout` 10s, `IdleTimeout` 60s) and a 5s graceful shutdown.
- `daemon.go` wires it up behind `cfg.Metrics.Enabled` (disabled by default).
- `poller.go` / `reporter.go` / `runner.go` instrument their existing hot paths with counters/histograms/gauges — no behavior change.
Metrics exported (namespace `act_runner_`):
| Subsystem | Metric | Type | Labels |
|---|---|---|---|
| — | `info` | Gauge | `version`, `name` |
| — | `capacity`, `uptime_seconds` | Gauge | — |
| `poll` | `fetch_total`, `client_errors_total` | Counter | `result` / `method` |
| `poll` | `fetch_duration_seconds`, `backoff_seconds` | Histogram / Gauge | — |
| `job` | `total` | Counter | `status` |
| `job` | `duration_seconds`, `running`, `capacity_utilization_ratio` | Histogram / GaugeFunc | — |
| `report` | `log_total`, `state_total` | Counter | `result` |
| `report` | `log_duration_seconds`, `state_duration_seconds` | Histogram | — |
| `report` | `log_buffer_rows` | Gauge | — |
| — | `go_*`, `process_*` | standard collectors | — |
All label values are predefined constants — **no high-cardinality labels** (no task IDs, repo URLs, branches, tokens, or secrets) so scraping is safe and bounded.
## Why
Teams self-hosting Gitea + `act_runner` at scale need to answer basic SRE questions that are currently invisible:
- How often are RPCs failing? Which RPC? (`act_runner_client_errors_total`)
- Are runners saturated? (`act_runner_job_capacity_utilization_ratio`, `act_runner_job_running`)
- How long do jobs take? (`act_runner_job_duration_seconds`)
- Is polling backing off? (`act_runner_poll_backoff_seconds`, `act_runner_poll_fetch_total{result=\"error\"}`)
- Are log/state reports slow? (`act_runner_report_{log,state}_duration_seconds`)
- Is the log buffer draining? (`act_runner_report_log_buffer_rows`)
Today operators have to grep logs. This PR makes all of the above first-class metrics so they can feed dashboards and alerts (`rate(act_runner_client_errors_total[5m]) > 0.1`, capacity saturation alerts, etc.).
The endpoint is **disabled by default** and binds to `127.0.0.1:9101` when enabled, so it's opt-in and safe for existing deployments.
## How
### Config
```yaml
metrics:
enabled: false # opt-in
addr: 127.0.0.1:9101 # change to 0.0.0.0:9101 only behind a reverse proxy
```
`config.example.yaml` documents both fields plus a security note about binding externally without auth.
### Wiring
1. `daemon.go` calls `metrics.Init()` (guarded by `sync.Once`), sets `act_runner_info`, `act_runner_capacity`, registers uptime + running-jobs GaugeFuncs, then starts the server goroutine with the daemon context — it shuts down cleanly on `ctx.Done()`.
2. `poller.fetchTask` observes RPC latency / result / error counters. `DeadlineExceeded` (long-poll idle) is treated as an empty result and **not** observed into the histogram so the 5s timeout doesn't swamp the buckets.
3. `poller.pollOnce` reports `poll_backoff_seconds` using the pre-jitter base interval (the true backoff level), and only when it changes — prevents noisy no-op gauge updates at the `FetchIntervalMax` plateau.
4. `reporter.ReportLog` / `ReportState` record duration histograms and success/error counters; `log_buffer_rows` is updated only when the value changes, guarded by the already-held `clientM`.
5. `runner.Run` observes `job_duration_seconds` and increments `job_total` by outcome via `metrics.ResultToStatusLabel`.
### Safety / security review
- All timeouts set; Slowloris-safe.
- Custom `prometheus.NewRegistry()` — no global registration side-effects.
- No sensitive data in labels (reviewed every instrumentation site).
- Single new dependency: `github.com/prometheus/client_golang v1.23.2`.
- Endpoint is unauthenticated by design and documented as such; default localhost bind mitigates exposure. Operators exposing externally should front it with a reverse proxy.
## Verification
### Unit tests
\`\`\`bash
go build ./...
go vet ./...
go test ./...
\`\`\`
### Manual smoke test
1. Enable metrics in `config.yaml`:
\`\`\`yaml
metrics:
enabled: true
addr: 127.0.0.1:9101
\`\`\`
2. Start the runner against a Gitea instance: \`./act_runner daemon\`.
3. Scrape the endpoint:
\`\`\`bash
curl -s http://127.0.0.1:9101/metrics | grep '^act_runner_'
curl -s http://127.0.0.1:9101/healthz # → ok
\`\`\`
4. Confirm the static series appear immediately: \`act_runner_info\`, \`act_runner_capacity\`, \`act_runner_uptime_seconds\`, \`act_runner_job_running\`, \`act_runner_job_capacity_utilization_ratio\`.
5. Trigger a workflow and confirm counters increment: \`act_runner_poll_fetch_total{result=\"task\"}\`, \`act_runner_job_total{status=\"success\"}\`, \`act_runner_report_log_total{result=\"success\"}\`.
6. Leave the runner idle and confirm \`act_runner_poll_backoff_seconds\` settles (and does **not** churn on every poll).
7. Ctrl-C and confirm a clean \"metrics server shutdown\" log line (no port-in-use error on restart within 5s).
### Prometheus integration
Add to \`prometheus.yml\`:
\`\`\`yaml
scrape_configs:
- job_name: act_runner
static_configs:
- targets: ['127.0.0.1:9101']
\`\`\`
Sample alert to try:
\`\`\`
sum(rate(act_runner_client_errors_total[5m])) by (method) > 0.1
\`\`\`
## Out of scope (follow-ups)
- TLS and auth on the metrics endpoint (mitigated today by localhost default; add when operators need external scraping).
- Per-task labels (intentionally avoided for cardinality safety).
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://gitea.com/gitea/act_runner/pulls/820
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Bo-Yi Wu <appleboy.tw@gmail.com>
Co-committed-by: Bo-Yi Wu <appleboy.tw@gmail.com>
## Summary
Many teams self-host Gitea + Act Runner at scale. The current runner design causes excessive HTTP requests to the Gitea server, leading to high server load. This PR addresses three root causes: aggressive fixed-interval polling, per-task status reporting every 1 second regardless of activity, and unoptimized HTTP client configuration.
## Problem
The original architecture has these issues:
**1. Fixed 1-second reporting interval (RunDaemon)**
- Every running task calls ReportLog + ReportState every 1 second (2 HTTP requests/sec/task)
- These requests are sent even when there are no new log rows or state changes
- With 200 runners × 3 tasks each = **1,200 req/sec just for status reporting**
**2. Fixed 2-second polling interval (no backoff)**
- Idle runners poll FetchTask every 2 seconds forever, even when no jobs are queued
- No exponential backoff or jitter — all runners can synchronize after network recovery (thundering herd)
- 200 idle runners = **100 req/sec doing nothing useful**
**3. HTTP client not tuned**
- Uses http.DefaultClient with MaxIdleConnsPerHost=2, causing frequent TCP/TLS reconnects
- Creates two separate http.Client instances (one for Ping, one for Runner service) instead of sharing
**Total: ~1,300 req/sec for 200 runners with 3 tasks each**
## Solution
### Adaptive Event-Driven Log Reporting
Replace the recursive `time.AfterFunc(1s)` pattern in RunDaemon with a goroutine-based select event loop using three trigger mechanisms:
| Trigger | Default | Purpose |
|---------|---------|---------|
| `log_report_max_latency` | 3s | Guarantee even a single log line is delivered within this time |
| `log_report_interval` | 5s | Periodic sweep — steady-state cadence |
| `log_report_batch_size` | 100 rows | Immediate flush during bursty output (e.g., npm install) |
**Key design**: `log_report_max_latency` (3s) must be less than `log_report_interval` (5s) so the max-latency timer fires before the periodic ticker for single-line scenarios.
State reporting is decoupled to its own `state_report_interval` (default 5s), with immediate flush on step transitions (start/stop) via a stateNotify channel for responsive frontend UX.
Additionally:
- Skip ReportLog when `len(rows) == 0` (no pending log rows)
- Skip ReportState when `stateChanged == false && len(outputs) == 0` (nothing changed)
- Move expensive `proto.Clone` after the early-return check to avoid deep copies on no-op paths
### Polling Backoff with Jitter
Replace fixed `rate.Limiter` with adaptive exponential backoff:
- Track `consecutiveEmpty` and `consecutiveErrors` counters
- Interval doubles with each empty/error response: `base × 2^(n-1)`, capped at `fetch_interval_max` (default 60s)
- Add ±20% random jitter to prevent thundering herd
- Fetch first, sleep after ��� preserves burst=1 behavior for immediate first fetch on startup and after task completion
### HTTP Client Tuning
- Configure custom `http.Transport` with `MaxIdleConnsPerHost=10` (was 2)
- Share a single `http.Client` between PingService and RunnerService
- Add `IdleConnTimeout=90s` for clean connection lifecycle
## Load Reduction
For 200 runners × 3 tasks (70% with active log output):
| Component | Before | After | Reduction |
|-----------|--------|-------|-----------|
| Polling (idle) | 100 req/s | ~3.4 req/s | 97% |
| Log reporting | 420 req/s | ~84 req/s | 80% |
| State reporting | 126 req/s | ~25 req/s | 80% |
| **Total** | **~1,300 req/s** | **~113 req/s** | **~91%** |
## Frontend UX Impact
| Scenario | Before | After | Notes |
|----------|--------|-------|-------|
| Continuous output (npm install) | ~1s | ~5s | Periodic ticker sweep |
| Single line then silence | ~1s | ≤3s | maxLatencyTimer guarantee |
| Bursty output (100+ lines) | ~1s | <1s | Batch size immediate flush |
| Step start/stop | ~1s | <1s | stateNotify immediate flush |
| Job completion | ~1s | ~1s | Close() retry unchanged |
## New Configuration Options
All have safe defaults — existing config files need no changes:
```yaml
runner:
fetch_interval_max: 60s # Max backoff interval when idle
log_report_interval: 5s # Periodic log flush interval
log_report_max_latency: 3s # Max time a log row waits (must be < log_report_interval)
log_report_batch_size: 100 # Immediate flush threshold
state_report_interval: 5s # State flush interval (step transitions are always immediate)
```
Config validation warns on invalid combinations:
- `fetch_interval_max < fetch_interval` → auto-corrected
- `log_report_max_latency >= log_report_interval` → warning (timer would be redundant)
## Test Plan
- [x] `go build ./...` passes
- [x] `go test ./...` passes (all existing + 3 new tests)
- [x] `golangci-lint run` — 0 issues
- [x] TestReporter_MaxLatencyTimer — verifies single log line flushed by maxLatencyTimer before logTicker
- [x] TestReporter_BatchSizeFlush — verifies batch size threshold triggers immediate flush
- [x] TestReporter_StateNotifyFlush — verifies step transition triggers immediate state flush
- [x] TestReporter_EphemeralRunnerDeletion — verifies Close/RunDaemon race safety
- [x] TestReporter_RunDaemonClose_Race — verifies concurrent Close safety
Reviewed-on: https://gitea.com/gitea/act_runner/pulls/819
Reviewed-by: Nicolas <173651+bircni@noreply.gitea.com>
Co-authored-by: Bo-Yi Wu <appleboy.tw@gmail.com>
Co-committed-by: Bo-Yi Wu <appleboy.tw@gmail.com>
## Summary
Adds a `container.bind_workdir` config option that exposes the nektos/act `BindWorkdir` setting. When enabled, workspaces are bind-mounted from the host filesystem instead of Docker volumes, which is required for DinD setups where jobs use `docker compose` with bind mounts (e.g. `.:/app`).
Each job gets an isolated workspace at `/workspace/<task_id>/<owner>/<repo>` to prevent concurrent jobs from the same repo interfering with each other. The task directory is cleaned up after job execution.
### Configuration
```yaml
container:
bind_workdir: true
```
When using this with DinD, also mount the workspace parent into the runner container and add it to `valid_volumes`:
```yaml
container:
valid_volumes:
- /workspace/**
```
*This PR was authored by Claude (AI assistant)*
Reviewed-on: https://gitea.com/gitea/act_runner/pulls/810
Reviewed-by: ChristopherHX <38043+christopherhx@noreply.gitea.com>
Co-authored-by: silverwind <me@silverwind.io>
Co-committed-by: silverwind <me@silverwind.io>
This PR
- adds the `cache-server` command so act_runner can run as a cache server. When running as a cache server, act_runner only processes the requests related to cache and does not run jobs.
- adds the `external_server` configuration for cache. If specified, act_runner will use this URL as the ACTIONS_CACHE_URL instead of starting a cache server itself.
Reviewed-on: https://gitea.com/gitea/act_runner/pulls/275
Co-authored-by: Zettat123 <zettat123@gmail.com>
Co-committed-by: Zettat123 <zettat123@gmail.com>
Follow #242, #244Fixes#258
Users could use `docker_host` configuration to specify which docker daemon will be used by act_runner.
- If `docker_host` is **empty**, act_runner will find an available docker host automatically.
- If `docker_host` is **"-"**, act_runner will find an available docker host automatically, but the docker host won't be mounted to the job containers and service containers.
- If `docker_host` is **not empty or "-"**, the specified docker host will be used. An error will be returned if it doesn't work.
Reviewed-on: https://gitea.com/gitea/act_runner/pulls/260
Co-authored-by: Zettat123 <zettat123@gmail.com>
Co-committed-by: Zettat123 <zettat123@gmail.com>
Follow #242.
Move `docker.host` to `container.docker_host`.
There are already some options for docker/container in `container`, so developers could get confused about where to add options.
It's breaking, but I think it's OK since `docker.host` was added just two days ago.
Reviewed-on: https://gitea.com/gitea/act_runner/pulls/244
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Close https://gitea.com/gitea/act_runner/issues/177
Related https://gitea.com/gitea/act/pulls/56
### ⚠️ Breaking
The `container.network_mode` is a deprecated configuration item. It may be removed after Gitea 1.20 released.
Previously, if the value of `container.network_mode` is `bridge`, it means that `act_runner` will create a new network for job.But `bridge` is easily confused with the bridge network created by Docker by default.
We recommand that using `container.network` to specify the network to which containers created by `act_runner` connect.
### 🆕 container.network
The configuration file of `act_runner` add a new item of `contianer.network`.
In `config.example.yaml`:
```yaml
container:
# Specifies the network to which the container will connect.
# Could be host, bridge or the name of a custom network.
# If it's empty, act_runner will create a network automatically.
network: ""
```
As the comment in the example above says, the purpose of the `container.network` is specifying the network to which containers created by `act_runner` will connect.
`container.network` accepts the following valid values:
- `host`: All of containers (including job containers and service contianers) created by `act_runner` will be connected to the network named `host` which is created automatically by Docker. Containers will share the host’s network stack and all interfaces from the host will be available to these containers.
- `bridge`: It is similar to `host`. All of containers created by `act_runner` will be connected to the network named `bridge` which is created automatically by Docker. All containers connected to the `bridge` (Perhaps there are containers that are not created by `act_runner`) are allowed to communicate with each other, while providing isolation from containers which are not connected to that `bridge` network.
- `<custom_network>`: Please make sure that the `<custom_network>` network already exists firstly (`act_runner` does not detect whether the specified network exists currently. If not exists yet, will return error in the stage of `docker create`). All of containers created by `act_runner` will be connected to `<custom_network>`. After the job is executed, containers are removed and automatically disconnected from the `<custom_network>`.
- empty: `act_runner` will create a new network for each job container and their service containers (if defined in workflow). So each job container and their service containers share a network environment, but are isolated from others container and the Docker host. Of course, these networks created by `act_runner` will be removed at last.
### Others
- If you do not have special needs, we highly recommend that setting `container.network` to empty string (and do not use `container.network_mode` any more). Because the containers created by `act_runner` will connect to the networks that are created by itself. This point will provide better isolation.
- If you set `contianer.network` to empty string or `<custom_network>`, we can be access to service containers by `<service-id>:<port>` in the steps of job. Because we added an alias to the service container when connecting to the network.
Co-authored-by: Jason Song <i@wolfogre.com>
Reviewed-on: https://gitea.com/gitea/act_runner/pulls/184
Reviewed-by: Jason Song <i@wolfogre.com>
Co-authored-by: sillyguodong <gedong_1994@163.com>
Co-committed-by: sillyguodong <gedong_1994@163.com>
Fixes#145
At present, the working directory of a work flow is a path like `/<owner>/<repo>`, so the directory may conflict with system directory like `/usr/bin`. We need to add a parent directory for the working directory.
In this PR, the parent directory is `/workspace` by default and users could configure it by the `workdir_parent` option.
This change doesn't affect the host mode because in host mode the working directory will always be in `$HOME/.cache/act/` directory.
Co-authored-by: Jason Song <i@wolfogre.com>
Reviewed-on: https://gitea.com/gitea/act_runner/pulls/154
Reviewed-by: Jason Song <i@wolfogre.com>
Co-authored-by: Zettat123 <zettat123@gmail.com>
Co-committed-by: Zettat123 <zettat123@gmail.com>
This adds a very simple Dockerfile and run script for running `act_runner` as a container.
It also allows setting `Privileged` and `ContainerOptions` flags via the new config file when spawning task containers. The combination makes it possible to use Docker-in-Docker (which requires `privileged` mode) as well as pass any other options child Docker containers may require.
For example, if Gitea is running in Docker on the same machine, for the `checkout` action to behave as expected from a task container launched by `act_runner`, it might be necessary to map the hostname via something like:
```
container:
network_mode: bridge
privileged: true
options: --add-host=my.gitea.hostname:host-gateway
```
> NOTE: Description updated to reflect latest code.
> NOTE: Description updated to reflect latest code (again).
Reviewed-on: https://gitea.com/gitea/act_runner/pulls/84
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Reviewed-by: Jason Song <i@wolfogre.com>
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
