Compare commits

..

3 Commits

Author SHA1 Message Date
bircni
68547886a5 feat: wait for healthy services and fill the job context (#1107)
Service containers were started and then left alone, so a job's first step could run while a database was still starting up. The runner now waits for every service whose image or `options` declare a healthcheck, as GitHub does. An unhealthy service fails the job with its container log, one that never becomes healthy fails it after `container.service_ready_timeout` (default `5m`, negative disables the wait), and one that exits without a healthcheck only gets its log and a warning.

The started containers also fill the `job` context, whose fields existed but were never populated: `job.container.{id,network}` and `job.services.<id>.{id,network,ports}`. `ports` is keyed by the plain container port, so `job.services.postgres.ports['5432']` resolves to the host port Docker picked.

---------

Co-authored-by: silverwind <me@silverwind.io>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1107
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
Co-authored-by: bircni <bircni@icloud.com>
2026-08-05 19:46:43 +00:00
bircni
8700adc933 feat: config command to edit config files (#1140)
Changing a setting after `generate-config` meant hand-editing YAML, which is awkward in provisioning scripts. `config` now edits an existing file in place:

```bash
./gitea-runner -c config.yaml config set runner.capacity 4
./gitea-runner -c config.yaml config set runner.envs.MY_VAR value
./gitea-runner -c config.yaml config add runner.labels 'ubuntu:docker://node:22'
./gitea-runner -c config.yaml config remove runner.labels 'ubuntu:docker://node:22'
./gitea-runner -c config.yaml config get runner.labels
```

Edits go through the YAML node tree, so comments, key order and the blank lines between top-level sections survive — a test asserts that appending a label to `config.example.yaml` changes nothing but the added line. Keys are resolved by reflecting over the `Config` struct's yaml tags, so an unknown key or a value of the wrong type is rejected before the file is touched. The write is atomic and keeps the file's symlink, owner, mode and line endings.

---------

Co-authored-by: silverwind <2021+silverwind@noreply.gitea.com>
Co-authored-by: silverwind <me@silverwind.io>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1140
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: bircni <bircni@icloud.com>
2026-08-05 16:48:25 +00:00
bircni
b70ff6893a feat: gate set-env/add-path and render annotation locations (#1109)
`::set-env::` and `::add-path::` let a step rewrite the environment of every later step from its own output, which the runner honoured silently. They are now refused, as GitHub has done since 2020, and `ACTIONS_ALLOW_UNSECURE_COMMANDS` opts back in per step or job. Support for that variable is new here too, and is the only opt-in, matching GitHub rather than adding a runner config key on top.

Annotations keep their source location: Gitea has no annotation store and its web UI strips command properties, so `::error file=main.go,line=12::msg` is rendered as `::error::main.go:12: msg`.

`DEVELOPMENT.md` writes down the log line encoding rules this relies on.

---------

Co-authored-by: silverwind <me@silverwind.io>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1109
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
Co-authored-by: bircni <bircni@icloud.com>
2026-08-05 16:43:17 +00:00
39 changed files with 2155 additions and 171 deletions

View File

@@ -8,6 +8,7 @@
- Never force-push, amend, or squash unless asked. Use new commits and normal push for pull request updates - Never force-push, amend, or squash unless asked. Use new commits and normal push for pull request updates
- Comments: write almost none, short and preferably same-line, explaining why for a future reader. Never narrate code, the change or the prompt. Preserve existing ones that still apply - Comments: write almost none, short and preferably same-line, explaining why for a future reader. Never narrate code, the change or the prompt. Preserve existing ones that still apply
- Add the current year into the copyright header of new `.go` files - Add the current year into the copyright header of new `.go` files
- Read `DEVELOPMENT.md` for internals and conventions
- Ensure no trailing whitespace in edited files - Ensure no trailing whitespace in edited files
- Run `make fmt` after `.go` edits, `make tidy` after `go.mod` edits, and `make checks` for the non-lint source checks - Run `make fmt` after `.go` edits, `make tidy` after `go.mod` edits, and `make checks` for the non-lint source checks
- Lint what changed with `make lint-go`, and `make lint-go-windows` for Windows and platform-split files - Lint what changed with `make lint-go`, and `make lint-go-windows` for Windows and platform-split files

32
DEVELOPMENT.md Normal file
View File

@@ -0,0 +1,32 @@
# Development
## Job log line format
Gitea stores one log row per line and its web UI decodes the payload, so getting the encoding
wrong never fails a test here, it only shows up in the browser.
**A row cannot contain a real newline.** `FormatLog` rewrites `\n` to a literal backslash-n and
truncates at 64 KiB on a byte boundary.
**The payload of a line starting with a recognised prefix is decoded**, with the escape set
depending on the prefix:
| prefix | decodes |
| --- | --- |
| `##[error]` `##[warning]` `##[notice]` `##[debug]` `##[group]` `##[endgroup]` `##[add-matcher]` | `%25` `%0D` `%0A` `%3B` `%5D` |
| `::error::` `::warning::` `::notice::` `::debug::` (with or without ` key=value` properties), `::group::` `::endgroup::` `::add-matcher::` | `%25` `%0D` `%0A` |
| `##[command]` `[command]`, or no recognised prefix | nothing |
### Rules
- **Emitting a command line?** Escape the payload with `runner.EscapeCommandData`. One escaper
covers both forms: it escapes `%` first, so a literal `%3B` becomes `%253B` that the extra
`##[…]` rules cannot match, and a raw `;` or `]` is never decoded. It is also what makes
multi-line work, `\n` becomes `%0A` and the UI turns it back into a line break.
- **Forwarding a command from step output?** Leave the payload alone, it arrived escaped and is
decoded once. Decoding here double-decodes and destroys multi-line.
- **No prefix?** Do not escape, and split multi-line values into one row each.
- **Interpolating a secret?** Masking runs after escaping, so `AppendSecretMasker` registers the
encoded forms too.
- Command *properties* also escape `%3A` and `%2C`, which the UI never decodes, so the reporter
decodes exactly those two when folding a location into an annotation.

View File

@@ -132,9 +132,11 @@ Same idea as `dind`, but built on `docker:dind-rootless` so the bundled daemon a
The runner is configured with a YAML file. Generate a starting point (this matches what ships in the tree): The runner is configured with a YAML file. Generate a starting point (this matches what ships in the tree):
```bash ```bash
./gitea-runner generate-config > config.yaml ./gitea-runner config generate > config.yaml
``` ```
> The top-level `generate-config` command still does the same thing, but is deprecated in favour of `config generate`.
Pass it with `-c` / `--config` on any command that loads configuration (`register`, `daemon`, `cache-server`): Pass it with `-c` / `--config` on any command that loads configuration (`register`, `daemon`, `cache-server`):
```bash ```bash
@@ -143,7 +145,26 @@ Pass it with `-c` / `--config` on any command that loads configuration (`registe
./gitea-runner -c config.yaml cache-server ./gitea-runner -c config.yaml cache-server
``` ```
Every option is described in [config.example.yaml](internal/pkg/config/config.example.yaml) (the same content `generate-config` prints). Every option is described in [config.example.yaml](internal/pkg/config/config.example.yaml) (the same content `config generate` prints).
#### Editing a config file
`config` changes an existing file in place, keeping its comments and key order, which is handy in provisioning scripts:
```bash
./gitea-runner -c config.yaml config set runner.capacity 4
./gitea-runner -c config.yaml config set runner.timeout 90m # written as 1h30m0s
./gitea-runner -c config.yaml config set runner.envs.MY_VAR value
./gitea-runner -c config.yaml config add runner.labels 'ubuntu:docker://node:22'
./gitea-runner -c config.yaml config remove runner.labels 'ubuntu:docker://node:22'
./gitea-runner -c config.yaml config get runner.labels
```
`-c` is optional for these subcommands: without it they use `config.yaml` (or `config.yml`) from the working directory, falling back to the directory of the `gitea-runner` binary, and print which file they picked to stderr.
Keys are the dotted YAML path and are validated against the known options, so a typo is rejected instead of being written. `add` and `remove` only work on list options such as `runner.labels` and `container.valid_volumes`, and fail if the value is already present or missing. `set` replaces the whole list when given several values.
The file is re-encoded on every edit, so indentation is normalised to two spaces and blank lines inside a section are dropped.
#### Without a config file #### Without a config file
@@ -209,6 +230,26 @@ Whenever the resulting labels differ from the ones in the registration file, the
> **Note:** A runner that only exposes `host` labels still needs access to a Docker daemon (e.g. a mounted `/var/run/docker.sock`) whenever a job uses a `docker://` action or a service container. `host` labels only change where the job's own steps run; container-based steps and actions are still executed with Docker. > **Note:** A runner that only exposes `host` labels still needs access to a Docker daemon (e.g. a mounted `/var/run/docker.sock`) whenever a job uses a `docker://` action or a service container. `host` labels only change where the job's own steps run; container-based steps and actions are still executed with Docker.
#### Service containers
A job's `services` are started before its steps run. When a service's image or its `options` declare a healthcheck, the runner waits for it to report healthy, so a workflow does not have to poll for its own services:
```yaml
services:
postgres:
image: postgres:17
options: >-
--health-cmd pg_isready
--health-interval 5s
--health-retries 10
```
A service that reports unhealthy fails the job right away, with its container log. One that never becomes healthy fails it after `container.service_ready_timeout` (default `5m`, negative disables the wait). A service that exits without declaring a healthcheck only gets its log and a warning.
A job in a container reaches a service by its id on the job network, on the port the service listens on, for example `psql -h postgres -p 5432`. The started containers also fill the `job` context: `job.container.{id,network}` and `job.services.<id>.{id,network,ports}`, where `ports` maps a container port to the host port Docker published it on, for the services that publish one.
Unlike GitHub, a job whose steps run on the host (a `host` label without `container:`) starts no service containers, so `job.services` and `job.container` stay empty. Give such a job a `container:` when it needs services.
#### Proxy #### Proxy
Set these variables in the runner's environment, with systemd `Environment=`, `docker run -e`, or Kubernetes `env:`: Set these variables in the runner's environment, with systemd `Environment=`, `docker run -e`, or Kubernetes `env:`:

View File

@@ -6,12 +6,14 @@ package container
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"io" "io"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
"github.com/docker/go-connections/nat" "github.com/docker/go-connections/nat"
"github.com/moby/moby/api/types/container"
) )
// ExitCodeError reports a non-zero process exit code from a container command. // ExitCodeError reports a non-zero process exit code from a container command.
@@ -57,6 +59,32 @@ type FileEntry struct {
Body string Body string
} }
// Container and healthcheck states, as plain strings so a caller of Info needs no docker
// SDK of its own.
const (
StateRunning = string(container.StateRunning)
HealthNone = string(container.NoHealthcheck)
HealthStarting = string(container.Starting)
HealthHealthy = string(container.Healthy)
HealthUnhealthy = string(container.Unhealthy)
)
// ErrContainerNotFound reports a container the daemon no longer knows. Its text is a
// fragment, missingContainerError composes it into the message every operation shares.
var ErrContainerNotFound = errors.New("does not exist")
// Info is a snapshot of a container, as of one inspect.
type Info struct {
ID string
State string // the docker container state: "created", "running", "exited", ...
ExitCode int
Health string // one of the Health* constants
// HealthOutput is the last healthcheck probe's output.
HealthOutput string
Ports map[string]string // container port ("5432") to the host port it is published on
}
// Container for managing docker run containers // Container for managing docker run containers
type Container interface { type Container interface {
Create(capAdd, capDrop []string) common.Executor Create(capAdd, capDrop []string) common.Executor
@@ -65,6 +93,8 @@ type Container interface {
CopyTarStream(ctx context.Context, destPath string, tarStream io.Reader) error CopyTarStream(ctx context.Context, destPath string, tarStream io.Reader) error
CopyDir(destPath, srcPath string, useGitIgnore bool) common.Executor CopyDir(destPath, srcPath string, useGitIgnore bool) common.Executor
GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error) GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error)
Inspect(ctx context.Context) (*Info, error)
DumpLogs(ctx context.Context) error
Pull(forcePull bool) common.Executor Pull(forcePull bool) common.Executor
Start(attach bool) common.Executor Start(attach bool) common.Executor
Exec(command []string, env map[string]string, user, workdir string) common.Executor Exec(command []string, env map[string]string, user, workdir string) common.Executor

View File

@@ -198,6 +198,109 @@ func (cr *containerReference) GetContainerArchive(ctx context.Context, srcPath s
return result.Content, nil return result.Content, nil
} }
// Inspect resolves the container by name when its id is not known yet. One the daemon no
// longer knows is reported as ErrContainerNotFound.
func (cr *containerReference) Inspect(ctx context.Context) (*Info, error) {
if common.Dryrun(ctx) {
return &Info{Health: HealthNone, Ports: map[string]string{}}, nil
}
if err := cr.connect()(ctx); err != nil {
return nil, err
}
if cr.id == "" { // a known id is trusted, find() would spend a call validating it
if err := cr.find()(ctx); err != nil {
return nil, err
}
}
if cr.id == "" {
return nil, cr.missingContainerError("inspect it")
}
result, err := cr.cli.ContainerInspect(ctx, cr.id, client.ContainerInspectOptions{})
if cerrdefs.IsNotFound(err) {
return nil, cr.missingContainerError("inspect it")
} else if err != nil {
return nil, err
}
return containerInfoFromInspect(result.Container), nil
}
// DumpLogs copies the container's log so far to its output writers.
func (cr *containerReference) DumpLogs(ctx context.Context) error {
if common.Dryrun(ctx) {
return nil
}
if err := cr.connect()(ctx); err != nil {
return err
}
if cr.id == "" {
return cr.missingContainerError("read its logs")
}
logs, err := cr.cli.ContainerLogs(ctx, cr.id, client.ContainerLogsOptions{ShowStdout: true, ShowStderr: true})
if err != nil {
return err
}
defer logs.Close()
return cr.copyOutput(logs)
}
// copyOutput writes a container stream to the writers the container was created with,
// demultiplexing it unless the container has a TTY, which sends a single raw stream.
func (cr *containerReference) copyOutput(stream io.Reader) error {
outWriter := cr.input.Stdout
if outWriter == nil {
outWriter = os.Stdout
}
errWriter := cr.input.Stderr
if errWriter == nil {
errWriter = os.Stderr
}
var err error
if !cr.input.AllocatePTY || os.Getenv("NORAW") != "" {
_, err = stdcopy.StdCopy(outWriter, errWriter, stream)
} else {
_, err = io.Copy(outWriter, stream)
}
// Flush any buffered, not-yet-newline-terminated trailing line so the final line of
// the output is not lost when it is not newline-terminated.
common.FlushWriter(outWriter)
common.FlushWriter(errWriter)
return err
}
func containerInfoFromInspect(inspect container.InspectResponse) *Info {
info := &Info{
ID: inspect.ID,
Health: HealthNone,
Ports: map[string]string{}, // an empty map, never null, in the expression context
}
if state := inspect.State; state != nil {
info.State = string(state.Status)
info.ExitCode = state.ExitCode
if health := state.Health; health != nil {
info.Health = string(health.Status)
if len(health.Log) > 0 {
info.HealthOutput = strings.TrimSpace(health.Log[len(health.Log)-1].Output)
}
}
}
if settings := inspect.NetworkSettings; settings != nil {
for port, bindings := range settings.Ports {
for _, binding := range bindings { // the last binding wins, a port maps to one host port
if binding.HostPort != "" {
info.Ports[port.Port()] = binding.HostPort
}
}
}
}
return info
}
func (cr *containerReference) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor { func (cr *containerReference) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor {
return parseEnvFile(cr, srcPath, env).IfNot(common.Dryrun) return parseEnvFile(cr, srcPath, env).IfNot(common.Dryrun)
} }
@@ -343,10 +446,10 @@ func (cr *containerReference) Close() common.Executor {
} }
} }
// missingContainerError is the shared "container X does not exist" error // missingContainerError is the shared "container X does not exist" error used by ops that
// used by ops that need a live cr.id. // need a live cr.id, wrapping ErrContainerNotFound so a caller can tell it from a failing daemon.
func (cr *containerReference) missingContainerError(format string, args ...any) error { func (cr *containerReference) missingContainerError(format string, args ...any) error {
return fmt.Errorf("container %q does not exist; cannot "+format, append([]any{cr.input.Name}, args...)...) return fmt.Errorf("container %q %w; cannot "+format, append([]any{cr.input.Name, ErrContainerNotFound}, args...)...)
} }
func (cr *containerReference) find() common.Executor { func (cr *containerReference) find() common.Executor {
@@ -737,7 +840,7 @@ func (cr *containerReference) exec(cmd []string, env map[string]string, user, wo
} }
defer resp.Close() defer resp.Close()
err = cr.waitForCommand(ctx, isTerminal, resp.HijackedResponse, idResp, user, workdir) err = cr.waitForCommand(ctx, resp.HijackedResponse, idResp, user, workdir)
if err != nil { if err != nil {
return err return err
} }
@@ -795,7 +898,7 @@ func (cr *containerReference) tryReadGID() common.Executor {
return cr.tryReadID("-g", func(id int) { cr.GID = id }) return cr.tryReadID("-g", func(id int) { cr.GID = id })
} }
func (cr *containerReference) waitForCommand(ctx context.Context, isTerminal bool, resp client.HijackedResponse, _ client.ExecCreateResult, _, _ string) error { func (cr *containerReference) waitForCommand(ctx context.Context, resp client.HijackedResponse, _ client.ExecCreateResult, _, _ string) error {
logger := common.Logger(ctx) logger := common.Logger(ctx)
// Buffered so the copy goroutine never blocks on send if the grace-period // Buffered so the copy goroutine never blocks on send if the grace-period
@@ -803,28 +906,7 @@ func (cr *containerReference) waitForCommand(ctx context.Context, isTerminal boo
cmdResponse := make(chan error, 1) cmdResponse := make(chan error, 1)
go func() { go func() {
var outWriter io.Writer cmdResponse <- cr.copyOutput(resp.Reader)
outWriter = cr.input.Stdout
if outWriter == nil {
outWriter = os.Stdout
}
errWriter := cr.input.Stderr
if errWriter == nil {
errWriter = os.Stderr
}
var err error
if !isTerminal || os.Getenv("NORAW") != "" {
_, err = stdcopy.StdCopy(outWriter, errWriter, resp.Reader)
} else {
_, err = io.Copy(outWriter, resp.Reader)
}
// Flush any buffered, not-yet-newline-terminated trailing line so the
// final line of a command's output is not lost (e.g. an error message
// printed without a trailing newline before the process exits).
common.FlushWriter(outWriter)
common.FlushWriter(errWriter)
cmdResponse <- err
}() }()
select { select {
@@ -1059,33 +1141,11 @@ func (cr *containerReference) attach() common.Executor {
if err != nil { if err != nil {
return fmt.Errorf("failed to attach to container: %w", err) return fmt.Errorf("failed to attach to container: %w", err)
} }
isTerminal := cr.input.AllocatePTY
var outWriter io.Writer
outWriter = cr.input.Stdout
if outWriter == nil {
outWriter = os.Stdout
}
errWriter := cr.input.Stderr
if errWriter == nil {
errWriter = os.Stderr
}
done := make(chan struct{}) done := make(chan struct{})
cr.attachDone = done cr.attachDone = done
go func() { go func() {
defer close(done) defer close(done)
var copyErr error if copyErr := cr.copyOutput(out.Reader); copyErr != nil {
if !isTerminal || os.Getenv("NORAW") != "" {
_, copyErr = stdcopy.StdCopy(outWriter, errWriter, out.Reader)
} else {
_, copyErr = io.Copy(outWriter, out.Reader)
}
// Flush any buffered, not-yet-newline-terminated trailing line once
// the stream reaches EOF, so the final line of the container's
// output is not lost when it is not newline-terminated.
common.FlushWriter(outWriter)
common.FlushWriter(errWriter)
if copyErr != nil {
common.Logger(ctx).Error(copyErr) common.Logger(ctx).Error(copyErr)
} }
}() }()

View File

@@ -25,6 +25,7 @@ import (
"github.com/moby/moby/api/pkg/stdcopy" "github.com/moby/moby/api/pkg/stdcopy"
"github.com/moby/moby/api/types/container" "github.com/moby/moby/api/types/container"
"github.com/moby/moby/api/types/mount" "github.com/moby/moby/api/types/mount"
"github.com/moby/moby/api/types/network"
mobyclient "github.com/moby/moby/client" mobyclient "github.com/moby/moby/client"
"github.com/sirupsen/logrus/hooks/test" "github.com/sirupsen/logrus/hooks/test"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@@ -548,7 +549,7 @@ func TestRejectsMissingContainer(t *testing.T) {
cr := &containerReference{cli: client, input: &NewContainerInput{Name: "job-1"}} cr := &containerReference{cli: client, input: &NewContainerInput{Name: "job-1"}}
check := func(op string, err error) { check := func(op string, err error) {
t.Helper() t.Helper()
require.Error(t, err, op) require.ErrorIs(t, err, ErrContainerNotFound, op)
assert.Contains(t, err.Error(), `container "job-1" does not exist`, op) assert.Contains(t, err.Error(), `container "job-1" does not exist`, op)
} }
check("copyContent", cr.copyContent("/var/run/act", &FileEntry{Name: "x", Mode: 0o644})(ctx)) check("copyContent", cr.copyContent("/var/run/act", &FileEntry{Name: "x", Mode: 0o644})(ctx))
@@ -557,6 +558,15 @@ func TestRejectsMissingContainer(t *testing.T) {
check("exec", cr.exec([]string{"echo"}, nil, "", "")(ctx)) check("exec", cr.exec([]string{"echo"}, nil, "", "")(ctx))
_, err := cr.GetContainerArchive(ctx, "/var/run/act/x") _, err := cr.GetContainerArchive(ctx, "/var/run/act/x")
check("GetContainerArchive", err) check("GetContainerArchive", err)
_, err = cr.Inspect(ctx)
check("Inspect", err)
// a known id the daemon has since dropped
client.On("ContainerInspect", ctx, "gone", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{}, cerrdefs.ErrNotFound)
removed := &containerReference{id: "gone", cli: client, input: &NewContainerInput{Name: "job-1"}}
_, err = removed.Inspect(ctx)
check("Inspect after removal", err)
} }
// End-to-end: a stale cr.id is cleared, repopulated from name lookup, // End-to-end: a stale cr.id is cleared, repopulated from name lookup,
@@ -825,6 +835,59 @@ func TestCheckVolumesRejectsEscapingHostPaths(t *testing.T) {
assert.Empty(t, hostConf.Binds) assert.Empty(t, hostConf.Binds)
} }
func TestContainerInfoFromInspect(t *testing.T) {
t.Run("reports no healthcheck when the image declares none", func(t *testing.T) {
info := containerInfoFromInspect(container.InspectResponse{
ID: "abc123",
State: &container.State{Status: "running", Running: true},
})
assert.Equal(t, "abc123", info.ID)
assert.Equal(t, "running", info.State)
assert.Equal(t, HealthNone, info.Health)
assert.Empty(t, info.Ports)
})
t.Run("reports the health status and the last probe output", func(t *testing.T) {
info := containerInfoFromInspect(container.InspectResponse{
State: &container.State{
Status: "running",
Health: &container.Health{
Status: container.Unhealthy,
Log: []*container.HealthcheckResult{
{Output: "first\n"},
{Output: "connection refused\n"},
},
},
},
})
assert.Equal(t, HealthUnhealthy, info.Health)
assert.Equal(t, "connection refused", info.HealthOutput)
})
t.Run("reports the published ports", func(t *testing.T) {
info := containerInfoFromInspect(container.InspectResponse{
State: &container.State{Status: "running"},
NetworkSettings: &container.NetworkSettings{
Ports: network.PortMap{
network.MustParsePort("5432/tcp"): []network.PortBinding{{HostPort: "49153"}},
network.MustParsePort("6379/tcp"): nil,
},
},
})
assert.Equal(t, map[string]string{"5432": "49153"}, info.Ports)
})
t.Run("tolerates a container without state", func(t *testing.T) {
info := containerInfoFromInspect(container.InspectResponse{ID: "abc123"})
assert.Equal(t, "abc123", info.ID)
assert.Equal(t, HealthNone, info.Health)
})
}
func TestMergeContainerConfigsVolumesReplaceRunnerMounts(t *testing.T) { func TestMergeContainerConfigsVolumesReplaceRunnerMounts(t *testing.T) {
logger, _ := test.NewNullLogger() logger, _ := test.NewNullLogger()
ctx := common.WithLogger(context.Background(), logger) ctx := common.WithLogger(context.Background(), logger)

View File

@@ -154,6 +154,14 @@ func (e *HostEnvironment) CopyDir(destPath, srcPath string, useGitIgnore bool) c
} }
} }
func (e *HostEnvironment) DumpLogs(_ context.Context) error {
return nil
}
func (e *HostEnvironment) Inspect(_ context.Context) (*Info, error) {
return &Info{Health: HealthNone, Ports: map[string]string{}}, nil
}
func (e *HostEnvironment) GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error) { func (e *HostEnvironment) GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error) {
buf := &bytes.Buffer{} buf := &bytes.Buffer{}
tw := tar.NewWriter(buf) tw := tar.NewWriter(buf)

View File

@@ -5,12 +5,18 @@
package model package model
type JobContext struct { type JobContext struct {
Status string `json:"status"` Status string `json:"status"`
Container struct { Container JobContainerContext `json:"container"`
ID string `json:"id"` Services map[string]JobService `json:"services"`
Network string `json:"network"` }
} `json:"container"`
Services map[string]struct { type JobContainerContext struct {
ID string `json:"id"` ID string `json:"id"`
} `json:"services"` Network string `json:"network"`
}
type JobService struct {
ID string `json:"id"`
Network string `json:"network"`
Ports map[string]string `json:"ports"` // container port to the published host port
} }

View File

@@ -186,10 +186,10 @@ func (rc *RunContext) compositeExecutor(action *model.Action) *compositeSteps {
err := rc.newCompositeCommandExecutor(step.main())(ctx) err := rc.newCompositeCommandExecutor(step.main())(ctx)
if err != nil { if err != nil {
logger.Errorf("%v", err) logger.Errorf("##[error]%s", EscapeCommandData(err.Error()))
common.SetJobError(ctx, err) common.SetJobError(ctx, err)
} else if ctx.Err() != nil { } else if ctx.Err() != nil {
logger.Errorf("%v", ctx.Err()) logger.Errorf("##[error]%s", EscapeCommandData(ctx.Err().Error()))
common.SetJobError(ctx, ctx.Err()) common.SetJobError(ctx, ctx.Err())
} }
return nil return nil
@@ -248,10 +248,10 @@ func newCompositeStepLogExecutor(runStep common.Executor, stepID string) common.
logger := common.Logger(ctx) logger := common.Logger(ctx)
err := runStep(ctx) err := runStep(ctx)
if err != nil { if err != nil {
logger.Errorf("%v", err) logger.Errorf("##[error]%s", EscapeCommandData(err.Error()))
common.SetJobError(ctx, err) common.SetJobError(ctx, err)
} else if ctx.Err() != nil { } else if ctx.Err() != nil {
logger.Errorf("%v", ctx.Err()) logger.Errorf("##[error]%s", EscapeCommandData(ctx.Err().Error()))
common.SetJobError(ctx, ctx.Err()) common.SetJobError(ctx, ctx.Err())
} }
return nil return nil

View File

@@ -6,6 +6,7 @@ package runner
import ( import (
"context" "context"
"fmt"
"regexp" "regexp"
"strings" "strings"
@@ -45,17 +46,24 @@ func (rc *RunContext) commandHandler(ctx context.Context) common.LineHandler {
return true return true
} }
if resumeCommand != "" && command != resumeCommand { if resumeCommand != "" {
// There should not be any emojis in the log output for Gitea. // There should not be any emojis in the log output for Gitea.
// The code in the switch statement is the same.
// Return true (not false) so the line still reaches the raw_output // Return true (not false) so the line still reaches the raw_output
// log handler; otherwise everything between ::stop-commands:: and // log handler; otherwise everything between ::stop-commands:: and
// its end token is silently dropped from the step log. // its end token is silently dropped from the step log.
logger.Infof("%s", line) logger.Infof("%s", line)
// Resumed here rather than from the switch, because the end token is arbitrary
// and a token naming a real command would otherwise never resume.
if command == resumeCommand {
resumeCommand = ""
}
return true return true
} }
arg = UnescapeCommandData(arg) arg = UnescapeCommandData(arg)
kvPairs = unescapeKvPairs(kvPairs) kvPairs = unescapeKvPairs(kvPairs)
if (command == "set-env" || command == "add-path") && rc.refuseUnsecureCommand(ctx, command) {
return true
}
switch command { switch command {
case "set-env": case "set-env":
rc.setEnv(ctx, kvPairs, arg) rc.setEnv(ctx, kvPairs, arg)
@@ -63,27 +71,20 @@ func (rc *RunContext) commandHandler(ctx context.Context) common.LineHandler {
rc.setOutput(ctx, kvPairs, arg) rc.setOutput(ctx, kvPairs, arg)
case "add-path": case "add-path":
rc.addPath(ctx, arg) rc.addPath(ctx, arg)
case "debug":
logger.Infof("%s", line)
case "warning":
logger.Infof("%s", line)
case "error":
logger.Infof("%s", line)
case "add-mask": case "add-mask":
rc.AddMask(arg) rc.AddMask(arg)
logger.Infof("%s", "***") logger.Infof("%s", "***")
// The raw line is still forwarded, carrying the secret: that is how the reporter
// learns the mask, and it drops the row rather than writing it out.
case "stop-commands": case "stop-commands":
resumeCommand = arg resumeCommand = arg
logger.Infof("%s", line) logger.Infof("%s", line)
case resumeCommand:
resumeCommand = ""
logger.Infof("%s", line)
case "save-state": case "save-state":
logger.Infof("%s", line) logger.Infof("%s", line)
rc.saveState(ctx, kvPairs, arg) rc.saveState(ctx, kvPairs, arg)
case "add-matcher":
logger.Infof("%s", line)
default: default:
// ::debug::, ::error::, ::warning::, ::add-matcher:: and anything unrecognised are
// passed through for the reporter and Gitea's web UI to render.
logger.Infof("%s", line) logger.Infof("%s", line)
} }
@@ -92,6 +93,52 @@ func (rc *RunContext) commandHandler(ctx context.Context) common.LineHandler {
} }
} }
const allowUnsecureCommandsVar = "ACTIONS_ALLOW_UNSECURE_COMMANDS"
// refuseUnsecureCommand reports whether a deprecated ::set-env:: or ::add-path:: command must
// not run, recording the error that fails the step. GitHub disabled both because a step that
// echoes untrusted content can use them to set NODE_OPTIONS or PATH for every later step.
func (rc *RunContext) refuseUnsecureCommand(ctx context.Context, command string) bool {
if rc.allowUnsecureCommandsOptIn() {
return false
}
// The step executor logs the failure itself, so keep this line's wording distinct.
common.Logger(ctx).WithField(rawOutputField, true).Errorf("##[error]%s", EscapeCommandData(fmt.Sprintf(
"The `%s` command is disabled: it can set the environment of every later step from untrusted output. "+
"Write to $GITHUB_ENV or $GITHUB_PATH instead, or set ACTIONS_ALLOW_UNSECURE_COMMANDS to allow it",
command)))
rc.unsecureCommandMu.Lock()
defer rc.unsecureCommandMu.Unlock()
if rc.unsecureCommandErr == nil {
rc.unsecureCommandErr = fmt.Errorf("the `%s` workflow command is disabled", command)
}
return true
}
// allowUnsecureCommandsOptIn reports whether the workflow itself asked for the deprecated
// commands, from any env scope, as it can on GitHub.
func (rc *RunContext) allowUnsecureCommandsOptIn() bool {
return isTruthyEnv(rc.currentStepEnv()[allowUnsecureCommandsVar]) ||
isTruthyEnv(rc.Env[allowUnsecureCommandsVar]) ||
isTruthyEnv(rc.GlobalEnv[allowUnsecureCommandsVar])
}
// isTruthyEnv mirrors GitHub's bool.TryParse: only "true", in any casing.
func isTruthyEnv(v string) bool {
return strings.EqualFold(strings.TrimSpace(v), "true")
}
// takeUnsecureCommandError returns and clears the error left by a refused command.
func (rc *RunContext) takeUnsecureCommandError() error {
rc.unsecureCommandMu.Lock()
defer rc.unsecureCommandMu.Unlock()
err := rc.unsecureCommandErr
rc.unsecureCommandErr = nil
return err
}
func (rc *RunContext) setEnv(ctx context.Context, kvPairs map[string]string, arg string) { func (rc *RunContext) setEnv(ctx context.Context, kvPairs map[string]string, arg string) {
name := kvPairs["name"] name := kvPairs["name"]
common.Logger(ctx).Infof("::set-env:: %s=%s", name, arg) common.Logger(ctx).Infof("::set-env:: %s=%s", name, arg)
@@ -161,9 +208,9 @@ var (
commandPropertyUnescaper = strings.NewReplacer("%25", "%", "%0D", "\r", "%0A", "\n", "%3A", ":", "%2C", ",") commandPropertyUnescaper = strings.NewReplacer("%25", "%", "%0D", "\r", "%0A", "\n", "%3A", ":", "%2C", ",")
) )
// escapeCommandData encodes the data part of a "::cmd::" or "##[cmd]" line the runner writes itself, // EscapeCommandData encodes the data part of a "::cmd::" or "##[cmd]" line the runner writes itself,
// so the log renderer decodes it back. Lines forwarded from step output are already escaped. // so the log renderer decodes it back. Lines forwarded from step output are already escaped.
func escapeCommandData(arg string) string { func EscapeCommandData(arg string) string {
return commandDataEscaper.Replace(arg) return commandDataEscaper.Replace(arg)
} }

View File

@@ -16,12 +16,18 @@ import (
"github.com/sirupsen/logrus/hooks/test" "github.com/sirupsen/logrus/hooks/test"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
) )
// unsecureRC opts into ::set-env:: and ::add-path::, which are refused without it.
func unsecureRC() *RunContext {
return &RunContext{Env: map[string]string{allowUnsecureCommandsVar: "true"}}
}
func TestSetEnv(t *testing.T) { func TestSetEnv(t *testing.T) {
a := assert.New(t) a := assert.New(t)
ctx := context.Background() ctx := context.Background()
rc := new(RunContext) rc := unsecureRC()
handler := rc.commandHandler(ctx) handler := rc.commandHandler(ctx)
handler("::set-env name=x::valz\n") handler("::set-env name=x::valz\n")
@@ -31,7 +37,7 @@ func TestSetEnv(t *testing.T) {
func TestStopCommandsKeepsSuppressedLinesInLog(t *testing.T) { func TestStopCommandsKeepsSuppressedLinesInLog(t *testing.T) {
a := assert.New(t) a := assert.New(t)
ctx := context.Background() ctx := context.Background()
rc := new(RunContext) rc := unsecureRC()
handler := rc.commandHandler(ctx) handler := rc.commandHandler(ctx)
// Stop command processing until the matching end token is seen. // Stop command processing until the matching end token is seen.
@@ -84,7 +90,7 @@ func TestSetOutput(t *testing.T) {
func TestAddpath(t *testing.T) { func TestAddpath(t *testing.T) {
a := assert.New(t) a := assert.New(t)
ctx := context.Background() ctx := context.Background()
rc := new(RunContext) rc := unsecureRC()
handler := rc.commandHandler(ctx) handler := rc.commandHandler(ctx)
handler("::add-path::/zoo\n") handler("::add-path::/zoo\n")
@@ -99,7 +105,7 @@ func TestStopCommands(t *testing.T) {
a := assert.New(t) a := assert.New(t)
ctx := common.WithLogger(context.Background(), logger) ctx := common.WithLogger(context.Background(), logger)
rc := new(RunContext) rc := unsecureRC()
handler := rc.commandHandler(ctx) handler := rc.commandHandler(ctx)
handler("::set-env name=x::valz\n") handler("::set-env name=x::valz\n")
@@ -119,10 +125,26 @@ func TestStopCommands(t *testing.T) {
a.Contains(messages, "::set-env name=x::abcd\n") a.Contains(messages, "::set-env name=x::abcd\n")
} }
// The end token is arbitrary, so one that happens to name a real command must still resume
// rather than being swallowed by that command's case.
func TestStopCommandsResumesOnCommandNamedToken(t *testing.T) {
a := assert.New(t)
rc := unsecureRC()
handler := rc.commandHandler(context.Background())
handler("::stop-commands::add-mask\n")
handler("::set-env name=x::suppressed\n")
a.NotContains(rc.Env, "x")
handler("::add-mask::\n")
handler("::set-env name=x::resumed\n")
a.Equal("resumed", rc.Env["x"])
}
func TestAddpathADO(t *testing.T) { func TestAddpathADO(t *testing.T) {
a := assert.New(t) a := assert.New(t)
ctx := context.Background() ctx := context.Background()
rc := new(RunContext) rc := unsecureRC()
handler := rc.commandHandler(ctx) handler := rc.commandHandler(ctx)
handler("##[add-path]/zoo\n") handler("##[add-path]/zoo\n")
@@ -218,6 +240,44 @@ func TestSaveState(t *testing.T) {
func TestEscapeCommandData(t *testing.T) { func TestEscapeCommandData(t *testing.T) {
a := assert.New(t) a := assert.New(t)
a.Equal("a%25b%0Dc%0Ad%250A", escapeCommandData("a%b\rc\nd%0A")) a.Equal("a%25b%0Dc%0Ad%250A", EscapeCommandData("a%b\rc\nd%0A"))
a.Equal("a%b\rc\nd%0A", UnescapeCommandData("a%25b%0Dc%0Ad%250A")) a.Equal("a%b\rc\nd%0A", UnescapeCommandData("a%25b%0Dc%0Ad%250A"))
} }
func TestUnsecureCommands(t *testing.T) {
tests := []struct {
name string
jobEnv map[string]string
stepEnv map[string]string
optedIn bool
}{
{name: "refused with no opt-in"},
// GitHub reads the opt-in with bool.TryParse, so "1" is not one.
{name: "refused for a value bool.TryParse rejects", jobEnv: map[string]string{allowUnsecureCommandsVar: "1"}},
{name: "opted in through the step environment", stepEnv: map[string]string{allowUnsecureCommandsVar: "true"}, optedIn: true},
{name: "opted in through the job environment", jobEnv: map[string]string{allowUnsecureCommandsVar: "TRUE"}, optedIn: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
a := assert.New(t)
rc := &RunContext{Env: tt.jobEnv}
rc.setCurrentStepEnv(tt.stepEnv)
handler := rc.commandHandler(context.Background())
handler("::set-env name=x::valz\n")
handler("::add-path::/opt/bin\n")
if !tt.optedIn {
a.Empty(rc.Env["x"])
a.Empty(rc.ExtraPath)
// The refusal fails the step that produced it, once.
require.ErrorContains(t, rc.takeUnsecureCommandError(), "set-env")
a.NoError(rc.takeUnsecureCommandError())
return
}
a.Equal("valz", rc.Env["x"])
a.Equal([]string{"/opt/bin"}, rc.ExtraPath)
a.NoError(rc.takeUnsecureCommandError())
})
}
}

View File

@@ -78,3 +78,14 @@ func (cm *containerMock) GetContainerArchive(ctx context.Context, srcPath string
} }
return args.Get(0).(io.ReadCloser), err return args.Get(0).(io.ReadCloser), err
} }
func (cm *containerMock) DumpLogs(ctx context.Context) error {
return cm.Called(ctx).Error(0)
}
func (cm *containerMock) Inspect(ctx context.Context) (*container.Info, error) {
args := cm.Called(ctx)
info, _ := args.Get(0).(*container.Info)
err, _ := args.Get(1).(error)
return info, err
}

View File

@@ -66,7 +66,7 @@ func reportStepError(ctx context.Context, rc *RunContext, err error) {
rc.markInterrupted(ctx.Err()) rc.markInterrupted(ctx.Err())
return return
} }
common.Logger(ctx).Errorf("##[error]%s", escapeCommandData(err.Error())) common.Logger(ctx).Errorf("##[error]%s", EscapeCommandData(err.Error()))
common.SetJobError(ctx, err) common.SetJobError(ctx, err)
rc.markFailed() rc.markFailed()
} }
@@ -260,7 +260,7 @@ func newJobExecutor(info jobInfo, sf stepFactory, rc *RunContext) common.Executo
logger.Infof("Cleaning up container for job %s", rc.JobName) logger.Infof("Cleaning up container for job %s", rc.JobName)
if err = info.stopContainer()(ctx); err != nil { if err = info.stopContainer()(ctx); err != nil {
logger.Errorf("Error while stop job container: %v", err) logger.Errorf("##[error]%s", EscapeCommandData("Error while stop job container: "+err.Error()))
} }
// For Gitea // For Gitea

View File

@@ -45,7 +45,7 @@ func (rc *RunContext) runJobHook(ctx context.Context, hookPath, name string) err
cmd, shell := hookCommand(hookPath) cmd, shell := hookCommand(hookPath)
rawLogger := common.Logger(ctx).WithField(rawOutputField, true) rawLogger := common.Logger(ctx).WithField(rawOutputField, true)
defer rawLogger.Infof("::endgroup::") defer rawLogger.Infof("::endgroup::")
rawLogger.Infof("::group::Run '%s'", escapeCommandData(hookPath)) rawLogger.Infof("::group::Run '%s'", EscapeCommandData(hookPath))
rawLogger.Infof("A %s hook has been configured by the runner administrator", name) rawLogger.Infof("A %s hook has been configured by the runner administrator", name)
if shell != "" { if shell != "" {
rawLogger.Infof("shell: %s", shell) rawLogger.Infof("shell: %s", shell)

View File

@@ -250,7 +250,7 @@ func AppendSecretMasker(oldnew []string, v string) []string {
ret = append(ret, tm, "***") ret = append(ret, tm, "***")
// command data reaches the log escaped, so "pass%word" also arrives as "pass%25word" // command data reaches the log escaped, so "pass%word" also arrives as "pass%25word"
if strings.ContainsAny(tm, "%\r\n") { if strings.ContainsAny(tm, "%\r\n") {
ret = append(ret, escapeCommandData(tm), "***") ret = append(ret, EscapeCommandData(tm), "***")
} }
} }
} }

View File

@@ -22,6 +22,7 @@ import (
"runtime" "runtime"
"slices" "slices"
"strings" "strings"
"sync"
"time" "time"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
@@ -34,6 +35,7 @@ import (
"github.com/docker/go-connections/nat" "github.com/docker/go-connections/nat"
"github.com/moby/moby/api/types/mount" "github.com/moby/moby/api/types/mount"
"github.com/opencontainers/selinux/go-selinux" "github.com/opencontainers/selinux/go-selinux"
"golang.org/x/sync/errgroup"
) )
// RunContext contains info about current job // RunContext contains info about current job
@@ -55,7 +57,7 @@ type RunContext struct {
IntraActionState map[string]map[string]string IntraActionState map[string]map[string]string
ExprEval ExpressionEvaluator ExprEval ExpressionEvaluator
JobContainer container.ExecutionsEnvironment JobContainer container.ExecutionsEnvironment
ServiceContainers []container.ExecutionsEnvironment serviceContainers []*serviceContainer
OutputMappings map[MappableOutput]MappableOutput OutputMappings map[MappableOutput]MappableOutput
JobName string JobName string
ActionPath string ActionPath string
@@ -83,6 +85,37 @@ type RunContext struct {
// failures. Those failures must still make success() false and failure() true for later // failures. Those failures must still make success() false and failure() true for later
// main-step if evaluation. // main-step if evaluation.
jobFailed bool jobFailed bool
// empty for a host-mode job, which starts no container
jobContainerID string
jobNetworkName string
// stepEnv is a copy of the running step's environment, so that workflow commands parsed out
// of the container's output can be judged against it. Written by runStepExecutor and read on
// the log-writer goroutine, hence unsecureCommandMu, which also guards unsecureCommandErr.
stepEnv map[string]string
unsecureCommandErr error // refused ::set-env::/::add-path::, turned into a step failure
unsecureCommandMu sync.Mutex
}
// serviceContainer pairs a service container with the workflow id that keys job.services.
type serviceContainer struct {
name string
image string
container container.ExecutionsEnvironment
logsDumped bool
info *container.Info // last poll, the source of the `job.services` entry
}
// setCurrentStepEnv records the environment of the step about to run.
func (rc *RunContext) setCurrentStepEnv(env map[string]string) {
rc.unsecureCommandMu.Lock()
defer rc.unsecureCommandMu.Unlock()
rc.stepEnv = env
}
func (rc *RunContext) currentStepEnv() map[string]string {
rc.unsecureCommandMu.Lock()
defer rc.unsecureCommandMu.Unlock()
return rc.stepEnv
} }
// markCancelled flags the job as cancelled so subsequent step `if` evaluations and the // markCancelled flags the job as cancelled so subsequent step `if` evaluations and the
@@ -488,7 +521,7 @@ func (rc *RunContext) startJobContainer() common.Executor {
Privileged: rc.Config.Privileged, Privileged: rc.Config.Privileged,
UsernsMode: rc.Config.UsernsMode, UsernsMode: rc.Config.UsernsMode,
Platform: rc.Config.ContainerArchitecture, Platform: rc.Config.ContainerArchitecture,
AutoRemove: rc.Config.AutoRemove, AutoRemove: false, // so a dead service's log survives, cleanupJobResources removes it
Options: rc.ExprEval.Interpolate(ctx, spec.Options), Options: rc.ExprEval.Interpolate(ctx, spec.Options),
NetworkMode: networkName, NetworkMode: networkName,
NetworkAliases: []string{serviceID}, NetworkAliases: []string{serviceID},
@@ -496,7 +529,7 @@ func (rc *RunContext) startJobContainer() common.Executor {
PortBindings: portBindings, PortBindings: portBindings,
AllocatePTY: rc.Config.AllocatePTY, AllocatePTY: rc.Config.AllocatePTY,
}) })
rc.ServiceContainers = append(rc.ServiceContainers, c) rc.serviceContainers = append(rc.serviceContainers, &serviceContainer{name: serviceID, image: serviceImage, container: c})
} }
rc.cleanUpJobContainer = rc.cleanupJobResources(networkName, createAndDeleteNetwork) rc.cleanUpJobContainer = rc.cleanupJobResources(networkName, createAndDeleteNetwork)
@@ -531,6 +564,8 @@ func (rc *RunContext) startJobContainer() common.Executor {
return errors.New("Failed to create job container") return errors.New("Failed to create job container")
} }
rc.jobNetworkName = networkName
defer printStartJobContainerGroup(ctx, image, name, networkName)() defer printStartJobContainerGroup(ctx, image, name, networkName)()
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
rc.pullServicesImages(rc.Config.ForcePull), rc.pullServicesImages(rc.Config.ForcePull),
@@ -538,9 +573,12 @@ func (rc *RunContext) startJobContainer() common.Executor {
rc.stopJobContainer(), rc.stopJobContainer(),
container.NewDockerNetworkCreateExecutor(networkName, rc.Config.ContainerNetworkCreateOptions). container.NewDockerNetworkCreateExecutor(networkName, rc.Config.ContainerNetworkCreateOptions).
IfBool(createAndDeleteNetwork), IfBool(createAndDeleteNetwork),
rc.startServiceContainers(networkName), rc.startServiceContainers(),
rc.reportUnstartedServices(),
rc.waitForServiceContainers(),
rc.JobContainer.Create(rc.Config.ContainerCapAdd, rc.Config.ContainerCapDrop), rc.JobContainer.Create(rc.Config.ContainerCapAdd, rc.Config.ContainerCapDrop),
rc.JobContainer.Start(false), rc.JobContainer.Start(false),
rc.captureJobContainerInfo(),
rc.JobContainer.Copy(rc.JobContainer.GetActPath()+"/", &container.FileEntry{ rc.JobContainer.Copy(rc.JobContainer.GetActPath()+"/", &container.FileEntry{
Name: "workflow/event.json", Name: "workflow/event.json",
Mode: 0o644, Mode: 0o644,
@@ -565,7 +603,7 @@ func (rc *RunContext) cleanupJobResources(networkName string, createAndDeleteNet
if removeJobContainer { if removeJobContainer {
errs = append(errs, rc.JobContainer.Remove()(ctx)) errs = append(errs, rc.JobContainer.Remove()(ctx))
} }
if len(rc.ServiceContainers) > 0 { if len(rc.serviceContainers) > 0 {
logger.Infof("Cleaning up services for job %s", rc.JobName) logger.Infof("Cleaning up services for job %s", rc.JobName)
if err := rc.stopServiceContainers()(ctx); err != nil { if err := rc.stopServiceContainers()(ctx); err != nil {
logger.Errorf("Error while cleaning services: %v", err) logger.Errorf("Error while cleaning services: %v", err)
@@ -662,21 +700,21 @@ func (rc *RunContext) stopJobContainer() common.Executor {
func (rc *RunContext) pullServicesImages(forcePull bool) common.Executor { func (rc *RunContext) pullServicesImages(forcePull bool) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
execs := []common.Executor{} execs := []common.Executor{}
for _, c := range rc.ServiceContainers { for _, svc := range rc.serviceContainers {
execs = append(execs, c.Pull(forcePull)) execs = append(execs, svc.container.Pull(forcePull))
} }
return common.NewParallelExecutor(len(execs), execs...)(ctx) return common.NewParallelExecutor(len(execs), execs...)(ctx)
} }
} }
func (rc *RunContext) startServiceContainers(_ string) common.Executor { func (rc *RunContext) startServiceContainers() common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
execs := []common.Executor{} execs := []common.Executor{}
for _, c := range rc.ServiceContainers { for _, svc := range rc.serviceContainers {
execs = append(execs, common.NewPipelineExecutor( execs = append(execs, common.NewPipelineExecutor(
c.Pull(false), svc.container.Pull(false),
c.Create(rc.Config.ContainerCapAdd, rc.Config.ContainerCapDrop), svc.container.Create(rc.Config.ContainerCapAdd, rc.Config.ContainerCapDrop),
c.Start(false), svc.container.Start(false),
)) ))
} }
return common.NewParallelExecutor(len(execs), execs...)(ctx) return common.NewParallelExecutor(len(execs), execs...)(ctx)
@@ -686,13 +724,159 @@ func (rc *RunContext) startServiceContainers(_ string) common.Executor {
func (rc *RunContext) stopServiceContainers() common.Executor { func (rc *RunContext) stopServiceContainers() common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
execs := []common.Executor{} execs := []common.Executor{}
for _, c := range rc.ServiceContainers { for _, svc := range rc.serviceContainers {
execs = append(execs, c.Remove().Finally(c.Close())) execs = append(execs, svc.container.Remove().Finally(svc.container.Close()))
} }
return common.NewParallelExecutor(len(execs), execs...)(ctx) return common.NewParallelExecutor(len(execs), execs...)(ctx)
} }
} }
const (
defaultServiceReadyTimeout = 5 * time.Minute
serviceReadyPollMax = 32 * time.Second
)
var serviceReadyPollInterval = 2 * time.Second // a variable so tests need not wait
// reportUnstartedServices logs a service that did not start. The steps that need it
// report it better than the runner can, so the job carries on.
func (rc *RunContext) reportUnstartedServices() common.Executor {
return func(ctx context.Context) error {
logger := common.Logger(ctx)
for _, svc := range rc.serviceContainers {
info, err := svc.inspect(ctx)
if err != nil {
logger.Debugf("unable to inspect service '%s': %v", svc.name, err)
continue
}
if info.State == container.StateRunning {
continue
}
svc.dumpLogs(ctx)
logger.Warnf("Docker container %s is not in running state: %s (%d)", info.ID, info.State, info.ExitCode)
}
return nil
}
}
// waitForServiceContainers blocks until every service that declares a healthcheck reports
// healthy, as GitHub does, so a first step cannot connect before the service listens.
func (rc *RunContext) waitForServiceContainers() common.Executor {
return func(ctx context.Context) error {
if len(rc.serviceContainers) == 0 {
return nil
}
timeout := rc.Config.ServiceReadyTimeout
switch {
case timeout < 0:
// disabled, but still describe the containers for `job.services`
for _, svc := range rc.serviceContainers {
if _, err := svc.inspect(ctx); err != nil && !errors.Is(err, container.ErrContainerNotFound) {
return err
}
}
return nil
case timeout == 0:
timeout = defaultServiceReadyTimeout
}
// the first error cancels the rest, so a failure does not wait out a sibling's timeout
group, groupCtx := errgroup.WithContext(ctx)
for _, svc := range rc.serviceContainers {
group.Go(func() error {
return svc.waitUntilHealthy(groupCtx, timeout)
})
}
return group.Wait()
}
}
// waitUntilHealthy waits on the healthcheck alone, so a container that declares none is
// ready at once and one that exited is left to the steps that need it.
func (svc *serviceContainer) waitUntilHealthy(ctx context.Context, timeout time.Duration) error {
rawLogger := common.Logger(ctx).WithField(rawOutputField, true)
interval := serviceReadyPollInterval
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
for {
info, err := svc.inspect(ctx)
if ctxErr := ctx.Err(); ctxErr != nil { // the wait ended, an inspect error only noticed it
if errors.Is(ctxErr, context.DeadlineExceeded) {
return fmt.Errorf("the service '%s' did not become healthy within %s%s", svc.name, timeout, svc.healthOutputSuffix())
}
return ctxErr
}
switch {
case errors.Is(err, container.ErrContainerNotFound):
return nil // gone, so there is no health left to wait on
case err != nil:
return err
}
switch {
case info.Health == container.HealthUnhealthy:
svc.dumpLogs(ctx)
common.Logger(ctx).Errorf("Failed to initialize container %s", svc.image)
return fmt.Errorf("the service '%s' is unhealthy%s", svc.name, svc.healthOutputSuffix())
case info.Health != container.HealthStarting:
rawLogger.Infof("%s service is healthy.", svc.name)
return nil
}
rawLogger.Infof("%s service is starting, waiting %d seconds before checking again.", svc.name, int(interval.Seconds()))
select {
case <-ctx.Done(): // reported at the top of the loop
case <-time.After(interval):
}
interval = min(interval*2, serviceReadyPollMax)
}
}
// dumpLogs writes the container's log to the job log once, however often it is reported.
func (svc *serviceContainer) dumpLogs(ctx context.Context) {
if svc.logsDumped {
return
}
svc.logsDumped = true
if err := svc.container.DumpLogs(ctx); err != nil {
common.Logger(ctx).Debugf("unable to read the log of service '%s': %v", svc.name, err)
}
}
// inspect also records the state for the `job.services` context.
func (svc *serviceContainer) inspect(ctx context.Context) (*container.Info, error) {
info, err := svc.container.Inspect(ctx)
if err != nil {
return nil, fmt.Errorf("failed to inspect service '%s': %w", svc.name, err)
}
svc.info = info
return info, nil
}
func (svc *serviceContainer) healthOutputSuffix() string {
if svc.info == nil || svc.info.HealthOutput == "" {
return ""
}
return ": " + svc.info.HealthOutput
}
// captureJobContainerInfo is a convenience: failing to describe the container must not
// fail the job.
func (rc *RunContext) captureJobContainerInfo() common.Executor {
return func(ctx context.Context) error {
info, err := rc.JobContainer.Inspect(ctx)
if err != nil {
common.Logger(ctx).Debugf("unable to inspect the job container: %v", err)
return nil
}
rc.jobContainerID = info.ID
return nil
}
}
// Prepare the mounts and binds for the worker // Prepare the mounts and binds for the worker
// ActionCacheDir is for rc // ActionCacheDir is for rc
@@ -1033,9 +1217,26 @@ func (rc *RunContext) getJobContext() *model.JobContext {
if rc.jobCancelled { if rc.jobCancelled {
jobStatus = "cancelled" jobStatus = "cancelled"
} }
return &model.JobContext{
Status: jobStatus, jobContext := &model.JobContext{
Status: jobStatus,
Services: map[string]model.JobService{}, // an empty map, never null
} }
if rc.jobContainerID != "" {
jobContext.Container.ID = rc.jobContainerID
jobContext.Container.Network = rc.jobNetworkName
}
for _, svc := range rc.serviceContainers {
if svc.info == nil {
continue
}
jobContext.Services[svc.name] = model.JobService{
ID: svc.info.ID,
Network: rc.jobNetworkName,
Ports: svc.info.Ports,
}
}
return jobContext
} }
func (rc *RunContext) getStepsContext() map[string]*model.StepResult { func (rc *RunContext) getStepsContext() map[string]*model.StepResult {

View File

@@ -13,6 +13,7 @@ import (
"runtime" "runtime"
"strings" "strings"
"testing" "testing"
"time"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/container" "gitea.com/gitea/runner/act/container"
@@ -22,6 +23,7 @@ import (
"github.com/docker/cli/cli/compose/loader" "github.com/docker/cli/cli/compose/loader"
log "github.com/sirupsen/logrus" log "github.com/sirupsen/logrus"
assert "github.com/stretchr/testify/assert" assert "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
require "github.com/stretchr/testify/require" require "github.com/stretchr/testify/require"
yaml "go.yaml.in/yaml/v4" yaml "go.yaml.in/yaml/v4"
) )
@@ -225,6 +227,12 @@ func (fakeContainer) Copy(string, ...*container.FileEntry) common.Executor {
return func(context.Context) error { return nil } return func(context.Context) error { return nil }
} }
func (fakeContainer) Inspect(context.Context) (*container.Info, error) {
return &container.Info{ID: "fake", State: "running", Health: container.HealthNone}, nil
}
func (fakeContainer) DumpLogs(context.Context) error { return nil }
// Regression test: a service without a `credentials:` block resolves to empty // Regression test: a service without a `credentials:` block resolves to empty
// credentials, which used to overwrite the job container's own credentials. // credentials, which used to overwrite the job container's own credentials.
func TestStartJobContainerKeepsJobCredentialsWithServices(t *testing.T) { func TestStartJobContainerKeepsJobCredentialsWithServices(t *testing.T) {
@@ -563,7 +571,7 @@ func TestCleanupJobResourcesCleansServicesWithoutJobContainer(t *testing.T) {
rc := &RunContext{ rc := &RunContext{
Config: &Config{}, Config: &Config{},
ServiceContainers: []container.ExecutionsEnvironment{service}, serviceContainers: []*serviceContainer{{name: "svc", container: service}},
} }
err := rc.cleanupJobResources("external-network", false)(context.Background()) err := rc.cleanupJobResources("external-network", false)(context.Background())
@@ -586,7 +594,7 @@ func TestCleanupJobResourcesContinuesAfterFailure(t *testing.T) {
Config: &Config{}, Config: &Config{},
Run: &model.Run{Workflow: &model.Workflow{Name: "wf"}, JobID: "job"}, Run: &model.Run{Workflow: &model.Workflow{Name: "wf"}, JobID: "job"},
JobContainer: jobContainer, JobContainer: jobContainer,
ServiceContainers: []container.ExecutionsEnvironment{service}, serviceContainers: []*serviceContainer{{name: "svc", container: service}},
} }
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
@@ -1045,6 +1053,199 @@ func TestRunContext_cleanupFailedStart(t *testing.T) {
}) })
} }
func TestWaitForServiceContainers(t *testing.T) {
origInterval := serviceReadyPollInterval
serviceReadyPollInterval = time.Millisecond
defer func() { serviceReadyPollInterval = origInterval }()
newRunContext := func(timeout time.Duration, services ...*serviceContainer) *RunContext {
return &RunContext{
Config: &Config{ServiceReadyTimeout: timeout},
serviceContainers: services,
}
}
t.Run("returns as soon as a service without a healthcheck runs", func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).
Return(&container.Info{ID: "id", State: "running", Health: container.HealthNone}, nil).Once()
rc := newRunContext(0, &serviceContainer{name: "redis", container: service})
require.NoError(t, rc.waitForServiceContainers()(context.Background()))
service.AssertExpectations(t)
})
t.Run("waits while a service is still starting", func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).
Return(&container.Info{ID: "id", State: "running", Health: container.HealthStarting}, nil).Twice()
service.On("Inspect", mock.Anything).
Return(&container.Info{ID: "id", State: "running", Health: container.HealthHealthy}, nil).Once()
rc := newRunContext(0, &serviceContainer{name: "postgres", container: service})
require.NoError(t, rc.waitForServiceContainers()(context.Background()))
service.AssertExpectations(t)
})
t.Run("fails with the probe output when a service is unhealthy", func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).Return(&container.Info{
State: "running",
Health: container.HealthUnhealthy,
HealthOutput: "connection refused",
}, nil).Once()
service.On("DumpLogs", mock.Anything).Return(nil).Once()
rc := newRunContext(0, &serviceContainer{name: "postgres", container: service})
err := rc.waitForServiceContainers()(context.Background())
require.Error(t, err)
assert.Contains(t, err.Error(), "the service 'postgres' is unhealthy: connection refused")
service.AssertExpectations(t)
})
t.Run("lets the steps run when a service exits without a healthcheck", func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).
Return(&container.Info{State: "exited", ExitCode: 2, Health: container.HealthNone}, nil).Once()
rc := newRunContext(0, &serviceContainer{name: "postgres", container: service})
require.NoError(t, rc.waitForServiceContainers()(context.Background()))
})
t.Run("proceeds when the container is gone", func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).
Return((*container.Info)(nil), container.ErrContainerNotFound).Once()
rc := newRunContext(0, &serviceContainer{name: "postgres", container: service})
require.NoError(t, rc.waitForServiceContainers()(context.Background()))
})
t.Run("fails right away when one service fails while another is still starting", func(t *testing.T) {
failing := &containerMock{}
failing.On("Inspect", mock.Anything).
Return(&container.Info{State: "running", Health: container.HealthUnhealthy}, nil)
failing.On("DumpLogs", mock.Anything).Return(nil).Once()
starting := &containerMock{}
starting.On("Inspect", mock.Anything).
Return(&container.Info{State: "running", Health: container.HealthStarting}, nil)
rc := newRunContext(10*time.Second,
&serviceContainer{name: "failing", container: failing},
&serviceContainer{name: "starting", container: starting})
done := make(chan error, 1)
go func() { done <- rc.waitForServiceContainers()(context.Background()) }()
select {
case err := <-done:
require.Error(t, err)
assert.Contains(t, err.Error(), "the service 'failing' is unhealthy")
case <-time.After(2 * time.Second):
t.Fatal("waitForServiceContainers did not fail fast; it waited for the starting service")
}
})
t.Run("gives up once the timeout expires", func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).
Return(&container.Info{State: "running", Health: container.HealthStarting}, nil)
rc := newRunContext(20*time.Millisecond, &serviceContainer{name: "postgres", container: service})
err := rc.waitForServiceContainers()(context.Background())
require.Error(t, err)
assert.Contains(t, err.Error(), "did not become healthy within")
})
t.Run("gives up with the same message when the deadline stops an inspect", func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).
Run(func(args mock.Arguments) { <-args.Get(0).(context.Context).Done() }).
Return((*container.Info)(nil), errors.New("inspect aborted"))
rc := newRunContext(20*time.Millisecond, &serviceContainer{name: "postgres", container: service})
err := rc.waitForServiceContainers()(context.Background())
require.Error(t, err)
assert.Contains(t, err.Error(), "did not become healthy within")
})
t.Run("does not wait when the timeout is negative", func(t *testing.T) {
service := &containerMock{}
// Still described once, so the `job.services` context is filled either way.
service.On("Inspect", mock.Anything).
Return(&container.Info{ID: "id", State: "running", Health: container.HealthStarting}, nil).Once()
svc := &serviceContainer{name: "postgres", container: service}
rc := newRunContext(-1, svc)
require.NoError(t, rc.waitForServiceContainers()(context.Background()))
service.AssertExpectations(t)
assert.Equal(t, "id", svc.info.ID)
})
t.Run("fails on an inspect error even when the timeout is negative", func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).Return((*container.Info)(nil), errors.New("daemon is gone")).Once()
rc := newRunContext(-1, &serviceContainer{name: "postgres", container: service})
err := rc.waitForServiceContainers()(context.Background())
require.Error(t, err)
assert.Contains(t, err.Error(), "failed to inspect service 'postgres'")
})
t.Run("is a no-op without services", func(t *testing.T) {
require.NoError(t, newRunContext(0).waitForServiceContainers()(context.Background()))
})
}
func TestReportUnstartedServices(t *testing.T) {
dead := &containerMock{}
dead.On("Inspect", mock.Anything).Return(&container.Info{ID: "dead-id", State: "exited", ExitCode: 1}, nil).Once()
dead.On("DumpLogs", mock.Anything).Return(nil).Once()
running := &containerMock{}
running.On("Inspect", mock.Anything).Return(&container.Info{ID: "run-id", State: "running"}, nil).Once()
rc := &RunContext{serviceContainers: []*serviceContainer{
{name: "postgres", container: dead},
{name: "redis", container: running},
}}
require.NoError(t, rc.reportUnstartedServices()(context.Background()))
dead.AssertExpectations(t)
running.AssertExpectations(t)
}
func TestGetJobContextReportsContainers(t *testing.T) {
rc := &RunContext{
jobNetworkName: "job-network",
jobContainerID: "job-container-id",
serviceContainers: []*serviceContainer{
{name: "postgres", info: &container.Info{ID: "svc-id", Ports: map[string]string{"5432": "49153"}}},
// A service that publishes no port reports an empty map, as GitHub does.
{name: "redis", info: &container.Info{ID: "redis-id", Ports: map[string]string{}}},
// A service that never reported is left out rather than reported as empty.
{name: "mailhog"},
},
}
jobContext := rc.getJobContext()
assert.Equal(t, "job-container-id", jobContext.Container.ID)
assert.Equal(t, "job-network", jobContext.Container.Network)
assert.Equal(t, map[string]model.JobService{
"postgres": {ID: "svc-id", Network: "job-network", Ports: map[string]string{"5432": "49153"}},
"redis": {ID: "redis-id", Network: "job-network", Ports: map[string]string{}},
}, jobContext.Services)
}
// A job that never started a container reports an empty context, not a placeholder.
func TestGetJobContextWithoutContainer(t *testing.T) {
jobContext := (&RunContext{}).getJobContext()
assert.Empty(t, jobContext.Container.ID)
assert.Empty(t, jobContext.Container.Network)
assert.Empty(t, jobContext.Services)
}
func TestImageOSFromImage(t *testing.T) { func TestImageOSFromImage(t *testing.T) {
for _, tc := range []struct { for _, tc := range []struct {
image string image string

View File

@@ -94,6 +94,7 @@ type Config struct {
InsecureSkipTLS bool // whether to skip verifying TLS certificate of the Gitea instance InsecureSkipTLS bool // whether to skip verifying TLS certificate of the Gitea instance
MaxParallel int // max parallel jobs to run across all workflows (0 = no limit, uses CPU count) MaxParallel int // max parallel jobs to run across all workflows (0 = no limit, uses CPU count)
AllocatePTY bool // allocate a pseudo-TTY for each step's process AllocatePTY bool // allocate a pseudo-TTY for each step's process
ServiceReadyTimeout time.Duration // how long a job waits for its service containers to report healthy (0 uses the default)
RunnerName string // name this runner registered with, reported as `runner.name`, defaults to the hostname RunnerName string // name this runner registered with, reported as `runner.name`, defaults to the hostname
JobStartedHook string // script run inside the job environment before the job's first step; ACTIONS_RUNNER_HOOK_JOB_STARTED is read from Env when empty JobStartedHook string // script run inside the job environment before the job's first step; ACTIONS_RUNNER_HOOK_JOB_STARTED is read from Env when empty
JobCompletedHook string // script run inside the job environment after the job's last step; ACTIONS_RUNNER_HOOK_JOB_COMPLETED is read from Env when empty JobCompletedHook string // script run inside the job environment after the job's last step; ACTIONS_RUNNER_HOOK_JOB_COMPLETED is read from Env when empty

View File

@@ -165,9 +165,22 @@ func runStepExecutor(step step, stage stepStage, executor common.Executor) commo
} }
_ = rc.JobContainer.Copy(actPath, files...)(ctx) _ = rc.JobContainer.Copy(actPath, files...)(ctx)
// The command handler needs the step's env to judge ACTIONS_ALLOW_UNSECURE_COMMANDS.
// Cloned: the step executor keeps writing to its own env map after this point, on a
// different goroutine from the command handler that reads it.
rc.setCurrentStepEnv(maps0.Clone(*step.getEnv()))
defer rc.setCurrentStepEnv(nil)
_ = rc.takeUnsecureCommandError() // a refusal from before any step belongs to no step
timeoutctx, cancelTimeOut := evaluateStepTimeout(ctx, rc.ExprEval, stepModel) timeoutctx, cancelTimeOut := evaluateStepTimeout(ctx, rc.ExprEval, stepModel)
defer cancelTimeOut() defer cancelTimeOut()
err = executor(timeoutctx) err = executor(timeoutctx)
// Always take it, so the job-scoped error cannot leak onto a later step. A refusal
// fails the step as it does on GitHub, but the executor's own error wins.
insecureErr := rc.takeUnsecureCommandError()
if err == nil {
err = insecureErr
}
if err == nil { if err == nil {
logger.WithField("stepResult", stepResult.Outcome).Infof("Success - %s %s", stage, stepString) logger.WithField("stepResult", stepResult.Outcome).Infof("Success - %s %s", stage, stepString)
@@ -181,7 +194,7 @@ func runStepExecutor(step step, stage stepStage, executor common.Executor) commo
} }
if continueOnError { if continueOnError {
logger.Errorf("##[error]%s", escapeCommandData(err.Error())) logger.Errorf("##[error]%s", EscapeCommandData(err.Error()))
logger.Infof("Failed but continue next step") logger.Infof("Failed but continue next step")
err = nil err = nil
stepResult.Conclusion = model.StepStatusSuccess stepResult.Conclusion = model.StepStatusSuccess

View File

@@ -18,6 +18,7 @@ import (
"gitea.com/gitea/runner/act/model" "gitea.com/gitea/runner/act/model"
"github.com/kballard/go-shellquote" "github.com/kballard/go-shellquote"
"github.com/sirupsen/logrus"
yaml "go.yaml.in/yaml/v4" yaml "go.yaml.in/yaml/v4"
) )
@@ -63,7 +64,7 @@ func (sr *stepRun) printRunScriptActionDetails(ctx context.Context) {
normalized := strings.TrimRight(strings.ReplaceAll(sr.interpolatedScript, "\r\n", "\n"), "\n") normalized := strings.TrimRight(strings.ReplaceAll(sr.interpolatedScript, "\r\n", "\n"), "\n")
rawLogger.Infof("::group::Run %s", escapeCommandData(sr.runScriptGroupTitle(normalized))) rawLogger.Infof("::group::Run %s", EscapeCommandData(sr.runScriptGroupTitle(normalized)))
if normalized != "" { if normalized != "" {
for line := range strings.SplitSeq(normalized, "\n") { for line := range strings.SplitSeq(normalized, "\n") {
@@ -90,12 +91,12 @@ func printRunActionHeader(ctx context.Context, step *model.Step, env map[string]
if step.Name != "" { if step.Name != "" {
title = step.Name title = step.Name
} }
rawLogger.Infof("::group::Run %s", escapeCommandData(title)) rawLogger.Infof("::group::Run %s", EscapeCommandData(title))
if len(step.With) > 0 { if len(step.With) > 0 {
rawLogger.Infof("with:") rawLogger.Infof("with:")
for _, k := range slices.Sorted(maps.Keys(step.With)) { for _, k := range slices.Sorted(maps.Keys(step.With)) {
rawLogger.Infof(" %s: %s", k, step.With[k]) logKeyedValue(rawLogger, k, step.With[k])
} }
} }
@@ -129,7 +130,17 @@ func printStepEnvBlock(ctx context.Context, step *model.Step, env map[string]str
if caseInsensitive { if caseInsensitive {
lookupKey = strings.ToUpper(k) lookupKey = strings.ToUpper(k)
} }
rawLogger.Infof(" %s: %s", k, envLookup[lookupKey]) logKeyedValue(rawLogger, k, envLookup[lookupKey])
}
}
// logKeyedValue prints one row per line of value: Gitea stores one log row per line, so an
// embedded newline would reach the user as a literal "\n".
func logKeyedValue(rawLogger *logrus.Entry, key, value string) {
lines := strings.Split(strings.ReplaceAll(value, "\r\n", "\n"), "\n")
rawLogger.Infof(" %s: %s", key, lines[0])
for _, line := range lines[1:] {
rawLogger.Infof(" %s", line)
} }
} }

View File

@@ -6,6 +6,7 @@ package runner
import ( import (
"context" "context"
"errors"
"testing" "testing"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
@@ -14,6 +15,7 @@ import (
log "github.com/sirupsen/logrus" log "github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock" "github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
yaml "go.yaml.in/yaml/v4" yaml "go.yaml.in/yaml/v4"
) )
@@ -354,3 +356,48 @@ func TestIsContinueOnError(t *testing.T) {
assertObject.False(continueOnError) assertObject.False(continueOnError)
assertObject.Error(err) assertObject.Error(err)
} }
// A refused ::set-env::/::add-path:: records a job-scoped error. When the step that
// produced it also fails on its own, the refusal must be cleared at the step boundary, so
// it fails only that step and never leaks onto a later step that runs anyway (if: always()).
func TestRunStepExecutorDoesNotLeakRefusalToNextStep(t *testing.T) {
cm := &containerMock{}
noop := func(context.Context) error { return nil }
cm.On("Copy", mock.Anything, mock.Anything).Return(noop)
cm.On("UpdateFromEnv", mock.Anything, mock.Anything).Return(noop)
rc := &RunContext{
Config: &Config{Env: map[string]string{}},
Run: &model.Run{
JobID: "1",
Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}},
},
Env: map[string]string{},
StepResults: map[string]*model.StepResult{},
JobContainer: cm,
}
rc.ExprEval = rc.NewExpressionEvaluator(context.Background())
// Dryrun skips reading the path file back from the (mocked) container.
ctx := common.WithDryrun(context.Background(), true)
// A refusal parsed out of the job container's own output belongs to no step, so the
// first step must not be failed by it.
rc.commandHandler(ctx)("::set-env name=setup::y\n")
stepSetup := &stepRun{RunContext: rc, Step: &model.Step{ID: "setup"}, env: map[string]string{}}
require.NoError(t, runStepExecutor(stepSetup, stepStageMain, func(context.Context) error { return nil })(ctx))
// Step A refuses a ::set-env:: and then fails on its own.
stepA := &stepRun{RunContext: rc, Step: &model.Step{ID: "a"}, env: map[string]string{}}
errA := runStepExecutor(stepA, stepStageMain, func(context.Context) error {
rc.commandHandler(ctx)("::set-env name=x::y\n")
return errors.New("boom")
})(ctx)
// The step fails with its own error, not the refusal.
require.ErrorContains(t, errA, "boom")
// Step B runs despite step A's failure (if: always()) and issues no unsecure command;
// it must not inherit step A's refusal.
stepB := &stepRun{RunContext: rc, Step: &model.Step{ID: "b", If: yaml.Node{Value: "always()"}}, env: map[string]string{}}
errB := runStepExecutor(stepB, stepStageMain, func(context.Context) error { return nil })(ctx)
require.NoError(t, errB)
}

View File

@@ -3,6 +3,7 @@ jobs:
_: _:
runs-on: ubuntu-latest runs-on: ubuntu-latest
env: env:
ACTIONS_ALLOW_UNSECURE_COMMANDS: 'true'
MYGLOBALENV3: myglobalval3 MYGLOBALENV3: myglobalval3
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4

View File

@@ -4,6 +4,8 @@ on: push
jobs: jobs:
build: build:
runs-on: ubuntu-latest runs-on: ubuntu-latest
env:
ACTIONS_ALLOW_UNSECURE_COMMANDS: 'true'
steps: steps:
- name: TEST set-env - name: TEST set-env
run: echo "::set-env name=foo::bar" run: echo "::set-env name=foo::bar"

View File

@@ -15,3 +15,9 @@ jobs:
echo "id: ${{ job.services.postgres.id }}" echo "id: ${{ job.services.postgres.id }}"
echo "network: ${{ job.services.postgres.network }}" echo "network: ${{ job.services.postgres.network }}"
echo "ports: ${{ job.services.postgres.ports }}" echo "ports: ${{ job.services.postgres.ports }}"
- name: The job context describes the started containers
run: |
test -n "${{ job.container.id }}"
test -n "${{ job.services.postgres.id }}"
test -n "${{ job.services.postgres.ports['80'] }}"
test "${{ job.services.postgres.network }}" = "${{ job.container.network }}"

View File

@@ -16,7 +16,7 @@ the runner as a background service on a systemd host.
`.runner` file ends up in the working directory: `.runner` file ends up in the working directory:
```bash ```bash
sudo -u gitea-runner gitea-runner generate-config > /etc/gitea-runner/config.yaml sudo -u gitea-runner gitea-runner config generate > /etc/gitea-runner/config.yaml
cd /var/lib/gitea-runner cd /var/lib/gitea-runner
sudo -u gitea-runner gitea-runner register --config /etc/gitea-runner/config.yaml sudo -u gitea-runner gitea-runner register --config /etc/gitea-runner/config.yaml
``` ```

View File

@@ -52,7 +52,7 @@ export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock
- Generate a `gitea-runner` configuration file in the data directory. Edit the file to adjust for the system. - Generate a `gitea-runner` configuration file in the data directory. Edit the file to adjust for the system.
```bash ```bash
gitea-runner generate-config >/home/rootless/gitea-runner/config gitea-runner config generate >/home/rootless/gitea-runner/config
``` ```
- Create a new user-level`systemd` unit file as `/home/rootless/.config/systemd/user/gitea-runner.service` with the following contents: - Create a new user-level`systemd` unit file as `/home/rootless/.config/systemd/user/gitea-runner.service` with the following contents:

2
go.mod
View File

@@ -37,6 +37,7 @@ require (
go.etcd.io/bbolt v1.5.0 go.etcd.io/bbolt v1.5.0
go.yaml.in/yaml/v4 v4.0.0-rc.3 go.yaml.in/yaml/v4 v4.0.0-rc.3
golang.org/x/net v0.57.0 golang.org/x/net v0.57.0
golang.org/x/sync v0.22.0
golang.org/x/sys v0.47.0 golang.org/x/sys v0.47.0
golang.org/x/term v0.45.0 golang.org/x/term v0.45.0
golang.org/x/text v0.40.0 golang.org/x/text v0.40.0
@@ -104,7 +105,6 @@ require (
go.opentelemetry.io/otel/trace v1.44.0 // indirect go.opentelemetry.io/otel/trace v1.44.0 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/crypto v0.54.0 // indirect golang.org/x/crypto v0.54.0 // indirect
golang.org/x/sync v0.22.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect
) )

View File

@@ -5,10 +5,8 @@ package cmd
import ( import (
"context" "context"
"fmt"
"os" "os"
"gitea.com/gitea/runner/internal/pkg/config"
"gitea.com/gitea/runner/internal/pkg/ver" "gitea.com/gitea/runner/internal/pkg/ver"
"github.com/spf13/cobra" "github.com/spf13/cobra"
@@ -23,7 +21,7 @@ func Execute(ctx context.Context) {
SilenceUsage: true, SilenceUsage: true,
} }
configFile := "" configFile := ""
rootCmd.PersistentFlags().StringVarP(&configFile, "config", "c", "", "Config file path") rootCmd.PersistentFlags().StringVarP(&configFile, "config", "c", "", "Config file path. `config` subcommands fall back to config.yaml in the working directory or next to the executable")
// ./gitea-runner register // ./gitea-runner register
var regArgs registerArgs var regArgs registerArgs
@@ -61,14 +59,12 @@ func Execute(ctx context.Context) {
rootCmd.AddCommand(loadBugReportCmd()) rootCmd.AddCommand(loadBugReportCmd())
// ./gitea-runner config // ./gitea-runner config
rootCmd.AddCommand(&cobra.Command{ rootCmd.AddCommand(loadConfigCmd(&configFile))
Use: "generate-config",
Short: "Generate an example config file", // ./gitea-runner generate-config
Args: cobra.MaximumNArgs(0), generateConfigCmd := loadGenerateConfigCmd("generate-config")
Run: func(_ *cobra.Command, _ []string) { generateConfigCmd.Deprecated = "use `config generate` instead."
fmt.Printf("%s", config.Example) rootCmd.AddCommand(generateConfigCmd)
},
})
// ./gitea-runner cache-server // ./gitea-runner cache-server
var cacheArgs cacheServerArgs var cacheArgs cacheServerArgs

116
internal/app/cmd/config.go Normal file
View File

@@ -0,0 +1,116 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package cmd
import (
"fmt"
"os"
"path/filepath"
"slices"
"strings"
"gitea.com/gitea/runner/internal/pkg/config"
"github.com/spf13/cobra"
)
func loadConfigCmd(configFile *string) *cobra.Command {
configCmd := &cobra.Command{
Use: "config",
Short: "Generate, read and edit config files",
Args: cobra.MaximumNArgs(0),
RunE: func(cmd *cobra.Command, _ []string) error {
return cmd.Help()
},
}
configCmd.AddCommand(loadGenerateConfigCmd("generate"))
configCmd.AddCommand(&cobra.Command{
Use: "get <key>",
Short: "Print the value of a config key",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
file, err := resolveConfigFile(cmd, configFile)
if err != nil {
return err
}
value, err := config.GetValue(file, args[0])
if err != nil {
return err
}
fmt.Fprintln(cmd.OutOrStdout(), value)
return nil
},
})
for _, sub := range []struct {
use string
short string
edit func(file, key string, values ...string) error
}{
{"set <key> <value>...", "Set the value of a config key", config.SetValue},
{"add <key> <value>...", "Append values to a list config key", config.AddValue},
{"remove <key> <value>...", "Remove values from a list config key", config.RemoveValue},
} {
valueCmd := &cobra.Command{
Use: sub.use,
Short: sub.short,
Args: cobra.MinimumNArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
file, err := resolveConfigFile(cmd, configFile)
if err != nil {
return err
}
return sub.edit(file, args[0], args[1:]...)
},
}
valueCmd.Flags().SetInterspersed(false) // so a value such as `--cpus 2` is not parsed as a flag
configCmd.AddCommand(valueCmd)
}
return configCmd
}
func loadGenerateConfigCmd(use string) *cobra.Command {
return &cobra.Command{
Use: use,
Short: "Generate an example config file",
Args: cobra.MaximumNArgs(0),
Run: func(cmd *cobra.Command, _ []string) {
fmt.Fprintf(cmd.OutOrStdout(), "%s", config.Example)
},
}
}
var defaultConfigFileNames = []string{"config.yaml", "config.yml"}
func resolveConfigFile(cmd *cobra.Command, configFile *string) (string, error) {
if *configFile != "" {
return *configFile, nil
}
var dirs []string
if wd, err := os.Getwd(); err == nil {
dirs = append(dirs, wd)
}
if exe, err := os.Executable(); err == nil {
if dir := filepath.Dir(exe); !slices.Contains(dirs, dir) {
dirs = append(dirs, dir)
}
}
for _, dir := range dirs {
for _, name := range defaultConfigFileNames {
candidate := filepath.Join(dir, name)
if stat, err := os.Stat(candidate); err == nil && !stat.IsDir() {
fmt.Fprintf(cmd.ErrOrStderr(), "using config file %q\n", candidate)
return candidate, nil
}
}
}
return "", fmt.Errorf("no %s found in %s, pass one with --config",
strings.Join(defaultConfigFileNames, " or "), strings.Join(dirs, " or "))
}

View File

@@ -0,0 +1,75 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package cmd
import (
"bytes"
"os"
"path/filepath"
"testing"
"gitea.com/gitea/runner/internal/pkg/config"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func runConfigCmd(t *testing.T, configFile string, args ...string) (string, string, error) {
t.Helper()
out, errOut := &bytes.Buffer{}, &bytes.Buffer{}
cmd := loadConfigCmd(&configFile)
cmd.SetOut(out)
cmd.SetErr(errOut)
cmd.SetArgs(args)
err := cmd.Execute()
return out.String(), errOut.String(), err
}
func TestConfigCmdGeneratePrintsTheExample(t *testing.T) {
out, _, err := runConfigCmd(t, "", "generate")
require.NoError(t, err)
assert.Equal(t, string(config.Example), out)
}
// The subcommands only wire arguments through, so one pass over all of them is enough.
func TestConfigCmdEditsTheFile(t *testing.T) {
file := filepath.Join(t.TempDir(), "config.yaml")
require.NoError(t, os.WriteFile(file, []byte("runner:\n labels:\n - self-hosted\n"), 0o600))
_, _, err := runConfigCmd(t, file, "set", "container.options", "--cpus 2")
require.NoError(t, err)
_, _, err = runConfigCmd(t, file, "add", "runner.labels", "ubuntu:docker://node:22")
require.NoError(t, err)
_, _, err = runConfigCmd(t, file, "remove", "runner.labels", "self-hosted")
require.NoError(t, err)
out, _, err := runConfigCmd(t, file, "get", "runner.labels")
require.NoError(t, err)
assert.Equal(t, "ubuntu:docker://node:22\n", out)
out, _, err = runConfigCmd(t, file, "get", "container.options")
require.NoError(t, err)
assert.Equal(t, "--cpus 2\n", out)
}
func TestConfigCmdResolvesTheConfigFile(t *testing.T) {
t.Run("falls back to the working directory", func(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.yaml"), []byte("runner:\n capacity: 2\n"), 0o600))
t.Chdir(dir)
out, errOut, err := runConfigCmd(t, "", "get", "runner.capacity")
require.NoError(t, err)
assert.Equal(t, "2\n", out)
assert.Contains(t, errOut, "using config file")
})
t.Run("reports that none was found", func(t *testing.T) {
t.Chdir(t.TempDir())
_, _, err := runConfigCmd(t, "", "set", "runner.capacity", "4")
require.Error(t, err)
assert.Contains(t, err.Error(), "--config")
})
}

View File

@@ -510,6 +510,7 @@ func (r *Runner) run(ctx context.Context, task *runnerv1.Task, reporter *report.
RunnerUUID: r.uuid, RunnerUUID: r.uuid,
}, },
ContainerOptions: r.cfg.Container.Options, ContainerOptions: r.cfg.Container.Options,
ServiceReadyTimeout: r.cfg.Container.ServiceReadyTimeout,
ContainerDaemonSocket: r.cfg.Container.DockerHost, ContainerDaemonSocket: r.cfg.Container.DockerHost,
Privileged: r.cfg.Container.Privileged, Privileged: r.cfg.Container.Privileged,
DefaultActionInstance: r.getDefaultActionsURL(task), DefaultActionInstance: r.getDefaultActionsURL(task),
@@ -607,7 +608,8 @@ func (r *Runner) registerExternalCacheJob(token string, cred artifactcache.JobCr
}); err != nil { }); err != nil {
log.Warnf("cache external_server register failed (%s): %v", base, err) log.Warnf("cache external_server register failed (%s): %v", base, err)
if reporter != nil { if reporter != nil {
reporter.Logf("::warning::cache external_server register failed (%s): %v — cache requests from this job will be unauthenticated and likely return 401", base, err) reporter.Logf("::warning::%s", runner.EscapeCommandData(fmt.Sprintf(
"cache external_server register failed (%s): %v — cache requests from this job will be unauthenticated and likely return 401", base, err)))
} }
} else { } else {
resultsURL, _ = body["results_url"].(string) // absent from a server too old to forward resultsURL, _ = body["results_url"].(string) // absent from a server too old to forward
@@ -617,7 +619,8 @@ func (r *Runner) registerExternalCacheJob(token string, cred artifactcache.JobCr
map[string]any{"token": token}); err != nil { map[string]any{"token": token}); err != nil {
log.Warnf("cache external_server revoke failed (%s): %v", base, err) log.Warnf("cache external_server revoke failed (%s): %v", base, err)
if reporter != nil { if reporter != nil {
reporter.Logf("::warning::cache external_server revoke failed (%s): %v", base, err) reporter.Logf("::warning::%s", runner.EscapeCommandData(fmt.Sprintf(
"cache external_server revoke failed (%s): %v", base, err)))
} }
} }
}, resultsURL }, resultsURL

View File

@@ -1,7 +1,7 @@
# Example configuration file, it's safe to copy this as the default config file without any modification. # Example configuration file, it's safe to copy this as the default config file without any modification.
# You don't have to copy this file to your instance, # You don't have to copy this file to your instance,
# just run `./gitea-runner generate-config > config.yaml` to generate a config file. # just run `./gitea-runner config generate > config.yaml` to generate a config file.
# Logging for the runner process itself (messages printed to stderr). # Logging for the runner process itself (messages printed to stderr).
# This does not control how workflow step output is streamed to the Gitea UI; # This does not control how workflow step output is streamed to the Gitea UI;
@@ -176,7 +176,7 @@ container:
# network is labelled com.gitea.runner.uuid=<this runner's uuid>, which is how the idle # network is labelled com.gitea.runner.uuid=<this runner's uuid>, which is how the idle
# cleanup tells its own leftovers apart from those of other runners on the same daemon. # cleanup tells its own leftovers apart from those of other runners on the same daemon.
network_create_options: network_create_options:
enable_ipv4: true # Omit to use Docker's default (IPv4 enabled). Set false to disable IPv4. enable_ipv4: true # Omit to use Docker's default (IPv4 enabled). Set false to disable IPv4.
enable_ipv6: false # Omit to use Docker's default (IPv6 disabled). Enabling it requires dockerd started with --ipv6. enable_ipv6: false # Omit to use Docker's default (IPv6 disabled). Enabling it requires dockerd started with --ipv6.
# Whether to use privileged mode or not when launching task containers (privileged mode is required for Docker-in-Docker). # Whether to use privileged mode or not when launching task containers (privileged mode is required for Docker-in-Docker).
privileged: false privileged: false
@@ -226,6 +226,9 @@ container:
# from the DinD daemon's filesystem. When enabled, ensure the workspace parent # from the DinD daemon's filesystem. When enabled, ensure the workspace parent
# directory is also mounted into the runner container and listed in valid_volumes. # directory is also mounted into the runner container and listed in valid_volumes.
bind_workdir: false bind_workdir: false
# How long a job waits for a service container that declares a healthcheck to become
# healthy. A negative value (e.g. -1s) starts the steps without waiting.
service_ready_timeout: 5m
host: host:
# The parent directory of a job's working directory. # The parent directory of a job's working directory.

View File

@@ -92,6 +92,7 @@ type Container struct {
RequireDocker bool `yaml:"require_docker"` // Always require a reachable docker daemon, even if not required by runner RequireDocker bool `yaml:"require_docker"` // Always require a reachable docker daemon, even if not required by runner
DockerTimeout time.Duration `yaml:"docker_timeout"` // Timeout to wait for the docker daemon to be reachable, if docker is required by require_docker or runner DockerTimeout time.Duration `yaml:"docker_timeout"` // Timeout to wait for the docker daemon to be reachable, if docker is required by require_docker or runner
BindWorkdir bool `yaml:"bind_workdir"` // BindWorkdir binds the workspace to the host filesystem instead of using Docker volumes. Required for DinD when jobs use docker compose with bind mounts. BindWorkdir bool `yaml:"bind_workdir"` // BindWorkdir binds the workspace to the host filesystem instead of using Docker volumes. Required for DinD when jobs use docker compose with bind mounts.
ServiceReadyTimeout time.Duration `yaml:"service_ready_timeout"` // ServiceReadyTimeout bounds how long a job waits for a service container that declares a healthcheck to report healthy. Negative disables waiting.
} }
type ContainerNetworkCreateOptions struct { type ContainerNetworkCreateOptions struct {

530
internal/pkg/config/edit.go Normal file
View File

@@ -0,0 +1,530 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package config
import (
"bytes"
"errors"
"fmt"
"os"
"path/filepath"
"reflect"
"slices"
"strconv"
"strings"
"time"
"go.yaml.in/yaml/v4"
)
type fieldKind int
const (
kindScalar fieldKind = iota
kindSequence
kindSection
)
var durationType = reflect.TypeFor[time.Duration]()
// GetValue renders a flat list or mapping one entry per line, and anything nested as YAML.
func GetValue(file, path string) (string, error) {
session, err := loadForEdit(file, path)
if err != nil {
return "", err
}
node, err := lookupNode(session.root, session.segments)
if err != nil {
return "", err
}
return renderNode(node)
}
func SetValue(file, path string, values ...string) error {
session, err := loadForEdit(file, path)
if err != nil {
return err
}
var replacement *yaml.Node
switch session.field.kind {
case kindSequence:
if len(values) == 0 {
return fmt.Errorf("%q needs at least one value", path)
}
items, err := session.scalars(values)
if err != nil {
return err
}
replacement = &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq", Content: items}
case kindScalar:
if len(values) != 1 {
return fmt.Errorf("%q takes exactly one value", path)
}
items, err := session.scalars(values)
if err != nil {
return err
}
replacement = items[0]
default:
return fmt.Errorf("%q is a section, set one of its keys instead", path)
}
node, err := ensureNode(session.root, session.segments)
if err != nil {
return err
}
replaceNode(node, replacement)
return session.write()
}
func AddValue(file, path string, values ...string) error {
session, err := loadSequenceEdit(file, path, values)
if err != nil {
return err
}
items, err := session.scalars(values)
if err != nil {
return err
}
node, err := ensureNode(session.root, session.segments)
if err != nil {
return err
}
if node.Kind != yaml.SequenceNode {
replaceNode(node, &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"})
}
for _, item := range items {
if indexOfValue(node, item.Value) >= 0 {
return fmt.Errorf("%s already contains %q", path, item.Value)
}
node.Content = append(node.Content, item)
}
return session.write()
}
func RemoveValue(file, path string, values ...string) error {
session, err := loadSequenceEdit(file, path, values)
if err != nil {
return err
}
items, err := session.scalars(values)
if err != nil {
return err
}
node, err := lookupNode(session.root, session.segments)
if err != nil {
return err
}
if node.Kind != yaml.SequenceNode {
return fmt.Errorf("%s is not a list in %q", path, file)
}
for _, item := range items {
index := indexOfValue(node, item.Value)
if index < 0 {
return fmt.Errorf("%s does not contain %q", path, item.Value)
}
node.Content = slices.Delete(node.Content, index, index+1)
}
return session.write()
}
func indexOfValue(seq *yaml.Node, value string) int {
for i, item := range seq.Content {
if item.Kind == yaml.ScalarNode && item.Value == value {
return i
}
}
return -1
}
// replaceNode assigns field by field, as *node = *with would drop the comments attached to node.
func replaceNode(node, with *yaml.Node) {
node.Kind, node.Tag, node.Style, node.Value, node.Content = with.Kind, with.Tag, with.Style, with.Value, with.Content
}
type editSession struct {
file string
path string
original []byte
root *yaml.Node
field *fieldInfo
segments []string
}
// loadForEdit validates the path and parses the file, so every caller fails before anything is written.
func loadForEdit(file, path string) (*editSession, error) {
if path == "" {
return nil, errors.New("no config key given")
}
segments := strings.Split(path, ".")
field, err := resolvePath(segments)
if err != nil {
return nil, err
}
content, err := os.ReadFile(file)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil, fmt.Errorf("config file %q does not exist, create one with `config generate`", file)
}
return nil, err
}
var root yaml.Node
if err := yaml.Unmarshal(content, &root); err != nil {
return nil, fmt.Errorf("parse config file %q: %w", file, err)
}
if root.Kind == 0 || len(root.Content) == 0 {
root = yaml.Node{
Kind: yaml.DocumentNode,
Content: []*yaml.Node{{Kind: yaml.MappingNode, Tag: "!!map"}},
}
}
if root.Content[0].Kind != yaml.MappingNode {
return nil, fmt.Errorf("config file %q is not a YAML mapping", file)
}
return &editSession{file: file, path: path, original: content, root: &root, field: field, segments: segments}, nil
}
func loadSequenceEdit(file, path string, values []string) (*editSession, error) {
session, err := loadForEdit(file, path)
if err != nil {
return nil, err
}
if session.field.kind != kindSequence {
return nil, fmt.Errorf("%q is not a list, use `config set` instead", path)
}
if len(values) == 0 {
return nil, fmt.Errorf("%q needs at least one value", path)
}
return session, nil
}
func (s *editSession) scalars(values []string) ([]*yaml.Node, error) {
nodes := make([]*yaml.Node, 0, len(values))
for _, value := range values {
node, err := scalarNode(s.field.typ, value)
if err != nil {
return nil, fmt.Errorf("%s: %w", s.path, err)
}
nodes = append(nodes, node)
}
return nodes, nil
}
func lookupNode(root *yaml.Node, segments []string) (*yaml.Node, error) {
node := root.Content[0]
for i, segment := range segments {
if node.Kind != yaml.MappingNode {
return nil, fmt.Errorf("%q is not set", strings.Join(segments[:i], "."))
}
value := mappingValue(node, segment)
if value == nil {
return nil, fmt.Errorf("%q is not set", strings.Join(segments[:i+1], "."))
}
node = value
}
return node, nil
}
func ensureNode(root *yaml.Node, segments []string) (*yaml.Node, error) {
node := root.Content[0]
for i, segment := range segments {
if node.Kind != yaml.MappingNode {
if node.Kind == yaml.ScalarNode && node.Tag == "!!null" {
node.Kind, node.Tag, node.Style, node.Value = yaml.MappingNode, "!!map", 0, ""
} else {
return nil, fmt.Errorf("%q is not a section", strings.Join(segments[:i], "."))
}
}
value := mappingValue(node, segment)
if value == nil {
value = &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!null"}
node.Content = append(node.Content,
&yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: segment},
value)
}
node = value
}
return node, nil
}
func mappingValue(mapping *yaml.Node, key string) *yaml.Node {
for i := 0; i+1 < len(mapping.Content); i += 2 {
if mapping.Content[i].Value == key {
return mapping.Content[i+1]
}
}
return nil
}
func renderNode(node *yaml.Node) (string, error) {
if !allScalars(node.Content) {
encoded, err := encodeYAML(node)
if err != nil {
return "", err
}
return strings.TrimRight(string(encoded), "\n"), nil
}
switch node.Kind {
case yaml.SequenceNode:
lines := make([]string, 0, len(node.Content))
for _, item := range node.Content {
lines = append(lines, item.Value)
}
return strings.Join(lines, "\n"), nil
case yaml.MappingNode:
lines := make([]string, 0, len(node.Content)/2)
for i := 0; i+1 < len(node.Content); i += 2 {
lines = append(lines, node.Content[i].Value+"="+node.Content[i+1].Value)
}
return strings.Join(lines, "\n"), nil
default:
return node.Value, nil
}
}
func allScalars(nodes []*yaml.Node) bool {
for _, node := range nodes {
if node.Kind != yaml.ScalarNode {
return false
}
}
return true
}
func encodeYAML(node *yaml.Node) ([]byte, error) {
var buf bytes.Buffer
encoder := yaml.NewEncoder(&buf)
encoder.SetIndent(2)
if err := encoder.Encode(node); err != nil {
return nil, err
}
if err := encoder.Close(); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
// restoreBlankLines re-inserts the blank lines between top-level sections that the encoder drops.
func restoreBlankLines(original, generated []byte) []byte {
spaced := map[string]bool{}
blank := false
for line := range strings.Lines(string(original)) {
line = strings.TrimRight(line, "\r\n")
switch {
case strings.TrimSpace(line) == "":
blank = true
case strings.HasPrefix(line, "#"): // the block belongs to the key below it
default:
if key, ok := topLevelKey(line); ok && blank {
spaced[key] = true
}
blank = false
}
}
var out []string
for line := range strings.Lines(string(generated)) {
line = strings.TrimRight(line, "\r\n")
if key, ok := topLevelKey(line); ok && spaced[key] {
start := len(out)
for start > 0 && strings.HasPrefix(out[start-1], "#") {
start--
}
if start > 0 && strings.TrimSpace(out[start-1]) != "" {
out = slices.Insert(out, start, "")
}
}
out = append(out, line)
}
if bytes.HasSuffix(generated, []byte("\n")) {
out = append(out, "")
}
return []byte(strings.Join(out, "\n"))
}
func topLevelKey(line string) (string, bool) {
if line == "" || line[0] == ' ' || line[0] == '\t' || line[0] == '#' || line[0] == '-' {
return "", false
}
key, _, ok := strings.Cut(line, ":")
return key, ok
}
func (s *editSession) write() error {
generated, err := encodeYAML(s.root)
if err != nil {
return err
}
// A file the runner already refused to load stays the user's to fix, only a regression is rejected.
if err := yaml.Unmarshal(generated, &Config{}); err != nil && yaml.Unmarshal(s.original, &Config{}) == nil {
return fmt.Errorf("the edit would produce a config the runner cannot load: %w", err)
}
content := restoreBlankLines(s.original, generated)
if bytes.Contains(s.original, []byte("\r\n")) { // the encoder only ever emits LF
content = bytes.ReplaceAll(content, []byte("\n"), []byte("\r\n"))
}
file := s.file
if resolved, err := filepath.EvalSymlinks(file); err == nil {
file = resolved // keeps a config linked in from elsewhere intact
}
var info os.FileInfo
mode := os.FileMode(0o600)
if stat, err := os.Stat(file); err == nil {
info, mode = stat, stat.Mode().Perm()
}
temp, err := os.CreateTemp(filepath.Dir(file), filepath.Base(file)+".*.tmp")
if err != nil {
return err
}
defer os.Remove(temp.Name())
if _, err := temp.Write(content); err != nil {
temp.Close()
return err
}
if err := temp.Sync(); err != nil {
temp.Close()
return err
}
if err := temp.Close(); err != nil {
return err
}
if info != nil { // before the chmod, as a chown can clear mode bits
if err := preserveOwner(temp.Name(), info); err != nil {
return err
}
}
if err := os.Chmod(temp.Name(), mode); err != nil {
return err
}
return os.Rename(temp.Name(), file)
}
type fieldInfo struct {
kind fieldKind
typ reflect.Type // the element type for a sequence
}
// resolvePath walks the Config struct through the yaml tags of a dotted path.
func resolvePath(segments []string) (*fieldInfo, error) {
typ := reflect.TypeFor[Config]()
for i, segment := range segments {
switch typ.Kind() {
case reflect.Struct:
field, ok := fieldByYAMLName(typ, segment)
if !ok {
return nil, fmt.Errorf("unknown config key %q, valid keys here: %s",
strings.Join(segments[:i+1], "."), strings.Join(yamlNames(typ), ", "))
}
typ = field.Type
case reflect.Map:
// The segment names a user-defined entry, so the walk ends here.
if i != len(segments)-1 {
return nil, fmt.Errorf("%q has no sub-keys", strings.Join(segments[:i+1], "."))
}
return &fieldInfo{kind: kindScalar, typ: typ.Elem()}, nil
default:
return nil, fmt.Errorf("%q is a value, not a section", strings.Join(segments[:i], "."))
}
}
switch typ.Kind() {
case reflect.Slice:
return &fieldInfo{kind: kindSequence, typ: typ.Elem()}, nil
case reflect.Map, reflect.Struct:
return &fieldInfo{kind: kindSection}, nil
default:
return &fieldInfo{kind: kindScalar, typ: typ}, nil
}
}
func fieldByYAMLName(typ reflect.Type, name string) (reflect.StructField, bool) {
for field := range typ.Fields() {
if yamlName(field) == name {
return field, true
}
}
return reflect.StructField{}, false
}
func yamlNames(typ reflect.Type) []string {
names := make([]string, 0, typ.NumField())
for field := range typ.Fields() {
if name := yamlName(field); name != "-" {
names = append(names, name)
}
}
slices.Sort(names)
return names
}
func yamlName(field reflect.StructField) string {
name, _, _ := strings.Cut(field.Tag.Get("yaml"), ",")
if name == "" {
return strings.ToLower(field.Name)
}
return name
}
// scalarNode types the value, so a bad one is reported instead of landing in the file as a string.
func scalarNode(typ reflect.Type, value string) (*yaml.Node, error) {
if typ.Kind() == reflect.Pointer {
typ = typ.Elem()
}
if typ == durationType {
duration, err := time.ParseDuration(value)
if err != nil {
return nil, fmt.Errorf("%q is not a duration such as 30s, 5m or 3h", value)
}
return &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: duration.String()}, nil
}
switch typ.Kind() {
case reflect.Bool:
parsed, err := strconv.ParseBool(value)
if err != nil {
return nil, fmt.Errorf("%q is not a boolean", value)
}
return &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!bool", Value: strconv.FormatBool(parsed)}, nil
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
parsed, err := strconv.ParseInt(value, 10, typ.Bits())
if err != nil {
return nil, fmt.Errorf("%q is not a valid %s", value, typ.Kind())
}
return &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!int", Value: strconv.FormatInt(parsed, 10)}, nil
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
parsed, err := strconv.ParseUint(value, 10, typ.Bits())
if err != nil {
return nil, fmt.Errorf("%q is not a valid %s", value, typ.Kind())
}
return &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!int", Value: strconv.FormatUint(parsed, 10)}, nil
case reflect.String:
return &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: value}, nil
default:
return nil, fmt.Errorf("unsupported config value type %s", typ)
}
}

View File

@@ -0,0 +1,13 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build windows || plan9
package config
import "os"
// preserveOwner is a no-op where a new file inherits its ownership from the directory.
func preserveOwner(_ string, _ os.FileInfo) error {
return nil
}

View File

@@ -0,0 +1,267 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package config
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const editFixture = `# A leading comment.
log:
# The logging level.
level: info
runner:
capacity: 1
envs:
EXISTING: value
timeout: 3h
labels:
- ubuntu-latest:docker://node:20
- self-hosted
`
func writeEditFixture(t *testing.T) string {
t.Helper()
path := filepath.Join(t.TempDir(), "config.yaml")
require.NoError(t, os.WriteFile(path, []byte(editFixture), 0o600))
return path
}
func TestEditValues(t *testing.T) {
tests := []struct {
name string
edit func(file string) error
assert func(t *testing.T, cfg *Config, content string)
}{
{
name: "set scalar",
edit: func(file string) error { return SetValue(file, "runner.capacity", "4") },
assert: func(t *testing.T, cfg *Config, _ string) {
assert.Equal(t, 4, cfg.Runner.Capacity)
},
},
{
name: "set duration",
edit: func(file string) error { return SetValue(file, "runner.timeout", "90m") },
assert: func(t *testing.T, cfg *Config, content string) {
assert.Equal(t, 90*time.Minute, cfg.Runner.Timeout)
assert.Contains(t, content, "timeout: 1h30m0s")
},
},
{
name: "set a pointer field in a missing section",
edit: func(file string) error {
return SetValue(file, "container.network_create_options.enable_ipv4", "false")
},
assert: func(t *testing.T, cfg *Config, _ string) {
require.NotNil(t, cfg.Container.NetworkCreateOptions.EnableIPv4)
assert.False(t, *cfg.Container.NetworkCreateOptions.EnableIPv4)
},
},
{
name: "set map entry",
edit: func(file string) error { return SetValue(file, "runner.envs.ADDED", "yes") },
assert: func(t *testing.T, cfg *Config, _ string) {
assert.Equal(t, map[string]string{"EXISTING": "value", "ADDED": "yes"}, cfg.Runner.Envs)
},
},
{
name: "set replaces a list",
edit: func(file string) error { return SetValue(file, "runner.labels", "one", "two") },
assert: func(t *testing.T, cfg *Config, _ string) {
assert.Equal(t, []string{"one", "two"}, cfg.Runner.Labels)
},
},
{
name: "add appends to a list",
edit: func(file string) error { return AddValue(file, "runner.labels", "ubuntu:docker://node:22") },
assert: func(t *testing.T, cfg *Config, _ string) {
assert.Equal(t, []string{"ubuntu-latest:docker://node:20", "self-hosted", "ubuntu:docker://node:22"}, cfg.Runner.Labels)
},
},
{
name: "remove drops a list entry",
edit: func(file string) error { return RemoveValue(file, "runner.labels", "self-hosted") },
assert: func(t *testing.T, cfg *Config, _ string) {
assert.Equal(t, []string{"ubuntu-latest:docker://node:20"}, cfg.Runner.Labels)
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
file := writeEditFixture(t)
require.NoError(t, tt.edit(file))
raw, err := os.ReadFile(file)
require.NoError(t, err)
content := string(raw)
cfg, err := LoadDefault(file)
require.NoError(t, err)
tt.assert(t, cfg, content)
assert.Contains(t, content, "# A leading comment.")
assert.Contains(t, content, " # The logging level.")
assert.Contains(t, content, "\n\nrunner:")
})
}
}
func TestEditValuesRejectsBadInput(t *testing.T) {
tests := []struct {
name string
edit func(file string) error
wantErr string
}{
{
name: "unknown key",
edit: func(file string) error { return SetValue(file, "runner.labl", "x") },
wantErr: `unknown config key "runner.labl"`,
},
{
name: "value is not a number",
edit: func(file string) error { return SetValue(file, "runner.capacity", "many") },
wantErr: `"many" is not a valid int`,
},
{
name: "value is not a duration",
edit: func(file string) error { return SetValue(file, "runner.timeout", "soon") },
wantErr: `"soon" is not a duration`,
},
{
name: "value is not a boolean",
edit: func(file string) error { return SetValue(file, "runner.insecure", "maybe") },
wantErr: `"maybe" is not a boolean`,
},
{
name: "set needs a single value",
edit: func(file string) error { return SetValue(file, "runner.capacity", "1", "2") },
wantErr: "takes exactly one value",
},
{
name: "set on a section",
edit: func(file string) error { return SetValue(file, "runner", "x") },
wantErr: "is a section",
},
{
name: "add on a scalar",
edit: func(file string) error { return AddValue(file, "runner.capacity", "4") },
wantErr: "is not a list",
},
{
name: "add a duplicate",
edit: func(file string) error { return AddValue(file, "runner.labels", "self-hosted") },
wantErr: `already contains "self-hosted"`,
},
{
name: "remove a missing entry",
edit: func(file string) error { return RemoveValue(file, "runner.labels", "absent") },
wantErr: `does not contain "absent"`,
},
{
name: "sub-key of a free-form map entry",
edit: func(file string) error { return SetValue(file, "runner.envs.A.B", "x") },
wantErr: "has no sub-keys",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
file := writeEditFixture(t)
err := tt.edit(file)
require.Error(t, err)
assert.Contains(t, err.Error(), tt.wantErr)
content, err := os.ReadFile(file)
require.NoError(t, err)
assert.Equal(t, editFixture, string(content), "a rejected edit must leave the file untouched")
})
}
}
func TestGetValue(t *testing.T) {
file := writeEditFixture(t)
value, err := GetValue(file, "runner.capacity")
require.NoError(t, err)
assert.Equal(t, "1", value)
value, err = GetValue(file, "runner.labels")
require.NoError(t, err)
assert.Equal(t, "ubuntu-latest:docker://node:20\nself-hosted", value)
value, err = GetValue(file, "runner.envs")
require.NoError(t, err)
assert.Equal(t, "EXISTING=value", value)
// A section has no single-line rendering.
value, err = GetValue(file, "runner")
require.NoError(t, err)
assert.Contains(t, value, "labels:\n - ubuntu-latest:docker://node:20")
_, err = GetValue(file, "metrics.addr")
require.Error(t, err)
assert.Contains(t, err.Error(), "is not set")
}
func TestEditValuesFileHandling(t *testing.T) {
t.Run("reports a missing file", func(t *testing.T) {
err := SetValue(filepath.Join(t.TempDir(), "absent.yaml"), "runner.capacity", "4")
require.Error(t, err)
assert.Contains(t, err.Error(), "does not exist")
})
t.Run("writes through a symlink", func(t *testing.T) {
dir := t.TempDir()
target := filepath.Join(dir, "real.yaml")
link := filepath.Join(dir, "config.yaml")
require.NoError(t, os.WriteFile(target, []byte(editFixture), 0o600))
require.NoError(t, os.Symlink(target, link))
require.NoError(t, SetValue(link, "runner.capacity", "4"))
info, err := os.Lstat(link)
require.NoError(t, err)
assert.NotZero(t, info.Mode()&os.ModeSymlink, "the symlink must not be replaced by a regular file")
content, err := os.ReadFile(target)
require.NoError(t, err)
assert.Contains(t, string(content), "capacity: 4")
})
t.Run("keeps CRLF line endings", func(t *testing.T) {
file := filepath.Join(t.TempDir(), "config.yaml")
require.NoError(t, os.WriteFile(file, []byte(strings.ReplaceAll(editFixture, "\n", "\r\n")), 0o600))
require.NoError(t, SetValue(file, "runner.capacity", "4"))
content, err := os.ReadFile(file)
require.NoError(t, err)
assert.Contains(t, string(content), "capacity: 4\r\n")
assert.NotContains(t, strings.ReplaceAll(string(content), "\r\n", ""), "\n")
})
}
// The example config is the file users edit, so it has to stay written the way
// the encoder emits it, down to the single space before a trailing comment.
func TestEditValuesKeepsExampleConfigIntact(t *testing.T) {
file := filepath.Join(t.TempDir(), "config.yaml")
require.NoError(t, os.WriteFile(file, Example, 0o600))
require.NoError(t, AddValue(file, "runner.labels", "ubuntu:docker://node:22"))
content, err := os.ReadFile(file)
require.NoError(t, err)
withoutAdded := strings.Replace(string(content), " - ubuntu:docker://node:22\n", "", 1)
assert.Equal(t, string(Example), withoutAdded, "only the appended label may differ")
}

View File

@@ -0,0 +1,25 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !windows && !plan9
package config
import (
"errors"
"os"
"syscall"
)
// preserveOwner keeps a config that root edits owned by the service user it was created for.
func preserveOwner(file string, info os.FileInfo) error {
stat, ok := info.Sys().(*syscall.Stat_t)
if !ok {
return nil
}
// A caller that may replace the file but not chown it is no worse off than before.
if err := os.Chown(file, int(stat.Uid), int(stat.Gid)); err != nil && !errors.Is(err, os.ErrPermission) {
return err
}
return nil
}

View File

@@ -255,6 +255,9 @@ func (r *Reporter) Fire(entry *log.Entry) error {
if step.StartedAt == nil { if step.StartedAt == nil {
step.StartedAt = timestamppb.New(timestamp) step.StartedAt = timestamppb.New(timestamp)
urgentState = true urgentState = true
// The runner's own handler is per step, so an unresumed ::stop-commands:: must not
// leave the reporter suppressed, and no longer registering masks, for the whole job.
r.stopCommandEndToken = ""
} }
// Force reporting log errors as raw output to prevent silent failures // Force reporting log errors as raw output to prevent silent failures
@@ -396,10 +399,9 @@ func (r *Reporter) Logf(format string, a ...any) {
func (r *Reporter) logf(format string, a ...any) { func (r *Reporter) logf(format string, a ...any) {
if !r.duringSteps() { if !r.duringSteps() {
r.logRows = append(r.logRows, &runnerv1.LogRow{ // Masked like any other row: these bypass parseLogRow, but a caller can still
Time: timestamppb.Now(), // interpolate a secret, such as a configured URL carrying credentials.
Content: fmt.Sprintf(format, a...), r.logRows = append(r.logRows, r.newLogRow(timestamppb.Now(), fmt.Sprintf(format, a...)))
})
} }
} }
@@ -700,66 +702,128 @@ func (r *Reporter) parseResult(result any) (runnerv1.Result, bool) {
return ret, ok return ret, ok
} }
var cmdRegex = regexp.MustCompile(`^::([^ :]+)( .*)?::(.*)$`) // A property value never contains a raw ':' (GitHub escapes it as %3A), so excluding ':' ends
// the property list at the first '::' as GitHub does; greedily would swallow a '::' message.
var cmdRegex = regexp.MustCompile(`^::([^ :]+)( [^:]*)?::(.*)$`)
func (r *Reporter) handleCommand(originalContent, command, value string) *string { // handleCommand takes value still escaped, so that the web UI decodes it exactly once. Only
if r.stopCommandEndToken != "" && command != r.stopCommandEndToken { // the branches that consume the payload here decode it.
return &originalContent func (r *Reporter) handleCommand(originalContent, command, properties, value string) *string {
if r.stopCommandEndToken != "" {
if command != r.stopCommandEndToken {
return &originalContent
}
// Resumed here rather than from the switch, because the end token is arbitrary and a
// token naming a real command would otherwise never resume.
r.stopCommandEndToken = ""
return nil
} }
switch command { switch command {
case "add-mask": case "add-mask":
r.addMask(value) r.addMask(runner.UnescapeCommandData(value))
return nil return nil
case "debug": case "debug":
if r.debugOutputEnabled { if r.debugOutputEnabled {
return &value return &originalContent // kept as ::debug::, so the web UI labels and decodes it
} }
return nil return nil
case "notice": case "notice", "warning", "error":
// Not implemented yet, so just return the original content. // Gitea has no annotation store, so the annotation is rendered into the log with
return &originalContent // its source location instead of being dropped: that location is the whole point
case "warning": // of the command for compiler and linter output.
// Not implemented yet, so just return the original content. annotation := formatAnnotation(command, properties, value)
return &originalContent return &annotation
case "error": case "group", "endgroup":
// Not implemented yet, so just return the original content. // Passed through: the web UI folds the log on these and decodes the payload itself.
return &originalContent
case "group":
// Returning the original content, because I think the frontend
// will use it when rendering the output.
return &originalContent
case "endgroup":
// Ditto
return &originalContent return &originalContent
case "stop-commands": case "stop-commands":
r.stopCommandEndToken = value r.stopCommandEndToken = runner.UnescapeCommandData(value)
return nil
case r.stopCommandEndToken:
r.stopCommandEndToken = ""
return nil return nil
} }
return &originalContent return &originalContent
} }
// formatAnnotation folds the file, line, column and title the command carries into its message,
// which the web UI otherwise drops along with the rest of the properties:
//
// ::error file=main.go,line=12,col=5,title=vet::undefined: x
// ::error::main.go:12:5: vet: undefined: x
//
// The ::-form prefix is deliberate, and value is not escaped here because it arrived escaped
// and must stay that way.
func formatAnnotation(level, properties, value string) string {
props := parseCommandProperties(properties)
prefix := props["file"]
if prefix != "" {
if props["line"] != "" {
prefix += ":" + props["line"]
if props["col"] != "" {
prefix += ":" + props["col"]
}
}
prefix += ": "
}
if props["title"] != "" {
prefix += props["title"] + ": "
}
return "::" + level + "::" + prefix + value
}
// parseCommandProperties parses the `file=main.go,line=12` part of a workflow command.
func parseCommandProperties(properties string) map[string]string {
properties = strings.TrimSpace(properties)
if properties == "" {
return nil
}
props := map[string]string{}
for pair := range strings.SplitSeq(properties, ",") {
key, value, ok := strings.Cut(pair, "=")
if !ok {
continue
}
// Only the property-list separators are decoded, the web UI decodes the rest.
value = strings.ReplaceAll(strings.ReplaceAll(value, "%3A", ":"), "%2C", ",")
// GitHub keys its property dictionary case-insensitively, so `File=` works there too.
props[strings.ToLower(strings.TrimSpace(key))] = value
}
// GitHub's toolkit emits `col`; accept `column` as well, which some tools write instead.
if props["col"] == "" {
props["col"] = props["column"]
}
return props
}
func (r *Reporter) parseLogRow(entry *log.Entry) *runnerv1.LogRow { func (r *Reporter) parseLogRow(entry *log.Entry) *runnerv1.LogRow {
content := strings.TrimRight(entry.Message, "\r\n") content := strings.TrimRight(entry.Message, "\r\n")
// cmdRegex only covers the ::cmd:: form, so the ##[add-mask] one would otherwise reach
// the log carrying its own secret. Registered and dropped like its ::add-mask:: twin.
if arg, ok := strings.CutPrefix(content, "##[add-mask]"); ok {
r.addMask(runner.UnescapeCommandData(arg))
return nil
}
matches := cmdRegex.FindStringSubmatch(content) matches := cmdRegex.FindStringSubmatch(content)
if matches != nil { if matches != nil {
if output := r.handleCommand(content, matches[1], runner.UnescapeCommandData(matches[3])); output != nil { if output := r.handleCommand(content, matches[1], matches[2], matches[3]); output != nil {
content = *output content = *output
} else { } else {
return nil return nil
} }
} }
content = r.logReplacer.Replace(content) return r.newLogRow(timestamppb.New(entry.Time), content)
}
// newLogRow applies the masking and validation every row must carry, whatever built it.
func (r *Reporter) newLogRow(t *timestamppb.Timestamp, content string) *runnerv1.LogRow {
return &runnerv1.LogRow{ return &runnerv1.LogRow{
Time: timestamppb.New(entry.Time), Time: t,
Content: strings.ToValidUTF8(content, "?"), Content: strings.ToValidUTF8(r.logReplacer.Replace(content), "?"),
} }
} }

View File

@@ -72,9 +72,12 @@ func TestReporter_parseLogRow(t *testing.T) {
"Debug enabled", true, "Debug enabled", true,
[]string{ []string{
"::debug::GitHub Actions runtime token access controls", "::debug::GitHub Actions runtime token access controls",
// Left escaped: the web UI decodes it, and a real newline would not survive storage.
"::debug::first%0Asecond",
}, },
[]string{ []string{
"GitHub Actions runtime token access controls", "::debug::GitHub Actions runtime token access controls",
"::debug::first%0Asecond",
}, },
}, },
{ {
@@ -86,31 +89,46 @@ func TestReporter_parseLogRow(t *testing.T) {
"<nil>", "<nil>",
}, },
}, },
// The three annotation levels share one code path, so the property shapes are only
// exercised under "error"; notice and warning just prove the level token round-trips.
{ {
"notice", false, "notice", false,
[]string{ []string{
"::notice file=file.name,line=42,endLine=48,title=Cool Title::Gosh, that's not going to work", "::notice::Gosh, that's not going to work",
}, },
[]string{ []string{
"::notice file=file.name,line=42,endLine=48,title=Cool Title::Gosh, that's not going to work", "::notice::Gosh, that's not going to work",
}, },
}, },
{ {
"warning", false, "warning", false,
[]string{ []string{
"::warning file=file.name,line=42,endLine=48,title=Cool Title::Gosh, that's not going to work", "::warning::Gosh, that's not going to work",
}, },
[]string{ []string{
"::warning file=file.name,line=42,endLine=48,title=Cool Title::Gosh, that's not going to work", "::warning::Gosh, that's not going to work",
}, },
}, },
{ {
"error", false, "error", false,
[]string{ []string{
"::error file=file.name,line=42,endLine=48,title=Cool Title::Gosh, that's not going to work", "::error file=file.name,line=42,endLine=48,title=Cool Title::Gosh, that's not going to work",
"::error::Gosh, that's not going to work",
"::error file=file.name,line=42,col=7::Gosh, that's not going to work",
// The message keeps its own '::', the property list ends at the first one.
"::error file=main.cpp,line=12::no member named 'foo' in 'std::vector<int>'",
// GitHub matches property names case-insensitively.
"::error File=file.name,Line=42,Col=7::Gosh, that's not going to work",
// Only the property separators are decoded here, %25/%0A are left for the web UI.
"::error file=a%3Ab.go,title=100%252C::still %25 escaped%0Aand multi-line",
}, },
[]string{ []string{
"::error file=file.name,line=42,endLine=48,title=Cool Title::Gosh, that's not going to work", "::error::file.name:42: Cool Title: Gosh, that's not going to work",
"::error::Gosh, that's not going to work",
"::error::file.name:42:7: Gosh, that's not going to work",
"::error::main.cpp:12: no member named 'foo' in 'std::vector<int>'",
"::error::file.name:42:7: Gosh, that's not going to work",
"::error::a:b.go: 100%252C: still %25 escaped%0Aand multi-line",
}, },
}, },
{ {
@@ -149,6 +167,24 @@ func TestReporter_parseLogRow(t *testing.T) {
"*** bar baz ***", "*** bar baz ***",
}, },
}, },
{
// a token naming a real command must still resume
"stop-commands with a command-named token", false,
[]string{
"::stop-commands::add-mask",
"::set-output name=x::suppressed",
"::add-mask::",
"::add-mask::masked",
"masked",
},
[]string{
"<nil>",
"::set-output name=x::suppressed",
"<nil>",
"<nil>",
"***",
},
},
{ {
"unknown command", false, "unknown command", false,
[]string{ []string{
@@ -179,6 +215,19 @@ func TestReporter_parseLogRow(t *testing.T) {
} }
} }
// Both add-mask forms must register the secret and drop their own row: the runner forwards
// the raw line, so failing to consume it writes the secret straight to the job log.
func TestReporter_parseLogRowAddMask(t *testing.T) {
for _, line := range []string{"::add-mask::supersecret", "##[add-mask]supersecret"} {
r := &Reporter{logReplacer: strings.NewReplacer()}
assert.Nil(t, r.parseLogRow(&log.Entry{Message: line}), line)
row := r.parseLogRow(&log.Entry{Message: "using supersecret now"})
assert.Equal(t, "using *** now", row.Content, line)
}
}
func TestReporter_Fire(t *testing.T) { func TestReporter_Fire(t *testing.T) {
t.Run("ignore command lines", func(t *testing.T) { t.Run("ignore command lines", func(t *testing.T) {
client := mocks.NewClient(t) client := mocks.NewClient(t)
@@ -1013,7 +1062,7 @@ func TestReporter_Result(t *testing.T) {
} }
func TestReporter_SetOutputs(t *testing.T) { func TestReporter_SetOutputs(t *testing.T) {
r := &Reporter{state: &runnerv1.TaskState{}} r := &Reporter{state: &runnerv1.TaskState{}, logReplacer: strings.NewReplacer()}
r.SetOutputs(map[string]string{"foo": "bar"}) r.SetOutputs(map[string]string{"foo": "bar"})
got, ok := r.outputs["foo"] got, ok := r.outputs["foo"]