docs: explain labels, the docker image cache volume, and the dind-rootless UID (#1086)

Fixes #106
Fixes #570
Fixes #627

Reviewed-on: https://gitea.com/gitea/runner/pulls/1086
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>

README.md

@@ -85,6 +85,8 @@ docker run -e GITEA_INSTANCE_URL=https://your_gitea.com -e GITEA_RUNNER_REGISTRA
 
 Mount a volume on `/data` if you want the registration file and optional config to survive container recreation (see [scripts/run.sh](scripts/run.sh)).
 
+> **`/data` does not hold the image cache.** It is the runner's working directory and contains only the `.runner` registration file and, optionally, your config file. Images pulled for jobs live in the *Docker daemon's* data root, which for the `dind` flavours is inside the container (`/var/lib/docker`, or `/home/rootless/.local/share/docker` for `dind-rootless`). To keep the image cache across restarts, give that path its own volume as well — otherwise every new container re-pulls the job images. With the `basic` flavour the images live on whichever daemon you point the runner at, so there is nothing extra to persist.
+
 ### Image flavours
 
 The image is published in three flavours, all built from the single multi-stage [Dockerfile](Dockerfile) in this repository. They differ only in how a Docker daemon is made available to the jobs the runner executes; the `gitea-runner` binary inside them is identical.
@@ -121,6 +123,8 @@ Two processes have to run side by side here (the Docker daemon and the runner),
 
 Same idea as `dind`, but built on `docker:dind-rootless` so the bundled daemon and the runner run as an unprivileged user (`rootless`, UID 1000) rather than `root`. `DOCKER_HOST` is preset to `unix:///run/user/1000/docker.sock` so the runner talks to the rootless daemon. This reduces the blast radius compared to the privileged `dind` flavour, but rootless Docker carries the usual rootless limitations (networking, cgroups, storage drivers, and some operations that need additional host configuration such as `/etc/subuid` / `/etc/subgid` mappings and unprivileged user-namespace support).
 
+> **The UID is fixed at 1000.** It comes from the `rootless` user baked into the upstream `docker:dind-rootless` base image, and the bundled daemon always listens on `/run/user/1000/docker.sock` inside the container, so running this flavour as a different user (`--user 1001`) does not work. If you need the runner to talk to a *host* rootless daemon that runs under some other UID, use the `basic` flavour instead and bind-mount that daemon's socket (see [examples/vm/rootless-docker.md](examples/vm/rootless-docker.md)); pointing `DOCKER_HOST` at a host socket from inside `dind-rootless` will not work. Changing the UID otherwise means rebuilding the image from a base with a different `rootless` user.
+
 > **Note on Podman:** these images target the Docker daemon. The bundled `dind`/`dind-rootless` daemons are `dockerd`, not Podman, and the `basic` flavour expects a Docker-compatible socket. Running them under rootless Podman is not a supported configuration, though pointing the `basic` flavour at a Podman socket that emulates the Docker API may work for some workloads.
 
 ### Configuration
@@ -147,6 +151,50 @@ If you omit `-c`, built-in defaults apply (same as an empty YAML document).
 
 Earlier releases let a small set of environment variables (`GITEA_DEBUG`, `GITEA_TRACE`, `GITEA_RUNNER_CAPACITY`, `GITEA_RUNNER_FILE`, `GITEA_RUNNER_ENVIRON`, `GITEA_RUNNER_ENV_FILE`) override parts of the default config. Those overrides have been removed — use a YAML config file for all settings instead. For the Docker images, the entrypoint still understands a separate set of variables (such as `RUNNER_STATE_FILE`); see [scripts/run.sh](scripts/run.sh) and the container documentation below.
 
+### Labels
+
+Labels decide **which jobs a runner accepts** and **how it runs them**. A job's `runs-on` is matched against the runner's label names; the first match wins and selects the execution environment for that job.
+
+A label is written as:
+
+```text
+<name>[:<schema>[:<args>]]
+```
+
+| Part | Meaning |
+| --- | --- |
+| `name` | The name a workflow refers to in `runs-on`, e.g. `ubuntu-latest`. |
+| `schema` | Either `docker` or `host`. Defaults to `host` when omitted. |
+| `args` | Only used by the `docker` schema: the image to run the job in. |
+
+Two schemas are supported:
+
+- **`docker://<image>`** — the job runs inside a container created from `<image>`:
+
+  ```text
+  ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest
+  ```
+
+- **`host`** — the job's steps run directly on the machine the runner is on, using the tools installed there:
+
+  ```text
+  macos:host
+  ```
+
+So with the labels
+
+```text
+ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,macos:host
+```
+
+a workflow with `runs-on: ubuntu-latest` is executed in the `runner-images:ubuntu-latest` container, and one with `runs-on: macos` is executed directly on the host.
+
+Names may themselves contain a colon (for example `pool:e57e18d4-10d4-406f-93bf-60f127221bdd`); only `host` and `docker` are treated as schemas.
+
+If a job's `runs-on` matches none of the runner's labels, the job still runs, in the default `docker.gitea.com/runner-images:ubuntu-latest` image. Images maintained for this purpose are listed at [gitea/runner-images](https://gitea.com/gitea/runner-images).
+
+Labels are chosen at registration time (`--labels`, or the interactive prompt) and can be changed afterwards by editing `runner.labels` in the config file, or in the Gitea UI under the runner's settings.
+
 #### Registration vs config labels
 
 If `runner.labels` is set in the YAML file, those labels are used during `register` and the `--labels` CLI flag is ignored.

act/container/docker_pull.go

@@ -72,7 +72,9 @@ func NewDockerPullExecutor(input NewDockerPullExecutorInput) common.Executor {
 
 				_ = logDockerResponse(logger, reader, err != nil)
 			}
-			return err
+			if err != nil {
+				return fmt.Errorf("failed to pull image '%s' (%s): %w", imageRef, input.Platform, err)
+			}
 		}
 		return nil
 	}

act/runner/expression.go

@@ -229,7 +229,8 @@ func (ee expressionEvaluator) evaluate(ctx context.Context, in string, defaultSt
 	logger.Debugf("evaluating expression '%s'", in)
 	evaluated, err := ee.interpreter.Evaluate(in, defaultStatusCheck)
 
-	printable := regexp.MustCompile(`::add-mask::.*`).ReplaceAllString(fmt.Sprintf("%t", evaluated), "::add-mask::***)")
+	// evaluated is an any: %t renders everything but a bool as "%!t(string=...)"
+	printable := regexp.MustCompile(`::add-mask::.*`).ReplaceAllString(fmt.Sprintf("%v", evaluated), "::add-mask::***)")
 	logger.Debugf("expression '%s' evaluated to '%s'", in, printable)
 
 	return evaluated, err

act/runner/run_context.go

@@ -339,6 +339,9 @@ func printStartJobContainerGroup(ctx context.Context, image, name, network strin
 	}
 }
 
+// newContainer is a variable so tests can substitute a container that needs no Docker daemon.
+var newContainer = container.NewContainer
+
 func (rc *RunContext) startJobContainer() common.Executor {
 	return func(ctx context.Context) error {
 		logger := common.Logger(ctx)
@@ -402,7 +405,9 @@ func (rc *RunContext) startJobContainer() common.Executor {
 			for _, v := range spec.Cmd {
 				interpolatedCmd = append(interpolatedCmd, rc.ExprEval.Interpolate(ctx, v))
 			}
-			username, password, err = rc.handleServiceCredentials(ctx, spec.Credentials)
+			// keep these local: reusing username/password would overwrite the
+			// credentials the job container is pulled with further down
+			serviceUsername, servicePassword, err := rc.handleServiceCredentials(ctx, spec.Credentials)
 			if err != nil {
 				return fmt.Errorf("failed to handle service %s credentials: %w", serviceID, err)
 			}
@@ -423,12 +428,12 @@ func (rc *RunContext) startJobContainer() common.Executor {
 			}
 
 			serviceContainerName := createContainerName(rc.jobContainerName(), serviceID)
-			c := container.NewContainer(&container.NewContainerInput{
+			c := newContainer(&container.NewContainerInput{
 				Name:           serviceContainerName,
 				WorkingDir:     ext.ToContainerPath(rc.Config.Workdir),
 				Image:          serviceImage,
-				Username:       username,
-				Password:       password,
+				Username:       serviceUsername,
+				Password:       servicePassword,
 				Cmd:            interpolatedCmd,
 				Env:            envs,
 				Mounts:         serviceMounts,
@@ -484,7 +489,7 @@ func (rc *RunContext) startJobContainer() common.Executor {
 		// For Gitea, `jobContainerNetwork` should be the same as `networkName`
 		jobContainerNetwork := networkName
 
-		rc.JobContainer = container.NewContainer(&container.NewContainerInput{
+		rc.JobContainer = newContainer(&container.NewContainerInput{
 			Cmd:            nil,
 			Entrypoint:     []string{"/bin/sleep", fmt.Sprint(rc.Config.ContainerMaxLifetime.Round(time.Second).Seconds())},
 			WorkingDir:     ext.ToContainerPath(rc.Config.Workdir),

act/runner/run_context_test.go

@@ -14,6 +14,7 @@ import (
 	"testing"
 
 	"gitea.com/gitea/runner/act/common"
+	"gitea.com/gitea/runner/act/container"
 	"gitea.com/gitea/runner/act/exprparser"
 	"gitea.com/gitea/runner/act/model"
 
@@ -202,6 +203,93 @@ jobs:
 	assert.Empty(t, password)
 }
 
+// fakeContainer turns every container operation into a no-op, so startJobContainer
+// runs without a Docker daemon. The embedded interface is nil, so any method the
+// test does not exercise panics rather than silently doing the wrong thing.
+type fakeContainer struct {
+	container.ExecutionsEnvironment
+}
+
+func (fakeContainer) Pull(bool) common.Executor  { return func(context.Context) error { return nil } }
+func (fakeContainer) Start(bool) common.Executor { return func(context.Context) error { return nil } }
+func (fakeContainer) Remove() common.Executor    { return func(context.Context) error { return nil } }
+func (fakeContainer) Close() common.Executor     { return func(context.Context) error { return nil } }
+func (fakeContainer) GetActPath() string         { return "/var/run/act" }
+func (fakeContainer) Create([]string, []string) common.Executor {
+	return func(context.Context) error { return nil }
+}
+
+func (fakeContainer) Copy(string, ...*container.FileEntry) common.Executor {
+	return func(context.Context) error { return nil }
+}
+
+// Regression test: a service without a `credentials:` block resolves to empty
+// credentials, which used to overwrite the job container's own credentials.
+func TestStartJobContainerKeepsJobCredentialsWithServices(t *testing.T) {
+	workflow, err := model.ReadWorkflow(strings.NewReader(`
+name: test
+on: push
+jobs:
+  job:
+    runs-on: ubuntu-latest
+    container:
+      image: registry.example/private:latest
+      credentials:
+        username: job-user
+        password: job-password
+    services:
+      redis:
+        image: redis:latest
+      db:
+        image: postgres:latest
+        credentials:
+          username: db-user
+          password: db-password
+    steps: []
+`))
+	require.NoError(t, err)
+
+	var inputs []*container.NewContainerInput
+	origNewContainer := newContainer
+	newContainer = func(input *container.NewContainerInput) container.ExecutionsEnvironment {
+		inputs = append(inputs, input)
+		return fakeContainer{}
+	}
+	t.Cleanup(func() { newContainer = origNewContainer })
+
+	rc := &RunContext{
+		Name: "test",
+		Config: &Config{
+			Workdir: "/tmp",
+			// no daemon: an explicit network mode creates no network, and
+			// reusing containers short-circuits the volume cleanup executors
+			ContainerNetworkMode: "host",
+			ReuseContainers:      true,
+			Env:                  map[string]string{},
+			Secrets:              map[string]string{},
+		},
+		Env: map[string]string{},
+		Run: &model.Run{
+			JobID:    "job",
+			Workflow: workflow,
+		},
+	}
+	rc.ExprEval = rc.NewExpressionEvaluator(t.Context())
+
+	require.NoError(t, rc.startJobContainer()(t.Context()))
+
+	credentials := map[string][2]string{}
+	for _, in := range inputs {
+		credentials[in.Image] = [2]string{in.Username, in.Password}
+	}
+
+	// the job container keeps its own credentials, whichever services exist
+	require.Equal(t, [2]string{"job-user", "job-password"}, credentials["registry.example/private:latest"])
+	// each service keeps its own, and a service without credentials gets none
+	require.Equal(t, [2]string{"db-user", "db-password"}, credentials["postgres:latest"])
+	require.Equal(t, [2]string{"", ""}, credentials["redis:latest"])
+}
+
 func TestRunContext_GetBindsAndMounts(t *testing.T) {
 	rctemplate := &RunContext{
 		Name: "TestRCName",

examples/kubernetes/README.md

@@ -6,6 +6,11 @@ NOTE: `dind-docker.yaml` uses the native sidecar pattern (init container with `r
 
 NOTE: A helm chart for `gitea-runner` also exists for easier deployments https://gitea.com/gitea/helm-actions
 
+Each example persists **two** things, and it is worth knowing which is which:
+
+- `/data` is the runner's working directory. It holds the `.runner` registration file and, optionally, the config file — so the runner re-attaches to the server instead of registering again.
+- The Docker daemon's data root holds the images pulled for jobs (`/var/lib/docker` for the dind sidecar, `/home/rootless/.local/share/docker` for `dind-rootless`). It is *not* under `/data`. If you drop this volume, the examples still work, but the image cache is discarded whenever the pod is recreated and every job re-pulls its images.
+
 Files in this directory:
 
 - [`dind-docker.yaml`](dind-docker.yaml)

examples/kubernetes/dind-docker.yaml

@@ -1,3 +1,5 @@
+# Holds the runner's working directory (/data): the .runner registration file
+# and, optionally, the config file.
 kind: PersistentVolumeClaim
 apiVersion: v1
 metadata:
@@ -10,6 +12,21 @@ spec:
       storage: 1Gi
   storageClassName: standard
 ---
+# Holds the Docker daemon's data root (/var/lib/docker), i.e. the images pulled
+# for jobs. Without it, the image cache is lost whenever the pod is recreated
+# and every job re-pulls its images. Size it for the images you expect to cache.
+kind: PersistentVolumeClaim
+apiVersion: v1
+metadata:
+  name: docker-vol
+spec:
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 20Gi
+  storageClassName: standard
+---
 apiVersion: v1
 data:
   # The registration token can be obtained from the web UI, API or command-line.
@@ -45,6 +62,9 @@ spec:
       - name: runner-data
         persistentVolumeClaim:
           claimName: runner-vol
+      - name: docker-data
+        persistentVolumeClaim:
+          claimName: docker-vol
       initContainers:
       - name: docker
         image: docker:28.2.2-dind
@@ -53,6 +73,8 @@ spec:
         volumeMounts:
         - name: docker-socket
           mountPath: /var/run
+        - name: docker-data
+          mountPath: /var/lib/docker
         startupProbe:
           exec:
             command: ["/usr/bin/test", "-S", "/var/run/docker.sock"]

examples/kubernetes/rootless-docker.yaml

@@ -1,3 +1,5 @@
+# Holds the runner's working directory (/data): the .runner registration file
+# and, optionally, the config file.
 kind: PersistentVolumeClaim
 apiVersion: v1
 metadata:
@@ -10,6 +12,21 @@ spec:
       storage: 1Gi
   storageClassName: standard
 ---
+# Holds the rootless Docker daemon's data root, i.e. the images pulled for jobs.
+# Without it, the image cache is lost whenever the pod is recreated and every job
+# re-pulls its images. Size it for the images you expect to cache.
+kind: PersistentVolumeClaim
+apiVersion: v1
+metadata:
+  name: docker-vol
+spec:
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 20Gi
+  storageClassName: standard
+---
 apiVersion: v1
 data:
   # The registration token can be obtained from the web UI, API or command-line.
@@ -43,7 +60,12 @@ spec:
       - name: runner-data
         persistentVolumeClaim:
           claimName: runner-vol
+      - name: docker-data
+        persistentVolumeClaim:
+          claimName: docker-vol
       securityContext:
+        # The dind-rootless image runs as the `rootless` user (UID/GID 1000);
+        # fsGroup makes both volumes writable for it.
         fsGroup: 1000
       containers:
       - name: runner
@@ -68,4 +90,7 @@ spec:
         volumeMounts:
         - name: runner-data
           mountPath: /data
+        # The rootless daemon keeps its images here, not under /data.
+        - name: docker-data
+          mountPath: /home/rootless/.local/share/docker
 

examples/kubernetes/statefulset-dind.yaml

@@ -44,6 +44,10 @@ spec:
         volumeMounts:
         - name: docker-socket
           mountPath: /var/run
+        # Keeps the images pulled for jobs across restarts. Without this, the
+        # daemon's data root is ephemeral and every job re-pulls its images.
+        - name: docker-data
+          mountPath: /var/lib/docker
         startupProbe:
           exec:
             command: ["/usr/bin/test", "-S", "/var/run/docker.sock"]
@@ -68,6 +72,8 @@ spec:
         - name: docker-socket
           mountPath: /var/run
   volumeClaimTemplates:
+  # The runner's working directory: the .runner registration file and, optionally,
+  # the config file.
   - metadata:
       name: runner-data
     spec:
@@ -77,3 +83,14 @@ spec:
         requests:
           storage: 1Gi
       storageClassName: standard
+  # The Docker daemon's data root: the images pulled for jobs. Size it for the
+  # images you expect to cache.
+  - metadata:
+      name: docker-data
+    spec:
+      accessModes:
+      - ReadWriteOnce
+      resources:
+        requests:
+          storage: 20Gi
+      storageClassName: standard