ink/act_runner
1
1
Fork 0
mirror of https://gitea.com/gitea/act_runner.git synced 2026-05-08 00:03:24 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix: Max parallel Support for Matrix Jobs and Remote Action Tests (#150)

Browse Source 
## Summary

This PR fixes the `max-parallel` strategy configuration for matrix jobs and resolves all failing tests in `step_action_remote_test.go`. The implementation ensures that matrix jobs respect the `max-parallel` setting, preventing resource exhaustion when running GitHub Actions workflows.

## Problem Statement

### Issue 1: max-parallel Not Working Correctly
Matrix jobs were running in parallel regardless of the `max-parallel` setting in the strategy configuration. This caused:
- Resource contention on limited runners
- Unpredictable job execution behavior
- Inability to control concurrency for resource-intensive workflows

### Issue 2: Failing Remote Action Tests
All tests in `step_action_remote_test.go` were failing due to:
- Missing `ActionCacheDir` configuration
- Incorrect mock expectations using fixed strings instead of hash-based paths
- Incompatibility with the hash-based action cache implementation

## Changes

### 1. max-parallel Implementation (`pkg/runner/runner.go`)

#### Robust Initialization
Added fallback logic to ensure `MaxParallel` is always properly initialized:
```go
if job.Strategy.MaxParallel == 0 {
    job.Strategy.MaxParallel = job.Strategy.GetMaxParallel()
}
```

#### Eliminated Unnecessary Nesting
Fixed inefficient nested parallelization when only one pipeline element exists:
```go
if len(pipeline) == 1 {
    // Execute directly without additional wrapper
    log.Debugf("Single pipeline element, executing directly")
    return pipeline[0](ctx)
}
```

#### Enhanced Logging
Added comprehensive debug and info logging:
- Shows which `maxParallel` value is being used
- Logs adjustments based on matrix size
- Reports final parallelization decisions

### 2. Worker Logging (`pkg/common/executor.go`)

Enhanced `NewParallelExecutor` with detailed worker activity logging:
```go
log.Infof("NewParallelExecutor: Creating %d workers for %d executors", parallel, len(executors))

for i := 0; i < parallel; i++ {
    go func(workerID int, work <-chan Executor, errs chan<- error) {
        log.Debugf("Worker %d started", workerID)
        taskCount := 0
        for executor := range work {
            taskCount++
            log.Debugf("Worker %d executing task %d", workerID, taskCount)
            errs <- executor(ctx)
        }
        log.Debugf("Worker %d finished (%d tasks executed)", workerID, taskCount)
    }(i, work, errs)
}
```

**Benefits:**
- Easy verification of correct parallelization
- Better debugging capabilities
- Clear visibility into worker activity

### 3. Documentation (`pkg/model/workflow.go`)

Added clarifying comment to ensure strategy values are always set:
```go
// Always set these values, even if there's an error later
j.Strategy.FailFast = j.Strategy.GetFailFast()
j.Strategy.MaxParallel = j.Strategy.GetMaxParallel()
```

### 4. Test Fixes (`pkg/runner/step_action_remote_test.go`)

#### Added Missing Configuration
All tests now include `ActionCacheDir`:
```go
Config: &Config{
    GitHubInstance: "github.com",
    ActionCacheDir: "/tmp/test-cache",
}
```

#### Hash-Based Suffix Matchers
Updated mocks to use hash-based paths instead of fixed strings:
```go
// Before
sarm.On("readAction", sar.Step, suffixMatcher("org-repo-path@ref"), ...)

// After
sarm.On("readAction", sar.Step, suffixMatcher(sar.Step.UsesHash()), ...)
```

#### Flexible Exec Matcher for Post Tests
Implemented flexible path matching for hash-based action directories:
```go
execMatcher := mock.MatchedBy(func(args []string) bool {
    if len(args) != 2 {
        return false
    }
    return args[0] == "node" && strings.Contains(args[1], "post.js")
})
```

#### Token Test Improvements
- Uses unique cache directory to force cloning
- Validates URL redirection to github.com
- Accepts realistic token behavior

### 5. New Tests

#### Unit Tests (`pkg/runner/max_parallel_test.go`)
Tests various `max-parallel` configurations:
- `max-parallel: 1` → Sequential execution
- `max-parallel: 2` → Max 2 parallel jobs
- `max-parallel: 4` (default) → Max 4 parallel jobs
- `max-parallel: 10` → Max 10 parallel jobs

#### Concurrency Test (`pkg/common/executor_max_parallel_test.go`)
Verifies that `max-parallel: 2` actually limits concurrent execution using atomic counters.

## Expected Behavior

### Before
- ❌ Jobs ran in parallel regardless of `max-parallel` setting
- ❌ Unnecessary nested parallelization (8 workers for 1 element)
- ❌ No logging to debug parallelization issues
- ❌ All `step_action_remote_test.go` tests failing

### After
- ✅ `max-parallel: 1` → Jobs run strictly sequentially
- ✅ `max-parallel: N` → Maximum N jobs run simultaneously
- ✅ Efficient single-level parallelization for matrix jobs
- ✅ Comprehensive logging for debugging
- ✅ All tests passing (10/10)

## Example Log Output

With `max-parallel: 2` and 6 matrix jobs:
```
[DEBUG] Using job.Strategy.MaxParallel: 2
[INFO] Running job with maxParallel=2 for 6 matrix combinations
[DEBUG] Single pipeline element, executing directly
[INFO] NewParallelExecutor: Creating 2 workers for 6 executors
[DEBUG] Worker 0 started
[DEBUG] Worker 1 started
[DEBUG] Worker 0 executing task 1
[DEBUG] Worker 1 executing task 1
...
[DEBUG] Worker 0 finished (3 tasks executed)
[DEBUG] Worker 1 finished (3 tasks executed)
```

## Test Results

All tests pass successfully:
```
✅ TestStepActionRemote (3 sub-tests)
✅ TestStepActionRemotePre
✅ TestStepActionRemotePreThroughAction
✅ TestStepActionRemotePreThroughActionToken
✅ TestStepActionRemotePost (4 sub-tests)
✅ TestMaxParallelStrategy
✅ TestMaxParallel2Quick

Total: 12/12 tests passing
```

## Breaking Changes

None. This PR is fully backward compatible. All changes improve existing behavior without altering the API.

## Impact

- ✅ Fixes resource management for CI/CD pipelines
- ✅ Prevents runner exhaustion on limited infrastructure
- ✅ Enables sequential execution for resource-intensive jobs
- ✅ Improves debugging with detailed logging
- ✅ Ensures test suite reliability

## Files Modified

### Core Implementation
- `pkg/runner/runner.go` - max-parallel fix + logging
- `pkg/common/executor.go` - Worker logging
- `pkg/model/workflow.go` - Documentation

### Tests
- `pkg/runner/step_action_remote_test.go` - Fixed all 10 tests
- `pkg/runner/max_parallel_test.go` - **NEW** - Unit tests
- `pkg/common/executor_max_parallel_test.go` - **NEW** - Concurrency test

## Verification

### Manual Testing
```bash
# Build
go build -o dist/act main.go

# Test with max-parallel: 2
./dist/act -W test-max-parallel-2.yml -v

# Expected: Only 2 jobs run simultaneously
```

### Automated Testing
```bash
# Run all tests
go test ./pkg/runner -run TestStepActionRemote -v
go test ./pkg/runner -run TestMaxParallel -v
go test ./pkg/common -run TestMaxParallel -v
```

## Related Issues

Fixes issues where matrix jobs in Gitea ignored the `max-parallel` strategy setting, causing resource contention and unpredictable behavior.

Reviewed-on: https://gitea.com/gitea/act/pulls/150
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Reviewed-by: silverwind <silverwind@noreply.gitea.com>
Co-authored-by: Pascal Zimmermann <pascal.zimmermann@theiotstudio.com>
Co-committed-by: Pascal Zimmermann <pascal.zimmermann@theiotstudio.com>
This commit is contained in:
Pascal Zimmermann
2026-02-11 00:43:38 +00:00
committed by Lunny Xiao
parent 495185446f
commit c0f19d9a26
10 changed files with 628 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
pkg/runner/runner.go
View File

@@ -74,6 +74,7 @@ type Config struct {
JobLoggerLevel *log.Level // the level of job logger
ValidVolumes []string // only volumes (and bind mounts) in this slice can be mounted on the job container or service containers
InsecureSkipTLS bool // whether to skip verifying TLS certificate of the Gitea instance
MaxParallel int // max parallel jobs to run across all workflows (0 = no limit, uses CPU count)
}
// GetToken: Adapt to Gitea
@@ -193,13 +194,21 @@ func (runner *runnerImpl) NewPlanExecutor(plan *model.Plan) common.Executor {
maxParallel := 4
if job.Strategy != nil {
// Ensure GetMaxParallel() is called if MaxParallel is still 0
if job.Strategy.MaxParallel == 0 {
job.Strategy.MaxParallel = job.Strategy.GetMaxParallel()
}
maxParallel = job.Strategy.MaxParallel
log.Debugf("Using job.Strategy.MaxParallel: %d", maxParallel)
}
if len(matrixes) < maxParallel {
log.Debugf("Adjusting maxParallel from %d to %d (number of matrix combinations)", maxParallel, len(matrixes))
maxParallel = len(matrixes)
}
log.Infof("Running job with maxParallel=%d for %d matrix combinations", maxParallel, len(matrixes))
for i, matrix := range matrixes {
matrix := matrix
rc := runner.newRunContext(ctx, run, matrix)
@@ -226,12 +235,39 @@ func (runner *runnerImpl) NewPlanExecutor(plan *model.Plan) common.Executor {
}
pipeline = append(pipeline, common.NewParallelExecutor(maxParallel, stageExecutor...))
}
ncpu := runtime.NumCPU()
if 1 > ncpu {
ncpu = 1
// For pipeline execution:
// - If only 1 element: run it directly (no need for additional parallelization)
// - If multiple elements: run them in parallel up to maxParallel or ncpu
if len(pipeline) == 0 {
return nil
}
log.Debugf("Detected CPUs: %d", ncpu)
return common.NewParallelExecutor(ncpu, pipeline...)(ctx)
if len(pipeline) == 1 {
// Single run/job: execute directly without additional parallelization wrapper
// This ensures max-parallel is the only limiting factor
log.Debugf("Single pipeline element, executing directly")
return pipeline[0](ctx)
}
// Multiple runs/jobs: execute in parallel up to maxParallel (if set) or ncpu
parallelism := runtime.NumCPU()
// If MaxParallel is set in config, use it
if runner.config.MaxParallel > 0 {
parallelism = runner.config.MaxParallel
log.Debugf("Using configured max-parallel: %d", parallelism)
} else {
log.Debugf("Using CPU count for parallelism: %d", parallelism)
}
// Don't exceed the number of pipeline elements
if parallelism > len(pipeline) {
parallelism = len(pipeline)
}
log.Infof("Executing %d pipeline elements with parallelism %d", len(pipeline), parallelism)
return common.NewParallelExecutor(parallelism, pipeline...)(ctx)
})
}