Steps in parallel?

[hugovk]

Is it possible to run given steps in parallel?

For example, we have a number of dependencies to build before we can build our component, and then test our component.

It would be nice if the dependency build steps could first run in parallel, followed by our component build step, and the test step.

[oldskool]

Not at this moment. Only jobs can run in parallel, but steps always run sequentially.

I don’t know if there are any plans to ever support this.

[epignatelli]

But jobs do not share workdirs.
So, if one wants to run two activities in parallel, they has to recreate the input state of each activity.
If this shared starting point is expensive to re-create, using jobs doesn't help.

It would be great if we could simply branch off a job (or a step), and then execute another one starting from there.

[epignatelli]

ps. visual programming would be a great approach here, in my opinion.

[epignatelli]

@community is there any update on this?

[ghost]

There's a feature request for it. See @zsolt-dev's comment

[loryruta]

if one wants to run two activities in parallel, they has to recreate the input state of each activity

This is so cringe. Please guys from Github/@microsoft, just copy what Gitlab is doing, they're x1000 steps further.

[chenrui333]

Hi, is this still the case that the steps within the job cannot run in parallel.

If that is the case, I wonder if there is any related documentation on best structuring the jobs to achieve the same goal (basically, turn the tasks into jobs) 

[amezick]

If your steps are shell commands can you run them as a background task using the ‘&’ at the end of the command line.  You won’t be told when they finish.  I also don’t know if the workflow run will know to wait for the job to finish before it kills the instance.

If you REALLY need parallelism, remember that an action allows you to write code.  That code could trigger tasks in multiple threads.  Seems like a lot of work, but the ability should be there.

[shakefu]

The wait command in bash waits for background jobs to complete.

[yss14]

As @amezick mentioned it is indeed possible via background tasks. And you can wait for them!

At our CI, we do something like this:

launch_as() {
  local cmd_name=$1
  shift
  (time $@ || echo $cmd_name >> fail.txt) 2>&1 > $cmd_name.txt
}
time npm ci

launch_as lint npm run lint &

time npm run build

launch_as build1 npm run build:build1 &

launch_as build2 npm run build:build2 &

launch_as build3 npm run build:build3 &
wait

cat lint.txt

cat build1.txt

cat build2.txt

cat build3.txt

if [ -f fail.txt ]; then

cat fail.txt

exit 1

fi

[eightnoteight]

but the logging ux for such background tasks is very bad :(

[felipecrs]

Or you can use GNU Parallel!

[mathieujobin]

I am looking forward seeing steps to be runnable in parallel. so that the setup steps can run once (installing dependencies, setting up database etc) but then everything else can be run in parallel

[gsheppard]

I'm confused why this is still not supported, a year later. This seems like an obvious use case, but all we have to support it is actions/upload-artifact which adds a substantial amount of time to the overall process even with compression.

[Pierstoval]

Same here, I just stumbled upon this feed, and I have a backend & frontend app running in Docker and since the frontend needs the backend to execute e2e tests, I want to install all dependencies once and then run backend & frontend tests in parallel (since they use different databases)

[jlaustill]

Same here, seems like a missing feature competitors have that would be high on the catch up list to me

[ScottRosenberg2]

I'm having the same issue with actions/upload-artifact not being parallel

[mhenrixon]

I just create a multitask rake task:

multitask ci: [
    :rubocop, 
    "bundle:audit", 
    :erblint, 
    :eslint, 
    :stylelint, 
    "brakeman:run", 
    :spec
  ]

[floralatin]

can show the detail of it, thanks

[mhenrixon]

require_relative "config/application"

Rails.application.load_tasks

if Rails.env.development? || Rails.env.test?
  namespace :brakeman do
    desc "Run Brakeman"
    task :run do # rubocop:disable Rails/RakeEnvironment
      sh("bin/brakeman --rails6 -A --no-summary --ensure-latest --no-progress")
    end
  end

  namespace :bundle do
    desc "Audit Bundler"
    task :audit do # rubocop:disable Rails/RakeEnvironment
      sh("bin/bundle-audit update")
      sh("bin/bundle-audit check")
    end
  end

  desc "Run ERBLint"
  task :erblint do # rubocop:disable Rails/RakeEnvironment
    sh("bin/erblint --lint-all")
  end

  desc "Fix ERBLint"
  task "erblint:fix" do # rubocop:disable Rails/RakeEnvironment
    sh("bin/erblint --lint-all -a")
  end

  desc "Run ESLint"
  task :eslint do # rubocop:disable Rails/RakeEnvironment
    sh("yarn lint:js")
  end

  desc "Fix ESLint problems"
  task "eslint:fix" do # rubocop:disable Rails/RakeEnvironment
    sh("yarn lint:js --fix")
  end

  desc "Run ESLint"
  task :stylelint do # rubocop:disable Rails/RakeEnvironment
    sh("yarn lint:css")
  end

  desc "Fix ESLint problems"
  task "stylelint:fix" do # rubocop:disable Rails/RakeEnvironment
    sh("yarn lint:css --fix")
  end

  require "rubocop/rake_task"

  RuboCop::RakeTask.new(:rubocop) do |task|
    # task.patterns = changed_files(5)
    task.options = %w[-DEP --format fuubar]
  end

  RuboCop::RakeTask.new("rubocop:fix") do |task|
    # task.patterns = changed_files(5)
    task.options = %w[-DEa --format fuubar]
  end

  RuboCop::RakeTask.new("rubocop:fixall") do |task|
    # task.patterns = changed_files(5)
    task.options = %w[-DEA --format fuubar]
  end

  # require "bundler-audit/bundler/audit/task"

  multitask lint: %i[erblint eslint stylelint rubocop]
  multitask "lint:fix" => %w[
    erblint:fix
    eslint:fix
    rubocop:fix
    stylelint:fix
  ]

  multitask "lint:fixall" => %w[
    erblint:fix
    eslint:fix
    rubocop:fixall
    stylelint:fix
  ]

  multitask ci: %i[
    brakeman:run
    bundle:audit
    erblint
    eslint
    rubocop
    spec
    stylelint
  ]
end

[paulz]

yes to parallel steps!

even as simple as true/false parallel steps would be huge time saving!

    steps:
      - name: Checkout
        parallel: true
        uses: actions/checkout@v3

      - name: Install Ruby Gems
        parallel: true
        run: bundle install

      - name: Run tests
        parallel: false
        run: rake 

[zsolt-dev]

Please vote for supporting parallel steps here: #14484

[qoomon]

Now it's possible, feel free to try my parallel-steps action https://github.com/qoomon/actions--parallel-steps

[qoomon]

The max parallel steps limit has been resolved. you now can run as many steps in parallel as you want.

Im curious about if it works for your use cases as expected?