Testing 3D applications with Playwright on GPU

Here at Promaton, we develop medical-grade web applications, which means that robust testing is a must. The most important component of these tests is the end-to-end testing of entire flows, and they are typically quite time-consuming, so we run these tests automatically with Playwright. We chose Playwright specifically because it comes out of the box with orchestrated test execution on several machines and because it uses native browser events, which is helpful for testing 3D applications that are drawn on canvas.

As we started expanding our test suite, we ran into a problem — the tests are fast and stable on a local ARM MacBook, but slow and falling apart on CI. There isn’t a lot of knowledge out there on how to test web 3D applications, so we spent some time investigating.

In the end, we were able to get all three major browsers to run with hardware acceleration, which significantly improved the runtime and stability of our E2E test suite. Here’s what we found!

TL;DR

• Having a performant GPU on the runner can significantly speed up CI, albeit at a higher cost
• Chromium on Linux needs a few flags to enable hardware acceleration in the new headless mode (available on the chromium channel since Playwright v1.49)
• Firefox has hardware acceleration enabled by default, but only uses it under Xvfb
• WebKit doesn’t support hardware acceleration in headless mode, so it must be run headed

The full command to run Playwright is:

# xvfb helps with WebGL in Firefox Headless
# --headed is required for WebKit, because its GPU acceleration doesn't work in headless mode
xvfb-run yarn exec playwright test --headed

And the browser profiles in playwright.config.ts are:

import { defineConfig, devices } from "@playwright/test"

// Source: <https://developer.chrome.com/blog/supercharge-web-ai-testing>
const chromiumGpuOnLinuxFlags = [
  "--use-angle=vulkan",
  "--enable-features=Vulkan",
  "--disable-vulkan-surface",
  "--enable-unsafe-webgpu",
]

export default defineConfig({
  // other irrelevant settings omitted
	use: {
		trace: "retain-on-failure"
	},
  projects: [
    {
      name: "chromium",
      use: {
        ...devices["Desktop Chrome"],
        // To enable the new headless mode; see <https://github.com/microsoft/playwright/releases/tag/v1.49.0>
        channel: "chromium",
        launchOptions: {
          args: [
            "--no-sandbox",
            ...(process.platform === "linux" ? chromiumGpuOnLinuxFlags : []),
          ],
        },
      },
    },
    {
      name: "firefox",
      use: { ...devices["Desktop Firefox"] },
    },
    {
      name: "webkit",
      use: { ...devices["Desktop Safari"] },
    },
  ],
})

How we got here

Not much is written on the internet about GPU-accelerated Playwright testing, but a great lead came from Puppeteer — another browser automation project, with less focus on testing. Its documentation has a few helpful sections about Chrome and GPUs:

chrome-headless-shell disables GPU compositing
chrome-headless-shell requires --enable-gpu to enable GPU acceleration in headless mode.

chrome-headless-shell is the old headless mode for Chromium, and it is described as the more lightweight, less precise option, whereas the new headless mode is more reliable and authentic. We chose the new mode to more accurately reflect real browsers.

Setting up GPU with Chrome
Generally, Chrome should be able to detect and enable GPU if the system has appropriate drivers. For additional tips, see the following blog post https://developer.chrome.com/blog/supercharge-web-ai-testing.

This article they mentioned turned out to be invaluable — it’s a detailed log of experiments to enable WebGPU on Chrome.

Time for our own experiments

We started trying to replicate it on an AWS machine with a GPU. It’s a fresh g5.xlarge instance with Ubuntu 24. You’d think it comes with pre-installed drivers for its GPU, but no :)

First, we need the CUDA keyring:

wget <https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb>
sudo dpkg -i cuda_keyring_1.1-1_all.deb
sudo apt-get update

Then we can install the NVIDIA driver:

sudo apt install nvidia-open

Then we restart the machine and verify with nvidia-smi:

|   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |

Got it! The GPU is detected, so the drivers must be working. Now, to get the browsers to notice…

Getting Chromium to use the GPU

In the “Supercharge Web AI model testing” article, there’s a useful snippet of code that uses Puppeteer. It runs Chromium in new headless mode with the given flags and prints the GPU report.

import puppeteer from 'puppeteer';

const browser = await puppeteer.launch({
  headless: 'new',
  // args:  ['--no-sandbox']  // without flags
  args: [  // with flags
    '--no-sandbox',
    '--headless=new',
    '--use-angle=vulkan',
    '--enable-features=Vulkan',
    '--disable-vulkan-surface',
    '--enable-unsafe-webgpu',
  ]
});

const page = await browser.newPage();
await page.goto('chrome://gpu');

await page.waitForSelector('text/WebGPU');
await page.pdf({ path: './gpu.pdf' });

await browser.close();

We can run this script on the remote machine with and without the flags to compare the outputs. These findings are just as relevant for Playwright, because both Playwright and Puppeteer control Chromium in a similar way.

Left: Without the flags (only `--no-sandbox` is included); Right: With all the flags (https://developer.chrome.com/blog/supercharge-web-ai-testing#enable-webgpu)

We can see that the flags make a big difference for Chromium. In particular:

• Canvas is hardware-accelerated
• OpenGL is enabled
• WebGL and WebGPU are hardware accelerated (even though it says “reduced performance”)

So that’s Chromium out of the way, but we still have Firefox and WebKit, and that helpful article doesn’t cover them. Dang. Back to searching for clues.

What about Firefox?

Searching around for WebGPU yielded a reasonably up-to-date page that tracks the implementation status of WebGPU across browsers: https://github.com/gpuweb/gpuweb/wiki/Implementation-Status.

Firefox
WebGPU is enabled by default in Nightly Firefox builds.
All the issues and feature requests are tracked by the Graphics: WebGPU component in Bugzilla. For more details, see the Mozilla wiki.

Servo
Work in progress, enabled by “dom.webgpu.enabled” pref.

Is Playwright using Firefox Nightly? Their docs were a bit confusing here. They claim that the version of Firefox they use matches the Stable build, however, when running Playwright tests in Firefox headed mode, the icon that shows up in the dock is Firefox Nightly.

We tried setting the dom.webgpu.enabled preference anyway. It didn’t help, regrettably — all the tests on Firefox were timing out. For future reference though, here’s how you can change user preferences in Firefox for Playwright:

import { defineConfig, devices } from "@playwright/test"

export default defineConfig({
  // other settings omitted
  projects: [
    {
      name: "firefox",
      use: {
        ...devices["Desktop Firefox"],
        launchOptions: {
          firefoxUserPrefs: { 
	          "dom.webgpu.enabled": true
	        },
        },
      },
    },
  ],
})

It would be great to have a way to print a GPU report for Firefox, the same way as we did for Chrome. Unfortunately, Puppeteer doesn’t work with Firefox, so the same code wouldn’t work. Firefox does have a similar page, though, located at about:support. Firefox also has a CLI that we can use to print screenshots of pages. Let’s see!

On our remote machine, we entered the folder where Playwright has downloaded the browsers (~/.cache/ms-playwright/) and ran the following command to get a screenshot:

./firefox -headless -screenshot ~/firefox-about-support.png about:support

Unfortunately, the report is rather underwhelming. It contained no information at all!

The “Graphics” section in the `about:support` report

However, it did refer to Firefox as “Nightly”, which confirms the theory — Playwright is indeed using the Nightly release of Firefox, so WebGPU must be enabled by default.

Another page to try when checking support for hardware-accelerated graphics is https://get.webgl.org/. Let’s take a screenshot!

Screenshot of the WebGL support page, taken with Firefox on the EC2 instance

$ ./firefox -headless -screenshot ~/firefox-webgl.png https://get.webgl.org/
*** You are running in headless mode.
JavaScript warning: resource://services-settings/Utils.sys.mjs, line 116: unreachable code after return statement
console.warn: services.settings: Ignoring preference override of remote settings server
console.warn: services.settings: Allow by setting MOZ_REMOTE_SETTINGS_DEVTOOLS=1 in the environment
Crash Annotation GraphicsCriticalError: |[0][GFX1-]: glxtest: Unable to open a connection to the X server (t=1.00299) [GFX1-]: glxtest: Unable to open a connection to the X server
Crash Annotation GraphicsCriticalError: |[0][GFX1-]: glxtest: Unable to open a connection to the X server (t=1.00299) |[1][GFX1-]: More than 1 GPU vendor detected via PCI, cannot deduce vendor
 (t=1.00599) [GFX1-]: More than 1 GPU vendor detected via PCI, cannot deduce vendor

Crash Annotation GraphicsCriticalError: |[0][GFX1-]: glxtest: Unable to open a connection to the X server (t=1.00299) |[1][GFX1-]: More than 1 GPU vendor detected via PCI, cannot deduce vendor
 (t=1.00599) |[2][GFX1-]: PCI candidate 0x1d0f/0x1111
 (t=1.00599) [GFX1-]: PCI candidate 0x1d0f/0x1111

Crash Annotation GraphicsCriticalError: |[0][GFX1-]: glxtest: Unable to open a connection to the X server (t=1.00299) |[1][GFX1-]: More than 1 GPU vendor detected via PCI, cannot deduce vendor
 (t=1.00599) |[2][GFX1-]: PCI candidate 0x1d0f/0x1111
 (t=1.00599) |[3][GFX1-]: PCI candidate 0x10de/0x2237
 (t=1.00599) [GFX1-]: PCI candidate 0x10de/0x2237

JavaScript warning: https://get.webgl.org/, line 193: Failed to create WebGL context: WebGL creation failed:
* WebglAllowWindowsNativeGl:false restricts context creation on this system. ()
* Exhausted GL driver options. (FEATURE_FAILURE_WEBGL_EXHAUSTED_DRIVERS)
JavaScript warning: https://get.webgl.org/, line 197: Failed to create WebGL context: WebGL creation failed:
* WebglAllowWindowsNativeGl:false restricts context creation on this system. ()
* Exhausted GL driver options. (FEATURE_FAILURE_WEBGL_EXHAUSTED_DRIVERS)
Screenshot saved to: /home/ubuntu/firefox-webgl.png

Doesn’t work, okay, but the logs are quite interesting. It says You are running in headless mode, but still tries to connect to the X server (and fails, because we’re on a server machine with no screen).

Another interesting part of the logs is WebglAllowWindowsNativeGl:false restricts context creation on this system. Looking up the error message yielded a question on the Firefox support forum, which suggests also setting the webgl.force-enabled preference. We tried running Playwright with both this preference and dom.webgpu.enabled, but the tests we still timing out.

If Firefox wants an X server, we could give it a virtual one using Xvfb. We only need to prepend xvfb-run to the command we run, and now this command should run with a temporary virtual screen. Taking a screenshot like that proved to be a success!

Screenshot of the WebGL support page, taken with Firefox on the EC2 instance, but under Xvfb.

$ xvfb-run ./firefox -headless -screenshot ~/firefox-webgl.png https://get.webgl.org/
*** You are running in headless mode.
console.warn: services.settings: Ignoring preference override of remote settings server
console.warn: services.settings: Allow by setting MOZ_REMOTE_SETTINGS_DEVTOOLS=1 in the environment
Screenshot saved to: /home/ubuntu/firefox-webgl.png

We added xvfb-run to the GitHub Actions workflow, and it worked! The tests started passing on Firefox, and were very stable.

Adding WebKit to the mix

With Chromium and Firefox working, it was finally time for WebKit. Tests on WebKit were also timing out, meaning that it probably wasn’t using the GPU. Unfortunately, the WebKit CLI doesn’t seem to have a screenshot option, so we were blindfolded here.

A promising lead came from a reply in an issue in Playwright’s GitHub repo about WebKit not rendering screenshots.

It’s export LIBGL_ALWAYS_SOFTWARE=true that forces CPU rendering, as described in the Mesa documentation at https://docs.mesa3d.org/envvars.html#envvar-LIBGL_ALWAYS_SOFTWARE.

It seems that libEGL loaded a device that’s incompatible with Webkit WPE in headless mode. According to the libEGL source code, available at https://gitlab.freedesktop.org/mesa/mesa/-/blob/mesa-24.0.9/src/vulkan/device-select-layer/device_select_layer.c?ref_type=tags#L492-526, it prefers GPU devices for rendering.

The LIBGL_ALWAYS_SOFTWARE environment variable didn’t bring us any closer, since our goal was hardware acceleration. However, this prompted us to try running WebKit in headed mode, since we had Xvfb from Firefox anyway. Adding --headed did the trick — the tests started running faster under WebKit, and crucially, finally stopped timing out.

However, now Firefox started misbehaving *sigh*. After some experimenting, we came to a bizarre realization that tests under Firefox headed mode were only passing when Playwright was “watching” — either by recording videos or traces. Not knowing what to make of it, we enabled traces for every run in the Playwright config:

import { defineConfig } from "@playwright/test"

export default defineConfig({
  // other settings omitted
  use: {
    trace: "retain-on-failure"
  } 
})

Result

In the end, we have managed to get our test suite to run swiftly and stable-y on CI. From a substantial 24 billable minutes to run the full suite on a default Azure runner (less in practice due to parallelization), we got it down to 3.5 minutes on a self-hosted AWS runner with a GPU. However, these tests still run twice as fast locally on an M3 MacBook Air, and given the amount of hacks and unprovable improvements, the end result doesn’t feel fully satisfying. Something to potentially explore in the future is why it’s still faster locally.

Another important thing to keep in mind is that the g5.xlarge GPU runner that we chose costs $1.06/hour. This is significant, although it’s still cheaper than GitHub’s GPU runners. To keep expenses manageable, we had to explore auto-scaling. Be mindful of the costs when setting this up for yourself!

Hopefully we managed to push the state of knowledge on Playwright testing Three.js applications. We also hope for a future when these flags and hacks are no longer necessary and testing 3D applications is fun and easy.

Christian and Lev of Promaton, signing out!