Playwright.Test.GitHubFormatter (Playwright v1.65.0-rc.3)

Copy Markdown View Source

Playwright Test's github reporter as an ExUnit formatter: GitHub Actions annotations for failed tests, errors outside a test, slow test files and the run's summary. It prints only workflow commands, so it runs next to ExUnit.CLIFormatter, on CI, as upstream's documented reporter: process.env.CI ? 'github' : 'list':

mix test --formatter ExUnit.CLIFormatter --formatter Playwright.Test.GitHubFormatter

--formatter replaces the configured formatters, so name ExUnit.CLIFormatter (and any of your own) too. A CI that reruns failed tests (mix test || { [ $? -eq 2 ] && mix test --failed; }) passes it to both runs, as upstream annotates every attempt. Or add it in test/test_helper.exs:

github = if System.get_env("GITHUB_ACTIONS"), do: [Playwright.Test.GitHubFormatter], else: []
ExUnit.start(formatters: [ExUnit.CLIFormatter | github])

It prints:

  • an ::error for each failure of a failed test, at the first frame of its stacktrace in your code (not Playwright's, ExUnit's, Elixir's, Erlang/OTP's or a dependency's under deps/; a path dependency is yours), else on the test's file, titled file:line › describe › test, with ExUnit's failure text, under Retry #1 on the rerun.
  • an ::error for each failed setup_all; its tests count as not run.
  • a Slow Test ::warning for each of the 5 slowest files whose tests took more than 5 minutes (upstream's reportSlowTests: config :playwright, :test, report_slow_tests: [max: 5, threshold: 300_000] in ms, false for none).
  • a 🎭 Playwright Run Summary ::notice: the failed and flaky tests, the counts of skipped, not run and passed tests, and the run's duration.

ExUnit has no retries in a run: a test under mix test --failed is upstream's retry 1 (as Playwright.Test.info/0's retry), so one that passes there is flaky. Each run prints its own summary, and a flaky test keeps the first run's ::error, as upstream's flaky test does. Unlike upstream, a test not run after a failed setup_all that passes on the rerun counts as flaky too: the failures manifest doesn't record why a test is in it.

Paths are relative to GITHUB_WORKSPACE. It reads only ExUnit's events: any ExUnit suite can use it, with or without Playwright.Test.

Summary

Functions

Returns a specification to start this module under a supervisor.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.