Playwright Test's per-test machinery on ExUnit: each test's result, its artifacts (traces,
screenshots, videos, error-context.md), Playwright Test's context defaults, soft errors,
snapshot and output paths, and steps. The browser comes from Playwright.Test.BrowserPool,
contexts and pages from the plain API; Playwright.Case puts them together.
# config/test.exs
config :playwright, :test,
trace: :on_first_retry, # traced under `mix test --failed`
screenshot: :only_on_failure,
base_url: "http://localhost:4000"
defmodule MyApp.CheckoutTest do
use ExUnit.Case, async: true
use Playwright.Test
alias Playwright.{Browser, BrowserContext, Page}
setup_all do
%{browser: Playwright.Test.BrowserPool.browser()}
end
setup %{browser: browser} do
%{page: BrowserContext.new_page!(Browser.new_context!(browser))}
end
test "pays", %{page: page} do
...
end
endThe runner is native, not a port of Playwright Test: it follows the runners of the official
non-JS clients (pytest-playwright, Playwright.NUnit, Java's @UsePlaywright) on ExUnit, with
Playwright Test's names and defaults where they exist (packages/playwright/src/index.ts).
What the ported matchers and traces need of Playwright Test's worker is ported: testInfo's
snapshot paths and errors (testInfo.ts), step/3 (test.step) and the test.trace of
trace.zip (testTracing.ts).
Registration
use Playwright.Test comes after use ExUnit.Case (or in a case template's using). It
imports a test macro, which hands the test to ExUnit.Case.test/3 with the body's end
marked, so the test's on_exit learns whether it passed, and assert/1,2 and refute/1,2,
which are also steps (see "Assertions"). And it adds a setup that
registers each test: its recording modes, its output dir and an on_exit (the first, so it
runs last) that keeps or drops the artifacts by the result, then releases the test's
BrowserPool.browser/1 drivers. Tests of other macros (doctests, a property) count as
failed for their artifacts, and a failure only in an on_exit callback counts as passed.
In a case template, use Playwright.Test may also be the first line of the template's
body: a template's own setup callbacks run before those its using brings in, so the test
is registered before them, and their contexts are recorded too. That use takes no options
(they go in using); a module's later uses (a template's using and the module's own) only
add their options (see "Artifacts"). With the use only in the template's body, the
setup_all timeout (see "Timeouts") covers only the template's setup_all callbacks. A
context that a test process creates, or a BrowserPool.browser/1 call it makes, before its
registration (a template's setup with no use in the body) makes the registration raise,
with that fix. It registers the module's setup_all process too, as upstream runs its
beforeAll hooks with its all-hooks-included fixtures: see "Contexts of setup_all". A
setup_all runs in that process, not a test's, so one that creates a context or calls
BrowserPool.browser/1 before the registration (a template's own setup_all, or one above
the module's use) makes the module's setup_all raise, with the fix, and fails all its tests. A context created outside
both (in an on_exit callback, in a module without use Playwright.Test, in a process with
neither in its $callers, on a driver you started yourself) gets no defaults and no recording.
The rest is called qualified, as upstream's test.info() and test.step(): info/0
(info/1 in on_exit, with the test's result), output_path/1, attach/2, snapshot_path/2, step/2,3 and step_skip/2,3. They work in the test's
process and in processes with it in $callers (a Task, and the client's callback
processes, see "Route handlers and listeners" in the Testing with ExUnit guide); outside a
test they raise. info/0, step/2,3 and step_skip/2,3 also work in the module's
setup_all: info/0 with the module's fields (see "Contexts of setup_all"), the steps
unrecorded (see "Steps").
An error a listener or route handler of the test raises fails the test and stops it at once,
as upstream's worker does (unhandledError): the test's process exits with the error, and
later such errors are dropped. A test that traps exits isn't stopped: ExUnit may report it
passed, while the error is in error-context.md and Playwright.Test.info/1. A test
defined with ExUnit.Case.test isn't a Playwright test: its listener errors are only logged.
Two deviations: the stopped test's failure screenshots are taken at teardown, as its contexts
close, after its on_exit callbacks may have run, where upstream takes them first
(_didFinishTestFunction); and once the body ended (in on_exit), such an error is logged,
where upstream still fails the test during its afterEach hooks and teardown.
Artifacts
Every context the test creates on a BrowserPool browser, in the test or a process with it
in $callers, is recorded. The recording starts with the test's first context (or at its
registration when it traces, for its steps); a test without contexts records nothing.
Contexts left open are closed after the test, last in its on_exit, and request contexts of
Playwright.APIRequest.new_context/2 left are disposed; a context closed earlier keeps its
artifacts. Each test writes to its own dir under output_dir, named as upstream's
testInfo.outputDir (checkout-pays-1a2b3 for the test "pays" in test/checkout_test.exs,
the suffix a hash of the module and its parameterize: parameters):
trace.zip- one trace of all its contexts, theirrequests and its request contexts (Playwright.APIRequest.new_context/2), recorded with screenshots, snapshots and sources unless the trace's options say otherwise. A failed test that keeps one printsmix playwright.cli show-trace <path>after the run, below ExUnit's summary.test-finished-N.png, ortest-failed-N.pngfor a failed test - a screenshot of every page open at the end of the test's body (before itson_exitcallbacks), then of those open when their context closes (timeout: 5000, caret: :initial, as upstream). A test that a listener's or route handler's error stopped (above) has no end of its body: its pages' screenshots are taken as its contexts close, after itson_exitcallbacks (upstream takes them before theafterEachhooks).video.webm,video-1.webm, ... - every page's video. A context created with its ownrecord_videokeeps it, and its video is copied here too.error-context.md- a failed test's errors, source and an aria snapshot (mode: :ai) of the first page of the first context closed after the failure (upstream'sdidFinishTestandbuildErrorContext), whatever the modes.
The modes are config, the project's over it (parameters/1), the module's use options over
those, and use/1 in tags over those (a test's @tag over its @describetag over the
module's @moduletag): trace and video :off
(default), :on, :retain_on_failure or a retry mode, screenshot :off, :on,
:only_on_failure or :on_first_failure (upstream's deprecated :retry_with_trace and
:retry_with_video are :on_first_retry; another value records nothing, as upstream's). Each
is also the :mode of a keyword list (or map) of options, as upstream's object forms, :off
without one: trace: [mode: :on, snapshots: false], screenshot: [mode: :on, full_page: true], video: [mode: :on, size: %{width: 640, height: 480}]. The trace's :snapshots (a boolean or %{dom: _, aria: _, screen: _}),
:screenshots and :sources are Playwright.Tracing.start/2's, true by default, as is
:attachments (those of attach/2 in trace.zip); the
screenshot's :full_page and :omit_background are Playwright.Page.screenshot/2's; the
video's :size and :fps are record_video's, and its :show is upstream's map:
%{actions: _} is record_video's showActions, %{test: %{level: _, position: _, fontSize: _}} overlays the test's title and its steps' (level "file", "title" or "step") on
each page of its contexts. With it, don't call step/2,3 in a route handler that holds a
page's navigation (a document request): the step's overlay waits for that navigation, so the
test hangs until its timeout, as upstream's does.
use Playwright.Test, trace: :on is upstream's file-level test.use({ trace: 'on' }) (the
modes are worker-scoped options there), with the values the config takes; an unknown option (or a
context option the facade rejects) raises at compile time. A module's uses merge per key, the later winning
(see "Context defaults").
A template's own use Playwright.Test, ... lines go before the forwarded options, which win per
option, as a test file's test.use beats a project's device.
ExUnit.Case's keys (:async, ...) are ignored, so a template forwards its options from using opts as is: use Playwright.Test, unquote(opts), as Playwright.Case does. A template
with options of its own pops them first ({sandbox, opts} = Keyword.pop(opts, :sandbox)),
as they raise here. Tags named trace, screenshot or video set no mode: only use/1's
keys are read.
The *_on_failure modes record every test and keep the files of failed ones, as upstream
(files you write to output_path/1 stay). A failing artifact step is logged; the test's
result stands.
The retry modes go by the test's retry (see below), as upstream's: :on_first_retry
records retry 1, :on_all_retries every retry, :retain_on_first_failure records the first
run and keeps a failure's, :retain_on_failure_and_retries records every run and keeps a
failure's and every retry's; the screenshot's :on_first_failure is :only_on_failure of
the first run.
A test's dir is deleted before it runs, as upstream's, so no file of an earlier run is left
in it. ExUnit has no retries in a run: a test under mix test --failed is upstream's retry
(info/0's retry, 1), and writes to <dir>-retry1, next to the dir of its first run. So
with trace: :on_first_retry,
mix test || { [ $? -eq 2 ] && mix test --failed; } traces the failures' rerun and keeps the
first run's files (exit status 2 means tests failed; anything else isn't retried). A run
without --failed deletes the test's -retry1 dir too, where
upstream deletes all of output_dir at the start of a run: here the dirs of tests that didn't
run stay. Config preserve_output is upstream's preserveOutput: :always (default) keeps
the files at the end of each test, :failures_only only a failed test's, :never none.
Context defaults
Every context and request context a test (or its module's setup_all) creates on a
BrowserPool browser (Playwright.Browser.new_context/2,
Playwright.BrowserType.launch_persistent_context/3, Playwright.APIRequest.new_context/2) gets Playwright Test's context defaults (locale: "en-US", a 1280x720 viewport, color_scheme: :light, ...), then the config's context options
over them, then the project's, the module's, the describe's and the test's (below), for each
option the call doesn't give, as upstream's _combinedContextOptions. viewport: :null is
none. An option replaces the one under it as a whole (extra_http_headers too, as upstream's
shallow spread).
The context options are options of their own, upstream's option fixtures, with the names
Playwright.Browser.new_context/2 takes them by: accept_downloads, base_url,
bypass_csp, client_certificates, color_scheme, contrast, device_scale_factor,
extra_http_headers, forced_colors, geolocation, has_touch, http_credentials,
ignore_https_errors, is_mobile, java_script_enabled, locale, offline, permissions,
proxy, reduced_motion, screen, service_workers, storage_state, timezone_id,
user_agent and viewport. use Playwright.Test, locale: "de-DE" (or use Playwright.Case, ...) is upstream's file-level test.use({ locale: 'de-DE' }): the module's options,
evaluated and validated at compile time (the values as new_context/2 checks them). A
function raises there: upstream's function-valued options, its option fixtures' functions,
aren't ported. A module's uses apply in order, each over those before it per option.
A device is a keyword list of these options and its default_browser_type (see "Browser"),
spread into the others as upstream's ...devices['iPhone 13']: use Playwright.Test, Playwright.device("iPhone 13") after use Playwright.Case, async: true, or in one use,
use Playwright.Case, [async: true] ++ Playwright.device("iPhone 13"). A later key wins, as in
JS's spread, so upstream's { ...devices['iPhone 13'], viewport } is
Playwright.device("iPhone 13") ++ [viewport: %{width: 800, height: 600}].
The options may be a map too.
context_options is upstream's contextOptions: new_context/2's options, for those that
are no option of their own (record_har, strict_selectors). It is one option, as upstream's
one fixture: the highest level that sets it replaces the others' as a whole. And it is under
every option of its own, whatever level sets either, as upstream's fixture of an option reads
contextOptions only when no level sets the option. context_options: nil falls back as any
option's nil (below), so does a misspelled Playwright.devices()["typo"], silently: use
Playwright.device/1, which raises for an unknown name.
Below the module, use/1 gives the options to ExUnit as a tag, for a describe block
(@describetag Test.use(locale: "de-DE"), upstream's test.use in a test.describe) or a
test (@tag); @moduletag works too, the worker options included, as upstream's top-level
test.use. A project's
are parameters/1 in the module's parameterize: (upstream's project.use).
The layers, each over the one before per option: Playwright Test's defaults, the config's,
the project's, the module's (use, then @moduletag), the describe's, the test's, then the
call's own. Any option a layer (the config, a project, the module's use, a tag) gives as
nil is unset there, as upstream's undefined option override, so the layer under it gives
it, in the end the default: the config's nil falls back to the default, the project's to
the config's, and a module's or a tag's to the project's or the config's, past the module's
own uses and tags, as upstream's test.use of undefined goes back to the config's.
context_options: nil is the bag of the layer under. A call's own options are no layer: a
context option's nil there is passed as given, over the layers, as upstream's
newContext({ locale: undefined }) spreads it over the defaults, so
Browser.new_context!(browser, locale: nil) gets the client's default, not the layers'.
BrowserPool.browser(connect_options: nil) launches; its browser_name and
launch_options take no nil. The config's, the project's and the
module's reach every context the module creates: a test's, those of its Tasks, and those of
setup_all, as upstream's beforeAll gets the use options. A describe's and a test's reach
the test's contexts only: ExUnit runs setup_all without them. The project's options are read
from the parameters ExUnit runs the module with, the others from the test's tags.
Browser
BrowserPool.browser/1 without a browser_name, as Playwright.Case calls it, launches the
module's browser, upstream's browserName: the browser_name of the module's use
(use Playwright.Case, browser_name: :webkit), else its project's (parameters/1), else the
config's if it sets one. Else their default_browser_type, in the same order, upstream's
worker option defaultBrowserType, which a device has (Playwright.device("iPhone 13") is
WebKit's) and upstream's browserName defaults to; else :chromium. The module's
launch_options (its use's, else its project's, else the config's, each as a whole) and
connect_options are that call's defaults too, upstream's launchOptions and
connectOptions: a project with launch_options: [channel: "chrome"] gets a worker of its
own. So are its headless and channel, upstream's options of their own, set over the launch
options' when given (headless: false launches headed whatever launch_options say); unset,
they are the launch options' (headless true when those have none). These six are the
worker's, as upstream's: the module's (its use, then all of its
@moduletag Test.use(...)s merged, the later per option, upstream's top-level test.use, for
every test of the module, also those above such a line) or its project's. Its other options
don't reach a test above the line, as ExUnit gives a test the @moduletags set before it:
under @moduletag Test.use(Playwright.device("iPhone 13")), such a test gets the device's
default_browser_type but not its viewport or user agent, so put the line above the tests.
In a @describetag or a @tag of a test of the test macro they raise when the module compiles, as upstream's test.use in a describe
when the file loads, a device's default_browser_type too; other tests (a doctest, a bare
ExUnit.Case.test) get the module's worker regardless. For a browser or a device per run,
make each a project (parameters/1) in the module's parameterize:.
Where each option can be set:
| Option | Config | Project (parameters/1) | Module (use, then its @moduletag use/1s, merged) | Describe, test (@describetag, @tag use/1) |
|---|---|---|---|---|
the context options, context_options, action_timeout, navigation_timeout | yes | yes | yes | yes |
trace, screenshot, video | yes | yes | yes | yes (an addition: upstream's are the worker's; they work fully per test, as the recorder records per test and touches no browser) |
browser_name, headless, channel, launch_options, connect_options | yes | yes | yes | raises when the module compiles a test under it, as Playwright Test's test.use in a describe |
default_browser_type (a device's) | picks the browser | picks the browser | picks the browser | raises, as the browser keys |
test_id_attribute | yes | no | no | no (a deviation: upstream's is an option of every level, global to a worker, which runs one test at a time; here it is global to the VM, which the tests running at once share) |
output_dir, preserve_output, workers, snapshot_dir, snapshot_path_template, driver, on_start | yes | no | no | no |
Contexts of setup_all
use Playwright.Test adds a setup_all (where the use is, so before the module's own) that
registers the module's setup_all process: every context and request context it creates, or
a process with it in $callers creates, gets the context defaults, the config's
context options (so base_url), the project's and the module's (its use and @moduletag,
see "Context defaults"), merged as a test's, and the timeouts of those same layers (see
"Timeouts"): a context's action_timeout and navigation_timeout, a request context's
timeout: the action_timeout (0, no limit, for :infinity) and no navigation timeout. Not
the video, as upstream's comes with the context fixture.
info/0 there returns the module's file, timeout and use (Playwright.Case reads the
browser_name from it): see info/0, also for reading the config outside these scopes.
Only processes with $callers count: a Task, the client's callback processes (route
handlers, listeners, exposed bindings registered in setup_all carry its $callers, see
"Route handlers and listeners" in the Testing with ExUnit guide), or a process that sets them
itself (Process.put(:"$callers", callers)). A spawn, a GenServer or Agent start, and
start_supervised don't set them, so their contexts get no defaults. Deviation: a long-lived
process or handler of setup_all's with them gets the module's defaults for every context it
creates later, during a test too, unrecorded. Upstream's runBeforeCreateBrowserContext
resolves the current test, so such a context gets that test's options and is recorded in it.
Not ported yet: a setup_all context isn't recorded per test (its actions aren't in a test's
trace, and it gets no screenshots or error-context.md), where upstream's ArtifactsRecorder
records every open context per test, beforeAll ones included: create the context per test,
as Playwright.Case does. And after a failed test, setup_all state (its pages too) carries
on, where upstream starts a fresh worker: bound calls on shared pages with action_timeout.
Timeouts
As in Playwright Test, a test's contexts have no action or navigation timeout by default: the
test's timeout bounds them. action_timeout and navigation_timeout (the config's, or an
option of any level over it; upstream's
actionTimeout, navigationTimeout; milliseconds, :infinity by default) set each context's
Playwright.BrowserContext.set_default_timeout/2 and set_default_navigation_timeout/2
(navigation_timeout alone governs navigations), and the action timeout is a request
context's timeout, when the call gives none. 0 is no limit too, not an immediate failure,
as timeout: 0 is everywhere in Playwright (see API conventions):
write :infinity to say so.
Shortly before ExUnit's timeout (1 s before, at most 5% of the timeout), the test's open
contexts are closed with upstream's reason, "Test timeout of 60000ms exceeded.": a call that
hangs until then fails with that message and its call log, after the close kept the artifacts.
A test stuck elsewhere (Process.sleep/1, a receive) still fails with ExUnit's
ExUnit.TimeoutError. The timeout is ExUnit's: the @tag timeout: (:infinity for none) or
mix test --timeout; mix test --trace has none. As ExUnit's (and upstream's, which counts
beforeEach hooks), the timeout counts the test's setup callbacks: the clock starts at the
module's first one, also one defined before use Playwright.Test or in its case template.
What ExUnit does before them (@tag :tmp_dir's cleanup, capture_log) is not counted, and
comes out of the margin.
A module's setup_all callbacks get the same timeout, the module's (@moduletag timeout:, else
ExUnit's timeout: ExUnit.start(timeout:) or mix test --timeout), as upstream's beforeAll
hook gets the test timeout: past it, the setup_all fails with upstream's message with
beforeAll renamed, "setup_all" hook timeout of 60000ms exceeded., and the stack it was at.
ExUnit has no timeout there, so the runner exits the setup_all process, which fails the
module as a crash does; its on_exit callbacks still run. The failure is ExUnit's for an exit
(** (EXIT from #PID<...>)), with the stack unpruned, and a hung call fails without its call
log: unlike a test's, its contexts aren't closed with a reason before the exit. Its pooled
browsers are reset at the module's end instead, as a failed test's, which closes its contexts:
a failed setup_all's are, as upstream stops a worker whose job failed. A setup_all process
that traps exits is killed 1 s after the timeout instead: it fails with :killed, without the
message. One timeout counts all of the module's setup_all callbacks, where upstream counts
each hook on its own. BrowserPool.browser/1 (the wait for a worker, its start) is not counted,
as upstream's queue has no timeout and its worker fixtures have a timeout of their own: there,
nothing bounds them. Its contexts get the action_timeout and
navigation_timeout of the config, the project and the module (:infinity by default, not the client's 30 s), as upstream's beforeAll
contexts get actionTimeout || 0.
Snapshots
In a test, Playwright.Expect.to_have_screenshot/2 and
Playwright.Expect.to_match_aria_snapshot/3 take upstream's name: and resolve it as
testInfo does, by upstream's default templates:
- screenshots:
{snapshotDir}/{testFileDir}/{testFileName}-snapshots/{arg}{-snapshotSuffix}{ext},test/checkout_test.exs-snapshots/home-darwin.pngforname: "home.png"intest/checkout_test.exs. - aria snapshots:
{snapshotDir}/{testFileDir}/{testFileName}-snapshots/{arg}{ext}.
Without a name, a snapshot is named after the test and its count in the test:
pays-1-<platform>.png for a screenshot, pays-1.aria.yml for an aria snapshot.
A name is sanitized for the file system, a list of path segments is not. snapshot_dir is
test/ by default; snapshot_path_template (or the matcher's path_template in config :playwright, :expect) takes upstream's tokens: {testDir}, {snapshotDir},
{testFileDir}, {testFileName}, {testFileBaseName}, {testFilePath}, {testName},
{arg}, {ext}, {platform} and {snapshotSuffix} (the platform, as upstream's);
{projectName} is the name of the module's project (parameters/1), empty without one. snapshot_path/2
returns a name's path. A failed comparison writes its -actual, -expected, -previous and
-diff images to the test's output dir.
A snapshot the matcher writes (a missing one, under the default update_snapshots) fails the
test at the end of its body, as upstream's soft error: the rest of the body runs, and ExUnit
reports all of the test's errors, also when another failure follows. path: in place of
name: is the snapshot file itself, also outside a test, where a written snapshot is printed.
Steps
step/3 is upstream's test.step: Playwright.Test.step "log in" do ... end, or
step(title, fun, options) for wrappers, with upstream's box, location, params and
subtitle (step_skip/3 is test.step.skip). Each matcher call in a test is a step too
(category: 'expect'), and a soft one's failure (Playwright.Expect.soft/1, upstream's
expect.soft) fails the test at the end of its body, as a written snapshot does. So is each
driver call (pw:api, titled as upstream's: Click getByRole('button')), in the step or
Playwright.Tracing.group/3 it runs in; a matcher's driver call is its step. The trace's
test.trace entry (upstream's TestTracing) has the steps, nested by the step they run in,
and the test's errors, so trace: :on gives a trace.zip also for a test without contexts.
attach/2 adds a test.attach step with the attachment (the trace's attachments: false
leaves the attachments out). Not ported: the test's stdout and stderr in the trace, hooks and
fixtures as steps, the step's TestStepInfo (the body takes no argument, so no
step.attach) and its timeout (the test's timeout bounds a step).
A step also works in the module's setup_all (and processes with it in $callers), as
upstream's in a beforeAll hook: it returns its body's value, nests, and raises its error on,
a boxed step's where its helper is called. Deviation: it is recorded nowhere (no trace, no
video overlay), also in a long-lived process or handler of setup_all's during a test, where
upstream records a beforeAll step in the first test (its trace's "Before Hooks") and a
handler's in the current test: ExUnit runs setup_all before any test, and hooks aren't
steps here.
Assertions
use Playwright.Test imports Playwright.Test.Assertions' assert/1,2 and refute/1,2 in
place of ExUnit.Assertions' (it imports that module again without them). Each is ExUnit's, also a
step of the running test, as upstream's expect(value).toBe(...) is: category: 'expect',
titled Expect "assert" or Expect "refute", with the expression as its subtitle and its
expression param, located at the call. Its expected param, as upstream's
toEqual(expected)'s, is the right side ExUnit's expansion binds (left == right and the other
operators, pattern = right, match?(pattern, right); a failed one's from its error), when it
is JS-truthy (if (args[0]): not nil, :null, false, 0, 0.0 or ""), previewed as upstream's
generatePreview but cut to 100 list elements and 1000 characters (…), as upstream's isn't
(Elixir only: bound trace size); a truthy assert x has none. A message (assert x, "msg")
is the title, as upstream's expect(value, message).
A passed one expands to ExUnit's assert as it is, then records its step, so it binds as
ExUnit's (assert {:ok, user} = create(), assert (n = count()) > 0), and its driver calls
come before it, as upstream's. A failed one raises ExUnit's error, with its message and
stacktrace; only a function that ends in an assertion keeps its frame in the stacktrace, as
the assertion is no longer its tail call (as JS keeps every frame). A failed assert/1 or
refute/1 is recorded once, with that error, where it ends: in a Playwright.Expect.to_pass/2
attempt, else at the end of the test's body, as a step of the test's (not of the step/3 it
failed in, which fails with the same error). assert/2 and refute/2 are functions in
ExUnit, so only their call is wrapped in a try, which records the failure there, in its
step, also when the test rescues it.
Outside a test, and in the module's setup_all (as a step there), nothing is recorded. The
other assertions (assert_receive, assert_raise, ...) are ExUnit's, no steps.
They are on by default. use Playwright.Test, assert_steps: false keeps ExUnit's own
assert and refute (its plain import ExUnit.Assertions) and records none in its tests,
passed or failed, a helper module's either; config :playwright, :test, assert_steps: false makes that the default,
which a module's assert_steps: true overrides. As it decides the import, it is read when the
module compiles: set it in config/config.exs or config/test.exs, not config/runtime.exs
(Mix recompiles the modules when it changes). The option takes a literal boolean.
Limits:
- A passed assertion is recorded only where the import is: a helper module opts in with
import Playwright.Test.Assertions, and, if it importsExUnit.Assertionstoo,import ExUnit.Assertions, except: [assert: 1, assert: 2, refute: 1, refute: 2]. A library with its ownassert/2excludes the arities it defines (except: [assert: 2]); its ownrefute/2also takes therefute(value, expr: ...)that ExUnit'srefute match?calls (seePlaywright.Test.Assertions). A failedassert/1orrefute/1is recorded wherever it was written. - A test module that imports
ExUnit.Assertionsagain afteruse Playwright.Testfails to compile, with an ambiguous call ofassert: drop that import, or add the sameexcept:. - A case template passes the import on with
use Playwright.Testin itsusing; the one in its body (see "Registration") doesn't. - A failed
assert/1orrefute/1is a step of the test's, not nested in thestep/3it failed in. One that the test rescues itself (a retry loop without steps) leaves no step, and so does one in the test's body undermix test --breakpoints, which runs the body without thetestmacro'scatch. One rescued and raised again withraise error(notreraise) is located at thatraise. - A failed
assert/1orrefute/1in a process of its own (aTask, a listener, a route handler) or in asetupcallback leaves no step: the test body's catch doesn't see it. - Each attempt of
Playwright.Expect.poll/3is its assertion's own steps, under the poll's step (as upstream's attempts are its matcher's); a poll with a message has the matcher's, titled by it, as upstream's. An attempt without one of theseasserts (assert_receive, a raise of its own) leaves no step when it passes, the matcher's when it fails. - A failed assertion's expression shows a capture's argument as ExUnit's error has it
(
_& == 2for&assert(&1 == 2)); a passed one's is&1 == 2. On Elixir 1.18 both show the argument ascapture(capture == 2).
Config
config :playwright, :test,
browser_name: :chromium, # BrowserPool: :chromium | :firefox | :webkit, or "chromium", ...
headless: nil, # BrowserPool: over launch_options' (nil: launch_options' headless, else true)
channel: nil, # BrowserPool: over launch_options' (nil: launch_options' channel)
launch_options: [], # BrowserPool: Playwright.BrowserType.launch/2
connect_options: nil, # BrowserPool: [ws_endpoint: _, ...] to connect instead of launching
driver: [], # BrowserPool: [env: _], Playwright.Supervisor's (the driver is config :playwright, :driver's)
on_start: nil, # BrowserPool
workers: max(div(System.schedulers_online(), 2), 1),
context_options: [], # Playwright.Browser.new_context/2's options, as upstream's contextOptions
# base_url: "http://localhost:4000", locale: "de-DE", ...: a context option, unset by default
default_browser_type: nil, # a device's: the browser, without a browser_name
action_timeout: :infinity, # ms, each context's default (and navigation_timeout's); :infinity or 0: none
navigation_timeout: :infinity,
trace: :off, # :off | :on | :retain_on_failure | :on_first_retry | ..., or [mode: _, ...]
screenshot: :off, # :off | :on | :only_on_failure | :on_first_failure, or [mode: _, ...]
video: :off, # as trace, or [mode: _, size: _]
output_dir: "test-results",
preserve_output: :always, # :always | :failures_only | :never, or "failures-only", ..., see Artifacts
snapshot_dir: nil, # upstream's testDir: test/
snapshot_path_template: nil, # upstream's defaults, see Snapshots
test_id_attribute: "data-testid",
report_slow_tests: [max: 5, threshold: 300_000], # Playwright.Test.GitHubFormatter's, ms; false: none
assert_steps: true # assert/refute as steps, at compile time (see Assertions)Each context option (base_url, locale, viewport, storage_state, ...) is a config key of
its own, over the context_options bag, as under every other level, and in config/0 only
when set. A test reads the resolved values with info().use.
The artifact modes are off by default and output_dir is test-results, as upstream's.
workers is half the schedulers, as upstream's default workers is half the cores ('50%').
test_id_attribute is upstream's testIdAttribute, set on each pooled driver: the client's
value is global to the VM, so set it here, not per module. config/0 returns the config
resolved. An unknown key raises at application start (and when the runner reads a config
changed since), and so does a bad value of the keys upstream checks: preserve_output,
output_dir, browser_name, report_slow_tests and the context options (as
Browser.new_context/2 checks them), and of workers (a positive integer: upstream's also
takes "50%" or a numeric string, which aren't supported), on_start and Elixir's
assert_steps (a boolean). browser_name and preserve_output take an atom (:webkit,
:failures_only) or upstream's string ("webkit", "failures-only"): an unknown atom raises
as any bad value, an unknown string with upstream's error. Other values pass as given, as
upstream's: an unknown artifact mode records nothing.
Summary
Functions
Upstream's testInfo.attach(name, options): attaches a file or a binary to the running test,
as a test.attach step of its trace, which shows it. Options
The resolved config :playwright, :test (see "Config" in the moduledoc), for libraries
that build on the runner: a map of the listed keys, each with its default filled in and its
value validated, as the runner reads it, and the context options that are set. The recording
modes are keyword lists with :mode first (trace: [mode: :off, snapshots: true, ...]), a
context option is as configured (none of Playwright Test's context defaults; one set only in
context_options is there only), and an unset action_timeout or navigation_timeout is
:infinity. A test's resolved options, over every level, are info().use. expect: %{timeout: ms} is the matchers' timeout of config :playwright, :expect (5000 by default, 0 for none). Works outside a test too (at startup, in a module
without use Playwright.Test); raises ArgumentError for an invalid config.
The running test of the calling process or a process in its $callers (a Task), as
upstream's test.info(): test_pid, file (absolute), test_name (without the macro's
test prefix), output_dir (absolute, with its -retry1 suffix under mix test --failed,
see "Artifacts" in the moduledoc), retry (upstream's testInfo.retry: 0, or 1
under mix test --failed) and timeout (upstream's testInfo.timeout: ExUnit's timeout of the test in
ms, the @tag timeout: or mix test --timeout, :infinity for none, as under mix test --trace) and deadline (upstream's internal testInfo._deadline(): monotonic milliseconds
(compare with System.monotonic_time(:millisecond)) by which the test must be done: after
it, the test's contexts are closed and Playwright calls fail; :infinity without a test
timeout; see "Timeouts"), from the test's setup to the end of its body. use is the test's
options as resolved from every level (the config's, its project's, its module's, its
describe's and its own), what upstream's option fixtures give a fixture: a map of the context
options every context of the test starts from (base_url, locale, extra_http_headers,
...; those set, and Playwright Test's defaults), action_timeout, navigation_timeout,
trace, screenshot, video, and the worker options of its BrowserPool.browser/1 calls
(browser_name, headless, channel, launch_options, connect_options). A library that passes options in a
call merges into it, as the call's replace the resolved ones as a whole:
Browser.new_context!(browser, extra_http_headers: Map.merge(Test.info().use[:extra_http_headers] || %{}, headers)). A library that waits or
retries on its own (a server-side backstop) should stop by deadline, as the matchers
stop 250 ms before it, as upstream's stop 250 ms before its test deadline.
info/0 of the test test_pid (the context's test_pid), from any process, from its setup to
the end of its on_exit callbacks. Once the test's process has exited, so in on_exit, it has
upstream's result fields too, as testInfo has in teardown
The path of path_segments (a string or a list of them) in the test's output dir (see
"Artifacts" in the moduledoc), as upstream's testInfo.outputPath(...pathSegments): creates the
dir, and raises for a path outside it. Files written
there stay when the test passes, under the default preserve_output.
A project for parameterize:, upstream's projects: [{ name, use: { ... } }]: name: (a
string, upstream's project.name), use/1's options and the worker's (browser_name,
default_browser_type, headless, channel, launch_options, connect_options), validated
as it does. A map of
one private key, :"playwright.project", whose value is a Playwright.Test.Project, which
ExUnit takes there (a tag takes use/1's keyword list).
The snapshot path of name for the running test, as upstream's testInfo.snapshotPath:
where Playwright.Expect.to_have_screenshot/2 (kind: :screenshot) and
Playwright.Expect.to_match_aria_snapshot/3 (kind: :aria) with name: read and write it
(see "Snapshots" in the moduledoc). name is a string or a list of path segments
(["home", "hero.png"], upstream's snapshotPath(...name)), which are not sanitized. Option
:kind: :snapshot (default, upstream's), :screenshot or :aria.
Upstream's test.step(title, body, options): runs the body as a step of the running test
and returns its value, also from a process with the test in its $callers (a Task); steps
nest. A step's error is raised on after the step records it. In the module's setup_all
(and processes with it in $callers) it runs too, recorded nowhere (see "Steps" in the
moduledoc); elsewhere it raises.
Upstream's test.step.skip(title, body, options): a step that doesn't run its body, marked
skip in the trace. Returns nil. The forms and options are step/3's.
ExUnit.Case.test/1: a test that fails as not implemented.
ExUnit.Case.test/3, with the end of the body marked: the test passed if its process then
exits as ExUnit ends a test, and fails there with the errors its matchers recorded (see
"Snapshots" in the moduledoc). A catch notes an error for error-context.md and re-raises it
unchanged, so ExUnit reports the failure as without this macro (with the recorded errors
before it, if any).
Options for a module's @moduletag, a describe block or a test: upstream's test.use. It
takes the options of use Playwright.Test (the context options, context_options,
action_timeout, navigation_timeout, trace, screenshot, video, and in @moduletag
the worker's), as a keyword list or a map, validates them as it does, and returns a keyword
list for an ExUnit tag. A device in @moduletag too; in a describe's or a test's tag only
without its default_browser_type.
Functions
Upstream's testInfo.attach(name, options): attaches a file or a binary to the running test,
as a test.attach step of its trace, which shows it. Options:
:path- a file, copied toattachments/<name>-<hash><ext>in the test's output dir, which keeps it also when the test passes.:body- the content itself.:content_type- by default the path's MIME type, ortext/plainfor a UTF-8body(application/octet-streamotherwise).
One of path and body, or neither for an attachment of just a name. Raises outside a test.
Playwright.Test.attach("screenshot", body: Page.screenshot!(page), content_type: "image/png")
@spec config() :: map()
The resolved config :playwright, :test (see "Config" in the moduledoc), for libraries
that build on the runner: a map of the listed keys, each with its default filled in and its
value validated, as the runner reads it, and the context options that are set. The recording
modes are keyword lists with :mode first (trace: [mode: :off, snapshots: true, ...]), a
context option is as configured (none of Playwright Test's context defaults; one set only in
context_options is there only), and an unset action_timeout or navigation_timeout is
:infinity. A test's resolved options, over every level, are info().use. expect: %{timeout: ms} is the matchers' timeout of config :playwright, :expect (5000 by default, 0 for none). Works outside a test too (at startup, in a module
without use Playwright.Test); raises ArgumentError for an invalid config.
Playwright.Test.config().workers
@spec in_registered_scope?() :: boolean()
Whether info/0 works in the calling process: a registered test or module setup_all, or a
process with either in its $callers. info/0 has test_pid only in a test. See info/0 for
reading the config elsewhere.
@spec info() :: %{ test_pid: pid(), file: Path.t(), test_name: String.t(), output_dir: Path.t(), retry: non_neg_integer(), timeout: timeout(), deadline: integer() | :infinity, use: map() } | %{file: Path.t(), timeout: timeout(), use: map()}
The running test of the calling process or a process in its $callers (a Task), as
upstream's test.info(): test_pid, file (absolute), test_name (without the macro's
test prefix), output_dir (absolute, with its -retry1 suffix under mix test --failed,
see "Artifacts" in the moduledoc), retry (upstream's testInfo.retry: 0, or 1
under mix test --failed) and timeout (upstream's testInfo.timeout: ExUnit's timeout of the test in
ms, the @tag timeout: or mix test --timeout, :infinity for none, as under mix test --trace) and deadline (upstream's internal testInfo._deadline(): monotonic milliseconds
(compare with System.monotonic_time(:millisecond)) by which the test must be done: after
it, the test's contexts are closed and Playwright calls fail; :infinity without a test
timeout; see "Timeouts"), from the test's setup to the end of its body. use is the test's
options as resolved from every level (the config's, its project's, its module's, its
describe's and its own), what upstream's option fixtures give a fixture: a map of the context
options every context of the test starts from (base_url, locale, extra_http_headers,
...; those set, and Playwright Test's defaults), action_timeout, navigation_timeout,
trace, screenshot, video, and the worker options of its BrowserPool.browser/1 calls
(browser_name, headless, channel, launch_options, connect_options). A library that passes options in a
call merges into it, as the call's replace the resolved ones as a whole:
Browser.new_context!(browser, extra_http_headers: Map.merge(Test.info().use[:extra_http_headers] || %{}, headers)). A library that waits or
retries on its own (a server-side backstop) should stop by deadline, as the matchers
stop 250 ms before it, as upstream's stop 250 ms before its test deadline.
In the module's setup_all (and processes with it in $callers), it returns the fields that
apply to the module, with the same meaning: file, timeout (ExUnit's timeout of the module,
@moduletag timeout: or ExUnit's, the one "Timeouts" in the moduledoc applies to its
setup_all callbacks) and use, the module's options as resolved from the config, its
project and the module itself (its use and @moduletag): a describe's and a test's options
never reach setup_all. The test's fields (test_pid, test_name, output_dir, retry,
deadline) are absent. Deviations: upstream runs beforeAll hooks with the
first test's testInfo, which ExUnit's setup_all, run before any test, has none of; and a
long-lived process or handler of setup_all's (with it in $callers) gets the module's fields
during a test too, as its contexts get the module's defaults (see "Contexts of setup_all"),
until the runner cleans up after it, where upstream's test.info() is the current test's. A
setup_all that runs before use Playwright.Test registers the module (a case template's
own, or one above the module's use) isn't registered: there it raises.
A library that also runs outside these scopes reads the config there:
alias Playwright.Test
use = if Test.in_registered_scope?(), do: Test.info().use, else: Test.config()config/0 has another shape: the config alone, with the context options set as keys of their
own (none of Playwright Test's defaults, nor those only in its context_options bag) and its
config-only keys (workers, output_dir, ...). Read the keys both have with Access,
use[:base_url] (nil when unset), not use.base_url, which raises for an unset key.
Raises elsewhere (see in_registered_scope?/0), and in on_exit callbacks (ExUnit runs them
in a process of their own, without $callers, setup_all's too): there, use info/1 for a
test.
Playwright.Test.info().test_pid
info/0 of the test test_pid (the context's test_pid), from any process, from its setup to
the end of its on_exit callbacks. Once the test's process has exited, so in on_exit, it has
upstream's result fields too, as testInfo has in teardown:
status-:passed,:failed,:timed_out(past ExUnit's timeout), ornilfor a test of another macro (a doctest).errors- upstream'sTestInfoErrors:%{message: _, stack: _}of an exception,%{value: _}of an exit or throw.expected_status-:passed(there is notest.fail()).duration- ms from the test's setup to its process's exit.
A failure in an on_exit callback doesn't change them.
setup %{test_pid: test_pid} do
on_exit(fn ->
if Playwright.Test.info(test_pid).status != :passed, do: ...
end)
end
The path of path_segments (a string or a list of them) in the test's output dir (see
"Artifacts" in the moduledoc), as upstream's testInfo.outputPath(...pathSegments): creates the
dir, and raises for a path outside it. Files written
there stay when the test passes, under the default preserve_output.
@spec parameters(keyword() | map()) :: %{ optional(atom()) => Playwright.Test.Project.t() }
A project for parameterize:, upstream's projects: [{ name, use: { ... } }]: name: (a
string, upstream's project.name), use/1's options and the worker's (browser_name,
default_browser_type, headless, channel, launch_options, connect_options), validated
as it does. A map of
one private key, :"playwright.project", whose value is a Playwright.Test.Project, which
ExUnit takes there (a tag takes use/1's keyword list).
use Playwright.Case,
async: true,
parameterize:
for device <- ["Desktop Chrome", "Desktop Firefox", "Desktop Safari"],
do: Playwright.Test.parameters([name: device] ++ Playwright.device(device))The project's options are under the module's use and over the config's, for every context
of the module's run, setup_all's included. Its browser is its browser_name, else the
config's, else its default_browser_type (a device's), see "Browser" in the moduledoc. A failed
test prints the project as its name (Parameters: %{"playwright.project": #Playwright.Test.Project<"Desktop Safari">}, [Desktop Safari] › in
Playwright.Test.GitHubFormatter); the name is also the snapshot templates' {projectName}.
An unnamed project, as upstream's, has no label: it prints as its options, so name each one.
For parameters of your own next to it: Map.put(Test.parameters(...), :role, :admin).
The snapshot path of name for the running test, as upstream's testInfo.snapshotPath:
where Playwright.Expect.to_have_screenshot/2 (kind: :screenshot) and
Playwright.Expect.to_match_aria_snapshot/3 (kind: :aria) with name: read and write it
(see "Snapshots" in the moduledoc). name is a string or a list of path segments
(["home", "hero.png"], upstream's snapshotPath(...name)), which are not sanitized. Option
:kind: :snapshot (default, upstream's), :screenshot or :aria.
Upstream's test.step(title, body, options): runs the body as a step of the running test
and returns its value, also from a process with the test in its $callers (a Task); steps
nest. A step's error is raised on after the step records it. In the module's setup_all
(and processes with it in $callers) it runs too, recorded nowhere (see "Steps" in the
moduledoc); elsewhere it raises.
step "log in" do
...
end
step "log in", box: true do
...
end
step("log in", fn -> ... end, box: true)Options:
:box- the step is located at the call of the function the step is in (a helper's caller), unless:locationis given, and an error inside it is reported there: its stacktrace starts at that call, instead of the frames inside it. The steps inside keep their own locations. A boxed step outside a helper (in the test's body) has no location of its own; passlocation:.:location-%{file: _, line: _, column: _}of the step in the trace. Thedoform passes its own, unless boxed.:params,:subtitle- shown with the step in the trace.
The body runs in the calling process. Upstream's timeout is not supported (it raises an
ArgumentError): a process can't be interrupted, and the test's timeout bounds the step.
The step, the matchers in it and the test's errors are in the test's trace (test.trace in
trace.zip). A step in a Task has no parent step: the Task doesn't carry the zone.
Upstream's test.step.skip(title, body, options): a step that doesn't run its body, marked
skip in the trace. Returns nil. The forms and options are step/3's.
ExUnit.Case.test/1: a test that fails as not implemented.
ExUnit.Case.test/3, with the end of the body marked: the test passed if its process then
exits as ExUnit ends a test, and fails there with the errors its matchers recorded (see
"Snapshots" in the moduledoc). A catch notes an error for error-context.md and re-raises it
unchanged, so ExUnit reports the failure as without this macro (with the recorded errors
before it, if any).
Options for a module's @moduletag, a describe block or a test: upstream's test.use. It
takes the options of use Playwright.Test (the context options, context_options,
action_timeout, navigation_timeout, trace, screenshot, video, and in @moduletag
the worker's), as a keyword list or a map, validates them as it does, and returns a keyword
list for an ExUnit tag. A device in @moduletag too; in a describe's or a test's tag only
without its default_browser_type.
# test.describe(() => { test.use({ timezoneId: 'Europe/Berlin' }) ... })
describe "in Berlin" do
@describetag Test.use(timezone_id: "Europe/Berlin")
...
end
# One test. Upstream nests a describe around it; ExUnit's can't nest.
@tag Test.use(trace: :on)
test "pays", %{page: page} do
...
end
# A device's emulation without its browser (a device in a test's or a describe's tag
# raises, as its `default_browser_type` is the worker's: make each device a project).
@tag Test.use(Keyword.delete(Playwright.device("iPhone 13"), :default_browser_type))Each option gets a private key of its own (:"playwright.use.trace",
:"playwright.use.context.locale"), so ExUnit, which merges a test's @moduletag,
@describetag and @tag per key, layers them per option, the test's over the describe's
over the module's: a describe's timezone_id and a test's trace both apply, as do several
calls on one test, and an option given as nil resets it to the project's, the config's or
the default. A plain tag named like an option (@tag trace: :on) sets nothing.
An unknown option raises ArgumentError, when the test module compiles. So do
browser_name, default_browser_type (a device's), headless, channel, launch_options and
connect_options in a @describetag or a @tag, when the module compiles a test of the test macro under it,
with upstream's message ("Cannot use({
browser_name }) in a describe group, because it forces a new worker.
Make it top-level in the
test file or put in the configuration file.", "in a test" for a @tag): they are the
worker's (a test the check doesn't see gets the module's worker regardless). They are options
of the module's use, its @moduletag and parameters/1. For a
project in parameterize:, which takes maps, see parameters/1. See "Context defaults",
"Browser" and "Artifacts" in the moduledoc for the layers.