Testing with ExUnit

Copy Markdown View Source

This library is the client, ExUnit is the test runner. Playwright.Test adds what Playwright Test's worker gives each test: a browser, a fresh context with Playwright Test's defaults, and traces, screenshots and videos of failures. This guide tests an app that runs somewhere and is reached through a base URL, whatever it is written in. Playwright.Test's moduledoc is the reference; the guides show how to use it:

Snippets alias Playwright.{Page, Locator, ...}.

Setup

Configure the runner in config/test.exs:

# config/test.exs
config :playwright, :test,
  base_url: "http://localhost:4000",
  trace: :retain_on_failure,
  screenshot: :only_on_failure

A test module uses Playwright.Case, which gives each test a context and a page:

defmodule MyApp.SignUpTest do
  use Playwright.Case, async: true

  test "signs up", %{page: page} do
    Page.goto!(page, "/sign-up")
    Page.get_by_label(page, "Email") |> Locator.fill!("ada@example.com")
    Page.get_by_role(page, :button, name: "Sign up") |> Locator.click!()
    Expect.to_have_text!(Page.get_by_role(page, :status), "Welcome, ada@example.com")
  end
end

Playwright.Case is a case template of three pieces, the same as:

use ExUnit.Case, async: true
use Playwright.Test
alias Playwright.{Browser, BrowserContext, Expect, Keyboard, Locator, Mouse, Page, Test}
alias Playwright.Test.BrowserPool

setup_all do
  %{browser: BrowserPool.browser(), browser_name: Test.info().use.browser_name}
end

setup %{browser: browser} do
  context = Browser.new_context!(browser)
  %{context: context, page: BrowserContext.new_page!(context)}
end

use Playwright.Test goes after use ExUnit.Case. It registers each test, so the contexts it creates get Playwright Test's defaults and are recorded, and imports a test macro that marks the end of each test's body, so the recording knows whether the test passed (ExUnit reports failures as before; see Registration).

Playwright.Test.BrowserPool.browser/1 is Playwright Test's browser fixture: called in setup_all, it gives the module a driver and browser of its own while it runs, as Playwright Test runs a file in one worker. A module that passed hands its browser on, warm, to the next; one that failed has it replaced (Workers). A context is a fresh browser profile with its own cookies, storage and cache, so tests don't share state. The drivers stop after the suite, so test_helper.exs needs only ExUnit.start().

A module's tests run one after another (ExUnit balances whole modules, where Playwright Test's fullyParallel balances single tests), so many small async: true modules parallelise better than a few big ones: split a big spec into a module per describe. Modules that share state in the app can stay async in one ExUnit group (use Playwright.Case, async: true, group: :tree, Elixir 1.18+): no two tests of a group run at once, while other modules keep running, which beats async: false. mix test --slowest N turns on --trace, which runs one module at a time, so it can't time a parallel run.

Options go in as arguments, over the config: BrowserPool.browser(browser_name: :webkit), Browser.new_context!(browser, locale: "de-DE"). Every context a test creates on a pooled browser starts from Playwright Test's defaults (locale: "en-US", a 1280x720 viewport, color_scheme: :light, ...), then the options of the config, the project, the module, the describe block and the test (Options, devices and projects).

With base_url, goto and the URL matchers take paths. To set it from the shell, read an env var in the config: base_url: System.get_env("BASE_URL", "http://localhost:4000").

As in Playwright Test, actions and navigations have no timeout of their own in a test: ExUnit's test timeout (60 seconds) bounds them; Playwright Test's is 30 seconds, so for the same behaviour, ExUnit.start(timeout: 30_000) in test_helper.exs. Just before it, the test's contexts are closed, so a stuck call fails with "Test timeout of 60000ms exceeded." and its call log, and the artifacts are kept. To fail sooner, set action_timeout and navigation_timeout in the config (milliseconds). :infinity, the default, is no limit, and so is 0, as in every timeout: option (API conventions). setup_all's timeout and the rest: Timeouts.

Each driver call logs pw:api ... at :debug (Logging), so give the test env a logger level (config :logger, level: :warning). ExUnit.start(capture_log: true) prints an in-VM app's logs only for failed tests, as Playwright Test hides the webServer's output.

Assertions

Playwright.Expect has the web-first assertions: they retry until the page matches or the timeout passes. The bang forms raise Playwright.ExpectError with the expected and received values and the call log:

Expect.to_have_url!(page, ~r"/dashboard$")
Expect.to_be_visible!(Page.get_by_text(page, "Saved"))
Page.get_by_role(page, :listitem) |> Expect.to_have_count!(3)
Page.locator(page, ".error") |> Expect.not_() |> Expect.to_be_visible!()

Prefer them to assert Locator.text_content!(...) == ..., which reads once and races the page. Their timeout is 5 seconds; set it for the suite in config/test.exs:

config :playwright, :expect, timeout: 10_000

A soft assertion (Playwright Test's expect.soft) doesn't stop the test: it fails the test once its body has run, and ExUnit reports all of the test's errors:

Page.get_by_text(page, "Saved") |> Expect.soft() |> Expect.to_be_visible!()
Page.get_by_role(page, :listitem) |> Expect.soft() |> Expect.to_have_count!(3)

Where an ExUnit.AssertionError is needed (a helper that rescues only that), wrap the matcher:

defmodule MyApp.Assertions do
  @doc "Runs `fun`, re-raising a failed matcher as an `ExUnit.AssertionError`."
  def assert_expect(fun) do
    fun.()
  rescue
    e in Playwright.ExpectError ->
      reraise ExUnit.AssertionError, [message: Exception.message(e)], __STACKTRACE__
  end
end
assert_expect(fn -> Expect.to_have_text!(Page.get_by_role(page, :status), "Saved") end)

Snapshots

In a test, Expect.to_have_screenshot!/2 and Expect.to_match_aria_snapshot!/3 take a name: that resolves next to the test file, as in Playwright Test:

Expect.to_have_screenshot!(page, name: "home.png")
# test/sign_up_test.exs-snapshots/home-linux.png (the suffix is the platform)
Expect.to_match_aria_snapshot!(Page.locator(page, "body"))
# test/sign_up_test.exs-snapshots/signs-up-1.aria.yml (no name: the test's name and count)

A missing snapshot is written, and fails the test once its body has run, so CI catches a snapshot nobody committed. update_snapshots in config :playwright, :expect (:all, :changed, :missing, :none) rewrites them. A failed comparison leaves -actual, -expected, -previous and -diff images in the test's dir under test-results/. Paths and templates: Snapshots.

Steps

Playwright.Test.step/3 groups a test's actions under a title, as Playwright Test's test.step: the trace shows the steps, nested, with each matcher call and each driver call as a step of its own, and the test's errors. A step returns its body's value and works in a Task of the test too:

test "signs up in steps", %{page: page} do
  Test.step "fill in the form" do
    Page.goto!(page, "/sign-up")
    Page.get_by_label(page, "Email") |> Locator.fill!("ada@example.com")
  end

  status =
    Test.step("submit", fn ->
      Page.get_by_role(page, :button, name: "Sign up") |> Locator.click!()
      Page.get_by_role(page, :status)
    end)

  Expect.to_have_text!(status, "Welcome, ada@example.com")
end

With box: true, the step and an error inside it are reported where the helper is called, which suits helpers, a library's too. The steps inside keep their own locations:

defp sign_up(page, email) do
  Test.step "sign up", box: true do
    Page.goto!(page, "/sign-up")
    Page.get_by_label(page, "Email") |> Locator.fill!(email)
    Page.get_by_role(page, :button, name: "Sign up") |> Locator.click!()
  end
end

A step runs in the calling process, so assert_receive works in it. It takes no timeout:, as a running body can't be interrupted: the test's timeout bounds it. step_skip declares a step without running it. In setup_all a step runs too, unrecorded (Steps), so the same helpers serve both. use Playwright.Test imports test and replaces ExUnit's assert and refute with Playwright.Test.Assertions' (the rest of ExUnit.Assertions stays imported); the rest of Playwright.Test is called qualified (Test.step, Test.info()), as Playwright Test's test.step(), and a wrapper library's own step doesn't clash.

Each assert and refute is a step too, as Playwright Test's expect: Expect "assert", with the expression next to it and its right side as expected, and a failed one with its error, at the test's level. They are ExUnit's otherwise, with the same bindings, messages and stacktraces (a function that ends in one keeps its frame). Don't import ExUnit.Assertions again after use Playwright.Test, which makes assert ambiguous. A helper module without use Playwright.Test needs import Playwright.Test.Assertions (and, if it imports ExUnit.Assertions too, except: [assert: 1, assert: 2, refute: 1, refute: 2] there), else its passed asserts silently leave no step. A library with its own assert/2 excludes exactly the arities it defines (import Playwright.Test.Assertions, except: [assert: 2]); its own refute/2 must also take the refute(value, expr: ...) that ExUnit's refute match? calls (Assertions). Assert steps are on by default: use Playwright.Test, assert_steps: false keeps ExUnit's own for a module, and config :playwright, :test, assert_steps: false (in config/test.exs, as it is read at compile time, not in runtime.exs) for all, which a module's assert_steps: true overrides.

Route handlers and listeners

Route handlers (Page.route/3, route_web_socket/3 and their WebSocket on_message and on_close), event listeners (Page.on/3), exposed functions, locator handlers and wait predicates run in processes of their own, on behalf of the process that registered them: they get its $callers, as a Task does. So the Ecto sandbox, Mox and Req.Test find the test in them without an explicit allowance, and matchers and step count for the test.

An error a route handler, listener or locator handler raises fails the test and stops it at once, as Playwright Test's worker does; once the body has ended (in your on_exit callbacks) it is only logged (Registration has the details). In a :dialog listener, accept or dismiss the dialog before asserting on it: a dialog left open blocks the page, so the screenshot and the snapshot of error-context.md wait on it until they time out.

An on_exit callback is Playwright Test's afterEach hook. ExUnit runs it after the test process has exited and its start_supervised!/1 children have stopped, but before the test's contexts close, which is last: it can still use page and context, and their trace records its calls. The test's listeners still run in your on_exit callbacks, but not as its contexts close. State linked to the test (an Agent.start_link/1 of it, a start_supervised!/1 child) is gone in on_exit, so a listener that uses it there logs "Uncaught in event listener: ... no process": send to the test process (a late message is dropped with it), or keep the state unlinked (Agent.start/1).

A check after the test

A fixture teardown that fails the test, such as LiveView's page fixture checking for page errors, is an on_exit: an assertion there fails the test, reported by ExUnit as an on_exit failure (the recording counts it as a pass). Its state must outlive the test process. Playwright Test's option fixtures (test.extend with { option: true }, set with test.use) are plain tags or parameter keys of your own: test.use({ ignoreJSErrors: true }) becomes @tag :ignore_js_errors (or @describetag, @moduletag, a key of a project's Test.parameters/1 map), which the setup reads:

setup %{page: page} = ctx do
  {:ok, errors} = Agent.start(fn -> [] end)
  Page.on(page, :page_error, fn error -> Agent.update(errors, &(&1 ++ [error])) end)

  on_exit(fn ->
    page_errors = Agent.get(errors, & &1)
    Agent.stop(errors)
    if !ctx[:ignore_js_errors], do: assert(page_errors == [], "unhandled page errors")
  end)
end

Polling and waiting

When the state shows in the page, assert on the page: Expect.* retries. For state elsewhere (an email, a background job, a record behind an API), retry the assertion with Expect.to_pass!/2 (expect(fn).toPass()), or poll a value with Expect.poll!/3 (expect.poll(fn).toBe(x)), whose matcher is an ExUnit assertion:

Expect.to_pass!(fn ->
  response = APIRequestContext.get!(BrowserContext.request(context), "/api/exports/1")
  assert APIResponse.json!(response)["status"] == "done"
end)

Expect.poll!(fn -> Repo.aggregate(Export, :count) end, &assert(&1 == 1), timeout: 10_000)

As upstream's, to_pass retries until the test's timeout (timeout: for less) and poll for the matchers' timeout, both at [100, 250, 500, 1000] ms intervals (intervals:). The callback runs in the test process, so self() and its mailbox are the test's; a step or a soft receiver (fun |> Expect.soft() |> Expect.to_pass!()) works as for any matcher.

As upstream's, a poll callback that raises fails at once. JS's (await rows())[0] is undefined while the list is empty and polling goes on, where hd([]) raises: read with Enum.at(rows, 0) and leave the check to the assertion.

Events forwarded to the test process arrive when the browser emits them, which can be well after the action that led to them (a request, a timer). assert_receive waits only 100 ms by default, so give it a timeout, or raise the default with ExUnit.start(assert_receive_timeout: 5_000):

test_pid = self()
Page.on(page, :console, &send(test_pid, {:console, ConsoleMessage.text(&1)}))
Page.get_by_role(page, :button, name: "Save") |> Locator.click!()
assert_receive {:console, "saved"}, 5_000

To check forwarded events afterwards, as JS collects them into an array, drain the mailbox. An event is handled before the reply of the call that caused it (Concurrency), so once that call returned (or a matcher saw its effect), its message is there:

defp received(tag) do
  receive do
    {^tag, value} -> [value | received(tag)]
  after
    0 -> []
  end
end

assert received(:console) == ["saved"]

See Events and waiting for one-off events (downloads, popups, dialogs).

Logging in once

Log in once per module and start each context from the saved cookies and local storage. A context in setup_all gets the defaults, base_url too, as a test's does, but no recording:

setup_all do
  browser = BrowserPool.browser()
  context = Browser.new_context!(browser)
  page = BrowserContext.new_page!(context)
  Page.goto!(page, "/login")
  Page.get_by_label(page, "Email") |> Locator.fill!("ada@example.com")
  Page.get_by_label(page, "Password") |> Locator.fill!("secret")
  Page.get_by_role(page, :button, name: "Log in") |> Locator.click!()
  Expect.to_have_url!(page, ~r"/dashboard$")

  storage_state = BrowserContext.storage_state!(context)
  BrowserContext.close!(context)
  %{browser: browser, storage_state: storage_state}
end

setup %{browser: browser, storage_state: storage_state} do
  context = Browser.new_context!(browser, storage_state: storage_state)
  %{context: context, page: BrowserContext.new_page!(context)}
end

A request context created in setup_all isn't disposed of by the runner either, as with upstream's manual newContext in beforeAll. Dispose of it yourself inside the setup_all: on_exit(fn -> APIRequestContext.dispose!(request) end).

To share the login between modules, BrowserContext.storage_state!(context, path: "tmp/auth.json") also writes it to a file, and storage_state: takes that path. An app that keeps its tokens in IndexedDB (Firebase Authentication) needs storage_state!(context, indexed_db: true): IndexedDB is not in the state by default. See Authentication.

To log in once per run, as Playwright Test's setup project or globalSetup does, log in from test_helper.exs, which runs before any test, on a driver of its own (one more driver start) that it stops before the tests start. Outside a test no defaults apply, so give the context its base_url. The config's storage_state then starts every test's context from the file:

# test/test_helper.exs, after starting the app
{:ok, driver} = Playwright.Supervisor.start_link(name: MyApp.GlobalSetup)
browser = Playwright.instance(MyApp.GlobalSetup) |> Playwright.chromium() |> BrowserType.launch!()
context = Browser.new_context!(browser, base_url: Playwright.Test.config()[:base_url])
page = BrowserContext.new_page!(context)
Page.goto!(page, "/login")
# ... log in, as above
BrowserContext.storage_state!(context, path: "tmp/auth.json")
Supervisor.stop(driver)

ExUnit.start()
# config/test.exs
config :playwright, :test, base_url: "http://localhost:4000", storage_state: "tmp/auth.json"

Any other once-per-run setup (Playwright Test's project dependencies) goes there too.

Several users

Each user is a context, and a user on another browser takes it from another browser/1 call, another worker that the module holds too (over workers, so it never waits for itself):

test "two users edit one document", %{browser: browser} do
  ada = Browser.new_context!(browser)
  bob = Browser.new_context!(browser)
  eve = Browser.new_context!(BrowserPool.browser(browser_name: :webkit))

  ada_page = BrowserContext.new_page!(ada)
  bob_page = BrowserContext.new_page!(bob)
  eve_page = BrowserContext.new_page!(eve)
  # ...
end

All three contexts are recorded into the test's one trace, and each gets its own video.

The app under test

test_helper.exs starts what a run needs, as Playwright Test's webServer does. To start the app server only when browser tests run (mix test --only e2e, also with --failed), check the run's filters, which Mix sets before it loads the helper:

# test/test_helper.exs
if :e2e in ExUnit.configuration()[:include], do: MyApp.E2E.start_server!()
ExUnit.start(exclude: [:e2e])

When the unit tests' helper doesn't suit a run against the real app (Mox globals, say), give the browser tests a dir and an env of their own: test_paths: if(Mix.env() == :e2e, do: ["e2e"], else: ["test"]) in mix.exs, with e2e/test_helper.exs and config/e2e.exs, run by MIX_ENV=e2e mix test. A dir under test/ must also be kept out of the other envs with test_load_filters (test_ignore_filters only silences the warning). Or make it a Mix project of its own that reaches the app over HTTP.

An app running as an OS process of its own needs no start in the test VM: mix test --no-start starts no application, so start this library's in test_helper.exs (Application.ensure_all_started(:playwright)). A server the tests start themselves inherits their environment, MIX_ENV included when the shell or CI job sets it: pass the child's explicitly (env: [{~c"MIX_ENV", ~c"dev"}] in Port.open/2).

Async modules run at once against one app and one database. Seed data each module owns (names with System.unique_integer([:positive])), or wrap seeding that can't be unique in :global.trans/2. For an Ecto app in the test VM, each test's browser must reach its sandbox: add Phoenix.Ecto.SQL.Sandbox to the endpoint and its LiveView hook (see its docs), and send the test's metadata as the context's user agent:

setup %{browser: browser} do
  owner = Ecto.Adapters.SQL.Sandbox.start_owner!(MyApp.Repo, shared: false)
  metadata = Phoenix.Ecto.SQL.Sandbox.metadata_for(MyApp.Repo, owner)
  context = Browser.new_context!(browser, user_agent: Phoenix.Ecto.SQL.Sandbox.encode_metadata(metadata))

  on_exit(fn ->
    BrowserContext.close(context)
    Ecto.Adapters.SQL.Sandbox.stop_owner(owner)
  end)

  %{context: context, page: BrowserContext.new_page!(context)}
end

Shared mode ({:shared, self()}) would need async: false. The on_exit closes the context first: the test's contexts close after your on_exit callbacks, by when the owner is gone and the app's LiveViews log ownership errors.

Accessibility testing

@axe-core/playwright isn't ported: install axe-core with your package manager, evaluate its script in the page (again after each navigation), as AxeBuilder does, and run it:

axe_path = "assets/node_modules/axe-core/axe.min.js"

Page.evaluate!(page, File.read!(axe_path))
result = Page.evaluate!(page, "() => axe.run()")
assert result["violations"] == []

axe.run(context, options) takes what AxeBuilder's include, exclude and withTags set: Page.evaluate!(page, "([c, o]) => axe.run(c, o)", [%{include: ["main"]}, %{runOnly: ["wcag2a"]}]). Take the axe-core your JS suite's @axe-core/playwright resolves, for the same rules. It checks the main frame only: AxeBuilder also runs axe in each iframe and merges the results.