Miniature Testers: On-Device QA for HTML5 Mobile Web

Intended Audience: Web Developers, QA Engineers

Estimated Reading Time: 12 minutes

In March 2013, I attended a conference and realized something didn't exist in the community yet: on-device UI capture for mobile web testing. Selenium's mobile support was spotty, Appium was brand new, and most teams relied on manual QA tapping through phones. On the Match.com mobile web team working on "Mobi" — our HTML5 dating app — we needed something better. So I built it.

Video

Four phones running different test plans simultaneously — Samsung and iPhones progressing through registration, profile creation, search, and login flows. Meanwhile, the server's log viewer streams results in real time, showing each action, its build version, device ID, and timestamp. You can even see the stuck test detection kick in: "Stopping TestRun because stuck 21 times on message."

The Problem

Mobi ran on easily 100 device/browser combinations. Testing the registration flow, profile swiping, and messaging features manually was slow and error-prone. We needed automated tests that ran on real devices — not simulators that missed real-world rendering bugs.

Tools like Selenium existed, but their mobile story in 2013 was weak. The "driver" wasn't a real browser — it was a separate app embedding a web control and puppeting it from outside. That embedded view didn't behave like an actual mobile browser. The exact bugs our team dealt with daily — swipe and drag interactions, history event handling, touch gesture edge cases — were precisely what those artificial environments failed to simulate correctly.

We needed something almost as good as a human tapping through the regression suite, but automated and running on the actual device.

The Solution: Test From Inside

RobiTest took a different approach: inject the test framework directly into the app. The tests run inside the browser, alongside the application code. No external drivers, no flaky WebDriver connections — just JavaScript executing in the same context as the app.

The entire client-side framework weighed in at 18KB. An entire on-device test framework in 18KB of JavaScript.

The framework consisted of two parts:

  1. Client-side JavaScript — A test engine injected into the app
  2. ASP.NET MVC Server — Collected results and served test definitions, hosted on Azure and deployed from TFS

I treated this as a real product — complete with a logo featuring a robot mascot on a circuit board background with a play-button icon.

The Fluent DSL

Test authoring needed to be simple. We built a fluent API that read like a script:

RobiTest.Define
    .createPlan("Registration Flow")
    .createTest("New User Signup")

    .go("/registration/")
    .waitUntilVisible("#regFormPage1")
    .changeValue("#emailAddress", "test@example.com")
    .click("#viewSingles")

    .waitUntilVisible("#regFormPage2")
    .changeValue("#zipCode", 75225)
    .click("#step2Button")

    .assert("Reached step 3", function() {
        return document.querySelector("#regFormPage3") !== null;
    })

    .finishTest()
    .finishPlan();

Actions included go, click, changeValue, waitUntilVisible, assert, and custom act blocks for complex interactions.

Domain-Specific Adapters

Beyond generic form filling, RobiTest included purpose-built adapters for Mobi's complex features:

These adapters abstracted away the complexity of touch gestures and swipe interactions that made Mobi's UX work. Test authors could write swipeProfile() without understanding the underlying touch event choreography.

Surviving Page Navigation

The trickiest part: tests had to survive full page reloads. When .go("/registration/") navigates away, the JavaScript context dies.

The solution was localStorage. The test engine persisted its state — current action index, test variables, active/paused status — before every navigation. On page load, it restored state and resumed execution. Tests could span dozens of page loads without losing their place.

On-Device Dashboard

Testers could trigger the dashboard by adding ?robitest to any URL. This overlay showed available test plans, run/pause controls, and real-time status. No app changes needed — the query parameter activated the testing UI.

Parallel Execution and DeviceAnywhere

A key capability: run the gauntlet after builds on several devices at once. The video shows four phones executing simultaneously, but this scaled further — tests could also run on DeviceAnywhere, a cloud device lab service that provided remote access to real physical devices. Farm tests out to devices you don't own, running in parallel with your local hardware.

Results Collection and Regression Tracking

Every action reported to a central server:

The real value: pinpointing when regressions were introduced. New failures were known with build number and date at a device-specific level. When a test started failing, you could identify exactly which build broke it and on which devices. The server's log viewer let us filter by device, build, or test run — essential for tracking down device-specific failures.

Local Development Mode

Test authors could write and debug tests locally while still reporting results to the central server. Test definitions could be temporarily driven from a local machine build, letting you iterate on selectors and timing without deploying changes. Once the test worked, commit it and the whole team could run it.

Stuck Test Detection

A common failure mode: waiting for an element that never appears. The reporting system tracked consecutive identical messages. After 20 repeats of "Waiting for #someElement to be visible", it automatically stopped the test run and logged a failure. No more phones stuck in infinite loops.

Adoption Arc

RobiTest started as spare-time work after that conference epiphany. It eventually got sanctioned Match time. I worked with our QA engineer to teach him DOM selectors and the fluent syntax. He produced test lists. RSI (another team member) wrote tests too — and their test-writing surfaced limitations that fed back into framework improvements.

Then it went dormant for 3-4 months. The slide deck I presented internally was a pitch to revive it. Match.com's mobile web was a high-stakes environment — the team was shipping constantly, and learning a new test authoring system while keeping up with feature work proved to be too much friction.

Management was pleased with the innovation and rewarded me with a VR development rig they knew I'd enjoy. But the project ultimately didn't take root. Sometimes good tools arrive at the wrong moment.

Lessons

Test from inside when outside control isn't available. Mobile browsers in 2013 didn't play nice with external automation. Running inside the browser sidestepped the problem entirely.

localStorage bridges page navigations. State persistence made multi-page test flows possible without complex server-side session management.

On-device dashboards beat remote control. Giving testers direct control on the device was faster and more reliable than coordinating through external tools.

Build-level regression tracking is the real value proposition. Knowing that a test passed yesterday and fails today is useful. Knowing which build and which device introduced the failure is actionable.

Adoption requires more than good architecture. A testing framework only works if people write tests. In a fast-moving environment, the friction of learning something new can outweigh the benefits — even when the tool is sound.

Originally created for the Match.com mobile web team.

Back to Home