Automated test suite to vet viewability solutions

[exiao]

Vision

Create an automated viewability test suite…to ensure OpenVV measurement is working properly and catch discrepancies between vendors:

From	To
Manual testing of viewability measurement, No coverage across browser versions	Automated QA across multiple browsers and viewability scenarios
Lack of alignment between different vendors outside of OpenVV	Align viewability numbers between vendors at an engineering level
Improper vetting of changes to source code that could greatly affect measurement	Ensure changes to OpenVV are properly vetted

Phases

1. Automation and test scenarios for internal testing use
2. Demo page to debug your own ad units for each test
   • Create a demo page that would be showcased at OPENVV.org. Select your test case from the dropdown and see the player fire and measure viewability. See www.openvv.org/ad-viewability-demo/

High level Implementation

Run a VPAID tag through a set of different scenarios that result in viewable, non viewable, or unmeasurable scenarios (i.e. full screen vs. 75% off screen). The flow of using this tool would be:

• Input a VPAID tag for tool to execute. VPAID tag would contain site macro that describes the test scenario
• Tool launches test pages using specific VPAID tag across different browsers
• Once test is complete, get reporting from vendor broken out by the site macro to see viewability results.
  • Example Test Name: BasicInView_mac_safari-V
  • Description: Basic In View Test on mac OS using Safari. Supposed to be Viewable

Details

Implementation Tools

• Basic website to input VPAID tags into queue for automated test suite
• Cron job & servers to continually run tests on demand
• Selenium to mimic visiting test pages using different browsers

Tests

Test Type	Description
Basic In View	Test basic functionality of whether we are measuring viewability correctly if it is >50% in the active browser window
Scroll	Test if viewability solution detects scrolling of browser window accurately to determine content coming into view or out of view
In-Focus/Out-of-Focus	Test whether viewability solution can detect if the video is on the active tab and not minimized
iFrame	Test whether viewability solution detects viewability correctly in iFrames
Cross-Domain iFrames	Same as above for cross-domain iFrames
FullScreen	Test for full screen players that they are considered viewable
Other	Any other tests (Overlapping elements, Invisible Elements, etc)

[exiao]

Low Level Steps to Execute Vision

1. Generate test pages for test scenarios
2. Put it on a production/host box so that we can see test pages and test manually
3. Create automation for test scenarios
4. Create input for any custom VPAID tag

[exiao]

Viewability Test Suite Technical Diagram