Today's web applications are not pieced together using only the standard HTML elements of yesteryear. We now have the ability to use Web Components as a way to encapsulate styling and functionality for particular components, which means we can easily share them and reuse them within and across apps.

ShadowDOM

This is great, but when it comes to testing this new breed of web app, things get more complicated. The way Web Components are implemented, the "interior" of a component does not appear in the DOM; instead, it appears in a separate tree called a "Shadow DOM", which is linked into the "main" DOM at the point we want it to appear. This separation is a good thing, because we don't want unrelated style definitions for our main document to be able to target the interior of a 3rd-party component. But what if we want to make use of an element located in the interior of a Web Component? We can't just access it as if it were part of the DOM, because it's hidden away inside its own special Shadow DOM.

Luckily, a JavaScript API is available for getting the root of the Shadow DOM attached to a given element:

element.shadowRoot

The shadowRoot of an element is itself an element, namely the root element inside the component. Using the executeScript method of our Appium driver, we can actually retrieve this shadow root as a standard WebElement:

// first, find the externally visible element
WebElement component = driver.findElement(By.tagName("custom-component"));

// now we can use it to find the shadow root
String getShadow = "return arguments[0].shadowRoot;";
WebElement shadowRoot = (WebElement) driver.executeScript(getShadow, component);

What's going on here? Basically we're constructing a little snippet of JS which will be run in the context of the currently-loaded page. But we're not running it free of any context---we're actually passing in an element we've found as an argument to the script. Appium knows how to convert your client-side WebElement into an actual HTML element in the context of the executing JS, which means we can access the standard properties available on HTML elements, including shadowRoot. Finally, we're simply returning the shadow root and casting it to a WebElement so we can work with it in our test script.

Now, if we were dealing with Selenium, we would be able to use this shadowRoot element just like any other element (for example in order to find an input field inside the component):

WebElement innerElement = shadowRoot.findElement(By.tagName("input"));

However, due to bugs and/or limitations in the technologies that Appium relies on (Chromedriver for Android, and the Selenium Atoms / remote debug protocol for Safari), this will not work. We'll get different errors on the two platforms, but either way we're stuck. What to do?

JavaScript to the rescue again! Let's use the web APIs for searching for elements, instead of relying on the WebDriver findElement command:

// first, find the externally visible element
WebElement component = driver.findElement(By.tagName("custom-component"));

// now use the 'querySelector' API to directly find the inner element we want
String getInnerEl = "return arguments[0].shadowRoot.querySelector('input');";
WebElement innerElement = (WebElement) driver.executeScript(getInnerEl, component);

With this little trick, we're able to now use innerElement just as if it were a regular old main DOM element, for example with commands like innerElement.isSelected() and so on. And that's how we can work with elements inside the shadow DOM! How did we know we were dealing with a shadow DOM to begin with? It's easy to tell by using the dev tools inspector in a browser like Firefox.

Closed Shadow DOMs

I've omitted one wrinkle, which is that shadow DOMs come in two flavors: open and closed. Closed shadow DOMs are unfortunately not accessible via the shadowRoot property, which means the strategy above will not work in accessing them. Luckily, closed shadow DOMs are considered unnecessary and are not too common. There are ways of dealing with them, nonetheless, which we will cover in another edition of Appium Pro.

Sorry, Safari

Due to the differences in how Appium supports testing website on Android vs iOS, there is another difference you have to consider in writing your tests, which is that for Safari, using any element found within a shadow DOM will always generate a StaleElementException when you try and use it. This means the strategy above (where we get a WebElement out of a shadow DOM using executeScript), is not going to work.

Instead, for Safari, we have to stay completely within JS-land if we want to successfully automate the shadow DOM. For example, let's say our goal was to check the state of a checkbox input inside of a custom component. With Chrome, we are able to simply find the checkbox using the strategy above, and call isSelected() on it to determine whether it's checked. With Safari, calling isSelected() will result in a StaleElementException. So what we must do instead is something like the following:

// first, find the externally visible element
WebElement component = driver.findElement(By.tagName("custom-component"));

// now find and determine checked status of element all using JS
String getChecked = "return arguments[0].shadowRoot.querySelector('input').checked;";
boolean checked = (boolean) driver.executeScript(getChecked, component);
// do something with checked value

This is a completely reliable automation technique, however it relies purely on JS and doesn't stay within the typical WebDriver API usage, which is not ideal.

It's worth pointing out that if you have to automate both Chrome and Safari, the pure-JS tactic I just described will work for both; so in cross-platform situations it's probably better to use this latter technique rather than mixing two different strategies together.

Have a look at a full sample which pierces the shadow DOM of a Material Web Component to determine the input state of a switch. (Of course the designers of this component were smart and made that checked state bubble up to a property on the component itself, so piercing the shadow DOM is not strictly necessary. It is however useful for instruction purposes here).

import io.appium.java_client.AppiumDriver;
import io.appium.java_client.android.AndroidDriver;
import io.appium.java_client.ios.IOSDriver;
import java.net.MalformedURLException;
import java.net.URL;
import org.junit.Assert;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

public class Edition044_Shadow_DOM {

    private static By SWITCH_COMPONENT = By.xpath("//mwc-switch[1]");

    private static String SHADOWED_INPUT = "return arguments[0].shadowRoot.querySelector('input')";
    private static String INPUT_CHECKED = SHADOWED_INPUT + ".checked";
    private static String CLICK_INPUT = SHADOWED_INPUT + ".click()";
    private static String URL = "https://material-components.github.io/material-components-web-components/demos/switch.html";

    @Test
    public void testShadowDom_iOS() throws MalformedURLException {
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("platformName", "iOS");
        capabilities.setCapability("platformVersion", "12.1");
        capabilities.setCapability("deviceName", "iPhone 8");
        capabilities.setCapability("browserName", "Safari");

        // Open up Safari
        IOSDriver driver = new IOSDriver<>(new URL("http://localhost:4723/wd/hub"), capabilities);
        try {
            testShadowElementsWithJS(driver);
        } finally {
            driver.quit();
        }
    }

    @Test
    public void testAppiumProSite_Android() throws MalformedURLException {
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("platformName", "Android");
        capabilities.setCapability("deviceName", "Android Emulator");
        capabilities.setCapability("browserName", "Chrome");
        capabilities.setCapability("automationName", "UiAutomator2");

        // Open up Chrome
        AndroidDriver driver = new AndroidDriver(new URL("http://localhost:4723/wd/hub"), capabilities);

        try {
            testShadowElementsAsNative(driver);
            testShadowElementsWithJS(driver);
        } finally {
            driver.quit();
        }
    }

    public void testShadowElementsAsNative (AppiumDriver driver) {
        WebDriverWait wait = new WebDriverWait(driver, 10);
        driver.get(URL);

        // find the web component
        WebElement switchComponent = wait.until(ExpectedConditions.presenceOfElementLocated(
            SWITCH_COMPONENT));

        // use it to find the inner control
        WebElement nativeCheckbox = (WebElement) driver.executeScript(SHADOWED_INPUT, switchComponent);

        // use the standard API to determine whether the control is checked
        Assert.assertEquals(nativeCheckbox.isSelected(), false);

        // use the standard API to change the checked status
        switchComponent.click();

        // and finally verify the new checked state after the click
        Assert.assertEquals(nativeCheckbox.isSelected(), true);
    }

    public void testShadowElementsWithJS(AppiumDriver driver) {
        WebDriverWait wait = new WebDriverWait(driver, 10);

        driver.get(URL);

        // find the web component
        WebElement switchComponent = wait.until(ExpectedConditions.presenceOfElementLocated(
            SWITCH_COMPONENT));

        // pierce shadow dom to get checked status of inner control, and assert on it
        boolean checked = (boolean) driver.executeScript(INPUT_CHECKED, switchComponent);
        Assert.assertEquals(false, checked);

        // change the state from off to on by clicking inner input
        // (clicking the parent component will not work)
        driver.executeScript(CLICK_INPUT, switchComponent);

        // check that state of inner control has changed appropriately
        checked = (boolean) driver.executeScript(INPUT_CHECKED, switchComponent);
        Assert.assertEquals(true, checked);
    }
}

Notice how both strategies are represented, and that the Android test runs both strategies in the same test, to prove that they both work for Android. Don't forget to also have a look at the sample on GitHub.