Integrating Qt6 and Unity applications in test automation

In the world of human-machine interfaces (HMI), integrating diverse technologies for seamless user experiences poses unique challenges. Let’s examine designing and testing multi-technology applications with separate safety-critical and non-critical functions. By leveraging specialised tools like Squish for Qt and AltTester SDK for Unity, we present a comprehensive approach to unifying test automation, ensuring efficient development and robust quality assurance. Discover how these methodologies can streamline your HMI projects, reduce overhead, and enhance overall software quality.

Introduction to automotive HMI and interface separation

Human-machine interface (HMI) design involves creating UIs that allow users to interact effectively with machines. In the automotive industry context, HMIs are particularly complex solutions because users interact with a car on multiple levels: driving, using entertainment services, adjusting the heating, and more.

Each service level carries varying importance; for instance, failing to respond to a voice command to lower a window is relatively insignificant, but not displaying an ABS warning can be hazardous. Hence, the automotive industry standards distinguish diverse types of applications and services. Safety-related applications are separated from non-safety applications, and signals for critical functions, like brakes, are transmitted through different wiring than those for non-critical functions, like navigation.

In such cases, the engineering team must develop separate applications for HMI and sometimes choose different technologies to achieve this goal. Finally, both applications are displayed on top of one another and deliver a complete user experience.

From a development perspective, this method is considered efficient – the system needs only to initiate and run two applications. User interface designers and developers ensure an aesthetically pleasing integration of these applications. Furthermore, this approach has additional advantages: applications utilising diverse technologies can be developed by independent teams, thus avoiding operational interference between them.

Challenges in testing multi-technology HMI applications

Let us now examine the testing process for such applications in greater detail. Each will be managed by distinct testing teams tasked with developing independent automation for these applications. Furthermore, test results and quality assurance will be conducted separately for each. The term “separate” is crucial here; while it facilitates the development phase, it proves to be less beneficial during tests. Such separation introduces additional efforts and overheads and leaves the question of the overall system’s behaviour unresolved.

Integrating Qt6 and Unity applications in test automation

We have a Qt6-based application that displays static content, like icons, warning messages, or menus. On the second layer, we have a Unity-based application. It shows animations of speed and RPMS, animated maps, and more, utilising a 3D scene approach.

The task is to create test automation for such a setup. Applications are delivered within a virtual simulator – it can be a QEMU or a Docker image. A must-have requirement is to combine test automation into one test framework and create test cases to test both applications in one shot.

Choosing the right tools: Squish for Qt and AltTester SDK for Unity

When it comes to Qt, the only reasonable approach is to use tools from the Qt portfolio to test the application. They are crafted to support their own technology in full. The official test framework designed for Qt is Squish. This solution is our first (and only) choice, as it provides comprehensive support and an out-of-the-box seamless test automation experience.

The choice of test framework for the Unity application has some constraints to meet:

• the possibility of connecting with Squish,
• reasonable price,
• use in headless mode (for CI purposes).

One of the preliminary evaluated tools was AltTester SDK – an open-source framework maintained by the Alttom company. Alttom provides a commercial version of the AltTester with full support and enhanced possibilities.

For our needs, the AltTester SDK was enough to start the initial implementation and covered all our requirements.

Designing a unified test framework architecture

After evaluating tools, we decided Squish would be our base for all test activities, such as running tests or preparing reports.

That’s because AltTester SDK is not a fully functional test framework but rather a driver, allowing one to connect to an instrumented Unity application and inspect its elements. It has no reporting options (except the screenshot method), and to run tests written for it, we would need to run Python, C# or JAVA scripts or use a test framework, which will deliver all test-related functionality (like the BDD approach or reporting).

Below, you can check the final architecture we have decided to implement.

The Unity application is instrumented during the build process with the AltTester SDK Unity plugin, Squish test runner, Python wrappers, and helpers written to provide desired functionality and files with object maps for both applications.

The runner executes tests by connecting to Qt (native connection) and Unity (via AltDriver SDK) applications. It executes steps written in the BDD approach, collects logs, screenshots, and other test products, and finally wraps everything up in a test report.

Making HMI applications testable with Squish and AltTester SDK

The test framework must connect to applications under test to make things happen.

Squish instrumentation can be done in two ways:

• hook into the application when it is starting,
• embed Squish hook into the application when it is building.

We decided to use the first option, as the test environment setup allowed us to do that.

AltTester SDK must be added during the Unity application’s build phase. It is impossible to instrument an already-built application, which introduces some difficulties. Most importantly, if we want to use AltTester SDK, we must have access to the Unity application’s build process and amend it to provide instrumentation.

The way the Unity application is instrumented is quite straightforward: the user needs to build the AltTester SDK plugin, add some dependencies to the Unity project, and import the plugin into it (documentation on importing the AltTester package in Unity Editor).

Of course, some other tweaks and adjustments may be needed, depending on the project specification. We needed to amend the build script for the Unity project to make things work. Here’s an example of command line instrumentation:

# Step 1: Set up the Unity project with dependencies.
python3 update_for_altester.py

# Step 2: Import the AltTester Unity plugin.
./Unity -quit -batchmode -nographics -disableaudio -importPackage AltTester-1.8.2.unitypackage -projectPath .

# Step 3: Build the project using a custom build method.
./Unity -quit -batchmode -nographics -disableaudio -projectPath . -executeMethod SpyroBuilder.BuildAltTester

Test framework implementation: object maps and more

At this point, we had Squish and Unity applications connected to our test framework. Tests could inspect objects in both apps, check their properties, interact with them, and perform all the test activities we designed.

Object maps

We created object maps for both applications for quick and simple access objects.
A Squish object map can be created automatically using Squish IDE and inspection tools. The objects are symbolic names, and their values are defined by a list of parameters, like “title,” “type,” etc. This is an example of a Squish object map:

# Main media player view
media_player_screen_main_view = {
    "title": "MediaPlayer",
    "type": "QQuickWindowQmlImpl",
    "visible": True
}

# Media player playlist objects
media_player_playlist_view = {
    "container": media_player_screen_main_view,
    "type": "ListsScreen",
    "visible": True
}

media_player_playlist_text = {
    "container": media_player_playlist_view,
    "text": "Playlist",
    "type": "Text",
    "visible": True
}

Having the object defined, Squish can reach it, check its properties, or interact with it. Here’s an example of the wrapper method for finding objects in Squish:

def find_object(obj, timeout=1000, visible_object=True):
    """Finds and returns an object. Does not log.
    Args:
        object_name (str or object): Name of the object to be found
        timeout (int): Search timeout in ms. Defaults to 1000.
        visible_object (bool): Is object visible? Defines Squish function to be used.
    Returns:
        obj: Found object, or None if not found.
    """
    try:
        if isinstance(obj, str):
            if visible_object:
                obj = squish.waitForObject(getattr(names, obj), timeout)
            else:
                obj = squish.findObject(getattr(names, obj))

    except LookupError as ex:
        return None

    except Exception as ex:
        raise Exception(ex)

    return obj

And inside the Squish IDE…

Unity object maps with AltTester SDK are a bit trickier. They can be created using Unity Editor (the easier way) or with the AltTester SDK Driver and manual inspection from the command line.

AltTester SDK allows the identification of objects by methods such as name, path, text, ID, and others. It can be used depending on the scene or object’s tree setup and gives flexibility to implement functions to find objects. Below is an example of our object map for the Unity application.

charging_toggle = {"by_selector": "NAME", "search_value": "Charging_Toggle"}
navigation_toggle = {"by_selector": "NAME", "search_value": "Navigation_Toggle"}
vehicle_toggle = {"by_selector": "NAME", "search_value": "Vehicle_Toggle"}

engine_economy_button = {"by_selector": "NAME", "search_value": "CarToggleForGroup (1)"}
engine_economy_label = {"by_selector": "PATH", "search_value": "//CarToggleForGroup (1)[1]"}

engine_balanced_button = {"by_selector": "NAME", "search_value": "CarToggleForGroup (2)"}
engine_balanced_label = {"by_selector": "PATH", "search_value": "//CarToggleForGroup (2)[1]"}
engine_performance_button = {"by_selector": "NAME", "search_value": "CarToggleForGroup (3)"}
engine_performance_label = {"by_selector": "PATH", "search_value": "//CarToggleForGroup (3)[1]"}

The function to get objects from the scene may look like this:

def get_object(object_name, is_enabled=True, timeout=0):
    """Finds and returns an object. Does not log.
    Args:
        object_name (str): Name of the object to be found.
        is_enabled (bool): is object active (true by default)
    Raises:
        Exception: When exceptions other than 'object not found' are encountered
    Returns:
        obj: Found object, or None if not found.
    """
    try:
        obj_data = getattr(names, object_name)

        if timeout > 0:
            return driver.wait_for_object(
                getattr(alttester.By, obj_data["by_selector"]),
                obj_data["search_value"],
                enabled=is_enabled,
                timeout=timeout,
                interval=0.2,
            )
        else:
            return driver.find_object(
                getattr(alttester.By, obj_data["by_selector"]),
                obj_data["search_value"],
                enabled=is_enabled,
            )

    except (alttester.NotFoundException, alttester.WaitTimeOutException) as ex:
        return None

    except Exception as ex:
        logger.info(f"Exception while looking for object: |{object_name}|")
        raise Exception(ex)

In the Python command line, finding objects looks like this:

In the Unity application, this is a sample object:

As already mentioned, AltTester SDK allows objects to be found using several methods. In this example, the By.NAME method was used, but a user can choose between PATH, TEXT, ID and more (documentation on finding objects).

When an object is found, both Squish and AltTester SDK allow the user to inspect its property, obtain the object’s text, click it, or do whatever else is needed to create a test step.

API capabilities of Squish and AltTester SDK

AltTester SDK exposes API to find objects, interact with them and inspect their properties. Some helpful features allow users to dig deeper into the Unity application, like calling Unity component methods or interacting with component properties. The scenes’ related functions are also in place, can be used to load or unload the desired scene, or assess, that the current scene is correct. Another method is screenshot capture. The function gets a screenshot and saves it as a PNG file, which can later be attached to test reports.

Squish API, except for test-related methods, delivers the functionality needed to organise the test flow. For instance, the user can take and wrap the test.fail()method and add other things needed by the test setup. In our case, we wrapped the test.fail() to provide additional functionality with the failing test step when testing the Unity application and added the AltTester SDK screenshot step:

def screenshot():
    """Takes screenshot
    Args:
        none
    Returns:
        none
    """
    dt = datetime.now()
    timestamp = dt.strftime("%Y%m%d_%H%M%S")
    path = Path(f'{os.getcwd()}/screenshots')
    
    if not os.path.exists(path):
        os.makedirs(path)

    scr_path = Path(f'{path}/screenshot_{timestamp}.png')

    try:
        driver.get_png_screenshot(path=scr_path)
        test.attachFile(str(scr_path), "Alttester screenshot")
    except:
        test.xfail("There was a problem while taking the screenshot")


def test_fail(fail_message):
    screenshot()
    test.fail(fail_message)

Interaction handling

Both tools provide interactions.

In Squish, there are several methods, like tap or click or even the Gesture Builder class, which allows the creation of simple and complex gestures.

AltTester SDK takes a similar approach, from simple press or move methods to complex multipoint interactions. It can create touch (BeginTouch(), MoveTouch(), EndTouch()). So, in both cases, users can easily and flexiblely simulate interactions and gestures.

The impact of integrated test automation on software quality

Using AltTester SDK and Squish together allowed us to create test scenarios that could test two applications developed in different technologies. The scenarios could switch context seamlessly and perform test steps. Test reports deliver all needed information in one place, and there is no need to create separate pipelines in CI to run all automation.

Are you looking for HMI specialists?

Learn how we can support your next product – check our HMI offering, and don’t hesitate to get in touch in case of any questions.

FAQ

What does integrating Qt6 and Unity enable?

expand_more

Integrating Qt6 and Unity allows developers to combine user interfaces built in Qt with 3D environments created in Unity. This enables more advanced applications, especially in simulation, HMI, or visualisation scenarios.

Why use Qt6 and Unity together in test automation?

expand_more

Using Qt6 and Unity in test automation allows teams to validate both UI logic and 3D interactions within a single workflow. This helps ensure that communication between components works correctly across the entire application.

How do Qt6 and Unity applications communicate?

expand_more

Qt6 and Unity applications typically communicate through defined interfaces such as sockets, APIs, or messaging mechanisms. This allows them to exchange data and synchronise behaviour during runtime and testing.

What are the main challenges when integrating Qt6 and Unity?

expand_more

The main challenges include synchronising communication, managing different runtimes, and ensuring consistent behaviour during automated tests. Proper architecture and clear interfaces help reduce these risks.

When is this type of integration useful?

expand_more

This approach is particularly useful in projects involving simulations, HMI systems, or applications where a 3D environment needs to be controlled or extended by a Qt-based interface.