Testing the integrated hyperconnected world

Integration testing in Internet of Things (IoT) means testing the interconnection of uniquely identifiable embedded computing devices that collect data or perform data-based actions through the internet.

IoT is characterized by:

• Smart Objects and Sensors
• Intelligence at the edge
• New communication paradigms
• Servitization & Virtual Continuum
• Massive/Big Data

The era of the hyperconnected world enabled by IoT through service integration is here upon us. Addressing the testing approach of a hyperconnected hybrid ecosystem with multi-layered connections having thousands of use cases, millions of apps, billions of users, and trillions of things is not a cakewalk.

In the hyperconnected system of systems, the most critical challenge is foolproof integration testing to ensure swift and quality deliveries, resulting in high customer satisfaction.

Need for Integration Testing in the hyperconnected world

There are many challenges that a testing professional would face while testing multilayered complexities, diversified platforms, devices, sensors, protocols such as:

• Diversity in apps, devices, networks/channels, and permutation combinations of real-time scenarios
• Wide range of protocols, interfaces, and services
• Due to many interconnected hyper systems, defining the testing scope is a big challenge
• Real-time data testing (field testing)
• Frequent changes in the environment require regular integration testing and end-to-end testing
• Close coupling of hardware and software components

Devising an efficient testing strategy is the key towards overcoming these challenges.

Goals of integration testing in the hyperconnected world

The six major goals for integration testing, which are necessary to ensure a well-integrated and well-tested application are:

1. High reliability – Integration testing ensures the system continues to perform correctly, and integrated modules work as intended to improve the reliability of tests.
2. Ability to test in isolation – Performing module testing in isolation helps the system to identify and detect module-level defects, especially when it is difficult to test from system testing level.
3. High scalability – As the system grows in data and traffic volume, or complexity, doing integration testing helps figure out the application health based on the performance with each integration.
4. Appropriate transitory – Integration testing helps assure the suitability of the modules and their results.
5. High speed and performance – Integration testing of each module ensures good performance under the expected load and data volume.
6. Cost-effectiveness – Integration testing is done after unit/component testing almost at the starting phase, so fixing the cost of a defect is lower compared to system testing.

Our proposed testing approaches

Unlike traditional software testing, the scope of testing hyperconnected hybrid solutions is much broader and more complex. Testing the hyperconnected solutions requires a more refined testing expertise and a robust QA validation process. The QA team must act from an IoT user's perspective and must follow the Test-As-A-User [TAAU] approach instead of following a conventional requirement-based testing approach.

We must radically transform how we test IoT-based applications – say hello and welcome to “IoT QA”! IoT QA should comfortably test not only hardware but also software, by testing in a defined environment and also in a dynamic environment. This requires strong domain knowledge.

Detailed approach for integration testing in IoT application

Nagarro follows a four-principle approach for specific, integration testing IoT applications:

1. Interoperability and conformance testing
2. Simulation
3. Service virtualization
4. Data validations

1. Interoperability and conformance testing

IoT is an integrated network of different software components, devices, access channels, protocols and network channels. Interoperability testing verifies the ability of all components to interact correctly, exchange data, and use information between software components or systems. It is important both when using a new IoT system and when changing services, protocols, or software components.

System Under Test (SUT) consists of Implementation Under Test (IUT) and the Means of Testing (MOT). The MOT includes at least one tester, depending on the architecture of the IUT and its interfaces. The MOT also handles activities such as coordination, logging, and reporting.

Ref: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8281514

2. Simulations

Since it is impossible to have diverse real devices, it is better to either build simulators to test certain aspects or use a simulation solution that provides access to connected things without the constraints of physical hardware. With a simulator, building a large network is relatively easy. This is achieved by writing code inside the simulator to provide and design nodes virtually over a virtual wireless network. With this simulator, you can simulate all aspects of your IoT design, such as the hardware setup of each node, processors, layers, wireless networks, etc. Changing the signal strength between a pair of nodes and randomly dropping packets, as a result, is something that can be set up within the simulation rules. Such conditions can frequently occur in the real world. For instance, when a train travels through the line of sight between two nodes, and the communication is interrupted. It would be difficult to create such a simulation in a laboratory with physical setup.

3. Service virtualization

Service virtualization provides a cost-effective way to perform interoperability testing. Service virtualization can speed things up significantly, enabling the testers to be less dependent on test infrastructures, evading the glitches of interconnectivity. For service virtualization, a lab setup for all simulated devices helps avoid any dependency constraint on the development and testing teams.

Methods for service virtualization:

1. Determine all the resources such as database system, routers, network, SaaS systems or complex test data, which require synchronization and integration
2. Identify the system that is unavailable for testing or has some other timeline or schedule
3. Create a realistic virtual model of dependent systems so that real systems are not required
4. Testing can be done by simulators with the help of the hardware/devices of the parent company to configure the virtualization product and by validating the input/output through automation script and utilities
5. Deploy the virtual model in a virtual environment to create a realistic environment. The configuration can be done for data or any other features, as required.

4. Data validations

Integration testing helps maintain data integrity, if there is any data loss during transition and transformation, when data flows from sensors, protocols, gateway, and each communication channel. Data validation testing is a cumbersome task, and we have created a data comparison framework based on source and destination, with minimal customization and configuration.

The key to achieving a successful integration is continuous automated testing. Identify the automatable areas and implement automation at different layers of IoT systems. By using the required simulators/emulators, one can identify different areas where automation can be applied like API testing, cross-browser and cross-device testing, device interaction testing, database testing or performance testing.

Scope of testing types applicable on hyperconnected IoT solutions

The following types of testing are usually performed across the integrated products before their release to the market:

How to ensure maximum coverage?

Integrated services emphasize a lot on robust testing competences to guarantee that the performance of the services meets the requirements. By adopting effective best practices, a user can successfully execute integrated testing. Ensure that the following points are addressed while testing integrated applications:

1. Use simulators to test deployment of hundreds or thousands of geographically-dispersed devices.
2. Use digital twins. With digital twins, problems can be predicted and various solutions can be virtually tested before actual changes to the physical counterpart are implemented.
3. Follow Exploratory Testing to discover, investigate, and learn more about the system while testing.
4. Use Remote Usability Testing to test the usability of websites when users are located in different geographical locations.
5. Use Combinatorial Testing for optimal representation of all the impacting factors with a good representative sampling of all the test cases.
6. Use service virtualization testing.
7. Well-defined requirements, comprehensive test plan, unit testing, integration testing, and effective communication should form the basis of integrated testing.
8. Use effective test data management tools that cater to:
   a. Data on demand
   b. Data integration
   c. Data security
   d. Data validation
9. Robust backend – if the mainstream functionalities are embedded into a robust backend, the backend functionalities can be tested thoroughly by the usual testing methods, tools, and approaches.
10. Use Risk-Based Testing to identify the features that are most critical to the business and prioritize features and co-related test cases according to their level of risk.

Test beds like Device Simulators, Network Simulators, Location Simulators are required, because it is not possible to perform testing on all the actual devices, actuators, and channels. It is also important to monitor the traffic between the devices/actuators and applications; hence, sniffers/analyzers are also required.