In this article I will explain some testing strategies for IoT solutions using devices like:  ARDUINO, Personal Medical Devices (PHD) and mobile phones.

Introduction

Some years ago I had an amazing experience while working as part of  the team challenged to create an open-source software implementation of the IEEE 11073 protocol, and at that moment, it  seemed to be the only open-source implementation of this standard. The protocol specifies how to exchange data for personal health devices (PHD) and it covers a huge kind of “health & fitness” devices like: pedometers, heart rate monitors, etc. The implementation should cover many different types of data format, communication flows and device specializations (families), and it does not rely on a single transport layer such as TCP/IP, BLE, Bluetooth LE, etc.

The Challenge

Update a device firmware or a protocol stack at operating system is a critical operation, you should do it only when needed, sometimes it is simply not an option to upgrade the firmware of so many devices in production.

At that time there were a few kind of devices available supporting just a small part of the protocol, so we had to create a strong testing strategy to keep the solution compatible with future releases. Otherwise, our product would not have the proper testing coverage, also each change we made to update the software later could potentially break the functionality for some of the target platforms already in use.

To mitigate this risk I recommended some development practices such as: Continuous Integration and different levels of automated testing, but for such low level programing (in C language) it sounded crazy at first time. However, we achieved interesting results.

Designing the testing strategy for your solution

1) Isolate the CORE and the INTEGRATION code

I recommend for IoT and embedded programs to use a Intelligent Software Agent design, such as Simple Reflex Agent (see picture)

The inputs are represented by sensors data and the reactions may be the state changes and output signals.

For this scenario the CORE is the reactive part of your application, it is the high-level abstraction responsible to respond to an INPUT and to produce an effect. If you isolate this logic you can validate it on multiple environments (development, testing…)  and platforms (PC, Mobile, etc.)  without rely on how the INPUT is generated (a sensor data, a hardware interruption, a network packet, a REST request or response). The input is dependent on the environment, so you can keep in the INTEGRATION code the control of data input.

2) Choose the testing strategy for each level

You can use regular unit tests to cover the CORE functions. I recommend to create pre-defined test scenarios that can be used in many test cases. For instance, I created some data files with the test input and the expected output test data to be compared with the real test output.

The integration area is where the things get complex, because it would be needed to use the real environment.

For this kind of application I opted for manual procedures when performing end-to-end testing with real devices. When real devices were not suitable, we created test agents at transport layer level (network, Bluetooth, etc.) to send the expected data units.

3) Validate it using a Continuous Testing process

The recommend test distribution, i.e., the quantity of tests by test categories, to achieve quality with a reasonable effort is to have cheaper tests to be maintained in automated test process.

Unit tests are simple and easy to maintain when comparing with other test level. So, as soon as you have unit tests enabled for you project you can enable a test suite to run in continuous integration server to validate that there is no bad side-effect for each new code change.

Final comments

What about other IoT gadgets like ARDUINO? You can use similar approach also. However, when your device does not have a LCD display, storage or network communication, it is hard to verify program output and you cannot use unit tests.

For this cases I use alternatives for system testing, like:

1. Write the program output to serial port for debugging the data using a cable connected to the device.
2.  Create a self-testing procedure on your app initialization. It is very common that this kind of embedded programs provide some checks during initialization on device, emitting signals like blinking lights or sounding beeps to notify about anything that breaks the standard functioning. For a recent project, I created an self-testing mode for an ARDUINO program to make it show one different agent reaction each 5 seconds, 30 seconds later I could validate all agent reactions for all expected scenarios in the sequence.
3. Do not let protocol be a restriction, use intermediate agents. When the device was not able to communicate directly to the server (using TCP, HTTP, etc.), you can use other devices like mobile phones  and applications to gather the information using NFC, Bluetooth, or other protocol needed.

I hope this information could be useful for you. If you want to find more about the open-source project described here go to https://github.com/signove/antidote.

Unfortunately, my contribution history was lost in the  migration process to github, but I have to thank the owners because they kept my name in source code.

You can see the agent code in: agent.c