Under its “Healthy People, Healthy Planet” mission, Aqua Libra Co. is working towards a sustainable future for workplace and retail drink dispensing.
Aqua Libra Co., owned by UK drinks giant Britvic, worked with Amazon Web Services (AWS) and Green Custard to develop the Aqua Libra Flavour Tap which eliminates the need for single-use packaging by delivering refreshing beverages straight to a reusable vessel. The tap can dispense still or sparkling water and a variety of flavours, including Raspberry & Peach; Orange & Yuzu; Blackcurrant; Cucumber, Mint & Lime.
Within the tap are several control systems, one of which reacts to user input and relays flavour cartridge information to AWS. Another directly controls the dispense, switching valves and solenoids for precisely controlled periods as well as driving pumps for the exact quantity of flavour for a specific drink recipe.
Testing the software and hardware running on the dispense control board is the subject of this success story.
THE CHALLENGE
The cost of finding bugs rises exponentially throughout the development and deployment process. Several studies have published the relative cost of resolving a bug, which can range from $100 for a change of requirements to many tens of thousands of dollars if fixed when the software is deployed.
Reproducing an intermittent bug seen in the field can be very difficult and unless the bug can be observed, it may well be marked as ‘unable to reproduce’ and ignored.
Reviewing code can uncover implementation errors which could manifest as a bug, but testing is the cornerstone of modern software development and takes many forms, including unit tests, integration tests etc. Testing software running on an embedded system is made more difficult because often the unit-under-test requires inputs from external sensors which provide signals which change over time.
Simulating the external signals allows the embedded software to function as if it were running within a complete system, not just an isolated test setup. We can simulate these inputs using a technique called Hardware-in-the-loop (HIL). With HIL, we can generate sensor inputs automatically, without user interaction. This helps us test how the software behaves under different conditions.
The Aqua Libra dispense controller is comprised of a microcontroller running a real-time operating system and is used to monitor weight, and fluid flow as well as drive stepper motors and electromechanical valves. Manual testing requires placing a vessel on a weight-measuring device to enable a dispense to be made. The subsequent valve switching and pump activity need to be instrumented to confirm that the expected drink is dispensed. Along with this is the need to simulate the flow of liquid by generating a sequence of pulses as if derived from an electronic flow sensor.
THE SOLUTION
The Raspberry Pi Pico is based on the RP2040 microcontroller and has a peripheral which is very useful for HIL testing. In this test tool, the Peripheral Input/Output (PIO) block generates high-speed serial data as if derived from a 24-bit ADC to simulate a strain gauge. The ADC output can be set arbitrarily and triggers events on the system under test.
The PIO is also used to simulate the output from a flow sensor, which is indistinguishable from the real hardware. The unit-under-test drives many stepper motors and the PIO is used to read the two-phase motor drive outputs, conditioned through optocouplers. This approach can measure every step applied to the motor, even when the step rate is in the kHz range. This provides unprecedented levels of instrumentation for a low-cost test tool.
The Pico is mounted at the top in the centre. IO is around the periphery of the board.
The board under test is in the centre, Pico-based HIL boards are left and right.
Similarly, solenoid- and valve-states are monitored as the test progresses. The RP2040 contains two PIO blocks and a single stepper channel uses all the programs for a single PIO. It is, therefore, necessary to daisy chain multiple RP2040s to enable monitoring of all the IO of the board under test. This requires a novel communication technique over serial that can address a unique board or all boards to configure at the onset of a test and recover the data as fast as possible during the test.
The states of the IO on the unit-under-test are sampled at 20Hz and the collected data is written to a time series database with a unique test identifier. Post-test, the collected data can be plotted using Grafana with a dashboard that is configured to represent a logic analyser. This is very useful for system development by the hardware and mechanical team.
The database can also be queried and depending on the type of test, the data is processed to signify a pass or failure condition. The whole system is fully automated and can run thousands of standalone tests without human interaction. The software running on the RP2040 is identical for each test node, GPIO is used via jumpers to provide a unique node identification which is read at runtime by the RP2040 application. This tailors the node functionality, loads the PIO programs for that node and means only a single image has to be maintained.
A PCB was designed and fabricated with the Pico on Node 0 acting as the main controller with the option to connect multiple nodes to provide more input monitoring.
THE RESULTS
The flexibility provided by the HIL testing tool allows for many combinations of dispensed recipes to be run in an automated environment. A single test can request many thousands of dispenses which take hours to complete, checking each and every pump step and valve open and closure cycles. This would be almost impossible to accomplish manually.
The test results can be viewed in one of two ways:
- As text-based results which are ultimately pass or fail conditions, or
- A graphical display, similar to a logic analyser.
Textual results
The test results are converted to records and written in bulk to an instance of an Influx time series database. The test input JSON file contains metrics which are used to determine if the test has passed or failed. The JSON file is parsed by a test runner Python script, and the relevant metrics for that test are extracted and stored ready for use. At the completion of the test, the Influx database is queried and checks for specific features can be made, for example, the number of steps of the pump motors, the valve opening time and duration to millisecond precision.
A successful test case.
A test failure, controller1 is out of tolerance.
Graphical results
A different way of displaying data is provided by Grafana. This view can be used to determine if the switching times and sequences are as expected by non-software development teams, for example, the people who develop flavours and liquid specialists.
Here we have three distinct charts sequentially generated during a dispense of four separate drinks.
- Top chart - this shows the data around the dispensing progress;
- Middle chart - here we have more data related to the dispense which changes over time;
- Bottom chart - this chart shows the opening and closing of control elements within the tap.
Grafana dashboard displaying system states.
The data descriptions have been obfuscated.
The Outcome
This level of detail and the ability to test many input conditions provide a very powerful testing tool.
In this application, Green Custard utilises the Raspberry Pi Pico to provide a low-cost solution to simulate different types of sensors and measure the resulting change in outputs. We can monitor the states of different components in real-time and write the collected data to a time series database. Finally, we can plot the data on a dashboard for further analysis. The whole process is automated and can run thousands of tests without human interaction.
Test cases can be developed that inject failure conditions that would be very hard or impossible to recreate on a real system and this will test the robustness of the software under test.
Without the Raspberry Pi Pico it would be very difficult, if not impossible, to identify intermittent bugs that may affect performance in the field. Using HIL, Britvic can be confident that the Aqua Libra Co ‘flavour tap’ will perform as expected.
Why Raspberry Pi Pico?
The Pico is attractive as a basis for this kind of testing for several reasons:
- Cost - the entire PCB with onboard power supply and flash storage can be purchased for less than £4. This is beneficial as several testing set-ups can be made for different sites and if a Pico is damaged during development, the replacement cost is not prohibitive;
- Form-factor - The Pico uses 40-pin DIL headers which can be soldered directly to a PCB or fitted with low-profile turned pin headers which allow for easy replacement;
- PIO - The PIO is a superb idea and gives such flexibility that it cannot be overstated how useful it is.
- Ecosystem - The whole Raspberry Pi ecosystem, from blog posts, GitHub repos, YouTube tutorials and the Pi forums provide a lot of useful information which reduces the development time significantly.
- Availability - The Raspberry Pi Pico has not been affected by the global silicon shortage which means that orders can be placed with the expectation that they will arrive.
"The Programmable I/O (PIO) is an incredibly flexible feature of the Pico. It allows for the development of hardware interfaces that would be difficult without a full-blown FPGA implementation. The parts cost and the power provided by the PIO made the Pico a compelling choice. The SDK is well-documented and there are many useful code examples in the accompanying GitHub repo. The Pico is my preferred part for low-cost, low-volume development. I am pleased with the HIL testing framework. It gives me confidence that any change made to the software running on the target hardware has not introduced any regressions and performs as expected."
Neil Bryan
Principal Consultant, Green Custard