A swarm of marbles that flows through your intestines in order to detect thickening. It sounds like science fiction, but it's closer to reality than you might think. Engineer Peter Baltus of Eindhoven University of Technology has developed, in collaboration with universities and the business community, sensors that are able to inspect hard-to-reach or even inaccessible environments. His primary focus is on large-scale pipeline networks such as drinking water systems, oil pipelines or transport systems for kerosene. But his dream is that these small balls will one day explore the inside of volcanos, or maybe even the human digestive tract. For more information see the IDTechEx report on Mobile Robots and Drones in Material Handling and Logistics 2018-2038.
Peter Baltus, professor of microelectronics at the department of Electrical Engineering at TU/e worked on the so-called Phoenix project in his role as project leader, together with project partners KU Leuven, Aachen University, Antea Group, the University of Groningen, VU Amsterdam, and the University of Trento. 'This multidisciplinary group, with universities as well as the industry, was essential to develop these smart marbles in such a short space of time.'
Financed with a €3.6 million grant from Horizon 2020 'Future and Emerging Technologies' (FET), an EU program aimed at developing science fiction-like technology, the group built a device that can explore difficult-to-access environments. And that's a real chicken-and-egg problem, because how do you design sensors for an environment you haven't seen? 'In order to collect good information, you need good sensors. But if you want to build good sensors, you have to know what you want to measure, and that requires knowledge of the environment you want to explore. An impossible task,' Baltus says, laughing.
The researchers specifically looked for socially relevant applications that address primary human needs such as safety, health, energy, and food and water. Think, for instance, of the pipeline network beneath our feet; drinking water pipes, the sewage system or pipes for transporting fuel. But also think of detecting underground storage places for CO2, chemical reactors, or even blood vessels or the human digestive tract.
After much trial and error, the research team eventually produced a collection of smart marbles. These are small balls with sensors inside, which flow through pipes filled with liquid in order to detect obstructions and leaks, or map a pipeline network. The balls are released manually into the liquid at the beginning of the pipeline system. They then float passively with the current until they reach their final destination, where they are extracted so that the researchers can read out the data. The sensors collect an enormous quantity of information along the way, which they store internally.
The marbles must be suited for measurements during a company's production process, without having to shut it down. Baltus: 'That's still frequently the case, for example during the inspection of the pipeline that transports kerosene between the port of Rotterdam and Düsseldorf airport. At present, a robot carries out this inspection, but in order to do so, the pipe needs to be emptied. A costly job, because time is money. Many production processes, for instance those in the chemical industry, simply can't be shut down at all.' The project team anticipates this by making the marbles resistant to all kinds of liquids and changes in temperature or pressure. Even the most corrosive liquid should produce reliable measurement data.
In the chemical industry, at an industrial scale, mixing liquids is often done with air bubbles. These bubbles rise to the top of a reactor vessel and properly mix the different liquids in the vessel this way. However, the measurement probes inside these types of tanks, whose purpose it is to monitor the mixing process, can seriously disrupt things. Testing has shown that the marbles have a much less disruptive effect, primarily because they float upwards with the air bubbles. Moreover, they collect more interesting information than the probes do. The balls, however, can only do this when they are smaller than the air bubbles. Stirring and mixing equipment supplier Jongia, specialized in industrial agitators among other things, has expressed an interest and it now wants to test the balls in their large silos. The company will soon start with this test.
In many cases, the pipes are too narrow to allow space for one super sensor that can do everything. The size of the ball, and of the battery in particular, is the limiting factor. That's because every sensor needs power. But the more intelligent you make the sensors, the more power they usually need. And a large battery simply won't fit into a small ball. That would exclude, for instance, cameras, light, microphones or drive system. The shell only fits simple sensors with low energy consumption, such as a thermometer, pressure gauge and speedometer.
That is precisely why Baltus and his project partners developed a swarm of marbles. 'There's not much they can do individually, but together, they collect all the data we need,' Baltus explains. But how do you ensure that each marble is optimally adapted to its specific task? The environments that Baltus wants to explore are so extremely inaccessible, that he can't adequately predict how the pipe will look like from inside. The researchers came up with an ingenious solution: they use Artificial Intelligence to let the marbles 'evolve' automatically on the computer during the design process, until they are optimally adjusted to suit their specific task. That has an additional advantage: it makes it possible to develop a generic system that can evolve into the best configuration for each application. 'The choice for Artificial Intelligence was born out of despair. If we knew how to design the marbles, we would have done it ourselves,' explains Baltus.
The exact design process of the balls works as follows: the generic balls, programmed with the scarce information and wishes of the client, traverse a company's pipeline system once. The sensors collect all kinds of information along the way, which they store in their internal memories. The researchers collect the data from the balls and develop an initial model of the unfamiliar environment. All the data collected by the marbles, including the first model of the environment, are subsequently copied to a virtual environment. An algorithm then starts to work on this during a series of simulations. It takes the algorithm about 500 simulation rounds to look for an optimal configuration that suits the specific implementation and wishes of the client. As soon as the new configuration doesn't fundamentally change compared to the previous couple of configurations, the researchers know that they are close to finding the solution. The software that was designed by Artificial Intelligence is then installed into the balls, after which they are once again released in the same pipeline system. This process is repeated three to five times. After this, the balls are finally ready to be sold to a company where they can carry out inspections.
Baltus:// 'I draw a parallel with evolution, because the process is so unpredictable. We don't have a solution, so we let an algorithm do it. This algorithm mixes the most successful balls and works towards an optimum. We don't know the results of that simulation process beforehand. Perhaps the algorithm decides to give each marble its own task, which could lead to location marbles and marbles that detect leakages. We haven't been able to test this extensively, so we have to wait and see. The fact that our first evolution experiment shows that the principle works is actually unbelievable, and they even evolve in the right direction as well!'
Once you have a working system with smart marbles that evolve automatically towards a certain application, you encounter one more problem: with all the data collected by the sensors, the client would like to know the exact location of the marbles at that moment. In case the sensors detected a pipe burst, you obviously need them to store the location. That is why the location sensor is extremely important, but GPS doesn't function underground. Luckily, there are alternatives. One of them is the use of acoustic waves, which can actually work underground. Think of an ultrasonic system for instance, comparable to the echos that bats use. This solution might sound promising, but there are still a number of obstacles to overcome. Baltus:// 'During an initial calculation, this system used no less than 5000 times more energy than all the other sensors and systems (including memory and the microprocessor) inside the small ball combined.'
The smart marbles will soon be tested at drinking water company Oasen in the province of South Holland. There, the marbles have to determine the precise location of the underground network, as the registration system does not always indicate the pipelines correctly. When carrying out maintenance work on the pipes, the drinking water company wants to prevent digging a few meters off. Moreover, the excavators must not cause damage to pipes and tubes. Technicians now often make small test slots, which they dig manually, to determine exactly where the main pipes are. The smart marbles offer a fast and efficient alternative without the need for digging. In the event of a leak, the speed and direction of the water flow changes, which a movement sensor or compass can clearly show. In addition, a gyroscope can measure the turbulence around a leak, and a microphone picks up the typical hissing noises that occur.
The researchers came up with an unexpected alternative for the smaller marbles: magnetism. Baltus: 'You can transport signals via the liquid in the pipeline network with this method. Reversing magnetic fields with low frequencies, for example. The ball can transmit its own serial number this way.' Additional benefit: the signal goes in every direction, which makes picking it up much simpler. 'A combination of magnetism and ultrasound might be a possibility. In that case, the systems calibrate each other, which enables you to make a reverse calculation of the sound velocity. You even collect new data that way, namely, the medium in which the balls float around,' Baltus explains.
The magnetic system, however, only functions over short distances. Luckily, that is not a problem: since the marbles travel in a large swarm, there's always a receiver nearby. The location is determined by their relative position to each other and by the entry point in the pipeline where the marbles were inserted. This provides the researchers with a very detailed map of the pipeline system once they read out all the information stored in the marbles.
Baltus fitted the marbles with an emergency system: 'When a marble gets stuck, it uses its last bit of energy to send out a distress signal to the other balls nearby. In doing so, the marble also sends all the information it collected, so that no vital information is lost.'
The Phoenix project has officially ended. Baltus: 'We have shown that the concept and the evolution of the marbles works, and we've developed the technology that will enable us to significantly reduce the size of the marbles in the future.' Baltus is clearly far from finished with the project. As a next step, he has started a spinoff under the name 'Smarble', together with the Antea group. The purpose is to market the balls, a project in which Antea will take the lead. Smarble received a €100,000 euro grant from the Launchpad program to develop a business case, in addition to the HORIZON 2020 FET grant. 'We want to produce robust and affordable swarms of marbles and put them into use. We don't need new inventions for that, but it will take a lot of hard work to solve all kinds of practical problems,' Baltus explains.
Besides Smarble, TU/e will continue with the research as well. Baltus: 'We want to make the marbles even smaller, so that we can use them in healthcare, for example. Imagine that they're small enough to flow through your bloodvessels. That would bring entirely new possibilities to the detection of obstruction. Researcher Giovanni Iacca from the university of Trento is starting up a consortium to reduce the size of the balls. He is currently looking for partners to apply for a new Horizon 2020 FET grant.' Baltus also hopes to perform simple routines in the future. 'Now, we only detect leaks, but in the future we also want to be able to close them right away with a kind of resin.' Baltus plans to apply for a grant for this as well.
Source and top image: Eindhoven University of Technology
Learn more at the next leading event on the topic: Healthcare Sensor Innovations USA 2020 on 18 - 19 Nov 2020 at Santa Clara Convention Center, CA, USA hosted by IDTechEx.