KINE Robotics Oy is a Finnish provider of custom turn-key robotics solutions. Their customers are in industries including food and beverage and logistics as well as semiconductor and general manufacturing. For more than 20 years they have designed, implemented, and delivered fully customized robot-based solutions to fulfil unique client requirements.

The team at KINE was asked to develop a conveyor tracking system with on-the-fly robot picking of bags of bread loaves and rolls for an industrial bakery. Product sizes are irregular, and shapes differ from one type of bread to another, making it a challenge to determine the location and orientation of the bags on the conveyor. The transparent plastic packaging likewise makes it difficult to detect with optical sensors, due to low contrast and partial reflections.

The 3D conveyor tracking vision system from KINE offers on-the-fly robot picking for cost-effective, reliable product detection and processing.

Prior to the upgrade to a high-performance vision system, the process involved manual removal and depositing of bread packages, which was potentially error prone and certainly not as efficient.

The team at KINE knew that machine vision was the only possible solution for this type of system. “Our key suppliers consider Matrox® Imaging the preferred option for machine vision solutions, and KINE has used Matrox Imaging products for many prior projects,” notes Kimmo Salonen, chief technology officer at KINE. “We were already familiar with the capabilities of Matrox Design Assistant® X vision software, which has proven to be a versatile tool for this project, offering exactly the kind of integration and control we wanted.”

KINE sought to develop the vision system around Matrox Design Assistant X software and a Basler Blaze 101 3D Time-of-Flight (ToF) camera as the sensor. The ToF camera was selected as it eliminated the need for separate lighting, and reduced concerns about contrast and reflections. “We really liked how smoothly the 3D ToF camera integrated with Matrox Design Assistant X software,” confirms KINE’s project manager, Tero Urponen. “In other projects, we have had multi-camera systems that can get a bit cumbersome, whereas in this case, it was very nice to have one system to control everything instead of having to program each camera separately.” Helping KINE through their development and acquisition process was OEM Finland Oy, trusted distribution partner of Matrox Imaging. A member of the one of the largest vendors of industrial components in northern Europe, OEM Finland Oy provides an exceptionally broad range of industrial automation solutions and electromechanical experience to machine and systems builders.

Bread rolls of various shapes and sizes are packaged in transparent plastic bags and move through the vision inspection system.

Rising to the challenge

For this food packaging installation, the products being examined posed some serious challenges for the vision system: different types of breads have unique shapes, sizes, and colors, and the semi-transparent packaging can obscure the contents, making it harder for the system to distinguish between products. A robotic solution with proper component selection would ensure a faster, reliable, and cost-effective manner of product detection and processing.

The packaged bread is viewed by the Basler 3D ToF camera, shown here mounted right next to the Stäubli robot, which sends the 3D data to Matrox Design Assistant X vision software for processing and analysis.

The inspection system is built around Matrox Design Assistant X flowchart-based software and the Basler 3D ToF camera, which affords real-time measurement of up to 30 fps and provides precise 3D data to the vision software. The camera is triggered from a photodetector by way of a B&R PLC X20CP1585 programmable logic controller (PLC) communicating with the vision program via a user datagram protocol (UDP). The PLC also synchronizes the robot by similar means. An industrial panel PC from B&R serves as the platform to run the Matrox Design Assistant X runtime environment and the associated vision program. Handling the picking and placing is a Stäubli TS2-60 SCARA robot, which communicates via transmission control protocol (TCP) with Stäubli VALtrack software library for conveyor tracking and picking. The robot needed to be able to get a good grip on the plastic bags without causing any damage to the delicate bread or its packaging.

Stationed near the operator on the production line, the main user interface provides an overview of the whole system. From here, the operator can control the whole production cell, defining and adjusting speed and timing. The operator can likewise access the Operator View of the deployed Matrox Design Assistant X project from this station.

Camera data analysis is implemented using Matrox Design Assistant X software, which also produces the central interface to the vision system. The 3D ToF camera records the surfaces as a point cloud with over 300,000 XYZ coordinates; Matrox Design Assistant X converts the point cloud into a depth map, which is analyzed using 2D vision tools to determine the grip points for the robot. The 2D tools provided by Matrox Design Assistant X work particularly well on depth-map data and allowed the team at KINE to effectively solve their application.

Expressly designed to require no operator interaction, the vision system is located directly atop the incoming product conveyor, just before the robot working area, and functions completely autonomously.

Proofing the system

Overall, implementation of the vision system went smoothly for KINE. “Basic implementation of the vision system took about one day,” Sami Sinisalo, robot specialist at KINE, reports, “and testing, adjusting, and fixing took about 80 hours, including time calibrating the robot. That’s a great turnaround.”

One challenge they faced involved issues caused by communication delays with the PLC trigger making synchronization difficult. The ToF camera does not have a physical trigger input, and use of the panel PC meant KINE did not have access to the I/Os that would be available on a vision controller, as an example. “This means that the communication between the photodetector and the Matrox Design Assistant X runtime involves a PLC and UDP communication,” Urponen reports. “The network communication does not occur in real time, which results in a variable delay.”

The transparent plastic bag is not seen by the ToF camera, and the random physical arrangement of the products inside the plastic bags is not useful for determining package position or pick height.

Another synchronization challenge stemmed from interface of the Basler ToF camera to the Matrox Design Assistant X runtime. The camera time varies from one acquisition to another by up to 50 ms, causing the product to move up to 20 mm (at 400 mm/s). KINE overcame this by using a trigger signal from the photodetector to determine the distance travelled on the conveyor; the same trigger is also used to trigger the ToF camera to commence image acquisition. The robot uses an encoder to track the conveyor movement after the trigger event, and the machine vision software then determines the transversal position and orientation before performing the analysis.

Sinisalo notes that, “initially, there was a plan to use 3D information from the Basler ToF camera to determine the pick height for the robot, but we ultimately decided to create a recipe for each product with Matrox Design Assistant X software.” As illustrated in the examples above, the transparent plastic bag is not visible to the ToF camera, and as the bread products within are arranged in random positions, the lack of a single, consistently flat surface makes it very difficult to determine the correct pick-height based on 3D depth-map information. KINE developed an identical vision system that only handles whole and sliced bread loaves; “in that case,” Sinisalo continues, “the height measurement could work but we still believe that a recipe-based solution is more reliable.”

The rolls inside the plastic bag are randomly arranged, making it difficult, if not impossible, to determine the correct pick height based on depth-map information as there is no single flat surface. Use of a recipe-based solution with Matrox Design Assistant X ensures consistent reliability for the pick-and-place vision system.

Recipes are used to configure select flowchart steps so that a flowchart can inspect different product variants. This allowed for the use a single project to inspect the different types of breads with their unique characteristics. Recipes can be created and configured in Matrox Design Assistant X vision software at both design-time and runtime.

Through use of the ToF camera and vision software, KINE saw a reduction in the overall costs related to machine vision by 75%, compared to compared to traditional camera and lighting options. “Typically,” Urponen notes, “there is a lot of trial and error with lights and camera settings to get proper contrasts for object detection. The ToF camera requires no external lights, and the 3D information it supplies to the IDE is superior to any traditional color/contrast-based detection technology.”

Conclusion

KINE and their customer are very pleased with the final vision system, with a lower error-rate than the prior manual handling and with a higher throughput of goods. With the new vision system, the robot grip can handle between 25 and 30 packages per minute. “We’re already considering the possibility of adding some measurement limits for the products to ignore or flag faulty products as they move through the system,” says Urponen.

The team at KINE can see many opportunities for future integration of Matrox Imaging products into their customed robot solutions, thanks in no small part to the success achieved with this 3D pick-and-place robot system. “I can see us breaking bread with Matrox Imaging for a long time to come,” smiles Salonen.