Skip to main content

Teaching robotic systems to read

As Industry 4.0 era takes hold, it’s important to understand the technologies that underpin industrial automation and productivity and also, how they help people work in harmony with the machines.

Large-scale production requires highly scalable assembly processes and detailed quality inspection. Fully automated robotics systems put parts together, guide them through the factory, and assure that quality standards are met. Algorithms relying on visually acquired information use modern machine vision and computer vision systems, where the combination of an industrial camera and a computer-processing unit simulates the function of human visual control. 

Machine vision-based robotic systems are able to ‘see’ and make decisions based on the visual input rather than just executing pre-determined actions like, for example, a static welding robot on a production line. By incorporating sophisticated text recognition Optical Character Recognition (OCR) technologies, machine vision systems can gain ‘reading’ capabilities. Text-based information printed on labels, displayed on control panels or visible on computer screens can be captured, interpreted and processed. While information on paper-based documents such as service reports or product documentation can be converted as part of a scanning process, automation systems capture this information with cameras that have been implemented directly in the production line or testing facility.

In both scenarios, the information is available as ‘image-only’ information. OCR technology is then needed to convert this and make it machine readable, searchable and usable.

What exactly can machines read?

Like humans, the systems can ‘read’ information and subsequently react. Such information includes text and barcodes on packaging as well as images of text on electronic displays. With integrated data capture technologies, robotics, computer vision and quality control systems can process the extracted data and compare it with entries in the database to initiate the system to take the appropriate action.

Converted information can be exported to applications and systems in the form of XML data and plain text. With these new capabilities, the robotics systems can conduct a range of completely new tasks. By implementing sophisticated technologies for text recognition, such as OCR, hand print extraction (ICR), barcode recognition (OBR) and optical mark recognition (OMR), independent software vendors and hardware providers for the manufacturing industry can significantly extend the functionality of machine vision systems – and offer more value to their customers.

Functionalities of these include:

  • OCR: Recognition of text on screenshots, photographed text or scanned documents. The text is converted into electronically readable information that can be reused by automation systems.
  • Advanced image processing technologies: Photographed images can be automatically optimised, for example, cropping image areas and improving the quality to reach high-quality recognition results, such as when photos are taken in very dark or very bright light conditions.
  • ICR: Recognition of hand-written letters in forms, such as maintenance checklists or questionnaires.
  • OMR: Recognition of checkboxes in forms, maintenance checklists and surveys.
  • Barcode recognition: Recognition of conventional 1D and 2D barcode formats as used for automated routing of parts through the production line.
  • Data extraction for predefined areas: Capture of information in specific areas of the page.

Armed with these abilities, manufacturers can put their machines to work to answer some of the more challenging questions that occur at the end of the production line.

Touch panel and device user interface testing

Touch panels and user interfaces in various consumer devices can be automatically tested by simulating the individual steps. During the screening process of the touch-enabled device, the individual areas of the screen are activated by the robotic platform and response messages are captured. The extracted text is automatically validated by a computer with integrated OCR technology.

Domestic appliances, such as washing machines, microwaves or dishwashers contain such panels, as do car infotainment systems, which are increasingly becoming the norm in modern vehicles. In order to enable communication between driver and vehicle to take place through the car’s infotainment system, a team of engineers ensures that these systems in the car functions accurately and reliably.

For Volkswagen, the testing department was looking to automate rigorous testing processes. To improve the automated testing processes, they introduced OCR technology (opens in new tab) to verify the content appearing on the screens – even for complex display messages and in various languages.

Routing parts on the production line

During the automated production process, individual parts or partially finished products need to be routed to different locations. By integrating text and barcode recognition technologies, automation systems in the production line can then be programmed to ensure that parts automatically follow pre-defined routes.

Checking of packaging information

By applying OCR to photographed labels, packaging or documentation, text contained in them can be extracted and checked against databases. This way, products with incorrect information can be singled out automatically before they leave the manufacturing site, thereby reducing the margin for error. This process can be applied to anything from information on pharmaceutical packaging, labels on beverage bottles or information about allergy or expiration dates on food packaging.

Automating service and maintenance processes

Even processes in which human interaction is necessary – such as during service and maintenance tasks – can be significantly accelerated by the help of text recognition and data capture technologies. These technologies automatically extract service steps information from scanned paper documents, for example notes and tick box information on repair slips, and make this data available in the central repository.

The benefits of teaching a machine to read and react for themselves across industries where quality and accuracy are key, will ultimately save time, money and help achieve customer satisfaction. All seen through the eyes of a robot.

Dr. Rainer Pausch, product group head, SDKs, ABBYY (opens in new tab)

Image source: Shutterstock/Sarah Holmlund