Since the earliest forays into optical character recognition (OCR) by Ray Kurzweil in the early 1970s, software developers have been on a mission to teach computers how to do the paperwork for us.
What if the computer could replace the interminable number of hours needed each day in offices around the globe for humans to read documents, understand the meaning, and extract the right data for the next step in a work process? What if the computer could also do the data entry?
This is the Holy Grail of knowledge worker productivity. It’s also the mission of intelligent document processing (IDP) software, which for 50 years has progressed very slowly towards the goal. AI has always been at the center of this quest. As the availability of cost-effective AI power has increased, so has the ratio of machine automation to human work.
In 2023 cost-effective AI power has gone to the next level. Foundational Large Language Models (LLMs) are now widely available to developers and – just as importantly – are now economically feasible for business applications. While not designed with IDP in mind, these LLMs have shown great promise to move us even closer to our Holy Grail. This marks the beginning of the 4th Wave of IDP.
For the first time, a machine can reliably classify documents and extract data without the need for training samples or prior knowledge. Our clumsy categorization of documents into structured, semi-structured, and un-structured does not matter to the LLM. Send it invoices, contracts, forms, emails, correspondence, or any other text. In AI terms this capability is known as zero-shot learning. It looks like magic to us.
When ChatGPT first landed, we fed it invoice data and asked it to extract company names, payment dates, discount data, product information from tables, and more. The results were astonishing. We knew then that LLMs could change IDP products forever – once skillfully integrated. This AI could replace current IDP needs for long and expensive AI model training work, coding, and arcane regular expression language. Users would need no data science knowledge or business intelligence experience to process documents.
