4 major challenges with RAG
Experience-based learning on what’s difficult with RAG in real life
Real data [Documents] are Messy
Most of the RAG examples out there deal with one type of doc. Either a PDF or Word doc. But in real life information is spread across formats like PDFs, PPTs, GitHub readme, RST files hosted on Sphinx server, Wiki pages on confluence, and so on. On top of this, each document does have images, tables, and code blocks in that. So parsing the right element in them becomes extremely challenging.
Unstructured is the answer as it tries to fix the ETL part in LLM apps
Getting the meaningful chunks is Hard
Most of the documents have a structure. They are divided into Sections, subsections, and so on. The flow of reading the docs is not generally sequential. We might need to jump between sections. For example, we might have an appendix section at the end of the doc that has vital info related to a section in the top half of the doc. If we do naive chunking based on sections or paragraphs we might miss the context and lose information.
Group the contents hierarchically.
Deciding the right context size is difficult
The length of the context window that the LLM should process is very critical. We can’t give 1000 docs and ask LLM to get the answers. Giving too much data will dilute the specificity and might result in erroneous and noisy repose. We need to send the most relevant docs to the LLM to get answers out of the data.
Embedding/similarity/relevance search etc specific to the application
Evaluation frameworks for the RAG are still evolving
There is a huge difference between generating a response vs generating a response that can be trusted based on the documents supplied. So faithfulness of the response needs to be monitored. But doing so in RAG is very challenging. There is no False-Positive-and-True-negative simplification out there for RAG yet. It is a very cognitive task.
RAGAS the to be de facto framework for evaluating RAGs
Read some ways to overcome these challenges in RAG here.
Thanks.
