What is the reason for the low accuracy of RAG? Improvements and evaluation methods for separating searches and answers
It would be premature to change to a larger model from the beginning just because RAG's answer is wrong. If the correct materials are not included in the search results, the cause will remain even if you change only the model that generates the answer. Observe material, search, answer generation, and change one condition at a time.
If you want to connect RAG improvements to building an in-house AI agent, please also take a look at ZETTAi's AI agent construction support.
ZETTAi's AI agent construction supporthttps://zettai-ai-agent-site-render.onrender.com/
Sort incorrect answers into four locations
| Observation results | Suspected locations | Things to check |
|---|---|---|
| The original document itself is old or contradictory | Materials | Revision date, official version, applicable target |
| The original is available, but it cannot be searched | Search | Question words, divisions, and candidate sequences |
| Correct part is obtained | Answer generation | Handling of instructions, length, and exception conditions |
| Reasons why it should not be shown | Authority control | User ID, search filter, connection settings |
Authority violations are not averaged together with the normal correct answer rate. If we find even one problem, we will restrict the target data and scope of use until the cause is resolved.
Fix the correct answer set first
We collect the questions that people in charge receive on a daily basis, dividing them into four categories: normal, exceptional, ambiguous, and unanswerable. For each question, attach the document ID of the main points and evidence of the correct answer. By separating the questions you use for improvement and the questions you use for final confirmation, you can easily identify adjustments that are specific to the example at hand.
Evaluation table CSV can record question ID, question, expected main points, basis ID, search pass/fail, answer pass/fail, and question classification. The attached lines are hypothetical examples.
Measure searches and answers separately
For the sake of explanation, let's assume that out of 20 questions, 16 had the correct basis in the search candidates, and 12 of those had correct answers. The rate of obtaining evidence is 16 ÷ 20 = 80%, and the overall correct answer rate is 12 ÷ 20 = 60%. For the questions that were retrieved, it is 12÷16=75%.
In this hypothetical example, there is room to both modify the search and modify the answer after it is retrieved. A small sample of 20 cases alone cannot guarantee the quality of the entire actual operation. Update evaluation targets during production, including important exceptions.
Order of improvement according to cause
If there is a problem with the material, priority will be given to excluding the old version, merging headings and tables, and checking the readability of the image PDF.If the search is missing, consider whether you need an exact match, such as a product code, or closeness in meaning. When changing the search method or number of candidates, make sure that the number of unrelated sentences does not increase.
If an incorrect answer is given even though there is a basis, the applicable conditions and exceptions will be required to be cited, and the design will return the question if something is missing. Simply adding a large amount of evidence may result in a mixture of contradictory material.
Record before and after improvement
Records document versions, models, search settings, question sets, and response times. In addition to pass rates, we also compare delays, usage fees, and human revision times. Even if the correct answer rate increases, if you have to wait a long time for a response, it may be difficult to use for that task.
Automatic scoring will be used to assist with sorting, and the person in charge will check any questions with serious errors or answers with mixed scores. Avoid the cycle of "it's right because the AI says it's right".
FAQ
Will the accuracy increase if I make the chunks smaller?
It doesn't necessarily go up. Conditions and tables may be split. See if the retrieved text actually contains information that answers the question.
Can I stop evaluating after improvements have been made?
Results will vary as materials, models, and settings change. We will continue to re-evaluate when changes are made and record incorrect answers during operation.
Before consulting ZETTAi
Please prepare materials to refer to in your answers and example questions that will help you determine correctness. Regarding ZETTAi's AI agent development and implementation support, you can consult about how to check citations and evaluate responses. The scope of individual support will be confirmed at the time of consultation.
After preparing evaluation questions and materials, you can check how to proceed with the consultation at AI agent construction and implementation support.
AI agent construction and implementation supporthttps://zettai-ai-agent-site-render.onrender.com/
Articles you may also want to read
- How to search internal documents using AI? Document organization and design procedure before RAG introduction
- Why does AI PoC fail? Evaluation and judgment criteria for proceeding to production implementation
Reference materials
Reference date confirmed: October 22, 2026. This document shows examples of evaluation indicators, and the scoring examples in the main text are not actual measured values of ZETTAi.