How to analyze free text in a survey using AI? Classification, aggregation, and confirmation procedures
Conclusion: In survey analysis using AI, the answer ID is left, the free text is classified, the denominator of the aggregation is specified, and then a person checks the original text and the numbers. If you draw conclusions based only on the AI summary, you may miss answers on multiple themes or a few important opinions.
What are the steps to analyze free text in a survey using AI?
| Procedure | Task | Things to check |
|---|---|---|
| 1. Decide on the purpose | Narrow down to one task you want to improve and one question you want to know | Don't mix things up that you can't judge from the answers |
| 2. Organize the CSV | Align answer IDs, questions, and free text for each line and answer | Check for blank spaces, duplicates, personal information, and garbled characters |
| 3. Define the classification | Humans read some answers and decide the boundaries of the classification | Leave "Other/Unknown" |
| 4. Temporarily classify using AI | For multiple themes, keep the answer ID and put them on separate lines | Leave the original text of the basis and the confirmation flag |
| 5. Verify the calculations manually | Compare the number, percentage, and minority opinions with the original data | Attach aggregation conditions and revision history to the report |
Before sending survey responses to AI, please check Q&A regarding input to AI generated by the Personal Information Protection Commission and confirm the purpose of use and data handling on the service side with the person in charge within the company. We will also review whether it is necessary to leave customer names and contact information as is in the CSV.
Q&A regarding input to AI generated by the Personal Information Protection Commissionhttps://www.ppc.go.jp/all_fad_index/fad2-3-3-q1/
How to create a classification table?
When improving services, consider categories that lead to action, such as price, operation, explanation, and response. First, the person in charge reads some of the responses and defines which expressions should be placed in which categories. If only the classification name is used, the judgments of AI and humans tend to deviate.
| Classification | Descriptions to include | Descriptions to not include |
|---|---|---|
| Ease of use | Screens and procedures are difficult to use | Dissatisfaction with only explanatory materials |
| Explanation | Guides and teaching materials are difficult to understand | Request that the function itself does not exist |
| Price | Opinions about price and fee structure | Dissatisfaction with response speed |
| Other/Unknown | Insufficient materials for judgment | Do not force assignment to existing classifications |
If you change the classification, the previously classified answers will also be reviewed using the same criteria.
How do you count answers for multiple themes?
``The screen is easy to understand, but the price is high'' includes a positive view of the operability and a negative view of the price. Information is lost when you combine it into one emotion. You can use a method that retains the answer ID and leaves the original text of the label and rationale for each theme.
It is possible to add multiple labels to 10 answers, resulting in a total of 14 labels. For example, if there are 6 responses regarding operability, 4 responses regarding price, and 4 responses regarding explanation, the percentage using the 10 responses as the denominator is 60%, 40%, 40%, for a total of 140%. Since this is the result of one person touching on multiple themes, the total should not be adjusted to 100%. Explanation of pie charts from the Statistics Bureau of the Ministry of Internal Affairs and Communications also explains that pie charts should not be used when the total of multiple answers exceeds 100%.
Explanation of pie charts from the Statistics Bureau of the Ministry of Internal Affairs and Communicationshttps://www.stat.go.jp/naruhodo/4_graph/shokyu/pie-chart.html
| Indicator | Value of this example | How to write it when reporting |
|---|---|---|
| Number of free text responses | 10 | Denominator. Do not include non-responses |
| Total number of labels added | 14 items | Total number of items with multiple labels allowed |
| References to operability | 6/10 = 60% | "60% of responses mentioned operability" |
If you want to calculate another ratio of ``6 out of 14 labels have operability,'' write the denominator as 14. Percentage of respondents and label mix are separate metrics.
What should be done before passing CSV to AI?
CSV saves the original file and creates a copy for processing. Based on each answer per line, check for blank spaces, the same answer ID, cells containing line breaks, and model-dependent characters. Even after expanding to multiple lines after classification by AI, the number of answers can be recalculated using the original answer ID. Excel maintenance and calculations are also summarized in Steps to advance Excel data analysis using AI.
Whether you use a free AI tool or a paid tool, the usage plan alone does not determine whether you can enter personal information or confidential information. Use the minimum amount of data necessary in a company-approved environment. When in doubt, check Information that should not be input to generation AI.
How to request classification to AI?
Please organize each answer according to the classification definition.
Please output as Answer ID/Classification/Affirmative/Negative/Neutral/Original text of basis/Confirmation required.
Please separate multiple themes into multiple lines and maintain answer IDs.
Please do not infer reasons or attributes not mentioned in the answer.
If you cannot determine, please mark it as "Unknown" and indicate the reason.
The instructions above are a starting point for classification. Just by erasing proper nouns does not necessarily mean that it is no longer possible to infer the individual. Please check your company policy and approved environment before sending live data.
Where should I check the AI classification results?
First, check for unknowns, multiple themes, strong negatives, and a few serious opinions. Next, we extract some of the common answers and check whether the classification is stable. Even if there are no problems with the classification of the extracted answers, we cannot guarantee that all answers are correct, so decide the scope of confirmation depending on the purpose.
You can leave the answer ID, original text, classification, emotion, rationale, confirmation required, and corrections by people in CSV for classification confirmation. The example lines are hypothetical answers. Even if the same answer ID appears in multiple lines, it will be counted as one when calculating the number of answers. Digital Agency's Notes on Using Generated AI also requires final human confirmation of AI output.
Digital Agency's Notes on Using Generated AIhttps://www.digital.go.jp/policies/genai/agentchat-intermediate
How to report the results of survey analysis?
The opinions of respondents may not be the same as those of customers as a whole. Explanation of the Survey Plan by the Statistics Bureau of the Ministry of Internal Affairs and Communications also explains that bias can occur depending on how survey subjects are selected. The response period, request target, number of responses, and handling of unanswered responses are included, and the trends found by AI are compared with inquiry records as a hypothesis for improvement.
Explanation of the Survey Plan by the Statistics Bureau of the Ministry of Internal Affairs and Communicationshttps://www.stat.go.jp/naruhodo/12_ppdac/plan/plan1.html
The report text can be written as ``A request was made to ○ people, ○ people responded, of which ○ free descriptions were made.Operability was mentioned in ○ comments/free descriptions in ○ cases.Multiple labels allowed'' will allow the reader to follow the population and counting method. Fill in your numbers with real data, don't supplement them with AI guesses.
FAQ
Is the positive/negative ratio enough?
It becomes difficult to know which specific points to improve. Leave the theme and rationale together.
Can minority opinions be excluded?
Don't exclude just the number of cases. We will individually check for safety, major defects, unusable conditions, etc., even if it is only a small number.
Can I use free AI to analyze surveys?
You can try the procedure with public information or fictitious data. When entering actual answers, whether free or paid, please check personal information, customer contracts, usage plan data handling, and internal approvals.
Before consulting ZETTAi
Please organize the questions, answer data, how to handle free text, and the purpose of using the aggregated results. Regarding ZETTAi's AI system development/contracted development, we can consult with you about data confirmation and aggregation methods before analysis. The scope of individual support will be confirmed at the time of consultation.
Articles you may also want to read
- How to advance Excel data analysis with AI? Maintenance and verification procedures before aggregation
- What information should not be input into the generation AI? Business data judgment criteria and actions
Reference materials
The following are published materials from public institutions that are referenced in the text. Enter the document name and link destination separately.
- Personal data input judgment: Personal Information Protection Commission "Q&A when inputting personal data to generated AI" https://www.ppc.go.jp/all_fad_index/fad2-3-3-q1/
- How to tally multiple answers and select a figure: Ministry of Internal Affairs and Communications Statistics Bureau "Handling of pie charts and multiple answers" https://www.stat.go.jp/naruhodo/4_graph/shokyu/pie-chart.html
- Confirmation of survey target and bias: Ministry of Internal Affairs and Communications Statistics Bureau "Creation of survey summary" https://www.stat.go.jp/naruhodo/12_ppdac/plan/plan1.html
- Reasons for humans to check AI output: Digital Agency "Notes on checking the output of generated AI" https://www.digital.go.jp/policies/genai/agentchat-intermediate
Reference date confirmed: November 14, 2026. The aggregation examples are fictitious values for illustrative purposes.