How to analyze open-ended survey responses

Find the product problems worth investigating, and keep the original answers behind every finding. This guide shows product and UX teams how to group written survey responses, check the counts, and choose a useful next step.

Start with one survey question and one decision. For example: which part of importing a first file should the team investigate? You can follow the workflow in a spreadsheet.

A worked example: where customers got stuck

Suppose you asked: “What, if anything, made importing your first file difficult?” The eight fictional responses below each represent a different customer.

The Assigned labels column shows the result of steps 2 and 3 below. R02 describes two obstacles, so it receives two labels. R04 records difficulty finding the button and a positive experience afterward. Keeping both parts preserves the mixed answer.

1. Keep each answer attached to its question

Export the responses and preserve an untouched copy. In your working sheet, keep the question wording, a response ID, the original answer, your labels, and a column for notes. Add only the context needed for the decision, such as setup stage or customer role.

Keep different questions separate while coding. “Nothing” answers a question about difficulties differently from a question about what worked well. If you later combine questions, retain their IDs and check whether the same person appears more than once.

Identical wording alone does not prove a duplicate. Remove confirmed duplicate records using a rule you can explain, and record what you excluded. A response ID may identify a submission rather than a person; when you cannot identify unique respondents, report answers or submissions instead of people.

Work in a tool your company has approved for customer data. Restrict access to the original responses and remove unnecessary identifiers from the working copy. Check the answer text too: people may type names, email addresses or account details. Redact identifying details before quoting answers or sending text to an AI service, and check that the permitted use of the responses covers that sharing.

2. Define labels that preserve the meaning

If the dataset is small, read it all before deciding on labels. For a large collection, start with a random selection, then add examples from different stages of progress, short replies, mixed experiences and unusual cases. Use this set to develop the codebook; apply the checked definitions to the remaining answers in step 3.

A code is a short label for an idea in an answer. Write down what qualifies for each label and what does not. This list of definitions is your codebook. For the fictional example:

• File requirements: uncertainty about the file’s required columns or format. Exclude trouble finding the Import control.
• Import location: trouble locating the Import control, including an answer that describes finding it after difficulty.
• Permissions: needing access or an administrator’s permission to import.
• Positive experience: an explicitly favorable account of part or all of the task. It can appear alongside a difficulty.
• No difficulty: an explicit statement that nothing made the task difficult. Do not combine it with a difficulty label. If an answer says “nothing” but then describes an obstacle, label the obstacle and note the contradiction.
• Unclear difficulty: a reported problem with too little detail to identify the obstacle.

Track Blank and Don’t know separately. Neither establishes that the customer had a smooth experience. Also keep separate statuses for refusals, not-applicable answers and irrelevant text. Use these exclusion statuses only when the whole answer gives no relevant account of the task. If “don’t know” is followed by a relevant description, code that description.

Apply the labels supported by the answer using these combination rules. R06 stays Unclear difficulty; it gives you no basis to decide whether the problem was permissions or file requirements. Flag unfamiliar or unresolved conflicting answers for review before counting.

For an example of explicit coding instructions, Pew Research Center’s published codebook explains how researchers handle categories and non-substantive responses.

3. Check the labels before you count

Try the codebook on a small set of varied answers. Where possible, have a colleague label the same answers without seeing your choices. Discuss disagreements using the original wording and revise definitions that led to different interpretations.

Then apply the definitions to every answer in scope. Review flagged cases, mixed answers, and replies that do not fit. If you add, split, or redefine a label, revisit earlier answers that could be affected. Record the final definitions and the decisions behind exclusions.

If you work alone, take a second pass through those cases and a random selection of the rest. Pew’s coding methodology shows how its researchers tested and refined a codebook through repeated review.

4. Count answers with a clear denominator

For each label, count an answer once even if it repeats the same complaint. An answer can still count toward several compatible labels.

For this example, count R01–R06: each reports a difficulty or explicitly says nothing made the task difficult. R06 counts because it reports difficulty, even though it does not name the obstacle. Use these six responses as the counting base. Keep blank, Don’t know, refused, not-applicable and irrelevant answers outside this base. Resolve flagged conflicts before counting; report any still unresolved outside the base. Report the count for each exclusion status separately.

• File requirements: 2 of 6 responses, R01 and R02.
• Import location: 2 of 6 responses, R02 and R04.
• Permissions: 1 of 6 responses, R03.
• No difficulty: 1 of 6 responses, R05.
• Unclear difficulty: 1 of 6 responses, R06.
• Positive experience: 1 of 6 responses, R04, alongside a difficulty.

Outside the base: 1 blank (R07) and 1 Don’t know (R08). Refused, not applicable, irrelevant and unresolved: 0 each. The six included responses and two excluded records account for all eight example records.

Define a broader Import clarity group as any response labeled File requirements or Import location. It contains three responses: R01, R02, and R04. Adding the two label counts would count R02 twice.

To count in a spreadsheet, keep one row per answer and give each label its own column. After coding, enter 1 where a label applies and 0 where it does not. Add an In base column with 1 for included answers and 0 for excluded answers.

For the eight example records in rows 2–9, put In base in column C, File requirements in D and Import location in E. =COUNTIF(C2:C9,1) gives the base of 6; =SUMIF(C2:C9,1,D2:D9) gives the File requirements count of 2. SUMIF applies the inclusion rule even when excluded rows remain in the sheet.

For the combined Import clarity group, use column J. Enter =IF(OR(D2=1,E2=1),1,0) in J2 and fill it down through J9. Then =SUMIF(C2:C9,1,J2:J9) returns 3, counting each qualifying answer once.

For percentages, divide the number of replies carrying the label by the number of replies in your stated base, then multiply by 100. Labels can overlap, so their percentages can total more than 100%. With six replies, counts such as “2 of 6” keep the small base visible.

State your inclusion rule beside the results. Apply the chosen rule consistently.

These counts describe what this group wrote. They do not establish how common the problem is among all customers. Pew’s research on open-question nonresponse shows why checking who answers, and who skips the question, matters.

5. Compare groups that could change your decision

A permissions issue may call for different help for an account administrator and a teammate. Compare those groups only if the survey or reliable records identify the roles.

Use the same definitions and counting rules for each group. Show its own base beside its count. Keep unknown roles visible as a separate group; do not infer a role from the complaint. With small groups, use apparent differences to choose follow-up questions; they may not support a reliable comparison.

6. Write a finding your team can investigate

A topic label such as Import clarity names a group of answers. A useful finding explains what the evidence suggests and what the team should check next. Keep those interpretations separate from the customer’s words, as the GOV.UK user-research guidance recommends.

Examine replies that challenge your first interpretation. R04 suggests the import process may work well once the control is found. That is a reason to inspect discoverability before assuming the entire import flow needs a redesign.

For the fictional example, a short handoff could read:

• Finding: Three of the six responses in our counting base describe trouble locating Import or understanding file requirements.
• Evidence: R01 and R02 mention required columns. R02 and R04 mention locating Import; R04 says the import itself was easy afterward.
• Limit: These are a few self-reported experiences. We have not observed the attempts or established how widespread the obstacles are.
• Next action: Watch customers locate Import and prepare a file. Ask what they expected to see where they hesitated. Then test a change that addresses that obstacle, such as a clearer control label or a sample CSV.

Assign an owner to the investigation and record what evidence you need before choosing a change. Keep severity alongside frequency: a single report of lost work can deserve attention even when another complaint is more common.

Select quotations that explain the finding, including meaningful exceptions. Preserve their meaning and mark omissions or redactions.

If you use AI, check the labels and the evidence

Give the tool the original question, your label definitions, and the counting rules. Ask it to return a response ID and supporting excerpt for each assigned label, and to flag uncertain cases.

Compare its classifications with a set you reviewed yourself. Include rare labels, mixed answers, and ambiguous wording as well as a random selection. Check for missing ideas and unsupported labels.

Fix the rules where needed and recheck affected answers. Calculate totals from the final, corrected label assignments across all answers in scope, using the stated counting base. Count the assignments directly rather than using a written summary from the tool. Inspect every quotation used in the final handoff. The AI coding section of a 2026 Pew methodology describes validation against human-coded responses, including rare categories.

Explore analysis for feedback collected in Validata

Validata’s survey analysis groups written answers into themes and connects findings to supporting responses. Use these patterns to choose what your team should investigate next.

Survey types in Validata

Validata’s 13 survey types:

• General Analysis — for surveys that don’t fit a named type
• Employee Engagement
• CSAT (customer satisfaction)
• NPS (Net Promoter Score)
• Product Feedback
• Onboarding and Activation
• Exit and Churn
• Market Research
• Event Feedback
• Lead Qualification
• Academic Research
• Form / Application Intake
• Healthcare

Join Validata to explore our analysis engine and uncover hidden patterns in your survey data.

If your replies are too vague to explain a problem, improve the next question before collecting more. Our customer onboarding survey questions include prompts for setup obstacles and relevant follow-ups. Use Validata’s survey builder to put your questions into a survey.

Create your feedback survey for free.