ChatGPT: Thematic Analysis on Steroids
Bulk verbatim coming through surveys, interviews, app reviews or feedback forms is one of the most labour-intensive data sets to synthesise. If you’re like me and do it on a monthly basis sifting through hundreds of cells or post-it notes over the course of days to distil meaningful themes, insights, or actionable problems can be both time-consuming and exhausting. This work is what Large Language Models (LLMs) like Chat GPT and Google’s Bard excel in.
After a bit of trial and error, I landed on an extremely effective prompt that has helped me power through a tonne of verbatim. It was almost awe-inspiring to watch the AI do in 30 seconds what took me three days to do. Don’t get me wrong, it’s not 100% perfect. Even with its capabilities, it's crucial to exercise caution and be prepared to double-check the AI's work. Trusting AI blindly can lead to errors, so cross-referencing the AI-generated analysis with the original sources is essential to ensure accuracy and identify any potential fake data. I’m sure you’ve heard about the New York-based lawyer who was fined for using ChatGPT for research for a personal injury case.
The prompt
Imagine you're a Product/UX designer working on [Context about your product]. You've received a list of customer verbatim from [sources].
Please conduct a thematic analysis of the data provided. Identify the main themes or insights from the verbatim, and list out the verbatim that supports each theme.
The thematic analysis should be conducted using an inductive approach. The themes should be identified from the data itself, without imposing any preconceived notions on the data.
[paste list items]
How to use it
Check with Your Employer: Before incorporating Generative AI into your workflow, ensure that your workplace permits its usage. Some companies may have policies in place regarding the use of AI tools for data analysis.
Clean Data and Protect Sensitive Information: Generative AI platforms might be moderated by humans, so it's essential to remove any references to your company from the data. If certain information is protected by NDA, it's best to avoid using these platforms altogether.
Fill in the gaps in the Prompt:
Context about your product - this could be something as simple as “a new social media platform” or something as complex as “a feature within a loyalty program app that allows users to make payments by scanning a QR code”. In my experience the more context you can give the AI the better the output of themes and insights.
Sources - to be honest, I am not sure why adding this information helped the AI pump out better results, but again the more specific you get the better.
Paste list items - I get all this data from my BA in a CSV format, I spend a bit of time tidying the data in google sheets by using the find all command to hide any information which may connect the data back to your workplace, however, I don’t even bother with spelling errors and grammar it seems to make no difference.
Press send and Be Prepared to Review AI's Work: While AI tools can significantly speed up the process, they might not be 100% perfect.
Cross-Reference AI Output: After obtaining the AI-generated analysis, cross-reference the themes and verbatim data with the original sources to ensure accuracy and identify any potential fake verbatim.
Here is an example of the prompt filled-in:
Imagine you're a Product/UX designer working on a new social media platform. You've received a list of customer verbatim from surveys, interviews, and social media posts.
Please conduct a thematic analysis of the data provided. Identify the main themes or insights from the verbatim, and list out the verbatim that supports each theme.
The thematic analysis should be conducted using an inductive approach. The themes should be identified from the data itself, without imposing any preconceived notions on the data.
Which platform should I use?
For conducting thematic analysis, I've achieved excellent results with both ChatGPT3/4 and Google Bard. If you're already subscribed to ChatGPT4, it's likely your best option. However, I recommend trying both platforms to determine which language model performs better for your specific needs.
The future of Generative AI and LLMs in the workplace
While AI can be extremely useful, there are concerns about trust and transparency, just read through their Open AI and Google’s data collection policies. Understandably, this is why my current employer recently blocked the use of generative AI from use in workflows. But what is the trade-off, analysis is still very manual and time-consuming. McKinsey projects that generative AI could add up to 4.4 Trillion to the economy annually through the automation of routine tasks. After experiencing the efficiency of these products that are very much still in beta I see why. We’re just at the beginning, there are already promising examples of research tech incorporating these technologies while also ensuring security and compliance. Check out Dovetail’s future of AI here.
Conclusion:
Large Language Models (LLMs) offer a promising solution to the tedious task of synthesising verbatim data. The prompt outlined in this article can be a useful starting point for conducting thematic analysis using an inductive approach, without imposing preconceived notions on the data. However, it's important to exercise caution when using LLMs, as they are not perfect, and cross-referencing the AI-generated analysis with the original sources is essential. As with any technology, the ethical implications of AI in the workplace must also be taken into consideration. Overall, LLMs offer great potential for enhancing the efficiency and accuracy of thematic analysis in the industry, and their usage should be explored further with due diligence.