Using AI to Create UX Research Studies

Academic Research

TL;DR

Skills: AI, Experiment Design, Data Analysis, Qualitative Analysis

Top Lessons Learned: This work is still in progress!

  • Tbd :)
  • Tbd :)

Please ignore incomplete sections as this work is still in progress!

Project Background

This project was part of a Directed Research Group (DRG) under the supervision of a UW PhD student with the intention of exploring the intersection of UX research processes and the capabilities of generative AI. We participated in a quasi-experimental study regarding the effect of avatars on group collaboration, then were instructed to create similar research studies and used Gen-AI extensively in the process.

Keep in mind that Gen-AI is moving very quickly and these findings and take-aways are a reflection of December 2023, I'm sure things will be very different not even 6 months later!

Project Overview

Research Question

How does creating custom avatars before a meeting impact creativity, confidence, and productivity during group meetings in the ideation stage of design thinking?
‍

Methods

We gave ChatGPT our research question and background, and asked it to generate a research study based on our restrictions (2 groups of 4 participants) including creating a script and ideation prompts, generating measures, and generating post-ideation session questions.

You can find some of our attempts here and here.
‍
After several rounds of prompting, ChatGPT had difficulty adding to the study it generated without changing aspects or removing parts, so we took from several different outputs to complete our study design.
‍

Study Design

Coming soon!

Analysis and Findings

Qualitative Analysis

Coming soon!
‍

Data Cleaning

Coming soon!

Data Analysis

Coming soon!

Our study has concluded and I am in the process of writing up our process, however I can share the what my group and other groups that created UXR studies with gen-AI discussed we would and would not use gen-AI for in the future.

What I would not use gen-AI for

What I would use gen-AI for

Idea Generation

Tasks or Prompts:

  • Pros: If your study is measuring something else but you need the participants to complete a task, ChatGPT can generate many different ideas.
  • Cons: May generate similar ideas after multiple uses.

Icebreakers

  • Pros: Besides quickly generating long lists of icebreakers, ChatGPT could also refine and specify ice breakers to your tasks and participants.
  • Cons: May generate similar ideas after multiple uses.
Refinement

Study Questions:

  • Pros: ChatGPT can easily refine study questions and be prompted to not prime participants, or change the wording for the audience.
  • Cons: Sometimes ChatGPT questions can be too direct, may take some prompt engineering.

Reporting:

  • Pros: UXR reports may have different audiences - ChatGPT can help take academic language and put it into layman's terms.
  • Cons: Will need to check all the content is there, should mostly pull from ChatGPT rather than taking whole output.
Coding Assistance

Generating R Code:

  • Pros: ChatGPT can quickly generate simple R code so the user doesn’t have to try to find the right stack overflow post. It often generates multiple ways of doing something to give options.
  • Cons: Can generate code that doesn’t work, you have to put in your variables in the right places, may suggest unsupported packages, requires double checking code.

Interpreting R Errors:

  • Pros: R errors are notoriously difficult to read and interpret, ChatGPT can often determine the most common reason an error may occur. It can also find missing commas, parentheses, or other easy-to-miss mistakes.
  • Cons: Can suggest more complicated fixes, or may not fix code, requires double checking code.
Reporting

Creating Report Structure:

  • Pros: Speeds up process of generating report, leaves enough to researcher that won’t influence findings and reporting.
  • Cons: May miss important sections, or be too generic.

Generating a Report to Edit

  • Pros: Speeds up research cycle, saves money, it’s quicker to edit and add/remove than generate form scratch.
  • Cons: If researcher relies on these reports they may leave out important findings, sometimes ChatGPT leaves things out or misunderstands what to emphasize.
Stakeholder Communication

Communicating with Stakeholders:

  • Pros: It can be time-consuming to write emails to different stakeholders, particularly when making difficult requests. ChatGPT can help generate these messages.
  • Cons: Can sound overly formal, or overly wordy.

Targeting Your Audience:

  • Pros: A report to the CEO will be different than a report to a designer, ChatGPT can help filter out unnecessary information depending on the audience.
  • Cons: Need to double check to make sure all the information is accurate and nothing is missing.

Reflections

Coming soon!

LinkedInInstagram