Clever Copy and Cutting Costs: AI in Advertising

Experimental Research - Academic Research

TL;DR

Skills: Experiment Design, Survey Design, Data Cleaning, Data Analysis, Qualitative Analysis, Literature Review

Top Lessons Learned: 

  • This was my first time conducting experimental research, so it was a great learning experience to run an experiment from start to finish. I was able to dive in deeper into some statistical concepts I haven't had a chance to explore yet.  
  • This project was also very informative when it comes to looking at how people view AI. AI is such a hot topic right now, and given how many different industries are looking to integrate AI into their products, having a foundational understanding of the variation of attitudes towards AI will be very useful.

Project Overview

Research Question

Does disclosing AI involvement in generating an advertisement on the Instagram feed have an effect on the perceived quality of the advertised product and the likelihood to purchase the product?
‍

Literature Review

We began by conducting a literature review of all academic work related to AI in advertising, particularly looking at purchase intent and perceived product quality. At this point, we realized there is not much literature out there that was directly related to this area.
‍

Hypotheses:

Following our literature review, our hypotheses were as follows:

  • H1: Disclosure of AI-generated content in an advertisement will have a negative effect on the perceived quality of the advertised product.
  • H2: Disclosure of AI-generated content in an advertisement will have a negative effect on the intent to purchase.
  • H3: Disclosure of AI-generated content in an advertisement will have no effect on the perceived product quality or intent to purchase when participants report high trust in AI.

Experiment Design

Our study was a between-subjects design with 2 conditions - the control group which had no disclosure of the use of AI, and the experimental group which had a disclosure. We conducted our experiment on Qualtrics, where participants would either receive version A (experimental) or version B (control) of our survey.

My teammate Marianna used AdGen AI to generate the advertisement we used for our stimuli. The ad featured a lotion from the brand Aveeno, which was chosen as a fairly neutral yet well-known brand. Marianna also designed the disclosure in Figma, which featured banners similar to current disclosures on Instagram of sponsored posts and ads.

Participants were recruited through word-of-mouth, Reddit, Facebook, Slack, and Discord, and were asked to complete a survey on advertising in social media on Qualtrics.

Project Overview

Survey Design

We had several main measures in our experiment:

  • Perceived Product Quality: 7-point Likert-type scale from Far Below Average (1) to Far Above Average (7)
  • Purchase Intent: 7-point Likert-type scale that ranged from Strongly Disagree (1) to Strongly Agree (7)
  • Attitudes Toward AI: 7-point Likert-type scale that ranged from Strongly Disagree (1) to Strongly Agree (7)
Data Cleaning

Before beginning to analyze our data I had to prepare it:

  • I profiled the data in excel and then used RStudio to clean the data.
  • I cleaned the columns by eliminating spaces and filtering out columns that were unnecessary for analysis (such as location, start date, etc). I also dropped unfinished responses, and anyone who completed the survey in less than 2 minutes.
  • I converted all variables to numeric and created composite variables of our survey measures.
  • I coalesced the data for our DVs into new columns in order to run our T-tests.
Quantitative Data Analysis

Following data cleaning, I began the analysis:

  • We ran descriptive statistics of all our measures and created histograms to visually examine our data.
  • Perceived Product Quality: Though the mean of the experimental group (M = 4.33, sd = 1.16) was lower than the control group (M = 4.38, sd = 1.25), the results of our independent samples T-test did not reveal a significant difference between the two groups, t(81) =  0.12, p = 0.90. Therefore, Hypothesis 1 is unsupported.
  • Purchase Intent: Though the mean of the experimental group (M = 3.08, sd =1.55) was lower than the control group (M =3.61,  sd = 1.34), the results of our independent samples T-test did not reveal a significant difference between the two groups, t(81) = 1.47, p = 0.15. Therefore, Hypothesis 2 is also unsupported.
  • After running linear regressions on our DV's with Attitudes Toward AI as a moderator, we found that there was significant interaction between the DV's and MV. Therefore, Hypothesis 3 was supported.  
Qualitative Data Analysis

At the end of our study, we included one free-response question investigating people's attitudes toward the use of AI in advertising.

I did not participate in the coding, but I did instruct my teammates on how to conduct the coding, as they had no experience with qualitative coding.  

  • We included a question that asked if they believed companies should use AI to create advertisements (yes/no), followed by a free-response question that asked them to describe in more detail why they felt that way.
  • One team member coded all of the free-responses in favor, and another the responses against. They first completed a round of open coding, followed by a round of closed coding.
  • Once all the codes were complete, they each created a codebook that included code names, definitions, and example quotes. We then switched, and used the codebook to conduct our own rounds of coding to check inter-coder reliability.
  • I used R to establish inter-coder reliability. The first round of coding for the responses against the use of AI in advertising had a low inter-rater agreement (κ = 0.44), which led to a discussion regarding the code definitions, after which the codebook was revised and the responses were re-coded (κ = 0.81). The first round of coding for the responses in favor of the use of AI in advertising had a moderately high level of inter-rater reliability (κ = 0.75) so it was unchanged. 

Findings & Discussion

Quantitative findings - attitudes towards AI amplify intent to purchase and perceived product quality

In conclusion, our results did not support a direct effect of disclosure of AI-generated advertisements on perceived product quality and intent to purchase (H1 and H2). However, we did find a moderating effect of attitudes toward AI between the disclosure and perceived product quality and intent to purchase.

In plain terms, people's individual beliefs and preconceptions of AI vary, and this has more of an impact on whether or not they think the product will be of good quality, and whether or not they plan to purchase the product when they see the disclosure. 

The disclosure seems to have prompted participants to think about AI, and their beliefs toward AI had an amplifying effect on how high or low quality they judged the product as well as their intent to purchase. If people had negative attitudes toward AI, they rated the product quality as lower and reported lower purchase intent when they saw the disclosure. However, when those with positive attitudes toward AI saw the disclosure, they were more likely to have an increased view of product quality and increased intent to purchase. 

‍

Qualitative findings - there's a large variety of how people view AI
  • AI Isn't Human: Responses highlight that humans are better than AI at making more creative and personalized advertisements. AI-generated content might end up being generic, lacking humanness, and failing to effectively attract the targeted users.
  • Job Loss: Responses have an opinion that AI seems dishonest, not trustworthy and no one can be held responsible for a task. They also highlight issues with crediting original creators.
  • Ethical Concerns: Responses represent the expressed fear or concern that the adoption of AI in advertising may lead to the loss of jobs and replace humans. 
  • Unrealistic: Responses highlight that AI advertisements look and/or feel fake and may not be up to the standards to believe it entirely. 
  • Efficiency & Ease: Responses focus on how using AI can save time, money, and/or resources. 
  • Confidence in AI: Responses communicate the belief that AI can produce results that are comparable if not better quality than what humans produce. They also posit that AI can be helpful and improve processes and/or outputs in some way.
  • More Personalization: Responses highlight how AI can generate advertising that is more personalized and fit for the target audience, which in turn successfully attracts consumers.
  • Neutral Stance Towards AI: Responses suggest a perception of AI that isn’t particularly positive or negative. Participants aren’t against the use of AI as a tool, assuming it’s used responsibly.

Final Paper

Reflections

Coming soon!

LinkedInInstagram