Study on ChatGPT by ChatGPT: The survey design experience and results analysis

AI has turned into an arms race in the world of big tech and right now, the app of the moment is undoubtedly ChatGPT – the conversational AI interface built on the GPT-3 large language model. Since it was made available in late November, ChatGPT has triggered a wave of headlines and stories, more so than anything that has come before it. ChatGPT can produce just about any kind of text or written output to a quality that, very nearly, appears as if it was written by a human. ChatGPT has prompted many discussions about its relevance at work, which of course, leads to the question of whether it’s going to put humans out of work. The World Economic Forum, however, has stated that while some jobs may become redundant, people won’t and instead, we should learn to work alongside AI. 

Could AI offer anything to a survey professional?

As a consumer research company, we simply had to put it to the test - can ChatGPT design a survey for companies? This is what we tried: we gave ChatGPT free rein to design an entire questionnaire, (which came to 19 questions) and have it analyse the results. 

We learned that there is great value in asking the right questions about how to ask the right questions. One of the most impressive features we found was how ChatGPT generated the framework of the study in seconds - the questions were properly structured and aligned with the research objectives. Although the draft impressed us, the final version required some human refinement to ensure that it aligned with the intended use of specific questions and met survey design best practices for data quality. So, here are a few tips that we’d recommend considering when using ChatGPT to assist in survey research:

• Take the time to come up with great questions - start broadly but include specific issues relevant to the question you want, then refine and narrow your follow-up questions and prompts. And you can always reference and cross check the best survey design tips here.
• Expect to go back and forth in a conversation.  
• Be mindful of the survey length  - long surveys lead to bad data and do avoid these common biases in your survey design.
• Ask for references when you feel they are needed. (This was a pretty lengthy side conversation that we had with the Bot, but it was able to provide references to support most of its suggestions.)
• And lastly, expect mistakes—do not accept its generated content “as is.” Always review its recommendations, or you can have Milieu’s research experts review it for you.

Results from the study

Let's take a closer look at the insights and data gathered from this latest 4-market study. We conducted this quantitative study via Milieu’s proprietary survey community with N=1,000 respondents in Singapore, Thailand, Malaysia and Indonesia to understand their perceptions and experiences towards ChatGPT. It also uncovers the levels of confidence in the technology and any concerns they may have, as well as whether they plan to use it for search purposes.

The results show that about 4 in 10 are very or somewhat familiar with ChatGPT and have interacted with it, with more than half (52%) rating their overall experience as good. This suggests that AI-powered conversational interfaces are becoming more mainstream, and people are getting comfortable using them. However, there are also concerns about the technology's impact on society and the potential for misuse. In this article, we analyse these findings and explore the implications of the survey results.

70% optimistic about ChatGPT's capability

When asked about the potential change in ChatGPT's capability in the future, 70% responded positively. The results also show that 43% believe the ChatGPT technology can positively impact society, and 38% are confident in its ability to understand and respond to queries, which suggests that there is a perceived value in the technology and its potential applications beyond its convenience and efficiency. 

6 in 10 are likely to use ChatGPT for search

The survey also found that 62% are likely to use ChatGPT for general searching in the future, such as looking for nutritional information on a particular food. This infers that AI-powered language models are increasingly becoming the preferred tool for information-seeking, and it is likely that their usage will continue to rise in the future.

‍

Above half of respondents express concerns or have reservations about ChatGPT’s technology

The top five concerns regarding ChatGPT are: 

• Dependence on instant information leading to a loss of critical and independent thinking (53%)
• Misinformation provided by ChatGPT due to machine-generated content (53%)
• Collection and use of personal data without consent (49%)
• Potential for ChatGPT to be hacked or used for malicious purposes (49%)
• Ethical issues regarding data privacy and intellectual property (42%)

These concerns highlight the need for proper regulation and oversight of AI technologies - the potential for misuse and harm is real. AI models can be vulnerable to attacks and can be used to spread misinformation or engage in malicious activities.

‍

7 in 10 support regulations to protect creators' rights whose content is used by AI models

It is encouraging that 72% of the respondents feel that there should be regulations in place to protect the rights of creators whose content is used by AI models. This shows that there is awareness of the ethical and legal implications of AI and the need for measures to ensure that these issues are addressed.

Lastly, 43% feel that AI-generated content based on content found on the internet can be considered as plagiarism. When it comes to AI-generated content, the lines can become blurry. Does the use of AI technology to generate content count as "copying" someone else's work? Or is it simply a tool that can be used to create new and original content? This is a complex issue, as AI models can generate content that is similar to existing material, leading to questions about the originality and authorship of the content. While AI-generated content has the potential to improve efficiency, it is important to establish clear guidelines and regulations around the use of AI-generated content. This includes proper attribution and credit for original creators, as well as measures to ensure that AI-generated content does not infringe on copyrighted material.

‍

‍