A3. Product A/B Test – Individual Report [CLO 2, 4, 5]

/ Weighting: 40% / Length: Max 2,000 words

Description

This assessment provides you with the opportunity to undertake product A/B test. Before launching new products, companies test their new products many times to optimize their new product features. In the previous group assignment, your group developed the 1^st MVP, your chatbot A. You will improve chatbot A by adding a new feature. This will be your Chatbot B. You will then test both Chatbot A and Chatbot B, with your classmates evaluating these two versions, and then decide which version is better. Finally, you will report the result of this product's A/B test.

Details

1. Benchmark other groups’ 1^st MVP

This task requires you to benchmark another group's 1st MVP to acquire new ideas. All slides and presentation recording are available in the OneDrive submission folder. Choose any group that you want to benchmark and answer the following questions in your report.

(1) What are the group’s tutorial session and group number?

(2) What is their project goal (Hills statement) regarding Who, What, and Wow?

(3) What are the TWO most important pain points that they found? Do you think that they are the biggest pain points? And why or why not?

(4) What are the TWO most important “need statements” that they found? Do you think they made the “need statements” properly using 5 whys?  And why or why not?

(5) What are their TWO most important solutions which they came up with?  Do you think that their solutions address their customer problems? And why or why not?

(6) What are their 3 chatbot scenarios? Do you think that their 3 chatbot scenarios reflect their solutions well? And why or why not?

(7) Among the 3 chatbot scenarios, which scenario would you give the highest accuracy score? Why?

(8) Among the 5 overall chatbot performance measures (Accuracy, Efficiency, Knowledge, Satisfaction, Willingness to Buy), which performance would you give the highest (or lowest) score? Why?

(9) In their chatbot, what would you like to benchmark the most? It could be a particular scenario, chatbot feature (e.g., slot, option), sentence style, or words (e.g., emotional words).

2. Hypotheses (Hs) for product A/B test

To complete Task 2, carefully read and follow the below guidelines before answering all questions in Sections 2.1 - 2.4.

Guidelines:

A. The requirement of your Hypothesis (H)

(1) Specific H: you need to make specific H, rather than exploration type of questions. Below is bad and good H.

a. Bad: Which chatbot’s communication style. is more effective? (It is a research question rather than H)

b. Good: Chatbot with a passionate tone increases user’s conversation satisfaction compared to chatbot without a passionate tone.

(2) Testable H: To test your H, you need to be able to measure the outcome. Thus, make sure whether the necessary outcome can be measured. For example,

a. Measurable: user’s communication satisfaction (via a survey)

b. Unmeasurable: chatbot’s communication satisfaction

(3) Important H: The company offers new products to solve customer problems. Come up with H which is relevant to either the pain points, solutions, or chatbot scenarios which your group figured out. Or you could identify other pain points and solutions.

(4) New H: “New” products need to be perceived as “new” ones. Try to make an innovative and creative product feature.

(5) Different H within a group: Each member within your group needs to come up with a different H. If you launch a similar product with your competitors or your own company, your new product might not attract many consumers.

(6) The hypothesis needs to have both (independent variable - new feature), Y (outcome variable). For example,

- H: A chatbot with an energetic tone (X) increases the user’s communication satisfaction (Y) compared to a chatbot without an energetic tone. Then,

§ Chatbot A: your 1^st MVP without energetic tone (X)

§ Chatbot B: your 1^st MVP with energetic tone (X)

- H: A chatbot that uses intelligent words (X) more easily persuades consumers to buy the product which the chatbot recommend (Y) compared to chatbot that does not use intelligent words.

§ Chatbot A: your 1^st MVP that does not use intelligent words (X)

§ Chatbot B: your 1^st MVP that uses intelligent words (X)

B. Potential X variables: Considering your customer’s problems and the requirement of H, you can simply change

● Conversational words, flow, style, tone, sequence, or the number of questions.

● Conversation styles (e.g., chatbots talk more positively or politely. Chatbots use emoticons frequently.)

● The sequence of conversation (e.g., when to mention “sell product”? early or later?)

C. Potential Y6 variable

● Considering your X, make a relevant Y6 in addition to the given Y1 to Y5 below.

D. A potential source of your H

(1) Customer’s pain points: Considering the customer’s pain points, your group has already made a product roadmap. You could make a hypothesis about your 2^nd MVP or Future Product.

(2) Market players: A company often benchmark other players in a market. You observed your competitors’ chatbots and other groups’ chatbots. You could benchmark them.

(3) Industry articles: Newspapers, blogs, industry reports

a. You can easily find many potential popular chatbot features by reading industry articles.

b. https://sproutsocial.com/insights/chatbot-marketing/

c. https://www.intercom.com/blog/chatbot-marketing/

d. https://www.revechat.com/blog/chatbot-marketing/

e. https://sproutsocial.com/insights/chatbot-marketing-examples/

f. https://www.hubspot.com/stories/chatbot-marketing-future

g. https://www.wordstream.com/blog/ws/2017/10/04/chatbots

h. https://www.artificial-solutions.com/chatbots

i. https://mobilemonkey.com/blog/chatbot-marketing/

j. https://sendpulse.com/support/glossary/chatbot-marketing

k. https://towardsdatascience.com/how-conversational-chatbots-marketing-is-the-future-of-ecommerce-6743268caa11

(4) Academic papers

a. Many academic papers have also tested chatbot features. You can benchmark those features.

b. Chatbot paper summary: Papers on Chatbot.xlsx, paper pdf

c.  Google Scholar (search with “chatbot” word): https://scholar.google.com.au/

d. https://medium.com/@ODSC/top-10-ai-chatbot-research-papers-from-axxiv-org-in-2019-1982dddabdb4

e. https://www.topbots.com/most-important-conversational-ai-research/

f. http://www.academia.edu/Documents/in/Chatbot

g. https://paperswithcode.com/task/chatbot

h. https://www.chatbots.org/papers/

i. https://link.springer.com/article/10.1007/s12525-020-00414-7

Answer the following questions.

2.1. Write down your Hypotheses (Hs).

You have been using 5 Ys for the market test of your 1^st MVP. For this product A/B test, your task is to make one X and Y6 in addition to the 5 Ys, which are already given below. Consider the requirements of H. Note that you use the same X across 6 Hs. Then, test the following 6 Hs.

H1: X increases the user’s perception of the chatbot’s accuracy (Y1).

H2: X increases the user’s perception of the chatbot’s efficiency (or speed) (Y2).

H3: X increases the user’s perception of the chatbot’s knowledge (or helpfulness) (Y3).

H4: X increases the user’s communication satisfaction (Y4).

H5: X increases the user’s willingness to buy the product which the chatbot recommends (Y5).

H6: X increases the user’s … (Y6).

2.2. State the source of your H6.

State the source of your H6. In other words, where did you get the idea of your H6? The example sources are competitors’ chatbot websites, another group’ tutorial session and group number, and references of industry or academic articles. If you make your H for yourself without referencing other sources, you don’t have to cite the source. You can just mention that it is your original idea.

2.3. Explain your H6 logically.

Explain why you are expecting that your X increases your Y6 in H6. Support your reasoning with some references.

2.4. Explain the importance of your H in solving your customer problem.

How does any of your H (H1 to H6) contribute to solving your customer problem? You can take any H among the above 6 Hs. Then, explain the relevance of your H to either the pain points, solutions, or chatbot scenarios that your group figured out. Or you could identify other pain points and solutions.

3. Conversational Flow and Chatbot

Carefully read and follow the below guidelines and complete task required.

Guidelines:

● The baseline chatbot is your group’s 1^st MVP. This is chatbot A.

● Then, export the JSON file of the 1^st MVP (your chatbot A). Then, you can import the JSON file of chatbot A to the Watson Assistant and then revise it. This will be your chatbot B.

● Note that a new feature for your chatbot B does not have to be complex. Depending on your H, you could change just some words. For example, you can add more polite sentences if your H is about a polite chatbot.

Complete the following tasks.

● Put screenshots of (1) conversation flow and (2) chatbot communication for both chatbot A and B. Please put only the key parts to show the difference between chatbot A and B.

● Then, describe the difference between chatbot A and B.

4. A/B Test Survey

Guidelines:

4.1. Online survey form.

● Create a link to the interface of your chatbot and add it to your survey form.

● Chatbot A/B test survey

o You can modify the survey form. of your 1^st MVP Market Test.

o How to make Google Form. https://www.youtube.com/watch?v=BtoOHhA3aPQ

o Submit your survey link before Tutorial.

o Classmates will answer your survey questions during the tutorial.

4.2. Outcome measures

Make relevant survey questions to test your Hs. You can modify the survey template for the 1^st MVP.

● Example of satisfaction question

▪ (Quantitative) How much are you satisfied with the conversation with a chatbot?

● Not satisfied at all (1 to 7) Highly satisfied

▪ (Qualitative) Tell me about your good or bad experience with the chatbot

● Example of a knowledgeable chatbot question

▪ (Quantitative) How was the agent's knowledge?

● Not knowledgeable (1 to 7) Very knowledgeable

▪ (Qualitative) Tell me about your impression of the chatbot’s knowledge.

Complete the following tasks.

● State all your survey questions here. Since the questions for chatbot A and B are the same, put only one version. Also put the links for your chatbots A and B, and your survey.

5. Analysis

5.1. Demographics

● Summarize the demographics (gender, ….)  of survey participants briefly.

5.2. Quantitative questions

● First, plot the survey participant’s answers by comparing the answers from two chatbots. You could use two box plots or other types of plots.

● Survey participants will talk with the two chatbots within the survey. Therefore, conduct the paired t-test.

● Interpret properly whether your 6 Hs are supported or not from their p-values. Due to the small sample size, it is not easy to get significant results. Although insignificant, there is no penalty.

5.3. Qualitative questions

· Quote some comments.

o E.g., “I like the chatbot because it gives me relevant career option”

· Visualize the answers using the Word Cloud.

· Compare the sentiment of the answers about the two chatbots. You could use a Python library. Or you can manually count the number of “positive”, “neutral”, and “negative” comments written by survey participants for each chatbot. Then, you can compare the sentiment of the two chatbots. You don’t need to do the paired t-test here.

6. Conclusion

● What are the key results? Does the result support your Hs? If not, explain (1) why your Hs are not supported and (2) what you would do differently so that your Hs are supported.

● Based on this result, what do you want to suggest for your company chatbot?

● What other Hs do you want to test in the future?

In completing this assignment, apply appropriate data analytics and consider the concepts introduced in class. Your report should not exceed the word limit, excluding the title page, relevant images, tables, charts, or reference.

Title page (1 page) includes (1) the title of your report, (2) Word count, (3) An executive summary (One paragraph) of sections 2, 3, 5, and 6, (4) Course name, tutorial session and group, and a tutor’s name, (5) Your first and last name & zID.

Reference: Cite academic papers, newspaper articles, blogs, or industry reports properly. Use APA (American Psychological Association) style. in-text citations and a reference list at the end. https://student.unsw.edu.au/apa

Format: Use Word file (.doc), 12pt, 1.5 lines spacing, at least 2.5cm margins on all sides. Make sentences rather than bullet points.

Appendixes (no page requirement): Do not put irrelevant or unedited raw results.

Submission instructions

Submit your report to Turnitin via Moodle.

- .doc contains your report. File name: Tutorial_Group_First and Last Name_A3.doc” (e.g., W12_1_Junbum_Kwon_zXXXX_A3.doc)

Submit other supporting files (conversation flow, chatbot, data, code, and papers) to Moodle submission folder.

1) .pdf has a conversation flow for the two chatbots A and B.

2) .JSON contains chatbots A and B.

3) .csv contains the dataset from your survey.

4) .ipynb contains all relevant Python code to get the results in your report.

5) .xlsx contains all the cited paper lists with a brief note about why you cited them.

6)  .zip contains all the cited papers. Submit a zip file of all the cited pdfs.

● For each missing file among the above (1) to (6), a -1 mark

● Late penalty: - 5% marks per day for the survey link (Week 10 tutorial) and your report, respectively.