How to Use Playtesting to Design Conversational Experiences

Conversation Is the Original Human Interface 

We have conversations every day—about work, family, hobbies, you name it. During an in-person conversation, we can tell if someone is confused through their facial expressions, body language, and the questions they ask. We can then course correct and adjust what we say to get our point across. However, we don’t have that luxury in most digital interfaces. 

For an organization, the failure to communicate clearly is a business problem. We lose sales, diminish the customer experience, and erode trust. Digital interfaces are no different. We may intuitively communicate with words in real life, but that same ability doesn’t always translate in our digital products. 

If every design problem is also a communication problem, why don’t we design more positive interactions that start with the conversation we want to have with our audience? 

Designing Conversation-first Builds Empathy 

When you start thinking about the conversation you’d have with a user, you step into their shoes and empathize with their biggest worries, fears, and anxieties. You can then anticipate their questions and answer them in their own words before they even ask. When you translate this into a user interface of some type, you end up with something that feels more intuitive and human. 

This is what I mean by designing conversation-first. Getting the words right makes it easier to get the design right. Words are the lowest-cost, lowest-risk way to design. 

Better yet, words are easier to change than design comps or code. You can act out the conversation—adapting a video game technique from the video game industry called playtesting—to think through the natural flow of questions and information. Playtesting helps you discover your users’ language to describe what they need from the interaction. It allows you to focus on the behaviors that your words can inspire. 

In the following section, I outline a three-stage process for translating a conversation framework into a digital experience. I adapted a version of this technique while designing for intelligent assistants and chatbots, but it works for web and app experiences as well. 

Design Your Conversation Before There’s a Digital Interface 

You don’t need a lot to get started. Grab a partner and open your favorite text editor. Your partner can be a designer, developer, or even your client. Lean towards someone with a completely different perspective! 

At the end of this process, you’ll end up with a content prototype that you can either test or flesh out into a digital interface. It’ll look like a script of question-and-answer responses between a user and your organization. If you’ve ever read Choose Your Own Adventure books, it’ll feel similar.  

Turning Conversation into Digital Experience in Three Stages: 

  1. Write down the conversation you would have with the user. 
  2. Read or act out the conversation with your partner.  
  3. Decide on a part of the conversation to sketch out or test. 

I’ll walk you through each stage in more detail below. 

Round 1: Write the Conversation 

Define the Problem  

What’s the specific challenge or use case you’re working on? You might already have this information written down as a user story or documented as a problem statement. Gather any relevant user research you can easily refer to. 

Define the Context  

Know who’s asking the question and why. Are they likely novice or power users? What is their level of knowledge? What are their biggest worries, fears, and anxieties? 

What situation are they in? Where are they and what environment are they in? What resources do they have ready access to? These factors influence the kind of questions a user might ask. 

Identify Goals 

Write down the intent or question that prompted the user to reach out. What do they want resolved at the end of the day?  What does the organization or company want to achieve through interacting with the user? 

A Hypothetical Example 

You’re a traveler looking to book a last-minute flight. You are price-conscious and want to arrive at your destination as quickly as possible with the fewest number of stops.  

Goals of the Traveler and Airline Company might look like this: 

“Traveler and Airline Company have different goals. Traveler’s goals are to find the cheapest price, get to the destination as quickly as possible, and travel on a certain date. Airline Company’s goals are to collect money, fill the plane, and optimize for profit.”

Let’s capture the conversation between a traveler who’s trying to book a flight with a [airline brand] call center agent.  

One person will play the role of the traveler, the other will play the role of the call center agent. 

Start with Mad Libs 

Write a sentence designed to capture travel details so you can provide a specific set of flights to choose from. 

Hi, my name is [first and last name] and I want to [do something]

Hi, my name is [Caty Herron] and I want to [book a flight to Sydney today]

Explore How to Be Helpful 

Record your responses line by line in your text editor. Ask clarifying questions if you’re not sure what the other person means or if you need more specific responses. 

Caty: I want to book a flight to Sydney today.
Agent: That’s Sydney, Australia?
Caty: Yep.
Agent: Ok. What is your departure date?
Caty: I’d like to leave on June 5th early in the morning.
Agent: And from which airport?
Caty: From Dulles please.


WHAT ELSE DO WE NEED TO KNOW?

Agent: Alright. Is this going to be a one-way trip or round trip?
Caty: One way.

WHAT WOULD YOU SAY TO SUBMIT YOUR REQUEST?

Agent: Got it. Anything else?
Caty: Yeah, can I get a list of the cheapest flights with the earliest departure times?
Agent: Sure thing. Here’s what we have right now.

Guide Toward Smart Decisions to Improve Customer Experience 

How might you make the conversation more contextually relevant? You can surface information the user cares about at the right time during the conversation. 

WHAT DATA WOULD YOU WANT TO SURFACE BASED ON THE USER PROFILE? 

  • Departure Time
  • Number of stops
  • Total Travel time
  • Price

WHAT OTHER DATA COULD THE USER ASK FOR? 

  • Airline company
  • Location of each connection
  • Arrival and departure time for each connection
  • Airport code (IAD, SYD, LAX, etc.)
  • Type of seat (economy, first class, etc.)
  • Number of available tickets
  • Amenities
  • Change fees
  • Checked bag policy

Show Users Flight Options

Agent: We have a flight at 5:30 AM for $3,320. The total travel time is 23 hours and 35 minutes with one stop in Los Angeles.
Caty: Ok. What’s the next cheapest flight?
Agent: The next cheapest flight we have is $3,650. That flight leaves at 6:15 AM but it has two stops, one in Seattle and the other in Tokyo. the total travel time will be 29 hours and 50 minutes.
Caty: Hmm. I think I like the first one better. Can you book that flight for me?

Close the Conversation 

The conversation is done when the user’s original question is answered. In Caty’s case, it’s booking a trip to Sydney, Australia. We skipped the payment process for brevity, but you’ll want to write that part out if you’re designing the entire booking experience. 

HOW SHOULD WE CONFIRM THEIR BOOKING? 

We can write a confirmation message designed specifically to make Caty’s trip more enjoyable.  

“Example of a confirmation message sent to the traveler after she books a flight. The confirmation message includes the destination, travel date and time, departure and arrival airports, and the estimated local time at the time of arrival. It also includes a confirmation code to check in at the airport. To make the traveler’s trip more enjoyable, the confirmation message displays the destination high and low temperatures and a tip to expect a longer security line than usual.”

Sometimes the conversation takes only a few exchanges. Other times, especially if the question is more complex, it takes more turns before you reach the answer. Some conversations are longer, some are shorter, just like in real life. 

You might summarize your conversation flow as: 

  1. Collect destination info
  2. Collect travel info
  3. Show tickets
  4. Process payment
  5. Send confirmation

Round 2: Playtest the Conversation 

Now that you have a conversation framework in place, you can review parts that need more specificity or clarity. Read the conversation script out loud from beginning to end with your partner.  

Highlight Unexpected Questions and Language 

Did any questions surprise you? You’ll want to highlight these scenarios and validate them later. They may be important to users for achieving their goals, and you may need to design for them. You’ll also want to note places that can benefit from more clarity or input. Maybe the user asked a follow up question because they didn’t understand your response. Either provide more relevant information or ask for more input from the user. 

Here are some changes that might make the conversation clearer and more contextually relevant: 

Agent: We have a flight at 5:30 AM for $3,320. The total travel time is 23 hours and 35 minutes with one stop in Los Angeles. The layover is 2 hours and 5 minutes.
Caty: What’s the next cheapest flight?

[Since Caty wants to arrive in Sydney as quickly as possible, it might be helpful to show her the layover time as well]

Revisit Hierarchy and Order 

Are you answering user’s burning questions upfront? Revisit your summarized conversation flow and see if the order of the questions make sense. Playtest the conversation if you need to. That’s what the words are there for! 

You may find that you missed a critical question. Here’s one I missed: 

Agent: Alright. Is this going to be a one-way trip or round trip?
Caty: One way.
Agent: Ok. And how many tickets are you looking to buy?
Caty: One. Just for me.

[We missed a crucial input that the system might need to present more relevant options] 

We can easily add this question into the conversation flow where it makes the most sense. If you’re not sure that it’s a burning question, write it out as an if/then statement. 

If [user wants to know about something], then [we tell them this]

If [Caty wants to know if she can use her travel voucher], then [we say this]

Asking “what if” helps you imagine other potential burning questions. Though, like fielding an unexpected question your partner might have asked, you’ll want to validate it with data before going down the rabbit hole. Consider that some interactions may not be necessary or add significant value. When you start with words, you discover this much sooner. 

Round 3: Prioritize and Prototype 

At this point, you can either test the conversation with users as a content prototype or turn the conversation into a digital prototype. Prioritize which part of the conversation flow you’d want to visualize or validate. 

Test the Content Prototype 

You can playtest with real users to understand how they think and feel. Are the choices presented in the flow realistic? Do users understand those choices? Do they feel confident enough to make a decision? 

Pay particular attention to their word choice and the kind of information they expect to receive. You may also uncover new questions you need to answer and missing steps or information you need to provide. 

Create a Digital Prototype 

This is where you express the information as design elements. Circle and star the nouns and verbs, respectively. Note how you might represent them. This could be a form field, call to action, lists, details page, etc.  

You and your partner might want to sketch separately and then compare ideas. You may even organize a sketch check-in to generate more robust ideas. You can then vote on the best ones to further flesh out. Make sure to capture any new scenarios to account for and any specific language that’s important in the experience. 

“Example of commenting on a conversation script to note how certain information should be represented as design elements on a web-based interface. The conversation script displays the dialog between the traveler and the call center agent explored in the previous paragraphs.”

Playtesting for Everyone 

One of my mentors has a habit of asking me, “We can, but should we?” From my experience, we don’t ask this enough. Or early enough. 

When we first decide what to say, when, how, and to whom—or if anything should be said at all—we leave the door open to ask, “Does this really serve the user? Will this change the outcome?”  

Designing a conversation together builds a shared understanding and ensures everyone is working from the same set of information. It’s a way to keep the whole team involved from beginning to end, not just their “slice of the pie.” You can playtest a conversation at any stage of the project lifecycle. This way, you start small, build trust with your stakeholders, and scale what works for delivering on your client’s outcomes.  

Communication is a team sport. Involve your content strategists, UX designers, and developers! Don’t forget your clients, product owners, and writers, too. Especially your writers—you can use their superpower to drive design. 

How do you use design to connect with audiences and improve customer experience? Let’s chat if you’re ready to get started. 

Get alerted to new job postings, events, and insights by registering for our monthly newsletter.