Interview with a conversation designer at Salesforce

Intro

I’ve always been interested in learning about what conversation design is, what projects conversation designers work on, and what role conversation design plays in a company, especially in the tech industry. Therefore, I reached out to a conversation designer at Salesforce and had a chat with him.

What role does conversation design play at Salesforce?

Right now, one of our main products that is conversational, industry-facing, and generally available at the moment is the Einstein bot builder. Essentially, it’s a chatbot-building tool. It’s like any other chatbot builder that you would see in other areas. The power behind Einstein is that it is connected directly to the Salesforce CRM platform. We can leverage all of the data about customers that is inside of CRM in chat with that customer.

Conversation design plays a part in how we constrain and put guardrails inside of the bot builder to make it easier to do the right thing conversationally. We make sure that folks try to train an intent on a wide variety of spellings, for example, because we know that spelling variation is often linked to sociolinguistic variation among users. We want to make sure that those users have their dialogues and ideologues represented in our models. As we train the intents, we encourage folks to use our conversation repair. So if you’re asking a user a question, we encourage variations among the conversation repair, so that way we can diversify the conversation a little bit to make it more natural as opposed to overly repetitive.

We put in a recommended research-informed bot response delay. We found out that about 1.2 seconds is a pretty strong or solid response delay before a bot takes a response. If the bot responds after zero second have a lapse, it come-acrosses as incredibly artificial. We also give our customers the ability to customize that time if needed.

So a lot of what I have done over the past four years in the Einstein bot space has been to inform that aspect about that bot builder. As of recently, I would say October of last year, that’s when we started putting in actual conversation-designed templates in the Einstein bot builder. At the moment, it’s currently in Beta. 

How do you measure success in the process?

I think it depends on what angle we are looking at because there are several different measures of success that I look at. One of which is as internal awareness around conversation design because it’s so new to Salesforce as a practice. Are we receiving engagement from different product teams who are touching the Einstein bot experience? Are they engaging with our guidelines on our internal website around best practices? Are they coming to our office hours? Do people have a clear understanding of what conversation design is and what we do? Does each product area know who of my team they should go to for a given specialization or solution?

In terms of the product work, I would say that a lot of it has to do with general feedback from the cross-functional stakeholders we work with on a given feature. Are products getting what they need in terms of the conversational experience, and are they getting it in a timely fashion? Are the engineers clear on why it is that we have designed a particular piece a copy or a piece of the flow in the way that we did? Is localization clear on why we phrase something in a particular way? Can we make sure that they understand it when they are translating that to another language for a different market? I’d say those are all of the internal metrics that I use.

What’s the most challenging part of the job?

I think the most challenging part of my job at the moment is scale. So how do we increase our brand awareness as a team? I feel like I’m working inside of a start-up inside of a large company. Something that I keep telling my directs is that I know it feels like we’re a bit of a broken record, but that’s what needs to happen at this point. We need to just repeat ourselves because people don’t really know what to do with conversation design yet because Salesforce’s never had it before. There’s a lot of patience involved. In terms of product challenges, the templates are set as ‘a template’ rather than a bot that we are making for a specific company. The challenge on that front is that you’re designing a conversational experience for an audience you don’t know. Salesforce is customizable from end to end. What if this particular object or record that they have in their system isn’t referred to the name that we’re referring it to in the template?

How do you apply ethical and inclusive design practice in conversation design?

I think what’s really fortunate at Salesforce is that we do have a strong value around things like equality and inclusion. We also have a dedicated office for ethical and humane use. That team of technological ethicists is really helpful to collaborate with in terms of how we can go about solutioning that. My academic background is in sociolinguistics, so for me, language, culture, and identity are tied together. I’m like we’re not designing conversations for 2006; we are designing conversations for the future. Future is inclusive and diverse. A black woman in the southern US doesn’t have to change how she talks to get what she needs out of a chatbot. A lot of that is around how do we go about sourcing a linguistically diverse pool of intent data and utterances in order to train the model, so it can understand various dialogues. Despite our values around equality and inclusion at Salesforce, I’d say it is certainly a tech company. I think it’s a very important thing to consider when we think about how we convey the brand persona in language. Are we doing it in a way that doesn’t sound like we’re appropriating from a particular group? We have to balance that input and output. In terms of training data, I think that’s what we have the most control over. We, as conversation designers, are actually telling people that you need this. We are trying to create standards about how we collect data in a way that is a linguistically diverse set of data.

How can a voice assistance/chatbot overcome gender and racial biases in hiring / in general?

One thing that I’m very firm on in Salesforce is making sure that we do not gender our chatbots. There are other people and conversation designers who say, “if you don’t give your chatbot a gender, your users will.” That tends to be more of a case with voice because the general public will associate a certain pitch value of a vocal quality with a gender. That I think it’s a separate thing. When it comes to text-based chat, there’s no tonal quality. If our stakeholders want to give a gender to a bot, I’m like, ”this thing is a machine.” What difference does it make if we make it a female or male? It’s not human. It doesn’t have biological sex; therefore, there’s no gender identity. So that is something I push really hard on internally to say like the bot does not have a gender. It’s going to have a name that is very clearly a bot name, usually with bot appended to the end of it, like Salesforce-bot. Internally, in terms of the training data, can the bot understand various gender identities? One thing I’m particular about is that there’s not just about male or female binary. What about non-binary users, as well? How do we make sure they get included? What if the user uses them pronouns instead of he/she pronouns? How does that change anything in terms of the model performance when the user is talking to the system? So those are the things we will account for on the model side.

At Salesforce, we have a firm stance on the fact that AI should not be screening or processing resumes. The way that we suggest chatbots to be involved in the hiring process is more as an extension of the company’s brand for the users to take turns with. For example, the user wants to find out their application process or to learn more about the company history. These things can be established through a chatbot because the turn-taking nature of conversations can create more of a relationship between the company and the applicants. There’s an opportunity there to leverage conversation as a way to generate more connections other than a form or a static website. But the bot itself should not be reviewing resumes and applications and whatnot. There’s a multitude of opportunities for bias there. I can see the bot is biased against gender due to the lack of intent training.

The way I advise other companies is by asking, “Where is your user base? Who are they? Who are they?” If the majority of your user base is of a particular identity, then yes, you want to make sure to include that, but that doesn’t mean at the expense of all other identities that are in your total addressable market. I want to make sure that people of all races are represented there, and are encouraged to use various languages that are inherited to that particular dialogue and ideolective English. I think it is really about going out and doing your due diligence and collecting data from the broad population rather than overly constraining to a subset.

What do you think it’s a common problem among current voice assistants or chatbot experiences?

There’s a conflation of intended entities. So I think that people often confuse intents for entities, and they will say the intents here are the issues. I’m like, that’s not an intent. The issue itself is the entity. Are you reporting an issue? Are you canceling a report of an issue? That’s the intent. That basic distinction between intents and entities causes a lot of trouble with chatbots because it makes it unclear for the users. Conversation is inheritably about actions and topics. When it comes to voice, that’s when I think gender is prevalent because people are quick to ascribe a gender identity to a given pitch quality.

What do you wish to see in a conversation designer’s portfolio?

Particularly because I work on chatbots, so I want to see the ability to write conversational copy conversationally. As a conversation designer, you should be able to explain every piece of text, every piece of copy that is in a single message in the same way that a UX designer explains the reason why this line is .5 pixels thicker than the other one. Conversation designers should also be able to explain why putting period here instead of an exclamation point because XYZ or spelling alright as one word instead of two because XYZ. Definitely the deep thinking and reasoning behind everything that’s in the copy. Also, within the portfolio, some sort of expression of an acknowledgment of like a variance of language and openness to that variance. We are not here to police people’s speech; we are here to embrace the way they communicate. Showing that that quality around celebrating the robustness of a language and accounting for it is what I look forward to in a portfolio.

Do you have a motto or guiding principle when you work?

I guess the motto I keep repeating these days is we are not designing for 2006, we are designing for the future, and the future is inclusive and diverse. I would say that the principles that I keep in mind go back to that diversity angle. It’s not shutting down different variations of language; it’s about accounting for it. Our job as a conversation designer is to account for the multitude of ways conversations could evolve, and part of that includes sociolinguistic variations. If the user types U instead of YOU or typing a thumbs-up emoji instead of OK, we need to be ready for that. If we can’t handle that, that’s a problem.

What do you think the trend is in terms of how users interact with conversational AI?

If anything, I’m expecting that we’re going to hit the ceiling in terms of user tolerance around how much the user has to modify the language in order to get the assistant to work for them. At some point, it’s going to be like, “There’s so much effort for me to change how I communicate in order to get this bot to respond or to respond appropriately. We’ll potentially see either a plateau or a drop-off for a little bit of engagement there, which is where I think the industry enters a phase of inclusion race if you will, where everyone will be focusing tapping into that market and that variation of language in order to unlock that user base for a particular company. I think that the diversity aspect opens a lot for this industry, and I expect we’ll start to move pretty shortly.