Podcast: Play in new window | Download
Subscribe: Apple Podcasts | Google Podcasts | Spotify | TuneIn | | More
My guest today is Emma Craig, UX Research Lead at Shopify.
Shopify was founded in 2004 by Tobias Lütke, Daniel Weinand, and Scott Lake after attempting to open Snowdevil, an online store for snowboarding equipment. Today, Shopify offers online retailers a suite of services for more than 1,000,000 businesses in approximately 175 countries, with total gross merchandise volume exceeding $42 billion.
Prior to joining the User Experience team at Shopify, Emma served as Manager Team Lead at Shopify and held the position of Investigative Interviewer at BackCheck.
Find Emma Online:
Twitter: https://twitter.com/emmaccraig
LinkedIn: https://www.linkedin.com/in/emma-craig-5b829920/
Find Jamin Online:
Email: jamin@happymr.com
LinkedIn: www.linkedin.com/in/jaminbrazil
Twitter: www.twitter.com/jaminbrazil
Find Us Online:
Twitter: www.twitter.com/happymrxp
LinkedIn: www.linkedin.com/company/happymarketresearch
Facebook: www.facebook.com/happymrxp
Website: www.happymr.com
Music:
“Clap Along” by Auditionauti: https://audionautix.com
This Episode’s Sponsor:
This episode is brought to you by Lookback. Lookback provides the tools to help UX teams to interact with real users, in real time, and in real contexts. It’s Lookback’s mission to humanize technology by bridging the gap between end users and product teams. Lookback’s customers range from one man teams building web and app experiences to the world’s largest research organizations, collectively ensuring that humanity is at the core of every product decision. For more info, including demos of Lookback’s offering, please visit www.lookback.io.
[00:02]
Jamin: Hey, everybody. My name is Jamin, host of Happy Market Research Podcast. Emma Craig, UX Research Lead at Shopify. Shopify has experienced massive, insane growth over the last five years – it’s unprecedented. Valuations are just amazing on that company. Congratulations, Emma, on all of the success there.
[00:23]
Emma: Thanks. I won’t take credit for it, but it’s exciting to be part of it.
[00:28]
Jamin: Yeah. No kidding, no kidding. So you’ve been at Shopify for a few years. What did you do before that?
[00:34]
Emma: So, before Shopify, I had a really interesting job, actually. I worked at a company that other companies outsourced this company to run their background checks on people that they were thinking of hiring. So, I was in the investigative interview team of that company. And I would phone people’s references, and the previous employers, or educational institutions, in some cases, and interview them about these people, basically trying to suss out what type of employee they would be, and if it was the type of employee that this future employer was looking for. So, yeah, it was really interesting. I think it caught a lot of people off when you’re more used to the average reference interview of like, “Were they productive? Were they nice?” And we really tried to get in there because these companies were very concerned about making sure they’re hiring the exact person that they’re looking for.
[01:26]
Jamin: So, our topic today is the art and science of a good question. It’s hilarious that you started your career in really framing out questions, and asking the right one in order to get the accurate response, I guess. Not like it’s leading, but you just needed to know if it’s a good fit or a bad fit. And then you moved into UX Research at Spotify. How did you wind up in research?
[01:49]
Jamin: So, how did you wind up in UX Research?
[01:51]
Emma: So, that was interesting, too, actually, because I started at Shopify in the Account Management sort of team. And that was the team that was spun up when we first launched Shopify Plus, now they’re called Merchant Success Managers. But, basically, your job was to work with these enterprise clients as their Account Manager and make sure that they were taken care of. You were answering their questions if they wanted to sort of stress the platform or take it to its most complex, interesting, unique performance. Then, you were the one that was trying to facilitate that between the Shopify Plus merchant and the engineers, or the data scientists, or whoever you needed to make it work in the back end. So, I learned a lot about Shopify and how it worked because most of these people were looking at how to make it work differently and in new ways. And, through that, I wound up speaking to our UX teams a lot because I was learning all of these things and there were really interesting insights and anecdotes, and I wanted to be able to share them. But I also knew that these were sort of our, they’re the loudest people because they have that direct line, so you’ll hear their voice the loudest. And it was just one-offs; none of these could really be generalized to a larger population. And I started to be more interested in finding patterns, and the themes, and talking to more people. And then I, I guess, outgrew my role and became much more interested in the UX research aspect of things. So, Shopify, being such a great company to work at, really kind of took me under that wing. I had UX Researchers mentor me and then I just naturally grew into the role.
[03:30]
Jamin: The mentorship part is really interesting to me. And I know it’s a little bit off-topic, but I just want to touch on that because I have actually seen that as a void or maybe an opportunity inside of consumer insights. Was there some formalized plan along that, or was it more just organic?
[03:46 ]
Emma: No, there was definitely a plan because I left my- It was an internal transfer, but I left my position and it was managerial at that point. So, I left my Manager position to become an intern on the UX Research team. To which my parents were like, “Oh, god, what are you doing? What is UX Research?” My mom, oh my gosh, she couldn’t stop calling it XXX research for the longest time because UX was this [CROSSTALK].
[04:09]
Jamin: Oh, my God, that is epic.
[04:12]
Emma: I know. It wasn’t in her vocabulary, so she was just telling people, “Oh, my daughter is, I think, it’s XXX research.” I was like, “Mom, stop.”
[04:18]
Jamin: Please stop saying that.
[04:19]
Emma: Stop it. [CROSSTALK]
[04:27]
Jamin: I have been in market research since 1996. And it wasn’t until, I swear to God, last year, so 2019, my dad got a survey, and he’s 82. And he says, “Oh, look, I got a survey,” in his email. And he says, “Is this what you do?” I am either like outsourced technology help, right. So, my computer won’t start, one of my family members calls me or it’s marketing.
[04:50]
Emma: Oh, that’s so funny. 20 years later, “Is this your job?”
[04:55]
Jamin: Yeah, right.
[04:57]
Emma: That’s really funny. I actually, so my degree is in Marketing, so I had some market research experience just from doing internships and stuff over university. And, yeah, it was really cool, I think, to start to learn the UX research aspects of things after some market research past.
[05:15]
Jamin: That’s very interesting. So, gosh, there’s a whole can of worms there on the difference between market research and UX research. Do you have just a short bit on that?
[05:24]
Emma: No, I am not the one to answer that. I have been- It was university days, I am so far gone from it. I am sure that it’s developed in so many ways, I would not want to give a definition of market research.
[05:37]
Jamin: You know what’s funny about that is I am actually doing an interview on that exact topic later this afternoon.
[05:41]
Emma: Oh, awesome. I will listen to that interview when it’s released.
[05:45]
Jamin: Yeah, it’s gonna be fun. So, questions, there’s this group. I don’t know if you’re a Doctor Who fan or listen to that. He always starts with, the hero of the show always starts with like, “Well, you’re asking the wrong question,” right. So, in order to get to the right answer, you need to have a framework of what the right question is. And, in this context, how I’m really thinking about a question, it’s centric to, at a tactical level, with a respondent, right. So, framing out for them, and I’ll just stop talking in one second. But just the difference of like, “Do you find this product interesting?” Versus maybe more coming around the back door of, “Which of these products do you find interesting,” or something along those lines. So, I am interested in, from your perspective, what do you see as the elements of a good question, a well-crafted question that a respondent can answer correctly?
[06:36]
Emma: So, I think good interview questions or these direct questions that you’re asking a participant or a respondent start with your bigger question, your research question. And I don’t want it to get confusing here of what’s what. But before you can start to formulate your discussion guide and understand exactly what it is you want to ask these people when you’re face to face with them, you have to have your research question and your research objective in mind. So, the research question here is, essentially, seeking to understand why something is happening. Or what is happening? You’re looking to uncover a process, or a need, or a challenge that someone is experiencing. So, an example would be, “What are the biggest challenges people experience when it comes to taking public transit?” And that would be your research question from which you derive all of your interview questions. And you had a really good point about not asking these pointed, direct questions that you just directly ask because, half the time, people won’t actually know the answer or they won’t have the answer. But I have learned over the years that if you ask somebody a question, they will answer your question. So, whether they make it up, or they exaggerate, or whatever it might be. If you ask them something directly, they’ll give you a direct answer. And you can’t always be certain but that is true or that they are not just telling you what they think you want to hear. So, your interview question, it’s there for you to collect evidence and you have to take different angles. You have to go sideways or, like you said, take the backdoor. If your research question is around the biggest challenges people experience when it comes to taking public transit, your interview question shouldn’t be just asking somebody if they like to take the bus, your interview question could be asking them to walk you through how they got to work last week. And to kind of take these roundabout ways to understand the environment that it is you’re researching.
[08:45]
Jamin: I really think that’s a nice counterpoint, “Do you like to take the bus,” versus “Tell me about how you get to work.” There’s this journey mapping that happens, at that point, which then places the participant or the respondent in kind of an emotional state similar to what they experience when they actually went through that activity.
[09:07]
Emma: Yeah, exactly. And then they’re actually thinking about the context. And they’re, hopefully, if you’re a really good interviewer they’re experiencing how it felt. So, you’re not always just asking the frontal lobe, top of brain, then, “Hey, do you like taking the bus?” “Sure, I take the bus. Yes, I like taking the bus.” You’re getting them back into their experience. You’re placing them situationally so that they can start to tell you about their experiences and their feelings with more texture.
[09:39]
Jamin: A good question places them situationally, that’s a pullout quote, for sure. The other side of it is, “Do you like to take the bus?” So, what are the common mistakes in framing interview questions?
[09:51]
Emma: So, again, asking directly. A common mistake, though, and I am sure if you’re interviewing other people for this, they’ll say it, too. But I am going to say it because I still see it all the time, and I still see myself start to do this. But it’s asking questions that aren’t open, and neutral, and non-leading. Which, of course, that’s Interview Training 101, don’t ask a loaded question, don’t ask a leading question. But I think I have started to see, I think it’s because we’re so naturally inclined to demonstrate that we understand another person or that we’re listening to this other person, or we’re connecting with them. And, in normal day to day conversation, we signal that to somebody else by expressing a shared or a mutual understanding. But when that happens in an interview, what winds up happening is you’re suggesting their answer or you’re putting words in their mouth. So, if the question is common mistakes, it would be asking those leading questions. And it can be really uncomfortable, but you have to strip your questions down as much as possible. And you have to get over looking smart in front of other people and be OK sitting there and asking just what might feel like a really dumb question, but giving them the opportunity to fill in the space with how they actually think, and feel, and process this.
[11:09]
Jamin: And the filling in the space part is really interesting because we have a tendency to want to do that in a conversation or in an interview because you want the other person to succeed, on both sides. How do you combat that as you’re doing interviews?
[11:30]
Emma: So, that, to me, I think the biggest struggle there is staying silent, being quiet and letting them answer the question, or elaborate on the question. A lot of people will jump to the next question to sort of get along with it or fill the space. We’re really uncomfortable with silent spaces. So, I have heard of a few ways of getting around it. I have some colleagues that will actually just count. I don’t know what that number is, but they’ll pick a number and they’ll sit and count to eight in their head, and then they’ll let themselves speak. I do a lot of meditation, so when I am in an interview, I’ll keep my feet planted on the ground. And, if it starts to feel uncomfortable or I start to feel like I just want to fill the space, I’ll actually just take some deep breaths and concentrate on the feeling of my feet on the ground, and it’ll really ground me. It’s like a little, mini, quick meditation. And then I also diffuse my energy out of that area. And, once again, it becomes space for my participants to fill themselves.
[12:34]
Jamin: The other challenge- By the way, I just grounded my feet.
[12:36]
Emma: Oh, good. Me, too.
[12:43]
Jamin: I love that. I love that. The other thing that you referenced is this desire, as humans, to connect with one another at an emotional level, right, so that’s very important to us. And yet, sometimes, that can get in the way of truth, right. Have you seen- In other words, we want to agree with one another is what I am trying to get to. Have you seen that play out in interviews, from a respondent perspective, where they’re just like trying to like the thing that you’re putting in front of them because they want you to feel good about yourself?
[12:51]
Emma: Yeah.
[13:16]
Jamin: And then how do combat that?
[13:20]
Emma: Of course, you see that all of the time. And it starts with you, the interviewer, I think, it’s easy to be unaware of the signals we send because they are so subtle. But just things like nodding your head, or shaking your head, or your facial expressions; every tiny, tiny movement, the other person, often unconsciously, is watching as a way of receiving feedback about how well they’re performing. So, to combat that, I make sure that I am very neutral, again, grounding myself really helps. I say thank you when they answer a question, which feels weird because I think we’re inclined to say like, “Oh, that’s super-interesting.” Or, “Oh, weird. OK, the next question.” But that’s influencing it, that’s giving them feedback about how well they have done or how interesting or correct they have been. So, if they answer a question, I just say, “OK, thank you,” and then I’ll move on. And you feel a bit like a robot, you feel a bit silly, but your job as the interviewer isn’t to be their friend or to really connect with them. Your job is to collect evidence to support the research question that you’re asking. [CROSSTALK] It helps to, I’ll add, it helps being a researcher and not the designer. And I’ll often start my sessions by saying, “This is a piece of work that I am looking into on behalf of the team here. I didn’t do this myself or these aren’t my sketches. So, there is no right or wrong answer and you can’t offend me.” I just kind of start it off with like, “You’re not going to upset me if you tell me this is garbage.” If anything, you are going to be helping us much more if you do.
[14:55]
Jamin: Tell me the baby is ugly.
[14:57]
Emma: Yeah, exactly.
[14:59]
Jamin: The worst question you’ve ever seen. We have all got them, right?
[15:01]
Emma: I know.
[15:04]
Jamin: And it could, I am assuming this is, there’s two lenses. One, it could be you have actually done or the other is, and probably more probable, is you have been exposed to as a respondent or a participant.
[15:13]
Emma: Oh, my gosh. I receive, I’d say, questions that I get in emails, I always send them to fellow researchers and be like, “Look at this garbage.” But I’ll speak about interviews because that’s what we’re talking about. And I’ll speak about myself because I did this a lot when I started out and I still, when I write my questions, will notice if I am doing it and try and undo it. But it’s anything in the future, so we really want to ask people to predict things in the future. It seems like a good way of finding out information. So, a really simple example would be, “How often do you picture yourself using this?” Maybe in the interview, you have exposed that this is something they are interested in, and they think it would be very helpful. It would ease all of these pains and challenges that they have. And then you want to say, “OK, well, how often do you think you would use it?” But people cannot give you a realistic idea about the future; they don’t know, they will make it up. Like I said, if you ask somebody a question, they will answer that question. But it probably won’t be true because they don’t know well enough if they’ll use something or if they’ll do something in the future. I think an example I use a lot is if somebody asked me what I was going to eat for lunch tomorrow, I can’t actually- I can give them a guess but I can’t actually tell them. But if they asked me what I have eaten for lunch every day this past week, I’ll give them a much better indication of what I might eat for lunch tomorrow or what my lunches look like. So, yeah, probably that, asking people to predict instead of basing the question in past behavior.
[16:48]
Jamin: And then having to use math and intuition to be able to get to future outcomes.
[16:54]
Emma: Yeah. In research, you hear this trope all the time, but it’s like don’t- God, now, I am going to screw up the quote. But it’s something like, “Don’t ask people what they do – watch.” You want to observe behavior in research. You don’t want to ask them to tell you their behavior, it makes so much more sense to observe it. So, in an interview, the closest you can get to that is to ask them to describe past behaviors that they have done because you know those to be true.
[17:17]
Jamin: Yeah, absolutely. Gosh, I really like that. And that caught me a little bit by surprise, I was not thinking it was going to go that way. That’s actually something that’s very, very insightful. And I have done that quite a bit, being completely transparent. I thought it was going to go a little bit more into grids or more into the- Maybe my bent is more quant, right, so my exposure is- I literally just saw this yesterday. I got a survey, it was on a scale of 1 to 10, how accurate was our delivery? And I swear to God, that is the question. And I am like, “It’s such a silly question [CROSSTALK].”
Jamin: It doesn’t.
[18:30]
Emma: No.
[18:31]
Jamin: It’s so interesting this and- But I’ll tell you what’s interesting, so this is what I find very interesting, and now we’re totally off-topic. But this inclusion of NPS at the activity level, where now we’re, everybody who’s buying stuff is being exposed to an NPS question, probably weekly. And it’s starting to inform our culture so that I asked somebody, it happened in our stand up two days ago. He happens to be Finnish. And then I said, “How was your day, or whatever, yesterday?” Just like the small talk before. And he goes, “I would rate it an 8.” And I am like, [LAUGHTER] I am like, “What happened? What happened?” But what was interesting about that is then, of course, I couldn’t help myself, I had to dive in. And, he said, “Well, the problem is that, in American culture, if you say good, that means something very squishy but, for me, it means something a lot more absolute. And so, I am trying to quantify it for you in a language that we’ll understand so that you can have an accurate idea.” Which is really hilarious, on so many levels. But I thought it was interesting that you are seeing this change in how we relate to one another at a human level really being dictated by this intrusion of NPS in our lives.
[19:58]
Emma: That’s so true. Yeah, that’s a light way to say it. It’s the sledgehammer of NPS [CROSSTALK]. The problem with NPS, though, we have actually kind of said the same thing here because NPS asks you to predict future behavior. “Will you recommend this to a friend?” Such a better question would be like, “Tell me about the last time you recommended a product to your friend.” And it feels super roundabout, you are like, “No, but I want to know about my product. And I want to know if they’re going to be loyal and a brand evangelist.” But they don’t know that, so you can’t ask them that, you need to infer it from their behavior.
[20:31]
Jamin: Oh, my gosh. There’s this, I’ll try and find it. I heard IBM gave a talk on NPS and the flaws, one of their head Researchers, and it was fascinating because they have been doing it for a long, long time and at a massive scale. And, anyway, so totally divergent, sorry about that. Emma, thank you so much for being on the Happy Market Research Podcast.
[20:45]
Emma: Hey, thanks for having me. This was fun.
[20:50]
Jamin: My guest today has been Emma Craig, UX Research Lead, Shopify. Everybody, have a great day. If you found value, please take time, screenshot, and share on social media. Tag HappyMR.com or whatever, enjoy your day.
