Do you make data driven decisions? Most web developers build websites with a combination of instinct, and relying on conventions established by other researchers. But truly proprietary, 1st party research on your sites and tools can provide a competitive advantage far beyond other metrics. Varun Murugesan from Apple & Banana joins us in this episode to talk about why UX research matters, and how you can tackle a seemingly daunting task with simple approaches.
- Apple & Banana
- Complete Beginner’s Guide to UX Research
- Creating A User Research Plan (with Examples)
- User Experience is Everyone’s Responsibility
- The UX Research Plan That Stakeholders Love
- Why You Only Need to Test with 5 Users
The following is a machine-generated transcript of this episode. It will contain errors until it has been reviewed and edited, and we apologize for the difficulty that may cause for screen readers. Do you want to help us speed up our transcribing process? Consider sponsoring an episode.
Thanks for tuning in to episode number 88 of the Drunken UX Podcast when we’re gonna be talking about how to create baby’s first UX research plan, whether you are a team of one or a team of a team. I am your host, The bearded Wonder, Sir Magnificent beardy beard person. He who shall not be shaved. I am the host with the most gray in his beard. Michael Fienen,
I’m your other host, also known as I wonder where his beard is and why isn’t he shaving that already? Um, and that doesn’t look like facial hair. Aaron
gratefully. He who remembered pants?
Unfortunately my pin tweet right now is actually I hate wearing pants, which is a fact.
That’s actually a very nice. I wish I had planned that better, but I’m going to take credit for that. Either way folks, if you enjoyed the drunken new X podcast, you can go check us out on twitter or facebook at slash drunken you X. If you are on instagram were slash drunken UX podcast and you can always come chat with us anytime at drunken uX dot com slash discord and that will get you an invite right into our lovely chatty channel thing.
Uh, that fancy Irc is really all we’re all we’re talking about this afternoon, evening and morning with a bottle of something very new for me. It’s royal salute 21, it is a blended Chivas regal uh, specialty that is in honor of the 21 gun salute. They’ve been making it since 1953, this is the signature blend. They’ve got three or four different bottlings they do of this in different fashions.
This is real nice, like it’s, I’m not like the biggest Chivas fan in the world, but whatever, like as a 21 year blend, that means everything in it is at least 21 years old, if not more, and I gotta give him credit, I don’t know what exactly it’s blended from region wise, but it’s light, it’s fruity. Um it’s got a very, like, delicate sweetness to it, that’s very nice. Um It takes a little bit of ice very well. I’m very glad with this.
And the bottle is, I’m a big fan of like scotch and good bottles bottle, because this is a keep bottle. Like once I’m done with this, this bottle is going on a shelf because it’s like one of those like blue earthenware, um english, what are the english, china, whatever the name, I should know what the name of that stuff is. But I don’t so that’s why I’m having this evening and Aaron I would love to know. Well I have
oh I’ve got the balcony american oak um had a
day that’s that’s not the caribbean cask.
it’s it’s nice.
Um It’s a nice smooth blend. It’s got like a little bit of like spice. I don’t want to call it cinnamon. Even
american oak is a little sharper, little little bit because you get that like fresh that that new oak flavor off of it, a little little harsher.
It’s it’s really
I don’t I don’t have the the alcohol vocabulary to describe it correctly but it’s the experience is less of like a like a burning feeling and but it’s got like some some extra to the flavor kind of like when you I guess the oak, the woody kind of oak flavored smell to it is similar to like smelling like a fresh cedar plank of wood or something like that. Same kind of earthy.
They say you’re getting there, I’m training you well, I feel very good about that. I’m just making
shit up man. I don’t know what the hell I’m saying.
We need some training on this subject as well because we don’t let Aaron out to talk with users ever. He’s he lives in his uh, in his cubicle there. I think I on the other hand, you’re saying
you’re not letting me out. I think that it’s like, I don’t want to come out and talk to users.
Well give me credit for having some power in this uh, in this relationship dynamic a little bit on the other microphone we have from the stunning metropolis of Minneapolis, maroon maroon Jason is joining us to talk about UX research. He is the head of research at Apple and Banana. Previously, he has worked in user experience at both Best buy and Facebook. Apple and banana is aux brand who is growing a library of resources with two main goals in mind.
They’re trying a to not dumb down research for people but also make that research accessible for everyone. Baron, thanks for joining us this evening. I know you’ve got a can over there that I saw and now I don’t remember what was printed on said can accept that it’s a or
or who are close but no cigar. Vanna White do not press that letter for this man here. Earth writer Brewery, North tower stout, excited to be here. I know because of covid no one has been able to go out and get talked to anybody. So this is a great way to get a little bear in, get a little conversation except to chat with you guys and really take that baby’s first step into UX research.
funny because we were talking about this before the show that like, it’s while we do try to like bring data and stuff into our our chat and everything. We haven’t talked specifically about UX Research since Annie lin was on back In the 50s episode, 50s, late 40s. Somewhere else In
I know I have a lot of gray in my beard, but not quite that much. Uh
six years. You’ve only got 88 episodes.
50 60. That’s my my show researcher is Aaron. So he’s always on tap to to find that guy. What are
the many jobs that perform here?
So, we’re talking about this idea of how how do you bring in uX research into your work and doing it from this idea that UX researcher is a roll us researcher is like a thing. You can do a thing you can specialize in. People. Pay
people. Yeah. There are there are actually companies out there that will pay you to go out and do this, but a lot of places don’t have that, they don’t have that dynamic or complexity to even distinguish sometimes between developers and designers at all. Um And so we’re gonna talk about this from sort of the level of where your responsibility lies as somebody who deals in web. Are you dev are you a designer or an accessibility person?
Are you a usability person? Um you know all of these roles touch you X in different ways. Um And so I say frequently like anybody who builds things for the web has a responsibility to you ex. Um Quentin Carlson has written over in an article on medium. He says often various factions inside technology companies divide and silo user experience responsibility shouldering it entirely on the product and design teams.
However, the best technology companies have a shared responsibility for their user experience. It’s all right to have people who are you ex pros whether that’s UX designer UX research or whatever you X tester. But the reality is you X is a responsibility of of all of our work at least. That’s the way I have approached it.
Um You know when we We’re developing websites, you know, 15 years ago you x wasn’t the field like I don’t think I don’t remember having those conversations with people. It was like when content strategy was kind of coming out of the woodwork back then
Have to that it’s kind of like you couldn’t have autism in like the 1800s because that wasn’t the thing that we labeled right. People were thinking about user experience even early on, right, putting a button in a place where you could click it or even notice it. We just didn’t maybe have that title as user experience up until later.
So to me that dynamic that shift Kids getting out and get diagnosed with autism, but maybe in the 1800s 1500s, you don’t have that thing. So we just kind of label them with something else. So
refining. We we were front refine our industry.
I don’t know when you X became or rather if you’re ever designing anything whether it’s software or hardware or a shovel or a car or anything at all, where a human has to interact with it. I can’t imagine not considering the context of a human using it, whatever it is that you’re making.
Although I’ve clearly seen products where that was not a consideration, but like I think kind of in mind what you’re both saying, like there there’s like you x professionals who are like really up on all the research that we’re gonna probably learn about tonight. Um and then there’s just like general consideration of like um is important, maybe you should be aware of it and consider the context of whatever it is you’re creating.
There’s like a wealth of information here that we aren’t tapping into.
That wealth of information though, that’s what actually stops a lot of discipline getting out. And I think about this as a researcher now. Good research. If you do it well it’s proprietary knowledge, it gives you a cutting edge. So if you’re giving out this information you actually give them a leg up in that sense. So you don’t see a lot of this evolution because it is sitting at the desk, long nights, dark rooms.
I’m sitting with the data and then Eureka, if you put enough time to it, you get a little breakthrough moment, then your VP takes it and then they’re like, this is product strategy, this is important. We’re going to develop and design a whole new innovation, a differentiation out of this. But that does not get out or if it does get out a lot of us people talk about why would just copy paste it? You don’t know what led to this finding or this design.
So you don’t want to just absorb and copy paste the good alongside the bed.
So, talk to me a little bit, you know, when we talk about UX research, especially I think as an industry, the workers, the people like us doing development, doing design, understand abstract lee what UX research can uncover. But if if I’m a stakeholder coming in, if I’m somebody, you know, if I’m a sea level or a marketing person that’s just wanting to get some work done and my designer says, hey, can we stop and maybe test some of this stuff?
Can we stop and and get some users lined up and run them through a script? What is it you X research is really designed to tell people
thing, what is designed to tell you specifically geared around the questions that you ask. That’s a more esoteric type answer. But what I tell the stakeholders all the way up to the C suite, all the way down to a designer that needs to maybe pixel push is good research as two things. Number one, it helps you identify bad ideas Early on in the process and the # two, it helps you kill that ideas faster. A lot of research at least at scale.
Is there on testing, right? Usability testing, testing, test it, put it in front of somebody, see what they do. That’s a great way to kill that ideas. But if you think about it that way you spent all of this time may be running down the wrong road, you’ve built all these prototypes. Maybe you have a new designer developer that fell in love with that kind of bad design and then you come out and you test it and you find out that zero out of 15 people can use it.
A negative one out of 15 people actually found it reasonable or valuable. Right? You can kill that ideas. But earlier on, good research is about finding and identifying bad ideas as early as possible. One of the things that tell stakeholders is if you do a good research early on, it starts to become part of the company’s folklore, what is true by the people that you’re building for?
So you start to recognize if you’re trying to build, if your company builds products for people that have difficulties with vision, you probably wouldn’t try to sell them three D glasses, right? If you have done a little bit of research early on, maybe they’re much more interested in premium wireless audio for really immersive movie type sound, that might be something that you learned earlier on and even you stop your entire production around um, three D glasses.
That’s a very archaic example. But if you do research early, you start to recognize that these are the directions that you don’t want to go. That doesn’t necessarily mean that we should be moving forward in this direction. That’s where people, like developers, designers, other smart people in the room can recognize to align our business strategy and the people that were trying to help for these are some possible ways.
Let’s go ahead and build something. Let’s test it. Let’s see if we can iterate from there
related to that. I like this idea of defining kind of the story of your users. Do you do you find that you ever like in marketing, the have like marketing persona? Do you find that you ever kind of developed these like, uh anthropomorphized ideas or concepts of who your users are giving them names and everything and identities?
To me, not necessarily the best way to go about it, simply because it needs. And there’s so many context, there’s different context that you want to keep in mind. And I think personas, what has happened, and I’ve seen it happen is it gets easy to say, well, we’ve got a persona. Why the fact that we tested, What do we gain from it? We spent all this time and money to make a nice little chart. Look at, look at chad, he makes $30,000 a year.
He drives a Subaru and he’s got pink hair. And then I think about does that actually help you make a decision if you need to know if the menu should be left to right? Because that help you figure out what your go to market strategy is. Does it really short answer is no long answer is it gets you actually further away from the people you’re building for.
Because any sort of research, if you talk to any researcher on the planet that will eventually come to an idea that I call shelf life personas have a very short shelf life simply because everything around everybody changes all the time. You actually spend more energy and on to build a reasonable persona that he spent maybe two weeks coming up with fun names, chad, timothy, brad butter knife, susie salad for it, and by the time you actually want to use it to make decisions, that persona is outdated.
Whereas if you did shorter, maybe frequent qualitative studies and actually get out into the field and watch people use what you’re building that knowledge consistently just refreshes. Right? So personas there is it is split. I’ve seen some people do them. Well, I lean in the camp of that’s not necessarily the best way to do it. You don’t necessarily want to abstract people.
You can get closer to the nitty gritty to the details. That’s where the best earnings come from, because then your stakeholders like why can’t he click the button?
okay, I see that. Right. That’s where the empathy comes. That’s where the really fun part of research comes is being wrong and being excited about being wrong
Aaron this is your chance to say the line,
keep your personas close and your users closer.
Yeah, exactly. My thing. And this goes back like to my college days of building websites in the early odds, I had this way of thinking about websites that was at the end of the day, every website you ever build or work on or designed for whatever the case is, is the result of somebody building something to communicate to another human being.
The further away you hold them, the farther away they are from the code you’re riding and the layouts and utilize your designing the worst your stuff is going to be because you’re not getting that data, you’re not generating the information to help you make good decisions. You’re working entirely on instinct. And most of us don’t have great instincts. Not at the end of the day, like
if I can add one thing to that though, what I’ve seen happen is if the more you invest in personas were talking all the way the whole nine yards, you’re making posters, you’re communicating them effectively. When you’re making decisions, you’re asking yourself what would susie salad fork do? And this is during this moment it becomes easier to not try other kinds of research.
You actually become sticky and people become less more resistant to change, right? Because somebody sold somebody to make personas and now you’re saying that might be outdated, maybe this is the wrong way to do it. And then what happens with personas that if your product changes fundamentally, you do it in the startup space, The pivot, your persona probably doesn’t capture the data, the variables that you really know you now need.
We’re not making wheelchairs were making skateboards. Well now it’s a different entire set of demographics and people you want to understand. So it’s the selling of personas that can be challenging as well.
So the big it sounds like what you’re saying is that the drawback of personas for you X is that it results in too much fixation and too much maybe stagnancy.
And it’s It’s not that it’s not helpful, but when it comes down to it, I have never seen, I’m not 1000 years old. So I can’t say I’ve been doing this for that long, but I’ve never seen a designer say, yeah, we should go this direction or that direction and then be able to attribute it back to a persona. What I have seen as people say, we ran some survey research, we learned X, Y and Z. That’s why we went this way.
We did some interviews and that one quote really put everything a cool part of research is the outlier or the anomaly as we sometimes call it. Which is a weird way of thinking about people. But the thing that stands out might be the one quote or the one moment that puts everything else into perspective, right? Of course. He said this boom, Everything else becomes a bit more clear. A persona by definition is a generalization.
It is supposed to take a bunch of variance and then abstracted down into something clean, something simple. Everybody has the same variables, right age, gender, what type of car you drive or how proficient in our technology. You miss out on the variability you miss out and you reduce the dimensionality and complexity people have.
It’s like, I don’t know like a fundamentally what I think about at least what we think about apple banana is is any of this data legitimately going to steer us in the right direction or make it easier for us to identify bad directions. And more often than not, we measure very few things and we have found the trial generation, the less you measure and the more confident we are, what we’re measuring, we make better decisions.
Let’s talk about. So I started this whole thing out by saying, you know, I, I understand that most web people is a general industrial term Workplaces that probably only have 1, 2 or maybe three people running their web team, so to speak. Mike most of those are probably armies of one. Um, you know, if, if you are lucky enough to work somewhere with like a genuine team or teams of teams, that’s really the rarity though.
Of course, places like facebook, certainly higher swathes and swathes of those people. But the country is a big place. The world is a big place. Uh, so let’s get into from here. Just a little bit about, okay, I don’t have anybody in aux role and I’ll buy into this idea that I’m building something for people and I would like to build the right thing for people and convince folks.
Or maybe I know um one of my favorite things in the world is when I get asked to make a carousel for something. And my favorite thing in the world is to say no because they’re terrible. And so it’s but but I have all the way I can do that is I have all the data that says, users hate these things. They don’t use them. They don’t convert, they don’t do anything. You think they do?
How do we go about building that initial case to say, okay, let’s slow down to speed up. We’re gonna slow down. We’re gonna research a couple things. We’re gonna run some tests, we’re gonna write some scripts, sit down with some of these users and figure out what we want to do where where is the starting point for somebody on that.
And how do you approach this subject without coming across as the person who says no to everything? Because that’s a real consequence sometimes, because there are a lot of there are a lot of bad ideas, there are a lot of bad ideas that will come across the desk. And so that’s that’s something I think people get concerned about is I don’t want folks to feel like they can’t come to me with suggestions because they feel like I’m going to shoot it down every time. I don’t
necessarily tell you that it’s a bad idea. It’s the person that you’re building for If you have launched something, so a great place to start this population, especially if you’ve never done research. Let’s go to your stakeholder. The person that you really want to champion and challenge them to do research is ask them one of the last time you got something wrong, there’s not a single stakeholder on this planet that has been 100 accurate in their predictions.
There is some idea that they had in the shower on the way here, uh in the in the fruit aisle at whole foods that they’re like, Eureka, I got it and they wanted to go put that out into the world and their entire market said, no, right, that will sting, that will sit with you in that moment. Let’s draw on those emotions, let’s be uncomfortable together. Let’s go and do a little bit of interview and a great way to.
Then pivot on top of that is what do we not want to be wrong about moving forward? That it’s OK being wrong? And that’s where a lot of innovation product differentiation can happen because you challenge your assumptions and you went out and said, hey, maybe the thing that we’re building isn’t for us and it’s for somebody else. And then that’s where a lot of exploration and cool things to start happening.
One of my favorite things, especially with senior stakeholders, the people that are the most resistant, right? The stubborn stegosaurus in the room as a stakeholder. One of my favorite place to pitch research even before is I sit down and have a frank conversation and I say, here’s what’s going to happen with research and typically you don’t want to predict what’s going to happen to research because if you know what’s going to happen, why go on researching, right?
Go get some new knowledge, I can change the way. But here’s what I say, two things are gonna happen as a result of this research. Number one, you’re going to be right, you have amazing product intuition that’s got to feel so good that you have a pulse on the people that are trying to build for. You are seeing something in these metrics in this industry that nobody else is hot. Damn. That’s gotta feel amazing to be right.
Or the second thing is you’re going to get smarter, you’re gonna be able to avoid obstacles. You’re gonna be able to duck dip, dive, duck dive and dodge or whatever All over some of those issues. And that’s Gotta feel good too, because then when you go out and you talk to your leader and you’re telling them, here’s the vision around it. You know, 10 different things and you can say, Hey, there’s actually a path to product success within here.
It’s a little trite of me to say this idea that, you know, the only failure is a failure. You, you know, don’t learn from, so to speak. But and I don’t know is if this is one of those made up quotes or not right whatever that this quote that’s attributed to Edison about, you know, uh The reporter that asked him about the 1000 ways he failed to make a light bulb and he said, uh, you know that I didn’t fail 1000 times.
The light bulb was an invention of 1000 steps. Like it’s that idea that any progress has to come from data and learning and this exercise, so to speak, all of this, our muscles for us and sometimes we try crazy things as part of that and sometimes crazy things that we know won’t work because we want to learn why they don’t work and use that to apply to other things that you know, building this body of knowledge of things that users don’t do or users don’t like is itself really valuable information that And to the way we started this whole conversation about like how valuable proprietary information is in proprietary research.
If you’re the company that knows all the things that people hate and can stay away from that, that’s incredibly powerful. That’s really, really useful information for building your next, you know, killer feature, whatever that may happen to be.
If you pull that thread even further, it doesn’t take a lot to figure out what people hate. We’ll all walk around saying I hate this. I hate traffic. I hate ads. I hate paying for things. We all know that it’s much more difficult. And I think that’s where researchers and design designers start to intersect and really build powerful relationships is what do people like? Right. Like, Oh, I don’t like that menu, it’s too crowded.
What level of menu can be crowded enough for you to find value for it to be interesting for to be visually esthetically pleasing.
Right, That’s so much more difficult. And I think good research into design can come out to do a lot of stakeholders, especially in research, immature companies, where it’s maybe some party just trying a little bit of testing every now and then a lot of stakeholders walk in and walk out of research sessions expecting, boom, this is what we’re doing, this is the research, this is exactly what we’re gonna do is guaranteed success.
And I think it’s fundamentally the wrong way of looking at it. It is a process of learning the process of research that helps stakeholders make sense of the world around them and the people that they’re building for. And then then if you empower your stakeholders, your product, people, your marketers, that’s where they get the coolest ideas.
The uh this all sounds like all of this stuff like we’re discussing this as a way of like extracting value from your work and creating value from the products you build. And obviously this is like a very deep, detailed well to get into the reality is it doesn’t have to be hard though. Like you can get a lot of value from very simple things in this discussion, um, to marry Sharon over at smashing magazine.
And this was quite a few years back wrote, you must be able to boil aux research plan down the one page. If you can’t or won’t then you won’t get buy in for the research and its results. This is I love this quote because it speaks to this idea of powerful people Want one page. They don’t want you to hand them a 30 page dissertation on why you need to do a thing. They they’re time is valuable, their time is money, They want you to make it simple.
And so one page explain why this research is valuable and the reality is you can do that. It’s very easy to write a basic research plan with a thesis. You know, a goal, a methodology and and a target for who you’re going to go after. Um I applied what we call smart methodology. When I think about this, it’s an acronym that means specific, measurable, actionable, realistic and timely. So is what you’re going after.
Like you want to learn a thing, not learn and more so like what I was just saying, like how much do people trust our website? I mean you can kind of learn some of that, but you definitely can’t get a number out of it by any means. It’s not measurable in that way. You can get feelings, you can get an emotional qualitative value and for some places that is something worth measuring. But if you’ve got limited resources limited target, maybe that’s not measurable enough.
If you think if you think about an e commerce, right, trust between sellers and within the platform, that’s huge, you’d want to make sure that I can actually trust uh this person out in Wyoming that’s going to sell me this part right, That they have to build that into it. But going back to what you said, can we measure Trust? My first question I ask every stakeholder is what happens when we’re wrap up the research? What’s your game plan? Right?
We measure trust, we got something, what are you gonna do now if your answer is a big fat zero and I’m better suited doing research on something else. Something that actually move the needle forward. There’s a lot of good interesting research questions. Um, would it be interesting to know that 99 of drunken us podcast listeners love blue dinosaurs. Shit.
Oh my God, How do you know that thing we didn’t publish? That’s so interesting. It’s so interesting. Here’s a fall question. Is that going to fundamentally change how this podcast is right?
I mean, probably it might change my sticker design. You guys
hesitated. Okay. You guys hesitated. That made me
know that That’s absolutely fair. That’s
working to build Michael. We have to build our entire, I’m gonna throw money at this whole thing is going to be built on a blue dinosaurs going forward. It’s
time to stop my redesign. I’ve got to you got a new gold blue dinosaurs
where we should have done research. You should call me earlier.
But is it fashionable? Right. Can we do something with what we’ve learned? Obviously
here’s the thing you could technically yes, you can rebrand into the blue dinosaur drunken UX podcast but that’s a mouthful and this is why we drink. That’s neither here nor there. But here’s what I think is interesting if you work at a toy manufacturer that made dinosaur toys, That 99 finding is incredibly powerful because you now know to increase production of blue dinosaur toys and maybe to understand the market that like 99 that likes that blue dinosaurs, Right?
And I understand why Hasbro pumping out blue dinosaurs. Maybe they did some great research. It’s the context that matters, right? Keep coming back to context. I can do research on pretty much anything, but not the world’s greatest research or the cheapest or the fastest I can get you findings on pretty much anything I can come up with something. Is that going to change how you do things here?
Is that fundamentally going to change and help somebody that you’re trying to help. If not, Let’s use our let’s use our research resources much more appropriately. If you think about a typical study and this, I’m using the word typical very abstractly. Let’s say it takes you three weeks to run a full study done really well. We’re actually challenges and helps you build better and faster. The first week you figure out what you’re going to study.
Maybe write down your interview questions and figure out who you want to talk to. The next week heads down, You’re talking to participants collecting quotes here is watching them interact with the product or whatever. Maybe start your analysis third and final week, wrap up your analysis and maybe make a short little power point or email report whatever. Three weeks start to finish you present to your team.
If you abstract that out, there’s 52 weeks in a year, that means if it takes you three weeks to run a single study that’s meaningful and valuable, You can only run 17 studies per year. That is assuming that you start and end the next study as soon as you’re done with your presentation you about face. And he jumped at the next one that discounts holidays, that discount sick days, that discounts your stakeholder, taking up two weeks to go to Cancun, right all of that.
So man, you hold that down, you probably have between 12-14 really good studies per year. And you’re asking me to go understand if people like blue dinosaurs when you frame it like that, you can start to recognize some questions, some studies we really want to run towards and some, we’re going to steer right clear.
Um, the other two realistic Is the goal that you want to get realistic. Like if somebody’s coming in and saying, Hey, we want to test this thing because we think it’s going to increase our conversions 100%. No, it’s not. I don’t need the research to tell you that probably what you’re applying to this is not going to double, you know, whatever it is unless we’re talking super low number of numbers, like it’s going to change from three conversions to six. Did It
50 baby. Yeah, let’s get a beer
and timely is, you know, a, can we learn it quickly enough to do something about that goes back to this whole thing about like the real time dashboard and all of that. Like can we learn something that we have time to do something about and can turn around before it’s not useful anymore.
A lot of initiatives, marketing issues and stuff are very time box or they’re based around the release or they’re based around something else that has a clock associated with it. And so knowing when even to research, we should be researching things not right before we’re doing them. Sometimes plan ahead, get, you know, 23 months of lead times that you have time to go out and check things before that process starts
Stakeholders one Question every month and it is found, I found to be successful at small companies and big companies. It’s, what do you want to learn this month that you can use next month? It sets everybody up to win. It tells me that, hey, that survey, we’re going to use that next month to avoid or move faster on something else. Usability testing is possibly the lowest rung on that because you can see no one can find it. No one can click it.
It clearly didn’t work on the network. That’s good. But then you jump back into shelf life the minute you change that button to red to blue, that research is outdated, you cannot use it and nor should you for additional kind of design and product about moving forward. If you did big scale strategic research on what is the future of remote health care look like there’s a good chance you’re gonna learn a ton that can influence a lot of years coming down the line.
So you and you mentioned something and I think it was just off hand, but I want to use it to form a question, which is this idea of like, okay, maybe it takes you three weeks to do research realistically, like when, when you’re doing, let’s let’s just take you know, like an average, very simple again, thinking about this idea from what kind of time do you find?
Or do you do you find like certain things take a certain amount of time and other things take this much time? How much time is reasonable for somebody to try to say, give me this to go find out something
time is, I don’t think is the biggest constraint. I think it’s finding people that are within your market, that you’re trying to understand, that is a bigger issue. So if you can find somebody that you want to be talking to, right, if you’re building an app for skateboarders and you can find 10 skateboarders, go ahead and do it quickly, do it early.
Do it, meet with me right down maybe some questions and think about how you can analyze that data, interpret it. That’s great. If it takes you forever to find participants, then time becomes an issue. It’s not about methods, it’s not about surveys or quantitative versus qualitative. It’s I have the most amazing set of questions I have a big fat zero of people to talk to. So they’re just sitting dead in the water.
So even three hours of just talking to someone that you’re trying to help can be meaningful. It doesn’t have to be three months, right? I think there’s a big issue, especially from academic research. They spend six months dwindling on one question. Whereas in real life, some of the studies I’ve done, I’ve ran an entire study in six hours. Why? Because I had enough support and I can find people that I needed to talk to very quickly.
If you can talk to somebody this afternoon and you can use that day to the next day, Do that because you will learn 10,000 things really quickly. The second one, if you are in a corporate type environment, it’s when you’re looking at a requirement or some sort of user story and then when you’re going to actually write some code that you’re finding yourself being in an ambiguous situation, what do they mean by that?
I could do it this way or I could do it that way. That might be another way to talk to a handful of people to get more clarity. Not your stakeholder, but the person that you’re building for, so time is one way of looking at it. But if you have a reliable way of getting in front of people, you can go so much faster, you can run way more studies in that 17 number per year.
We have a talked about this in the past and side of this. Uh Jacob Nielson has this habit of ice a habit. He wrote an article two decades ago. Yeah. That he says you can do you X research with five people agree or disagree.
I mean, I don’t think it’s binary. I think there’s more abstract. This is how, you know, I’m a researcher. I’m not gonna be like f No, Here’s what I will say. I think that research is foundational. I think it is important what he says though for that 80%. I think about in terms of effect size in psychological research, we talk a lot about effect sizes, which is essentially the relationship between variables.
For example, I would probably have to talk to maybe 56 people to figure out a relationship and a pretty strong one of height and weight, meaning the next person’s height. I could probably use to estimate their weight pretty reliably right, because that’s a very obvious pattern. But if I could only measure somebody’s height and the number of lines on their palm, it’s gonna be very difficult for me to make any sort of prediction meaningful.
So when you’re looking at the 80% those people are probably going to capture your biggest issues. There’s no back button, there’s no save button, There’s a safe and a confirm button, those kinds of issues. Whereas if you think about your power user, your new user, they’re probably you’re probably not going to find those more hidden issues, those smaller effect sizes with just fact.
Yes, sir, a lot of people ask me how many people, how many people do I talk to? I don’t know, like tell me more about what you’re trying to do, Tell me more about who you’re trying to help. Tell me about the product and what’s important to the person. I think a lot of those questions can change who you talk to and how quickly you can access the most folks.
So let’s talk about like sizing this stuff right? Because obviously it’s easy to ask hard questions and it’s very easy to realize that oh the thing I want to know is may be much harder to get at that I think or cars more time or people resources whatever. What about this idea of like okay so I’ve got the drink a new X. Website. I’m in the middle of a redesign of this Aaron and I are the only two guys working on it.
I don’t have a U. X. Person, I don’t have an accessibility person, I don’t have a usability person have nothing else. And I’m thinking you know I want to I want to figure some stuff out about how people listen to podcasts. You know I’ve got some data, I’ve got some rough data about like people don’t use the website much to listen to podcasts.
They use apps but I don’t know if that’s because my site sucks or because people just don’t use sites. What’s how do I start thinking about like writing that test plan to get that specific measurable data? What would be like a good starting point? No I I swear I’m not asking for free consulting on this question. He’s asking for free consulting.
I’m gonna give you some free consulting. The first thing I’d ask you is figure out what are you willing to change if you are a smaller team? You tend to have less bandwidth if you’re doing this on the side, you got a full time job, you got a wife and kids and you got this close up magic that you really like to practice. What are you willing to change?
You want to change the tight? Are you willing to change the layout where you want to actually support the entire application and put them in the app store? So what is your kind of friction point? What is you going, what are you going to change? And then you can start to structure research and then if you actually start to some people and maybe you ask people at the end of this podcast, drop a link to a calendar late Hey, sign up for 20 minutes with the two hosts to get a conversation going.
Or one of the things that I like to think about is incentivizing and attracting different kind of people. Maybe they don’t want to talk to you, Maybe they’re introverts, maybe they’re developers and they love the content. They don’t want to go that deep in the conversation and schedule that time. Can you say, Hey, send us a voice memo, our voice mail to this email and tell us about what is it like listening and asking some of those questions.
There’s numerous ways to collect that data, but at the end of the day, you still have to take action on it. So what are you willing to change within that dataset? And that will change the way that you kind of structure your research. You’ll ask less questions if you’re not going to change the name of the podcast, right. Would that be something you go out to study? Probably not.
But if you are willing to optimize the landing page to be more seamless or get them to sign up for maybe a Patreon account or sign up for your email wait list or whatever. Maybe those were fun times that maybe those are the questions that you’re actually going to try to ask. So what are you going to change? And that will definitely influence what kind of questions you ask? Not the most best answer answer, but for free consulting it’s not, it’s a good one.
I’m not gonna complain, I’m gonna take it. I actually like the idea of like just throwing a thing out there and saying like yeah, send us a voice memo. Yeah, I know you guys are busy, we’re busy too. I like
that too. Use that at best buy. And what happened was if I can add a little anecdote, a little storytime here, please pandemic happened. And I pivoted onto a new team and my team is focusing on the in home services. So how do we deliver? How do we install repair, get things to you? And that was an entirely new team and that was an entirely new context.
Right in the home, in this truck, getting in there, they’re stuck in traffic and I was asked to kind of do some foundational research of like how do we build a baseline of knowledge, how to build on top of it? How do you validate some of the things that you know, and some of the things that we know that just aren’t true and I was stuck with, How do I do field research in a remote only work environment?
Ah employees are super fantastic and they’re so, they’re so inundated with a whole lot of service calls, how do I get ahead of them and how do I not slow them down? Right? Hey take time to do an hour long interview. And so what I did was I took my team’s account because ever now has a team’s account and I set up my call whenever you call me and we’ll go straight to voicemail and I left exactly a three second message.
This is a maroon, leave your message if your in home agent by hang up and I just train them over time. But whenever you have an issue, whenever you just want to talk about your day, you want to tell me a small detail that someone might not see in one of those dashboard here on those metrics you were told the room and burn would sit there and listen to it.
And what happened was In three days I got over three hours worth of data of voice and you’ll be surprised how much data, how much richness and how much detail someone can pack into a 32nd sound bite. And a lot of these employees are telling me that they were starting to some of them one employee specific like this is like therapy.
Now I’m excited when there’s an issue because I know I’m gonna leave a voicemail about it right after and that touched me and my stakeholders. They were skeptical. They’re like voicemails. What what what are you doing with voicemails?
And I showed them this data, I let them listen to it and let them listen to the transcripts and I built this entire report out and there are like there’s something here. So now I always have my, if you call me on teams set to go to voicemail and I’m just ready for that kind of data
which can show like blind spots too right with that opening there, that’s a good way to find out about things that the unknown unknowns in the equation, right. The things that you don’t know our problems that people are experiencing. But if you’re not tuned into their networks or their channels, you may not be hearing those complaints in in real time.
Let me dial this down to like the big question, the elephant in the room sort of to wrap up on and you hit on this already. Covid is a frustration for all of us and um, that also makes it hard to get people into a research lab, get people, you know, into an eye tracking lab, things like this. Can you tell us about any tools that you have found useful? Um, over this last year for getting valuable research. When you can’t be in the room with the person,
one is kind of going back to that voice mail that I think scales nicely to anything. I don’t think good researchers should be, um, kind of limited by the tools that you have access to. Some of the big name companies have million dollar budgets that yes, they can throw their money at a lot of different tools and use them.
I think if you’re crafty, I think that is so much more effective in terms of getting environment people during Covid, I think you learned from a very different set of person. Now, Covid and some for some people has completely drastically changed their lives for the worst. They’re much harder. They’re struggling with income or anything like that. And for some people and I’m in maybe the other side, Covid hasn’t necessarily changed my life too poorly.
What I think about that from a research perspective is that maybe now you can attract and learn from different kinds of people that are struggling that you can get in front of them, even a phone call, even a google meet or a zoom can be powerful. I don’t think the best fancy tools are going to get you into the home, help you understand what the context is for someone that you’re building for. So I don’t think I answered your question.
Can you ask you ask your question?
So we we’ve talked about some tools, Right? So whether that’s google voice or whatever voice mailing system you have setting up things using things like calendar to set up a voice chat, we’re sitting here recording, you know, I’ve got a remote team, so to speak. I’m in Kansas, I’ve got my co host Aaron is over in new york europe in Minnesota. So this like using a voice chat system um is useful.
I’ve talked to folks who have found uh we were talking earlier about the tim Broadwater at the end of season three and in his book, uh You x fuck marry Kill. Um and I think it was him, we were talking to that. He’s gotten really into like using zoom, just using zoom to do user testing. And so just thinking about like what tools have you found useful in in that regard to kind of fill some of that gap of just being able to sit next to somebody and run through a script with them.
I’m going to be just not very exciting. I try to use the tools that I have better yet. I try to use as little tools as possible because then I can optimize what I use them for. What I have noticed though, being able to do things remotely, I’m able to interact with a different portion of the population that I might not have otherwise.
For example, if you don’t live close to where I live, where I’m testing or I’m running research, guess what I won’t hear from you, wow, look now, I can just do a video call, you can show up whenever you need to. That’s awesome. I’m learning from you, your voice has been heard and we’re having a meaningful participant experience. The other
thing that like consumers in Minneapolis might be different from consumers in Los Angeles.
I think people are the same. I think just one shade of weird, great.
then then then on the other side, what you see is maybe I’m nervous to participate, maybe I don’t want to come in and be grilled with all these types of questions. Maybe I am more introverted now, I could do something remote and I can learn from a different set of people. I think this remote working environment opens up different people to participate. And I think being in context and some reasons, yes, it is the right thing.
If you’re building high tech machinery on the supply chain, uh, in a supply chain factory, you probably want to be there. Because I don’t think people can bring large pieces of machinery home to work from home. I could be wrong. Last time I checked, you can’t take a giant multimillion dollar shorter home.
There was that guy that took the bulldozer like through his town a few years ago. That was that was the thing that happened,
That was me, that was Mardi Gras 2000
badly for everybody.
The tools is the wrong way of looking at it. One idea I think about is the best basketball player. isn’t the one with the most shoes, right? The best engineer isn’t the one that can use the most A. P. I. S. Right? It changes the way you look at it. Can you use the tools that you have and can you optimize them in the right way? Once you start to recognize what tools are working then you can start to mix and match with limited changes.
I don’t think adding more and more user usability testing tools and survey platforms leads to better research. It’s when you understand what your goals are. They can understand how you want to use certain tools. If you don’t need to be in the home, guess what? You don’t need to be in the home. If you don’t have a travel budget, if people can’t get to where you need to get them, you’re not actually better served by going into the home. So I think there’s pros and cons
and I think regardless of like the tools that you’re using or things like that, you know, especially if you are somebody who is on a small team or no team and you’re trying to set up these cases for for research and whatnot, it can kind of be beneficial maybe to even start with setting up like maybe even research things that you know, if nothing else just to start yourself off with those winds right?
Like get yourself and not just to build your own confidence, but to build out like confidence in you from other people, so to speak. Or or at least even if you’re disproving things like you’re disproving them in in ways that builds confidence in that process, right? And whether it and it doesn’t matter what tools you use at that point, as long as those tools build confidence,
I think that’s a great place to start. Can you take a finding, where can you take something that everyone is a company knows? Yes, people do blank and blank. Can you go out and validate that even through the process of validating people don’t like X, y, Z functional design, you’re still going to learn, have learned something, you’re going to deepen that knowledge and find a whole new set of questions to kind of go ahead and tackle.
I think I think talking about confidence is powerful because once you do one study and one and even one stakeholders, half a light bulb moment, what that does is that reinforces that maybe we should go ahead and do this again. I think about good research and even at a small company, even if it’s just figuring out if people like red or blue on some sort of design what that serves as a benchmark for future relationships, I can build on top of that.
Remember last year when we did that really quick piece of research and we learned those two things. What if we did that a little bit different this year can be built on top of that, wow, we’re actually building and adding to the knowledge that we have and we’re learning to refine that over time. I think confidence is very powerful word and a lot of junior researchers or if you are a team of zero or a team of one, that’s a tough place to be.
But at the end of the day, you will have to reach at some point good researchers in the fall out of the sky. You will have to invest. But it’s having these conversations being uncomfortable thinking about how you’re actually going to use this data and recognizing do we want to burn one of our limited research opportunities on this interesting question, knowing that we’re missing out on more valuable
and I I think to sort of sum this up to the other, I think to me anyway, one of the good uses of research is to question the things that you know, as well as the things you don’t know because what you knew last year may no longer be true. Users change.
Users get acclimated to a design and will begin to behave in different patterns. Um, and so the things that you learned last year, two years ago, that truth may not fall the way you think it does anymore. And that’s okay. Like that’s why I go back to this idea. Like it’s not a failure that your users have evolved or gotten better. Yeah. In fact,
product innovation, wow. You can react and learn with them if they change consistently, your product should be changing and you might be attracting an entirely new set of target consumers that you can build cool stuff for instead of just saying they like blues who are going to do everything. Blue.
I think that’s how we ended up with birthday cake flavored everything. I don’t know why they decided to try that in the first place, but it’s terrible
why that is a flavor.
I don’t know. I had the birthday cake flavored Oreos and they were awful. Yeah, It was, this was like seven years ago that I had them and I still think about it and I still remember them being horrible.
I mean, I think Oreos doing too much. That’s my two cents. I don’t need like mexican hot chocolate flavored oreo or like a chicken teriyaki oreo flavor. Like I get your market research validated that somebody might buy this. I think you’re doing too much.
And if that’s not a commercial break, I don’t know what is
it took me about eight takes to get your name right and I apologize for that, but I think it was all in good service because this was an incredibly enjoyable conversation. I hope everybody learned um at least a little bit about UX research and got some advice on how to go about it or how to approach it or how to argue for it in there. Um Take a second though as the payoff because I literally have no other way to pay you.
Uh take the microphone for a couple minutes here and talk about what you got going on, where you’re at, what you got doing apples, bananas, breadfruit, pineapples. Mhm. Yeah,
no, first off, this is awesome. I really appreciate getting a chance to sit down and go deep. Some other podcasts I’ve been on, we keep it high level, which is important, right? That makes it more accessible. But I like being able to go deeper into the weeds and hopefully your audience, your listeners were like, hey, this is nice. We’re hearing stuff that you might not see in a list. Ical top 10. That’s research. We go to
That’s too much. That’s too high brow.
Apple and banana is uh you X brand that we’re building right now. We’re currently working on our book we wanted out later in the summer. I am trying to have a full time job. I just got a puppy. We just turned a year old were training him. I have friends and family and relationship and trying to work out and try to find meaning as the world slowly descends into chaos.
But we are writing the book and then also supporting a blog at the same time. Fun fact in pro tip. Don’t do two things when you have no time. Don’t think about supporting a blog and writing action meaningful content. Not just list tickles and actually trying to work on a meaningful facebook.
All right. I got some bad news. Worry about what I’m doing
Way don’t do two things. Wait,
Does it get better if you do nine things
barely? I try to do half a thing if I can’t. That’s where I’m trying to get to. Uh huh. But no, the book is going well. We have a lot of feedback coming in. People are very excited about what’s happening, which is fantastic. They’re actually building something meaningful. If you look at our site Apple and banana dot org, you will be hard pressed to find a single picture of my face simply because we don’t want it to be about marketing and sales.
What we want is to you to focus on the content and feel like anyone can do great research because fundamentally that is what we believe. And one day after we’re done with the book later this summer, really interested in identifying new needs around the research and the U. S. Community. And a lot of people have been talking with us about community how they don’t feel like they have a group of people that they connect with.
And then we’re also interested in something like this audio video type content where it a little bit more engaging than just maybe something passive as reading dry text on a page. Fun fact we’re not dry. You should check out illustrations because we invest heavily into our visual design. That’s the cute
little apple and banana playing like on the swing. Oh I saw them. You’ve got
to check it out. Well you checked it out but maybe your list haven’t. So
uh in the show notes. Don’t worry.
100%. Yeah. I think research at the end of the day, I think it’s really creative. I think it’s really fun. I think you murder but don’t think that should be a barrier. If anything it should teach you where you want to focus your efforts. Maybe like qualitative versus quantitative, something like that. I think you should get excited about going out and being wrong and having been and being okay with that. That’s what we’re trying to do. What happened banana.
I’m a big fan of being wrong.
Do you do something after though? Just being wrong is not that great. But if you’re wrong and you learn from
Oh yeah. Well that’s well I would say that’s a given but I guess not everybody does that but I I think um
here actually used to wear a lot of Hawaiian shirts and I have
never really know why insurance. I’m not a fan of color. Um Well come and count down the episodes to 888 with us on social at twitter and facebook dot com. So I struck a new X and on instagram dot com slash junkanoo X podcast and come talk with us while you still can for free, not saying it’s going to cost money in the future, but who knows
But I mean if it did
then you’ll be glad you came and did it well now when it
was free at junkanoo X
dot com slash discord
come talking about but it will be free but if it wasn’t, but if it wasn’t in the future,
if it hypothetically wasn’t in the future, you’ll be glad that you came and talked to us now when it’s still free. Just same case. I’m not saying it will cost anything in the future, but if it did, you’ll be glad that you came back when I was free.
I will say like the field of UX research is like rich and growing. Um I think as we better define our entire industry, you know, you will see even more specification and growth in in that area if you’re interested in being the UX researcher while like it is a more narrow field Introductory salary start at around $8,000. They top out around $121,000.
Like it’s not like the least lucrative field you can go into because there is an increasing demand and increasing value in that whole area. We’ve talked to a lot of people over the seasons. I’ve enjoyed every conversation that we’ve had to this date. I don’t say that lightly. I genuinely, you know, go through all of the folks we have on. I’ve enjoyed this absolutely thoroughly.
If for no other reason than because it has thoroughly and completely validated the idea that you should keep it over something as close and your users closer erin, Goodbye.