User Research: How to Get Out of Our Bubbles by Talking to Users (Devcon5)

[Applause] the difference of disciplines expertise in the room who here is a designer or a researcher also dude does something he's done research before or tried it great it's okay all right good so you got some some vague experience in it are you all working on it small to use on new products yeah okay well that's great that's good to know okay so this the sister unit so we're going to talk about what we use the research is definitely what it is not to try and glass some myths why you should do it especially was going to give us some information really that about cognitive biases and how how we work as humans especially we do our research and talk to users as well so what kind of what might be going on in the users mind at the same time as you interview them well give you tips to doing it yourself afterwards and depending on the output site in the room we may focus more attention on that so this isn't you know what we could teach you also get all these take parts and research with us at the end of purpose Vasiliy this isn't a session on how to do a usability you're not going to come away from this where they templates or questions to ask people the reason for that is it doesn't exist that anyone who gives you that is not doing their best work it's always different people are messy this is about helping you get the skills the confidence to try to ask better nitrogen to get better results out of it so what we ask is that jamison workshop that we wait to participate fully in the activities i learn to keep an open mind and interrogate your own ways of thinking as well so just a little bit about us i'm georgia raucous and i design research immediate consensus with a company for a year DEFCON markzware bursaries so it's nice to be here again and i do use a researcher about seven years now i works at number different ecommerce financial government projects and before that would Europe's leading usability testing company as well a consensus I work across lots of different spokes helping do research at all the different users including with Sasha and I also help train designers to do research as well so hi I'm a special message right now I have a designer event different senses and enviable there [Music] I shine all the time user research and this is why sometimes I nice Bernie's think my opinions about users and why they have their own right okay so there are actually enough people's do this to pass which is a bit better than race so you can just grab somebody just to find out a little bit about them of their experience of doing research afterwards to each other eight years - [Music] as I have experiences with doing a one-year system did some of you already know that is that up - yes I did another industry set up a system what Dan was really growing in Dan's money get on that my Expressionism NBA's summer program the major Association is America well she's really American poets hear us graduation [Music] projects thank you it's my appreciate I just love tonight if anyone would like to share any any of their biggest pains doing user research or the biggest problems that they've encountered and then what we could do we can try and focus on that towards the end of the session it's a Taylor series so it's anyone got anything a dying to know about I think you said that you were interested in finding people to talk to how to go finding users anyone else with anything else that they're interested in all of it amazing okay cool all right so what is user research we're not going to go into too long about this you've all got a sense of it but just want to explain this is Margaret means she's a very famous was a very famous anthropologist and she famously wrote this what people say what people do and what they say they do are entirely different things and she discovered this by observing people in the natural environment and interviewing them separately and doing surveys to train loads of different methods this was in the thirties and it was quite influential at the time and also quite would come true controversial too because this was around the time where man was meant to be master of his mind right so so what consumer behaviour the science around that has proven this to be the case so people not very insightful about their own rather habits behaviors motivations they need to help eliciting that out of them and that's the job of the use interviewer so what is user researchers we're talking about qualitative research here you could go and do big data surveys we'll talk about that in a minute but it's usually interviewing or observing real people and what we're looking for is identifying these behaviors goals motivations pains all that good stuff which tells you what you should be building and understanding the why behind the what they do right what they do is interesting but it's hard to make informed decisions based on that so what we want to decide is what is the intrinsic motivation behind people's families and then the whole purpose of research is to have an action plan off the back of it right you can't be seeing these paths and themes across users and now what you're going to want to do is come up with some recommendations and actions and why should talk at the end about how you can you how you can help do that with your team as well so what user research is not it is not about people's opinions it is not about whether they would prefer the yellow object or the black objects right that it does kind of market research it's as well market research is about asking many people the same set of questions and actually you ended up learning a little about a lot of people the benefits of qualitative user research interviews is you will learn a lot from a small number of people and data analytics you know site usage what's happening with your product that is very interesting what data ok but there's two problem is first thing the project is that most apps or doubts or services don't have many users so you don't have data then how else you do find out what people are doing a lot of companies are actually not intrinsic intentionally not building it any analytics products into their product of the basis that they want to keep it pure I'd like to make sure that the users behaviors are not be tracked and things like that by any third party like Google Analytics so data analytics is useful but you need like a lot of users for that to make sense otherwise you're basically just looking at what all your development team is doing on your website and it's not something which I I've experienced before I thought we've had a hundred users 70 so something you only do once you build the thing right it's this is the wrong time to be doing research we need to be doing research at the start of the thing really before we build anything there is a bias towards action and a bias towards technology within blockchain within the ecosystem which is let's get building something okay but I question the value of building the one thing I really like to encourage people just to at least do some research to start things off yeah so why is being able to interview users important to you you probably heard the statistic somewhere between 50 and 90 percent depending on what article you read between 50 90% of all startups fail in their first year right and then the remaining last start failing and only a small portion last two year five okay so based on that information most people in this room are working on a failing product and one of the people just to think about that for a second right and like let's let's be open to this because we we live in a world where we have to with cheerleaders for our products all the time we're in our own little echo chambers and our bottles are yeah it's gonna change the world well we don't know that to be the case and the way we could do risk is doing research early so just get some done first right before we build the product to find out it's the wrong thing and that's the whole point of saving time and money by doing this it could really help align people around the problem as well that they might have a worry in this early stage of this pioneering technology and deserting the way the people of businesses interact with one another but because crypto got change the world is relatively small and highly technical is easy for us to think that our users are just like us right oh we're building a product who's the product for those people liking okay that might be true but it's get is a guesswork it's guesswork until you're gonna find it home right so I'm not saying that you're wrong but maybe just going about verify that because not everybody speaks in the same languages us and I mean like technical language as well as nationality and other language as well so that's really important and probably you don't have a dedicated user researcher on your team I there are many most teams don't have someone like meeting your session to come and do all the research for you so don't just leave it until you you know six months down the road and then you are you your designer then you get them to do some research you do it first I think it's a really great of an empowering activity so I can I know that is a still faster tech failures okay so how many on product many how many of you have done user research for product not so many it's okay you're not alone many tech companies forget to do that and they ignore user research and this is called tunnel vision and that's like the deadly scene design ignores users and users if users feel that they are ignored by design [Music] design so I will show you some heartbreaking testimonials so this is from a crowd they say we spend way too much time building it for ourselves and not getting feedback from prospects it's easy to get tunnel vision I'd recommend not going more than two or three months from the initial start to get in the hands of product and prospect so if you are working on your product and it wasn't liked more than three months it's alright you can feel it now and this is from Voltaire type we didn't spend enough time talking with customers and we were rolling out features that I thought were great but we didn't gather enough input from clients we didn't realize it until it was too late it's easy to get tricked into thinking you're your thing in school you have to pay attention to your customers and adapt to their needs so probably we all have been there [Music] so what does bad user research look like okay so the first time provide user research is no news of research you may be familiar with this quote from Henry Ford got a question mark there because actually the first attribution today was in 1970 this quote so who knows we actually did say so that I've heard this from people in love chain as the reason not to speak to customers okay there's a few problems with the history it's also like it's used to argue that like true innovation from comes from signaling gifted individuals who know better than anybody else right I'm sure we all we all know people who kind of behave it like that but the problem with using this quote as an argument is that this is not user research so what he's actually talking about if I had asked people what they wanted we don't ask people what they wanted user research that's capturing opinions people are very poor predictors of their future behavior okay so this is this is just bad research that he's talking about doing okay in addition Ford the most car company due to their lack of innovation so so they obviously created great products but then all the innovation that happened after that was about the production line to save money to build cars cheaper you may remember you it's also apparently said that you can have any color you want except black so what happened was for the cars motive the share of the market dropped from 66% in 21 to 15% in 1927 six years a massive drop in market share so you can use that as a fact next time that somebody says that we don't need to speak to customers the competition was innovating in the space and Ford was not and that's the reason why that happened maybe after today's workshop you will think twice about putting this out okay I look we've all done this at some point right I made a thing please give me feedback yeah that would be amazing so what happens what sort of feedback do you get who answers everybody people who are your fans are these people who already know what your product is right you're not really what about people who refuse to click on it maybe you should find out what they think because also talking to detractors is really really valuable and then this brings us on to surveys so most surveys are absolute garbage I'm really sorry to explain this to you now there's a few reasons why this is the case they feel like think they're a good tool that's because they do make good sense every company is putting them out there all the time you always get requests to fill out a survey right so by frequency it brings validity the problem is is they're very easy to create a bad survey survey design is actually a science right they're statistics that goes into it in order for it to really help you you need to think put a lot of effort into it but it's really easy just to throw one up or what should we ask them what you liked what you dislike lalala send it out okay and I think is often used in instead of user research because talking to people is scary right so it's like what the survey to do the talking for us and we expect the survey also for everybody to fit into these neat boxes that we've already determined for them but people are messy and Amorphis okay so if so it's not as simple as that and I'd like to just change this a little bit further it's also to think about your own use of surveys so when you get sent a survey do you fill it out why do you why do you not is it because you have a particular relationship to that brand what happens when the survey is too long do you abandon it halfway through what happens when you see a question that you're not really sure where it's asking you and you don't feel that surveys asking you the question you really wish it was asking you and then you get asked to rate something between 1 and 10 like what's the difference between a 7 and 8 someone's actually gonna do some number packaging around that they haven't thought about the science of it either right so they're not that useful ok I hope you put that across and also we are default to we have confidence in large sample sizes right so we think that them oh good well it's better than speaking to a few people when you've got loads of survey responses and now give us confidence about the answers that are there so firstly if you're how do you get large responses a big question around that but loads of surveys where you get like 15 responses doesn't really help you make decisions B question around that and also if they were to see is only true that they are give you more certainty if you're doing a robust recruitment method to recruit a very large group that represents the people in your users and then randomly select from them that's the way to have accuracy at large data no one does that when they send our surveys they just blast their customer base and hope that some people reply so we guarantee such your I put like put money on it that if you go to speak have deep interviews with five people you will learn a lot more than you will surveying and get no response for twenty people I promise okay so another type of bad user research is doing it with those who are easiest to access the people who love us they're great they're like our cheerleaders they want us to do really really well alright the problem with this is that they're probably too close to the products they may speak the same language as us in terms of the terminology that we use for things they probably have an idea of who you are and they want you to do well so they're probably not going to be very honest about it's okay and and and but there is a there is a use case for doing this practice your interview with them they're great people to practice questions with so don't feel scared to do it it's just these aren't your users ok these this is just other people's opinions as well so so yes I'm going to hang status to talk about them brain hacking yes so what we were that you know brain hacking we really get to learn down our greatest vulnerabilities and how to turn them into opportunities and into to put them on to our advantage so I will run you through some cognitive biases and feel free to raise your hand if you spot them and you have experienced them so what are cognitive biases they are mental shortcuts our brain thing this means that they are unconditioned and we can't avoid them but what we can do is learn how to spot them by knowing more about this matter and because cognitive bias is loving us all also researchers are there winking so why it's so important to know about them is because as researchers we we might spoil our our world so let's let me give you an example you are doing research you have some assumptions you are superb like you you're super super thinking that that assumption and because you want to really prove that you are formulating your questions like you know just a program so you are influencing and biasing your user by articulating not very well the questions so but not only researchers are the only ones affected by biases users are humans too so there you have to bear that in mind and when you do user interviews don't take their words like for granted just try to attend tonight so we'll start with the response bias and I'm quite curious how many of you have experienced it I handed myself many times the response wise is it's caused by the way we collect data this means that you can bias your entire research by the way you articulate the questions sometimes this happens when the development team conducts the research it's very hard to step away the objective especially the responses you are getting are not what you want to hear have you experienced experienced it have you went through that domain put your hands up okay a very severe drought I love it [Music] like everyone who works on the product this is the development Ian I'm not showing okay so get into our next bad boy this is the desire for harmony in a group which results in a dysfunctional decision-making we want to minimize conflict and reach consensus so we might suppress opposing viewpoints and isolate our team from outside influences so we love our product so much we don't want to hear anything like outside confirmation bias this is the tendency to search further interpret focus and remember information in the way that confirms our beliefs so this happens when you do user research I don't know you only hear the answers you want to hear so even though 95% of the users that use something else you only want to hear the 5 percent so you get some supervised information that that happens the million Alessio and we were working on our blockchain Explorer we are so convinced that our core users were just power users so we only wanted to hear that curse of knowledge effect when better informed people find it extremely difficult to think about problems from the perspective of lettering form people this is what we don't really use we we use the terminology and think that everybody should really understand about the think of it you're you're working in blockchain right and you talk to everyone with the terminology and expect them to really know what you are saying but it's not all right you don't have to make your user feel dumb don't challenge your user even if they are an expert [Music] social desirability bias yeah I mean this is actually very you can encounter it all the time thing when you're doing the research but I'm against blockchain me who can you exploit now here you go to rekall buicks and if you have a user who has never heard of go chain you confront them with play all the apps onboarding we'd make something sense this new user they have no clue what I don't know yeah that's alright nothing given I think that's one of the biggest challenges for us and what changes if you have people that know who okay let's make it successful I think this is our role to do education and social desirability bias is about users you interview might give you the answers they think you'd like to hear over there true beliefs because they want to plead you so this always happens to me and in order to avoid it when I do a user interview I tell them I never worked on this sign don't worry about the people who are working on these are not part of their research so you use them in so you make them feel more comfortable and be honest Hawthorne effect this is when people know they are being observed they might change their behavior they might think they are being tested in an interview instead of just listen to so you tip for this to be to tell them all the time and remind them every now and then during the interview you're not tested only the product is tested and they're there once there once happened to me when I was testing with the user he didn't know what an avi code was and I was started to be very reluctant to answer to my questions and I didn't want to feel bad so you can avoid them IKEA effect I think over designers and the development team is affected by this we place a high value on products we created or worked hard for let your users do something as part of the comporting process not too hard but rewarding so they can connect with your product but let's stick to that we place a high value of products we created every time you do a research or you can get back because you love so much so that will you tend to not hear what people are telling you and but there's also a reverse type of thing actually you can make it make these biases of an advantage as a researcher or a designer you're the designated messenger for bad news so you get the bad feedback and you have to send tell it to your development team and in order for them to feel positive okay I went somewhere I have invented this magic they're all left out you you invite them to take part in the user interviews whenever they can or witness everything that that's happening because UX has to be transparent we love transparency in blockchain right so we have to make all the processes transparent and what this is about the fact that everyone can do user research it's a skill that you can learn you're not born with it and you only have to get in the mindset of a user researcher and that means that you have to be a detective think of it you're the Sherlock Holmes of your product so you have to pay attention to all the details dig deep ask all the WH questions you think you need to own in order to get all the answers and the innocent also think that you have to be a psychologist you have to understand Wow how people are working and why are they doing that and just try to see what tickles them and yeah and also you have to be like on a safari you have to observe people but not to be an active observer just try to take a step back but to influence them with your opinions your biases this is not a heavy petting zoo it's just a safari [Music] okay so the first thing that we're going to do that this is where we can start getting into activities now so who all have your coffee and woken up so what we do to start any research is stating our assumptions and this is a really valuable thing to do with your team so the three assumptions are basically what do we think we're going to find out when we start interviewing people that's what we're trying to that's what we're gonna put down okay they don't have to be right you don't know the answer to these things but there are lunch there it gets like I think probably this will happen all right I think X because Y it's really good to do this as a whole team exercise so using that IKEA effect again get people involved early then they will feel heard and care about the outcome as well and then also when you do as a team exercise you start servicing where there is disagreement where there are conflicting views about beliefs that become the people in the company have those are great places to focus the research on so if you've got a debate in your organization about which way should we go with this particular product feature or whatever then interview people can help you know solve that question really so it states it like this I think X because X 1 and it's fine not to agree about them so it's just a basic example Sasha's gonna announce two things to you as well so an example research question right so let's say you're building a centralized exchange where's the gap in the center it changes now which could be an opportunity for us right quite straightforward so an assumption might look like this I think what's missing is a mobile app because if users can access the decks on the go this will lead to more usage for liquidity on the exchange okay so Sasha's freaking out some things here we need to get you two groups of three so if you can organize yourselves in groups of three please so we're just going to spend probably only about ten minutes on this activity and I want you just to work together to state what your assumptions are about the thing I've put in front of you okay so in your group you should all have the same thing yeah you don't have the same question right so how do we the research question and just write your assumptions down you don't have to agree with each other just work them out okay [Music] which is so trying to copy by the list at stake that because as well right so what we're thinking is people are going to interview someone about this what would we expect to find out [Music] the usual I got to obsess more yes they travel journey from the heart to the attention and I'll be sunset we know the forecast all users [Music] [Music] [Music] [Music] [Music] is what we think all right we can have the user perspective that I have that also we can have a researcher protects against yes yes power W over session will be very of us except Kennedy that is what I understood [Music] the user respect experience is your experience [Music] yes [Music] anyway what a teenager our cars beautiful ways that people made this a better experience means that expects for people so if you're gonna ask people [Music] experience I have this issue that everyone else that the same machine as well so that's a good assumption I got every season we where it goes first your muscles done yeah it's okay our solution interesting that's a weird question it's great when your establishments I think you felt about the expo is these other tracks not that the [Music] like magic just Helen yes I was speaking out here when I wrote my favorite cement food but then we [Music] and here it's a right away once you buy the Wi-Fi but otherwise I'm you have a little mouse yeah we always be like you [Music] [Music] the directions so I think that the costumes I was more focused on building companies to build travel for what comforts that most basic federal powers to commercially to this compressor customer reaching out there partner today I'd like to know what America yes yeah whatever II know what you mean it is not yes 5:27 yes maybe [Music] [Music] [Music] okay last minute last assumption and I know and maybe you're thirsty to face the different questions about you yeah I must've arranged everybody you can wrap up now some assumptions aren't down with you brilliant so we're going to go through now sort of the anatomy of an incident all right you will have in front of you some stickers so it's usually three roles in an interview facilitator a user and an observer use observers really useful as the person who can take notes the facilitator the observe it doesn't talk during it it's important not to have a three-way conversation you want a long conversation between the user and the facilitating separate so you know you probably guessed by now you need to assign it yourselves with the stickers so doing that now quickly just to stick them on you usually be a facilitator who's gonna be nice okay so so I'm going to talk through some good versus bad interview questions oh wait right that gets not surprised you're going to be doing some interviewing in a minute all right you have not prepared for this so it's going to be uncomfortable and awkward that's fine again it's just it's just getting a little bit of the experience of it okay so we've got handouts which we're going to head out in a minute which have all of this stuff on my cheat sheets for how to interview people what the questions are about questions so good questions they're open-ended they're things like tell me about subscriber time tell me more about this thing okay story time is quite good like walking me through the last time you did X when you started to the end but you need to be following up with why why is that how come tell me more about that right these are all very like curious questions that allow the user to answer from their own experience without recommending them right you're not telling them what do I hear they're very much like you tell me right and then when you hear something that's particularly interesting you can ask why about that particular thing so on the flipside bad questions are closed and with yes or no future casting everyone is dying to ask this question would you pay for this right you can ask this question if you want right but the answer you get will not reflect reality people are very poor predictors of their future behavior okay this is you can quite advanced market research to to work out what people are going to for your product and how they'll pay for it and whether they will pay for it and also the best thing to do is to understand the pain so if you actually really want to find out someone's going to pay for something try and find an analogous situation where they they've moved from a free product or paid products in the past what made them move you know what what was enough to incentivize them to start paying for something when you can understand their past behavior then you can start predicting their future behavior but don't rely on them to tell you because they will not necessarily get it right okay I'm leaving questions so we talked about this already quite a bit their questions which suggests to the user what you have in mind right and a very simple example of this right is do you like that do you like that is is a leading question okay it's also a closed question so enter the yes or a No and it's it suggests for the user that they shouldn't like it right then there are some varying degrees of improving on this question until you get to a really good one so this is kind of a bit weird do you like that or not you started out balancing that I there's two ways that could go and that this is like really do you like that or not like that you probably see survey questions phrase like this trying to not leave the user and be balanced but the best question is this what do you think of that allowed them to use their language to describe what it is that they're experiencing okay so this is the way to get really good stuff out of Google yeah sure [Music] instead of like it is nothing so you give the option to it also the negative but what if she specifically want to have like positive ones but if somebody's like always like negative things yes what an apostle but you know still you speak good good question so you can do that later on so the idea is that you start quite broad right and get information from them and then start going in it's a good question that might be what was your experience of that what do you think of that okay and then they start going negative negative negative negative was there anything you liked about that yeah what did you like about that that's perfectly reasonable but it's just best to staff allowing them to tell you yeah yeah so we'd be grateful for their time and honest with the purpose of the research research is not the same as selling your product and the amount of calls are being on where the product team is trying to pitch at the same time as doing user testing these are not compatible at all right when you're pitching to a client they want you to do the talking they're not going to be honest about their experiences to you why would they do that keep this thing separate make sure they know is this customer you're not selling than anything you want their feedback their their responses to the questions that you've got you need to be friendly but not friends when we're friends with people and we've all experienced this I love having great chats with my friends right but this has come to the social desirability bias they you will have a wonderful chat you'll all feel good at the end but you probably are not being totally truthful so it's about creating a bit of - about honesty to be there as opposed to just be chummy and agreeing with everything everyone says all the time let them do the talking as much as possible it starts slow down here okay let's compromise we're influencing them and like we said before don't receive technical or knowledged technical knowledge use plain language where you can that's quite important so we're actually going to demonstrate what human research interview might feel like so I'm gonna have a word to Sasha because she's going to be so we're going to do the playing so I will be the researcher I'm researching Fitness out and George is my user that I am going to interview I will start by asking her broad questions about her just to ease her in and find out about a bit about her context then I will start to ask her more questions about Fitness brought questions to see what comes out and listen and pick out some of the things she says and then start getting narrower and asking her more granular questions about the Fitness Africa this is my end goal but I don't start directly by asking her for that because I might not get the so many site that she might give me for I'm biasing her somehow she will think that oh I'm only used in Fitness I don't want to know anything about her or her life or anything like that so she will stick only to that subject so okay hi Georgia thank you for accepting my invitation for the interview I am Sasha I'm a design researcher and I would love you to tell me a bit about yourself I'm talking to some content about me I'm Georgia I work in travel and travel insurance I live in London I've got a husband and cats okay nice but what do you do when you're not working so I well I work from home so I try to get out and about as much as possible so long lazy at home I do a lot of reading you guys do like walking I hang out with friends and I also go do yoga as well sometimes so now she has mentioned yoga so I will speak on that okay some of you have mentioned yoga well can you tell me more about that yeah I go to a place called yoga space London it's not very far from my home and I kind of like that maybe once or twice a week like I like I buy a book the sessions online I just like buy a whole bulk of tokens and then I can book or the sessions I want so you're using a website for that London yoga studio you mentioned but what do you like about the website I really love actually the thing I really like about it is that I can cancel the sessions with like no hardly any notes it's just like two hours before because like what gets in the way a lot and I'd like people just dump meetings and my calendars so often have to move move my exercise sessions around there okay and now I'm getting narrower and where do yoga I don't do get a great around Soviet studio why is that I've tried doing at home but I can't really do this okay not sure about space in the house and also it's just hard to get motivated obviously I get many big words from a partner she doesn't have space but they not do because I but she also said she doesn't get motivated so now I'm picking on that and have you explored any ways to get more motivation so you're hurt on that before yeah I used to like fitness apps before but it's really such a send us stop I just don't really stick with them and it's very good to ask many wise try to find everything about what they say you know I'll ask her why why did you stop using that to be honest I thought like not a very modern phone and all the apps are just too large so answers end up deleting them starting to understand like some pains potential opportunities as well okay so they recently caught a pretty light weight so people don't have loads of room on their phone it doesn't get automatically deleted which is what Apple does now so just simply facilitates tips of the Train actually we're going to share these with you so the Firewise this is root cause analysis and you may be familiar with it it's comes from a new battery industry I think and the idea is something came off the production line that's Godfather's why is that we made a piece of machinery was working correctly why is that management had changed why is that but so by asking why many times you start to get to the real reason now obviously you're not going to ask the user why five times okay and you don't sound like a robot is supposed to be this sort of a conversation but maybe once or twice it will get to the real reason if you remember that users don't have this why always there at their mind at this top level it's not readily accessible so you have to ask it they're not going to necessarily tell you the reason why and that's one is the echo so you know then you can encourage them to say more perhaps they've said something a lot we surely don't really understand you could just kind of repeat it back to them and they'll hopefully say some more kind of on a similar thread the boomerang so this is really good in user testing and if song goes that they're trying out your app and they're like really I click this is the kind of thing to a confirmation page and yeah they're asking you a question so your automatic responses to answer right don't answer the question just ask it back what do you think will happen when you press that one then they will explain okay so just so like any questions that come back at you throw it back at them silence I'm really bad at this that you composite could probably tell I like talking or tuning in or quit silences but the more silence you can give the more other people will do that so they use and they will say something that extra little nuggets and there are many times of user research interviews I've done where I just use a technique where I'll just have my back that said something they've paused I don't just go straight to the next question sounds like there's something else there so I just write some notes and just look busy and then they just continue talking and it's a real waste of McArthur which is really exciting but that does what and then this is the question right about you know when you when you're just not getting what you need out of them right you use the Columbo method alright so right okay so would you ever use it like what what about these fitness apps you know right just like get me out of them right at the end because you the purpose of the research interview is to have some insights okay if it's not going anywhere that they're not seeming to get the questions that you're asking you can ask a more direct question but just maybe try the other techniques first before you get here and everything so Sasha is giving you copies of all of this stuff you can take away with you it uses your cheat sheets for when you're running user research interviews but so you're listening for the reason behind the answer not just the answer that they give you habits behaviors issues annoyances opportunities all of that stuff is real gold dust and in this session that we're about to do now in the workshop we're gonna interview one user right but really you do more than one user so then you're looking for themes and patterns across those users I will will talk about how many users and a little bit later on so I know there's some newcomers here hello you've missed out on the first activity that that's fine why I suggested that you join a group and be an observer in what's going to happen in the okay so let's try it out users you got your user stickers on yeah okay I want you to a platinum band you guys aren't going to stick us up and please please open it up because you're going to be reading around so it's easy for people to identify it so you're not going to be in your groups right the user the user from each group abandon your abandon abandon ship you don't need your assumptions with you move to another group okay so the users are gonna move around so the facilitators Center lat so facilitate sandals you're gonna do some question are statements I haven't prepared you with questions it's not going to be easy right this is just an experience for speaking to somebody who probably and does not see topic that you're going to ask questions that maybe I thought okay so just go ahead let's do it for 10 minutes remembers let's start with those broad questions get to know them start narrowing in on what you want to find out observance please listen carefully and note where suchan's are incorrect or to enter okay it's [Music] I would say we're working on the title makes an int or which is what you it's not like that we have a conference everything is happening so I've evolved directly into a fight [Music] sometimes hours to recharge I mean what's that feels like it's gonna think it was hired I have to go back to my hero I just said with these or there's no the compression might have to be conquered and that's the don't worry in ever [Music] I mean not certain that world to be available so you don't have to be around to watch they enough to talk I know that Melony but still think the secular is the better yeah by water experience pangs in the end with people [Music] that's like two minutes what [Music] yeah [Music] [Music] okay everybody thank you thank you for so willingly getting to the rolls I really appreciate you that's nice so we were originally going to do this so we were when you get back into original groups but I don't know whether it's to necessary I'd love for the people I'd love to go around each group remember to share the experience I'm interested first of all for each team did you discover whether any of your assumptions were true or not what about you guys over here and it always they're all surprising experience [Music] [Music] [Music] [Music] they're like a lot of assumptions that we wrote down and especially to be both like the reason I did we use and you felt like beep touch both of them but for me that's the server I think the six example that the speed-dating coming in give it also your attack reaction you're like what the really notice W was like you so I think that also if you're asking questions you should really keep it like yourself and see if you can try to that validate them exactly this is why it's helpful not to do it by yourself is to have an observer gotta keep an eye because you need that that person who's not trying to stall the information in their head I've also asked but not leading questions it could be quite trying to do it here but in the end to say like okay did we check our assumptions I don't think we checked our assumptions better you didn't rise in a knowing you require a discussion guide which is and I have the questions in it that will help address the assumptions right I just learned it was on you cold so it's no surprise something everybody writes about but well done you okay yeah so I submit my head what about team but did you have the same question say makes us question about that come attendees yeah okay I'm done within your group and that's good right but I mean obviously you wouldn't then just take the information from one you say you would do this across many and as you start to hear the same things that raises the confidence very high yeah and what about over there and back [Music] or research question was about buying experiences that accompany this and like what the assumption which we didn't like that I said everyone bought the ticket just like here's the way let's let's act like just the ticket page but I'll use it actually take it from another person former sponsor that what you think of responsibly by the selling services this is an example of getting out of your problem right this is where we think everyone's has the same experiences us but we can be directly challenged doing this kind of thing yes and over here and this team yeah so what was your research question I think you know my listen westeros what does that experience like for someone traveling to another country for a world conference and what might make that experience better so I mean first of all something you have experienced is that asking open-ended question is very challenging especially if you don't have that experience you will tend to ask do you would you yeah exactly it was the over or leaving video so it's a lot of practice and also I found like going with the assumption also not straightforward because the user could also steal the conversation because there's a second there's a you you have some assumption but the user could deviate so keeping the conversation in the control aligned with the assumption also I found it a bit yes that excite me because there's no Experian I mean wait yeah I mean you you've come not prepared at all so this interview right so in real life there would be some preparation for this you'd have a kind of a script of questions they also you do have to be flexible enough to go off-script because you know if the users tell you something absolutely fascinating basis I mean follow that you know go down that rabbit hole it's worth it's worth doing it but yeah it's practice and practice with people you know the last point in terms of assumption hit rate I would say we were like fifty percent were fifty percent like correct about our assumption 50 percent indeed that the user has different different thoughts and different opinions importantly so we were not like completely yeah it's pretty fine maybe politicians so I think we it's anyone got any particular so I think nice example with us was watch the questions was like okay what do you think would help you meet more people would you like don't I open and questions Fitzgerald in the right direction and then directly after was like the bias instead of like answering the question was like the best for example we feel the workshops are helping so and then directly you're feeding him basically okay out what if you love warships so and I think that's important like if you sort of like ask a question and there's this silence it might be two separate versions thinking back to think and then that's the Columbo question at the end right so what about workshops do they help you write you know and you're like trying to find this app but to do it give try the other options first exactly yes I think so my language like be patient curious based on your experience how do you analyze that would you get here so are you describing the observer should take all the notes I also take notes as a facilitator because it helps me kind of process parts of the interview too like I might like underline something I'll come back to that things like that you can we'll talk about analysis in a minute but it's a matter of plotting your insights across your users right so any kind of matrix like a spreadsheet is good for this but then also we'll talk about this if you're we have distributed teams so you want to collaborate things like mural is great for that as well right so thinking about where everybody can see all the information especially if more than one person is doing the exodus right be able to put it all in one place and actually getting the insights into that thing is the long it takes the longest coming to the conclusion can be pretty quick when you see it all there it's sometimes if I go it's light bulb moment yes that advantage to do it got the conversation yes it is talked about that you may say let's just I just want to hear from the users what your experience was like and then we'll have asked about questions about how to do about this so what was it like this all right I mean I use a researcher so I could I could see the Sun is still I think it was I was not living by facilitator in any moment maybe this specific question and then some clothes issues also work where the [Music] time yeah what about you use them I think I had a little bit of part of additional perspective because I'm in this session and I actually I didn't fully process what was happening I thought that they had the same topic yeah and so I was waiting to get those questions which maybe is an insight about really users experiences they come with opinions time's yeah and they think they know what you're gonna ask them and they're excited to talk about it and then this is like a completely different conversation she's already the topic great okay any other insights from users what what was the experience like no [Music] okay great so we hear a lot from people that they want the tap play on how to do user research right and it feels a more personal and scientific because it's dealing with real people people are messy so often companies and I've worked in some of these companies as well they cling to tools and software user testing platforms hoping that if they use the right tool then they will get the right results out of their research right and there are loads of platforms and tools out there for doing this stuff companies is a building on demand right but user research doesn't need software to be done effectively sorry to tell anybody here so overtime it but in my experience my talking as the munition a lot so I usually use of these platformers and stuff but now it's just quite simple so I'm going to talk about how we do research at consensus and I think maybe we could just make this an opportunity for questions and answers right as we go through right so you might want some tips um or ask for more information on air as we do and so you know if you can get the interview part right then you really don't need to rely on any of the other tools and software because you know you can use you can use a user testing platform but if you ask the wrong questions then you get garbage outs so it's about being able to do the interview as well it's the first important bit it's also the scariest bit and the hardest bit so that's what people like to avoid and hope that the software doesn't for them so we'll just show you how we do it here so there's a first of all is about who to talk to and where to find these people we get this question all the time you really need to be people outside with your work or your bubble being totally honest that consensus is a really big company and there are still some projects who only test with other people in consensus right these are not our end users or they might be but how could they possibly be all of the users you know that's pretty neat audience that you're building for so finding people to interview is really important so for example say you're building a product for dark developers like recruiting those people at a conference like this is a really great thing to do it's a good way to do it because probably there already you know it's close to your target user and you have a conversation with them to establish whether they are really what the right kind of person and then you do you don't do the interview at the conference people not in the mindset for taking part user research they'll say yeah I guess and then they won't turn up for the thinker's only something else more important that they want to do right so just get the contacts make the connection and then do follow on with it afterwards we use I would not paid but to promote them but I love the respondent do also using it interviews comm they you create the screen and you specify the type of user that you want they go and find those people present them to you in a list with how qualified they are you can check their credentials check them out on LinkedIn they're real people you can message them to us and clarifying questions and it's a scheduling tool for all your sessions as well all right so and they process the payments as well for the user so this is indispensable while successfully use this to find space enthusiasts in Kenya soft commodities traders in Brazil that developers people who use maker down right you can find these people at these platforms so give them a try it's also free to just launch a project project you don't pay for anybody until they've completed it so is even you can just test to see whether you can find the types of people that you're interested in talking to or just give it a go so really all you recommend that snowball recruiting is a method where okay you found that perfect the perfect user after that at the end who else should I don't talk to it get them to do the improvement for you right so just passing out handling that honors and pay them for their time there's two benefits to this right some people say oh well then they're not really interested in my giving feedback itself absolutely that's exactly what you want you want somebody objective who doesn't care around your project right they need to just be it's them establishes the terms of a relationship right this is a trial action between your your sharing your knowledge and I'm asking you questions in addition paying them means they will probably turn up this is the biggest pain in the ass for a researcher for your research participants to not turn up you can't just magic another one out of the air for that power slot you've got your observers lined up you know you made booked a meeting room somewhere as well so is well worth it yeah five dollars no is absolutely not enough money then that's just the failure that's just the reality of some people right you can't necessarily access every single person so understanding what the incentive is that was going to attract that person is really important I've done research with like CEOs of major big or corporations they don't want money they want a charity donation in their name to think about what the alternatives are be aware if you use bouncies then you're only going to get users likely to the Bounty's platform we already going to get people who use the right so it's quite small and its really annoying because so when I don't be to be interviews I sort of a phrase it like I want you meet your expert opinion right but people love being heard and they want to be considered the you know the industry somehow have to react to their world they're you know pearls of wisdom right but but it burps but yeah it shouldn't be a two-way thing right it needs to be I'm researching you it's not useful to you if there does not come out with those questions for you so it's about establishing that the rules of engagement right this is a research session I'm going to be asking new questions about this I pay people like five hundred dollars for an hour at their time depending on who they are right and you can find you you can find people in specific roles and specific companies through these platforms as well so it depends on the other thing as well there's always going to be a bit of bias as to why people are motivated to take part that's that's just reality that's just real life you know but speaking to a few of them is going to raise the confidence and responses there any other questions about recruiting or paying people okay just be aware of you do insist on pay people have done or easy you're only gonna get people who are happy with that right so just consider that also I I don't do that because I just I live just next level I couldn't get it financed to be able to work out how to receipt it properly so just like you never seen it announcing it coming why don't you hate the company or not how many people so you probably heard five is the magic number for users right there's actually some science behind it and it's it but it's quoted incorrectly I'm not going to go to loads of detail about it now five it's it's like how long it's a piece of string five is a good number to start with per user time right it's not just five one want that developer one crypto trader one asset manager you need to do five of each of those right to understand the themes and patterns okay but you really do it enough that you stop learning right no users no insights one user loads of insights two users not twice the insights because there's probably going to be some overlap between them so you're going to get diminishing returns and I'm working on a study at the minute about I defy users people using maker and compound and things like that and we've got two about twelve and now I'm not learning anything new right so that's really a good indicator that I can stop now formulate these insights we make the recommendations to the business and it's about doing this little and often so if you imagine that your company is like a ship right you've got your end pointers in sight or on the back right but you need this constant course correction it's much much better to do a few users often than it is like 20 users studied at the start and they're never doing it again right because you'll probably be well off course by the time that you get to launch your product so just doing a little bit often helps correct and gives the people purpose and alignments around what they should be building - any questions on numbers cool okay so the set up alright we've just aimed in person if you do it in person that's great there's pros and cons for both of these things so I do most of my by video call the reason is that like we're distributed company I work from my home and sometimes I get like Sasha such a world dial in with a camera off you know and sound off and she'll be the observer right and I'll explain to the user I've got my colleague Sasha on the call she's just taking notes for me and so it's not a distraction having like loads of faces popping up right summation of videos on yes so why would you have those second first anything at all it's already recording it I challenge you to go rewatch your videos it's very time consuming it most people don't watch videos because it doubles my time so I do the interview and then I would have to watch the interview again so if you do a synchronous it's also and people don't watch but people just don't watch the videos I know we said some people might some people might in my experience I think you know the CEOs weren't really interested on how to do it send them links they don't they don't watch the videos right so the video is there as an artifact for me to rely on in case where you've it up right in case I can go back to them what the hell were they saying when I my notes don't make any sense in this part I can go back to that you might make like a highlights real for people if you really want to do that that's quite that's very time consuming but also very powerful as well willing to behave differently when he knows that his life never that doesn't record absolute they pay attention right so when you're watching your video but you only ever just watching a video you're on slack your email texting whatever right The Observer has to pay attention exactly when I get there and there's some training that there are some positives to doing video calls as well because it requires less less of the user to take part they can do it in a time and place that since then they're booking with you and it's it's easier for them to take part if you require them to be in a location you have to pay that loads more money to turn up and also there are some problems with inviting them to your office so let's say you work it's really cool tech company godlike it's alright really cool post-it notes everywhere and like cool furniture and stuff that can all influence the users perspective of your ground and your company I've used like like look like meeting rooms by the hour and like anonymous sort of places I've done that before meet them in like a coffee shop I guess as long as it's quiet enough for you to have a conversation without it being too disruptive that could work as well but vehicles they work and we work we've got global audiences the old customers all right so the days of the usability lab of well over right that's old-school approach to doing testing and interviews we don't need those anymore there's better ways to do this that's much much cheaper and it's doing this well there's a difference though between you doing usability testing a very structured sequence and you're also eye tracking of you're seeing where well button they're looking at before they click that would be the old-school usability microscope compared to this is more subjective fluid is that correct yes what we focusing more on that on the dialogue element but I have you know my tracking looks cool but I that tells you what not why right so I tracking says other user look to that and they look to that limited than that but you need to ask them questions to listen to why behind it so you still need all that other stuff I personally not a fan of some of those things as if you're marketing of course at UCL in London around this where there's like movements into you know by putting my by putting people through MRI scanners or stuff to try and understand their brains and their behaviors and it's still guess it's guesswork at this stage it's just you know went to the course and say well I can't to be honest just interview get the sacred sites service yeah but no we there was a place for usability studies because there was an easier technology to do it so is it trade-off right you might get the most in-depth usability report possible in a lab right but it's that worth in it than just doing you know a few users often there's a trade-off for things to balance this one question is you know I've been both worth it user experience design management and so on and I've also been a stealth subject for a big software company as I was a I was the guinea pig but didn't abuse those people with the whitecoats and so on yes scary though I went through a sequence and I did my own thing and then you know said goodbye their product was for yes this is that over reliance on technique and science and software I think we put a tracker on they will know what's going on in their brains it's not necessarily true you know it's just having able to have the be able to ask these questions as well yeah things like watch are there maybe perhaps well so kind of things firstly the hot jar you need users so you mean as it known as it not like it gets a test you actually need a large song poor people to be doing it so one of the problems with that is they've you're early in the crypto space you may not have many users in which case you just not gonna get a lot of insights out of it basically the same problem as Google Analytics using about your platform is watching one session which is extremely useful so you have like basically one year ago to a global command and seeing how they do things and if you watch like them like when you session to get a better feeling of how people are yeah yes so the real benefit of that is that those are real people doing real things at your site it's not a made-up user test right that is absolutely beneficial the only thing I would say is it needs supplementing because it's still up or what it's still one more else are they doing normal I yeah but absolutely I agree that the question is you know because you don't understand their context you don't know what else is going on in their life and I've watched some of those videos and then then they're not doing anything for a while what are they doing they got a toilet like you know what's going on in that time I don't know why they make pauses when they do why they click on things that they do what they're looking for when they start but they can be interesting I think this is like that's what they write and or elements of it sort of bridges that it's like you can dive into that data to seal the room you know level but also the collection I think all successful companies these are both of those things that you could you can make your assumptions based on information right he's like Oh everyone goes to this page and then they do this thing that's weird I wonder why they do that and then go find out yeah exactly yeah so I did this proof above a shop so the last thing is about couple more things is video conferencing right so we zoom but the amount of target service generally easy but the amount of times that I had a session where the end user has not been able to use the software that you demanded that they use right is like it's it's just after that what's easiest for them and you do that right so the easier for you to set it up at your own to make sure the the you know spend the first half of the interview helping the work out where the share button is right so this is quite a good tip it's just to to do go where they are in terms of the software that's all use record it but do always ask permission beforehand so this comes to that is really beneficial to record it so you can come back right don't just ask them at the start of the session but by the way I'm recording in ask them like when you're recruiting them okay because then they'll always say yes if you land it on them last minutes then people don't have being surprised about that kind of thing at so it's a it's a good idea not to just yeah like it on them like that you might have consent forms or nba's that's down to you as a company I mean so I think some companies really overhype their own the the information that they think they're going to be sharing with this the user and rightly you're asking the user all into questions hopefully you're probably not going to tell them loads of stuff that you're doing if it's secret right so just like it's a maybe bit heavy-handed in some context some companies insist on it speak to your legal team if that's the case most of the projects I work on we don't use FDA's so the last bit we're nearly done I think yep so it's just about making sense of the findings right so you've got all these patterns of things now that are coming out you're going to create recommendations and actions so you go to come up with I think we should be doing this that's and it's good to do this as a team as well but what you need to do is prioritize them because you're like especially if this is your first research project you know it's like low-hanging fruit you're gonna find our loads and loads and loads and loads of stuff you want to do or change about the product in which case is worth plotting it on something like this so this x-axis is like technical difficulties and up there use the value or remedy so what you can do with your recommendations is as a team decide where what quadrants the recommendation goes right so if it's low technical difficulty but really high using our annual revenue it's in the top left easy and important those are may be the things you should work on first stuff there's high technical difficulty and high value important that's big so maybe work on that in the background and you've got stuff that's low value and low tech difficulty not important to be working on like wait there and then you can go to the CEO who's been demanding this feature here in the high technical difficulty learn user value and say look we've plotted it's it's not worth doing right so let's focus on the top of their things so this is just it's really simple activity you can do in like an hour from the research that you've done and it starts getting all the T might go alright that's the stuff we're gonna focus on you can even do it in that session who's gonna work on what create your tickets for it right further for whatever you're doing so it's it's not rocket science it's just getting to this point and remember that it's not it's not certainty it's always about course correction okay you you're gonna have new assumptions and new hypotheses that come out of the research that you just done oh god we only thought that's no way to explore this to research usually leads to more research but so so yes so we've got like 10 minutes left the last things just want to do is just give you some what we recommend as great books so the first one Daniel Kahneman's Thinking Fast and Slow it's all about cognitive behavioral science the cognitive biases that we have of why people do the things that they do if you can if you can read this and you'll hopefully have a slightly better understanding of how people respond to your product use a person and how you respond to the world the book in the middle is an excellent resource for learning to do UX research especially if you don't have a researcher and it's great it's easy to read it's quite entertaining and I recommend it and then speed or particles interview users is more detailed about uncovering insights through into young people or techniques and the very last thing is we need participants consensus research that we like that you can't hide so if you fit any of the following criteria that's all please email design research at consensus don't mess or scan the QR code I'll take you to it it's pretty fun that we got out of our bubbles okay I guess this is sort of a bubble and Epcot but this is the most bubbling it gets but you know this isn't specifically looking for people in like this region or who are defy users or adapt developers you don't have to be all three of these things right just any of them drop us an email and that would be really good so I think we could maybe just take some questions in the last couple of minutes if anyone has a name are you going to share this slide this one the whole tag and we can do but I don't know how what the method is for that sure if you want yeah everyone's copy than yet so again based on your experience and well I think what can you work and in your experience was different research methods you can you see specifically for work free if you see a different surgeon to see a mere human see I can go first and then you yeah so my personally no I think the attitude the attitude the attitude problem thought the teams is different but the techniques are the same the youth the people might do the same as well and we don't need to reinvent everything that we do right there are some try to tested things that work or technical comfort or big tech companies doing this so I what my experience is and that's why we're doing this workshop here is we've seen lots of crypto project teams just not not be down to try and doing this because it's not in their wheelhouse they haven't had experience doing it before and they're on the assumption that it's blockchain is so growing you no one knows what they need so there's no point in asking them right and that comes to that this problem what I know teams will be building things for two years and haven't found a use case for it right what a waste of money and effort time [Music] I mean that's what I get don't you think it is it is still viable to conduct user research in each stage of the product life cycle yes absolutely so we're encourage you to do it early but it's like I guess better late than never right and and also we you know enough promotes of products with like really big you says that turning over 200 million yeah but we do testing all the time one big challenge you need to there's a lot of friction with project managers with developers with all those type of people what is the best way you could confess the team that we really need to do that user search how to is yeah it's about you starting the first time is difficult I yeah because people are like we've talked about the biases people are resistant to hearing feedback or they they're not quite sure how it works and what they're going to get out of it my advice is definitely to include them don't just do it on the side because that's going to make you look like you don't trust any of their opinions right and then you're just going off and do some secret research and then you're like wow I suppose these users like you so that's not to be an approach this book the one in the middle addresses this exactly right it's got like that you can get this free or - it's like right but by a book and he's it's also forgetting buying absolutely the most compelling experiences that I've seen are insisting that the person who has the scoop is resistant to research observes all the sessions right get them don't rely on them to watch any video they won't watch the videos I get your the observer comes in they're here and once they start hearing from people they can't button and not allowed to say anything during the session they're just really like okay Rapids do more of this right but it's getting to that point which is tricky we're hoping that by you know what what in consensus design there's quite a few researchers we're trying to do this with all companies so I like to I do like a little bit of like I'm going to talk to or spoken to gnosis before about doing research to help them understand like some of the benefits of doing other they're already doing in so that's good so it's just about education in being district and I think a lot of sort of trouble I think a lot of teams are you know lots of people made lots of money and I SEOs and it's got nothing to do with their ability to run a company and as a result they're necessarily are the right questions so I'd like to add to that that you know companies were first of all division is not clear that the strategy like they're the main objective or the main key technical some of those they're just ready how to get there and then if you go further further than like use cases and also like non-existent yeah building very cool yeah it's really cool but cool is also do discomforts a lava and feeling else has to do a bit with the maturity of like a maturity level exactly yeah really makes sense to me because mothers are a lot of young people yeah so we're gonna I'm gonna waited to create a consensus Academy like the modules I'm doing user research up and trying to get out in front of people so so they can just try and you know these skills and sales you know the UNIX researcher your job was to educate doctors and educate your team and then why is so important so I just wanted to dub it down to more course to do you say if you have a resistant or that existing team member you can put him to start as an observer yeah so that's observer is like the most comfortable position among the three exactly and they don't have to do anything like take notes and then just listen I take that it's yeah so it's true twice I'm no taking is that when it down right everything yeah and I just write it all down every single thing that said because I'm you can interpret it later yeah just take her all day after like after position you didn't observe or do they do they do they like ease by beginning they they are very resistant but after being observers they change my advice is to get them to listen to - or home because one of the things that they can do is just latch on to that one user that they permit of God right we need to change everything right because it likes a really good session and my answer was ready for both but of course that's not use research used to be to a few people so just like more than one very original bias they might hear do you remember that user he said it's okay no I mean you know at least I want to listen to use this as a star right even if they're hearing what they want to hear it's a really good start to get them interested in use they're doing the sessions - okay any other questions for me rather particular what he comes to need but learning for me is where I should not really hurt people in this room even though I love you to think we are what's here what works for you guys in terms of like to finding you people around thieves like general consumers I'd be morally in favor of you know using products that 2x might say as opposed to others more like recruiting people around things really yeah so ignore demographics demographics are not really overrated their proxies for for that what you're actually interested in who cares how old that person is right and you know how much the household income is if they are behaving in the way that you're trying to find right so within my platform like this or with any recruitment use a screener a screen and a series of questions to determine whether they are the right person for you so for example I was finding defying users this is an example I mean people who use make and down stuff like that I ask them first of all do you do you have cryptocurrency yes or no like like a kind of starting for award getting more specific just this as you doing interview itself and then like what coins do you hold allow them to like type them all in so so you can tell whether that sounds like they do know what they're talking about or not and they like which of these things if you use before a long list of all kinds of different apps and there are the defy products in there and they have to have selected one of those in order to qualify for the research so that's the behavior behavior is how they've done this thing in the last six months that's what I care about I don't care how old they are you know a lot city they live in that's not really very useful information again anything else well one thing is in issue is you know I think user research is great primary source understanding you know there's a separate function lots of companies is totally silo to break and that's really frustrating I had to build relationships with the data analytics teams and other company that worked out probably you're probably on a smallish team that it's not such a problem to be able to reach out to that person but we should be working together don't really see why why that hasn't been the case probably if you work encrypter or depends on what your product is if it's a private product after launch you probably don't have our analytics of but I kind of a say they need to be working together you need to perform your assumptions your hypothesis from that you know from what you're seeing where people what countries people visiting for are mapped you know where they click and how often they dwell time of things like that or things like hot you are as well and they can help form your hypothesis but it won't tell you what the intrinsic leaned is of the user which is the thing you can improve on Center you can build onto a prenup right okay yeah just drop myself just drops me you know you're interested in buying research come and talk to me afterwards if you like I may be around for an hour so I gotta fly to avoid my foods [Applause]