In-car user testing. How to test a mobile app with drivers?

In-car user testing. How to test a mobile app with drivers?

Natalia Pięta
January 03, 2018  | 11 min read

Parkanizer is an awarded parking application for which we conducted very interesting in-car user tests. In this article I would like to share my knowledge with you for the sake of next user-friendly mobile applications for drivers.

If you have any questions along the way, don't be afraid to ask in the comments!

In-car user testing. Preparation phase

Do you know the Pareto rule? 80% of preparation and 20% of work give great results. When conducting the in-car tests with real users preparations are the essentials. One needs to use the imagination to ask the right questions in the right momnet.

So before research, you need to have an overall overview on what you want to study. In your imagination, try to recreate the natural circumstances and route for your scenario as much as possible.

Try to imagine how would the test looks like in details

Here are some questions I was asking myself before conducting in-car tests for Parkenizer application.

  • Will I conduct the full test in the car or you can I start it in the lab, office, café or at user’s home before we hit the road? Maybe I should invite the user for a cup of hot coffee after the drive to sum up the test?
  • What route would I like the user to take? Do I want her/him to drive the usual way somewhere or go to some specific location? Will the user drive me back? If not, how will I get back home/ office from the test?
  • Will the whole test be recorded? What will I record? Will it take place at day or after dusk? What is the weather, so also the light, going to be like on the day of the test?
  • What recording tools will I need? Where will I place them? How will I make sure they will not run out of power or storage space during the test? How will I carry them before and after the test?
  • Will I need internet/GPS access during the whole test or only at some stages? How will I make sure it is available in different places? What will I do when the access is lost?
  • Will I use my phone or the user’s smartphone? Will it be paired with the in-car screen? Do we need any backup? What happens if the smartphone battery goes low?
  • Where will be the smartphone located? Do I want to record the way the user uses it on everyday basis (so sometimes against the law) or would I rather concentrate on the safety requirements?
  • Where will I sit in the car? When to ask which questions? When it will be best not to ask questions?

As I mentioned, these are just some questions ;)

In-car user testing. Where to start and finish the tests

There are different logistic, but also technological factors that will determine the best place to start and finish the usability test with a driver. I recommend considering at least 4 of them:

  • In what circumstances would the user usually use the app?
  • How would the user normally start using the app?
  • What is the stage of development of the app for drivers you want to test?
  • How much time and effort will it take for you to adjust to the user’s natural use scenario?

The design of mobile application should be tested in variuos circumstances due to gaining real-life knowledge in different situations. The key is to recreate the natural circumstances and route for your scenario as much as possible.

Example: If the user should use the app at the way to work, try to accompany him/her some day in the morning, when he/she actually drives to work. Then it would be natural to start the test at the user’s home and have a cup of coffe before hitting the road. Insights about the context of the service you are providing may turn out invaluable.

Pay attention to details!

Again, your application must work smoothly in different driving situations. Observe what activities a driver does when driving. He or she can e.g.:

  • turn the engine on/off,
  • find the nearest gas station,
  • stop a car at the lights.

Of course, there is much more. If the app is usually searched for or turned on in the car – start by sitting in and let the driver show you how she/he does it her/his way.

But if you want to test something that would naturally be done out of the car, you can do it in the lab or any calm place where you can sit down with the driver.

The place your test ends also influences what information you get.

Driver's focus and satisfaction

Another important factor is driver's focus. In-car user testing may include situations whit talking, not talking, searching for a specific place in a hurry, using the app in the traffic jam and so on.

Collect some opinions about the satisfaction with using the app. Talk with the driver for several minutes in some quiet place and sum it up. If he/she is in hurry, ask the user if you can call him/her in the afternoon or tomorrow to ask what he/she thinks after several hours.

 

  1. What is the stage of development of the app for drivers you want to test?
    If you have the functional app, do not hesitate to adjust 100% to user’s habits when possible. If you are checking some functionalities with existing users – ask them where and how they use the app when you recruit them and try to shadow them.
    If your application is by now only a bunch of sketches or mockups, it may be wiser to start in some calm environment to teach the user a little bit about these artifacts and instruct her how to think about and interact with them. When the user is ready, it is high time to move to the car.

     

     in studio user testing app If you test the mockups, think about introducing the user to the research protocol in some peaceful place.

     

  2. How much time and effort will it take for you to adjust to the user’s natural use scenario?
    Think about the user’s comfort, but do not forget about your time and budget. If it is not a problem to get to the user’s home or office to start a test from there – do it. But since she drives a car and you are going to go to some place she does not know either way, maybe she can pick you up from the office or drop you off there at the end? Or maybe she can imagine, for the sake of the test, that she does not work in another city, but in the office centre just several streets away from your place? One thing is for sure – at the end of the day you do not want to land in the middle of nowhere in the night with no taxi or bus home ;-)
    You are always trying to balance pros and cons of some research decisions – just make sure you pick the right combination of factors that make it possible to validate the product and get a valuable feedback from the driver. For me – it usually means getting as close to the natural use of the app as possible, on the routes that the drivers would pick themselves.

Pick the right tools for in-car user testing

Think if you really need a video recording of the whole test. If the test is short and you will check with the users only one or two functionalities, it may be sufficient to record the interview with the voice recorder on your smartphone and use your smartphone as a video recording device in the crucial moments.

Just make sure your battery is high, and you have quite a lot of storage space for saving the videos. You do not want your smartphone to black out just in the middle of the test ;-)

 

in car usability testing with phoneThe most important con of this approach is that when recording the video with a smartphone, you set some kind of a device-barrier between you and the user. You also start concentrating on recording with your phone – it consumes part of your attention, which makes you a less concentrated and less attentive researcher.

From my experience it is better to record the user tests with drivers (and user tests in general ;-)) with a small portable camera used together with several types of grips.

user testing go pro on tripod Tripod is only one of many grips available for Go Pro — it is perfect for recording at studio.

 

At Skyrise we have used a Go Pro HERO 4 Silver for the last 18 months – it is just sufficient for our needs. It is light and can be used in every circumstances – that is especially important if you conduct the test outside the lab and as a single researcher.

The quality of videos is decent and easy to control (from 720p to 4K, we usually use 1440p and medium or ultra wide field of view), one charge of the battery lets us conduct up to 3 1-hour tests without recharging, and the camera is small and quite easy to use, which is extremely important when you travel or need to use it in unusual circumstances.

Extra tip: Pick the lowest quality possible which looks OK for you for the recordings– you will save storage space and energy of the camera. If you need, you can use the mode which includes both recording video and taking pictures every several seconds. Sometimes the set of pictures may be easier and faster to use in the report than cutting the parts out of the video.

Remember about the grips

Especially in the car, the camera is nothing without the grips. You do not need to spend a fortune on them. Use the grips that can be easily matched and reshaped. Most of them is small and will fit in most of the laptop backpacks.

When the table is available, the tripod works just great. When I am in the car with a driver, I usually use the multi-part grip with the sucker to place the camera near the rear-view mirror, viewing a full windscreen with the road in front of the car and the top of a dashboard – this is usually the area where the drivers keep their smartphones. If I want the camera to capture also the driver or other parts of the car, I can easily change the angle by manipulating parts of the grip or put the camera in the top right corner of the windscreen.

 

in car usability app Grips can really help when installing a camera.

 

Extra tip: with Go Pro you can use the upside-down mode when recording if you need to. The option becomes handy if you do not want to waste time to rotate the video after downloading or uploading it to your PC.

Remember about the thorough preparation

At least one day before you take off for the test with a driver:

  • check what kind of frames you get when placing the camera in different places in your own car. Experiment also with the placement and position of the smartphone. That will help you make a decision and set up the camera fast when you enter any car.
  • try recording in-car in the different times of the day and different lighting conditions. The settings you need for the night or cloudy January afternoon will be different than for a sunny July noon. In case of a lot of sunlight, you may need some sun shield for a smartphone’s screen. Otherwise nothing will be visible on the video.
  • check how much time will the battery in your device last and how much video recordings will your micro-SD card (or any other storage device) hold.
  • at the end of the day check if the camera is charged with the storage card inside ;-)

As a practitioner who always tries to counteract the worst scenarios before they become reality, I would also recommend that you take your laptop to the field with you. Thanks to that you will be able to download the files immediately after recording and recharge the camera a bit before another test. If you also have a webcam camera and a roll of office tape with you, as well as some screen recording software on the computer, then you have an emergency kit of recording devices ready to use if needed ;-)

I recommend also that your smartphone is fully charged before you start the user testing – you may need it for communication, but also as a hotspot if there is a problem with the internet connection on the user’s smartphone.

Extra tip: Check the video settings of the camera once again just before you start recording. Then start recording as soon as possible and screen the camera from time to time to check if the light that means recording is on. If you have a problem of forgetting about the technicalities when conducting the tests, put the big question “Am I recording?” at the very top of every page of the list of research questions you have with you.

Learn as much as possible about the driver in advance

Use recruiting the drivers and setting to get as much information as possible.

Ask about driver’s habits, how do they drive, where they put their smartphone in the car, do they pair it with any devices in the car and how much they use it during the drive and during the day. Ask whether they have a smartphone car charger – if not, bring one with you.

One of the most important things you have to learn is whether the driver uses any kind of car phone holder. It may occur that the user has it, but does not use it or does not have one at all. This is the place where the research gets really tricky: you have to ask yourself if you want to test the application with the minimal intervention and maximum realism or do you put safety first.

 

in car usability testing detail distraction Ask the drivers in advance if she uses or at least has a car holder.

 

Either way, I recommend you always have some car holder with you when you go on tests with drivers.

For many reasons, I very rarely make the decision that the need for full realism of the research context beats the need for responsible behaviour behind the wheel. Here is why:

Safety first

When you work on applications for drivers and have a chance to test your products with them, you fast become more aware that what you develop may influence the lives and health of many people, even those outside your target group.

You see drivers distracted by talking to you. You see drivers so concentrated on the app that they stop looking at the street. You see people holding their smartphones in their hands or on their laps, looking down on them, losing their concentration and awareness of the surroundings of the car. You see people behaving irresponsibly and irrationally, forgetting about the pedestrians, other drivers, cyclists. And you definitely do not want your product to be the cause of it, neither a part of it.

Distracted driving, especially the one caused by using mobile devices, has been a hot topic for several years and using mobile devices while driving is banned to some extent by law in many European countries and United States. At the same time it is still a huge problem – e.g. the U.S. Center For Disease Control and Prevention estimates that 9 people are killed every day in the U.S. as a result of crashes involving a distracted driver. It is though very important to reduce the distractions to minimum – design the application so it requires minimum attention and interaction (would be great if it had a voice interface!), but also make sure the cell phone is located in the spot where having a look at it does not require the driver to take her eyes off the road.

If the driver you are meeting with is still unaware of the possible effects of being distracted by her phone, it is your role to educate her. I do it by shortly describing that the safety is most important (not only) during the research, that is why I ask her to concentrate mainly on the road and the traffic, and only secondary on the app. I underline also that looking down the phone when driving is dangerous - this social ad may be useful for building the awareness:

[embed]https://youtu.be/O51f1BZKPoo[/embed]

Avoid distracting the driver

But not only the electronic devices may distract the driver during the test. It can be also you and your questions that cause tension, pull the driver's attention off the road and make it difficult to stay concentrated on driving. That is why you should:

  • ask most of your questions before you start a ride or after you finish it
  • ask the driver to talk-aloud what is she doing when driving, but try not to ask further questions or make her concentrate on the app. If you see it requires driver’s effort, tell her to stop. You will see how much attention does the app draw anyways by observing what is happening.
  • ask some questions carefully when you stand in the traffic jam or stop at a red traffic light, but make sure answering them does not require more than 1-2 clicks in the app or much thinking.

Remember – a small talk between a passenger and a driver is not a crime, but you must be sensitive to the capabilities of the person you are interviewing in this specific circumstances.

Extra tip: If you can see that some functionality of the app causes distraction and may be a source of a dangerous behaviour on the road, make sure your team redesigns or removes it from the app.

Be an agreeable guest in somebody’s private zone

People treat their cars as extensions of themselves or their homes – bubbles of privacy in the public space. They decorate it, they keep their things inside, they play their favourite music or audiobooks on the car audio, they let it be maximally messy or keep it as tidy as they like. Each car is not only a utility, but also a real vehicle of memory for its owner (the fact well caught in the amateur ad that went viral earlier this year).

[embed]https://youtu.be/4KlNeiY4Rf4[/embed]

This is great news for you, because by visiting the car you can learn a lot about the driver – your customer, and use what you see as a starting point for the meaningful conversation. But beware not to cross the line – the driver may also feel very uncomfortable with the unfamiliar nosy person on the board.

As Michael Margolis remarks in one of his latest posts on Medium - when you are conducting a test with a user, you should behave like a good host - do your best to make her comfortable and build a rapport by being open, attentive and encouraging, but not overwhelming.

The “Gracious Host” rule applies equally well when you, the researcher, are the guest. A gracious guest arrives on time (the fashionably late rule does not apply here!) and is flexible, willing, appreciative and courteous.

So if I can give you just the last extra tip: when conducting the mobile app test with a driver, try to do nothing more and nothing less than be a gracious guest.

Keep reading

tech development

Building a custom Prometheus exporter

In this post I'd like to guide you through the process of ...

Marcin Budny

Dec 02, 2019

6 min read

business

4 Business Model Canvas Tips from the Strategyzer Masterclass in London

In May, 2019, I had the pleasure of attending the ...

Jarosław Pilarczyk

Sep 11, 2019

4 min read

tech development

Bluetooth with Raspberry Pi and bleno - part 1: iBeacon

Check how easy it is to create a Bluetooth peripheral ...

Marcin Budny

Dec 14, 2017

2 min read

tech ux design mobile

How to make progress animation with Lottie?

Animations created with Lottie save a lot of effort and ...

Kamil Chlebuś

Mar 12, 2019

3 min read

Show More Posts