Nine Questions on Virtual World Market Research

Mary Ellen Gordon of Market Truths is by far the single most knowledgeable and experienced person (and avatar) doing market research in Second Life.  She is highly respected by both the business and academic research community in Second Life - and she has been very generous with her knowledge within the SL Business Communicators community in-world.  I've been after her for a while to allow me to interview her for this blog - and she has finally agreed.  What follows are nine questions I posed to her about her work and research in SL. 

By way of disclosure, Mary Ellen has become a friend and a colleague through our association in SL. She and I collaborate from time to time on business projects and my respect for her expetise continues to grow. I'm delighted she has agreed to share this space with me briefly.  I hope you find it beneficial:

1) Mary Ellen, tell us a bit about your background as a market researcher.

I've been involved in research for twenty years now.  I have a PhD in marketing, and have Professional Researcher Certification from the Marketing Research Association.  The first project I did was the first study of the US IVF market back when I was working in the pharmaceutical industry. I subsequently completed my PhD program with a focused research component, and then spent six years teaching (marketing research, among other things) and doing academic marketing research.

The idea behind Market Truths was to blend the type of rigor that is demanded by academic research with the real world relevance demanded by applied market research. We've been going since 2001, and have done a variety of different types of research projects for clients in a range of industries in the US, UK, and New Zealand.

2)  How long have you been conducting research in Second Life - and what type of research have you done?

We started doing virtual world research in Second Life in October 2006. Since then, our SL research has included surveys, interviews, focus groups, and observational research. We maintain our own panel to use as a sampling frame for SL research, but we also survey visitors in client locations, at events, or utilize "real world" participants.  Our experience in SL has allowed us to create some unique methods of gathering and validating research there, as well.

3)  Why do research in virtual worlds at all?  What characteristics of the environment of Second Life - or virtual worlds in general -  makes them valuable to conduct research via those environments?

There are several things that make virtual worlds valuable from a research perspective:

a. They make it possible to evaluate reactions to three dimensional objects easily, quickly, and (relatively) inexpensively. For example, if a company wants to test a prototype of a product or a package, everyone can see that in 3D and you can even manipulate it in real time in 3D to interactively create the ideal version in cooperation with the research participants.

b. Access to target markets. At present the SL environment is quite unique in that it is very international and it has a high concentration of people in technical, creative, and marketing jobs. Technical and creative people are generally not the types of people who jump at the opportunity to participate in market research, yet we find that they are willing to cooperate in SL. That's certainly important for companies targeting those types of people, and can also be important for companies that need technical or creative input into new product development. The international aspect is important for companies targeting customers in more than one country since it makes it possible to get information from people in a lot of different places in a short time.

c. It removes RL geographical constraints. For example, it's not uncommon for companies to make decisions for all of the US based on focus groups conducted in just a few cities. Obviously that's not appropriate if there are regional differences, but in RL you just can't easily get people from Boston, Iowa, Alabama and Los Angeles together for a focus group. In the virtual world you can get people from all over the globe together at once. While it's been possible to do that sort of thing with other forms of online communication for a while, there's definitely something more immersive about being "together" in the same room in a place like SL -- and that influences the quality of data.

d. The quality of data collected tends to be high. People are really forthcoming in SL, and comments made in focus groups, responses to open-ended survey questions, etc. all tend to be thoughtful and helpful. This is probably attributable to a combination of the types of people who are involved, the fact that they tend to be relaxed while in SL (they don't have to think about how long it's going to take to drive home after being interviewed, etc.), and (in the case of any research that relates directly to SL) because many people are really passionate about SL.

4)  There seems to be a general misunderstanding about sample size relative to the validity of research.  Can talk a bit about sample quality and quantity?

Yes, there's a huge amount of misunderstanding about sampling in general and sample size in particular!

To start at the beginning, it's important to remember that most statistics were developed for use with a "probability sample." That means that every member of the population has a known - non-zero - chance of being included. That's fine for crops or lab rats, but is problematic when it comes to humans because any research that enables people to opt in or out will have some people with a zero chance of inclusion. That, combined with things like variation in the number of phone lines in a single house or the number of avatars associated with a RL person, means the "known" criterion is also seldom fully satisfied.

While almost no market research can fully meet the gold standard of a true probability sample, with everything we do we should be trying to approximate that to the extent that we can and be aware of potential problems and biases to the extent that we can't. So, for example, one way to maximize the size of a sample would be to just let everyone who wants to do a survey, and do it as often as they like (like the audience voting someone in or out on a reality show).

I've seen a lot of SL surveys being done that way, but from a sampling standpoint that's not good as the type of people who participate are unlikely to be representative of the whole population that you're interested in. Also, the observations are unlikely to be independent (e.g., if the same person responds to a survey with three different avatars, you are really just getting one person's views three times rather than three independent opinions), which is another one of the fundamental assumptions underlying most statistics. So knowing the absolute size of a sample really doesn't tell you much of anything unless you know how it was selected and what was done to try to control for things such as self-selection, and non independent observations (and in particular in the case of SL, alts - multiple avatars owned by the same person).

Assuming a sample has been drawn in a way that's likely to reduce those problems, the size that's required to get good information is smaller than people often think. Larger samples tend to make estimates more precise; however, most marketing decisions don't require the same degree of precision as, say, determining the appropriate dosage of a drug. The requirement for precision is important because once you pass a certain threshold the sample size needs to increase a lot to get a meaningful increase in precision.

For example, the following graph shows how the maximum margin of error associated with the estimate of a proportion (based on a population of 5,000,000 and a 95% confidence level) changes based on the sample size:

   

The exact numbers change, but a similar pattern holds for estimating different types of statistics, and with different population sizes, confidence levels, and sampling methods.

For many marketing decisions - especially in an area that's evolving as rapidly as virtual worlds, a 5 or even 10% margin of error is fine. All other things being equal, it might be nice to go for even greater precision, but all other things are not equal. As shown in the graph, going down to something like a 1% margin of error requires an extremely large sample. Getting that sample increases the cost of the project - particularly if adequate controls are put in place to reduce the chances of the types of sampling problems described previously (and if those controls are not in place the larger sample size is pointless). In an environment that's evolving as rapidly as virtual worlds, clients would often get more information and insights for their money if they conducted a survey at regular intervals with a smaller sample than with a one shot survey with a larger sample.

4A).  So, then small samples (if chosen correctly) can be more "accurate" than large samples, and that the size of the sample isn't quite as important as the method used to include the participants?  It sounds like you are also telling marketers to at least look at how a the data was gathered before taking a survey as relevant.  Right?

Exactly. Quality is more important than quantity as far as samples go. If a poor quality process has been used to generate the sample, then it’s pretty useless no matter how big it is. If, on the other hand, great care has been taken in the sampling process, then you can get a lot of information from even a relatively small sample.

5)  What is a surprising or unexpected issue, outcome, or finding you have run across?

Perhaps the most surprising thing is the extent to which people's behavior in world corresponds to their behavior in RL. Prior to beginning research in SL I had read a lot of speculation about people using avatars to enact some sort of fantasy life, and I still hear and read that sort of conjecture fairly frequently (though typically not from people who spend a lot of time in virtual worlds themselves). This is an issue we've looked at in a number of different contexts, and what we've found is that some people may use avatars to enact fantasies some of the time, but that does not apply to most people most of the time in SL. Role playing may be more common in other virtual worlds, but at least in SL there is a fairly close correspondence (in attitudes and behaviors - not necessarily physical appearance) between the majority of RL people and their avatars.

6)  There seem to be a lot of companies and universities beginning to conduct research in SL with varying degrees of success.  What is the most common "mistake" you see when conducting research in this environment?

The most common mistake is conducting research before really understanding the SL environment. That can manifest in different ways. One is not putting in place any controls for alts when the unit of analysis is supposed to be the RL person. It seems to me that dealing with alts is the #1 issue facing SL researchers. We spent hundreds of hours on it last year, finding solutions to keep our research valid.

Lack of understanding of SL can also manifest itself in the conceptualization and design of the research. For example, there was an article by researchers at well known universities that was published in an academic journal and that was cited in a lot of mainstream media that investigated eye contact among avatars without addressing things like the fact that the RL person may be using the camera to look somewhere entirely different from where the avatar is looking; there are SL settings that make avatars' heads automatically turn to whoever is speaking; and to manually get one avatar to have eye contact with another relies at least as much on dexterity with a mouse as it does on intention, empathy, etc.

Lack of understanding can also come through in companies not applying the same standards for privacy protection in SL as they would when collecting data in RL because they're under the (false) impression that they don't have to.

7)  We see a lot of different kinds of  "mainstream" research companies like comScore, Nielsen, etc., releasing research about virtual worlds.  What is useful about these tools for marketers?

They're useful in terms of getting a general sense of what proportion of the population participates in virtual worlds because they're including virtual world questions in surveys of the general population. They're much less useful in terms of uncovering the details of what virtual world participants think, do, etc. because the proportion of the general population that participates in virtual worlds is still quite small, so even if their general population survey starts with a big sample the number of virtual world participants that they end up with tends to be too small to use for detailed analyses of just that sub group.

8)  What are three things you recommend a marketer consider about vw research?

First, just that they consider it at all. As described above, done properly, it does yield information that's relevant to RL marketing decisions and has some benefits even compared to more traditional forms of research.

Second, that you don't need to have a virtual world presence, or even be considering one, to benefit from virtual world research. It can just be the venue for gathering data to make decisions about marketing strategies and tactics that will be executed through other (more traditional) channels.

Third, that marketing is changing. While none of us know exactly where that change will lead, it seems clear that traditional channels are diminishing in importance and virtual worlds and other social media are increasing. There may not yet be a lot of "eyeballs" present in virtual worlds, but if this is indeed the general direction that marketing is going, it may pay to consider some exploratory research now to provide a foundation for future decisions and to be prepared if / when virtual worlds do become a more dominant channel. 

9) What trends do you see with marketing in virtual worlds this year?

I think we'll see more variety in who is using virtual worlds and for what purpose. Up until about the middle of last year it seemed that most companies with a virtual world presence were big consumer brands there for PR reasons. Toward the end of last year when our most recent RL Brands in SL Report was published, we noticed that some brands (particularly high tech companies) were using SL not for general PR purposes, but as a way of communicating with a very specific target audience (in the case of the high tech companies, that was software developers). I suspect that trend will continue this year, probably extending into other populations that are disproportionately represented within SL and to other purposes (e.g., new product development, product customization, etc.). I also think we'll see more small and B2B brands coming in to SL and that their aims will also be more specific and varied than just generating buzz.

January 23, 2008

Comments

[Tom Novak]

Very good overview of market research issues in virtual worlds! I'd like to comment on your point 6, where you critique the paper of Nick Yee and his colleagues, "The Unbearable Likeness of Being Ditigal." This paper looks at interpersonal distance and eye contact among avatar pairs. You cite as examples of "lack of understanding of SL" in conducting research that cameras can be used in SL, avatar heads can be made to turn automatically, and some dexterity is required to maintain virtual eye contact. However, none of these observations in any way invalidate the findings of Yee et al. They found for example, that 1) avatars looked less directly at each other the closer together they stood together, and 2) male-male avatar pairs were less likely to maintain mutual gaze than female-female or mixed pairs. The arguments you raise might possible *reduce* the magnitude of these effects - but nonetheless, significant results were obtained. One way to look at it is that even with the concerns you note, the results observed by Yee et al were strong enough to still appear.

It is always possible, of course, that the objections you note do not have a very strong influence in reality, and your concerns may not impact the results at all. Even if avatars use cameras to look around, they may well simultaneously position their avatars to face each other. Even though SL setting can turn avatar heads automatically, how many people actually use this option? And, while dexterity is required, the fact that avatars who were further apart looked at each other more directly than those closer together suggest dexterity may not be an issue, because presumably it is easier to manipulate avatars who are close together - you have a greater control over fine movements.

It is always good to think of competing explanations and influences. In this case, however, the competing explanations cannot be used to discount the findings of the research that avatars behave much the same as real people in terms of maintaining interpersonal distance and eye contact.

[Mary Ellen Gordon]

Oh, I’m not disputing what they found. They measured what they measured very accurately – the ability to use scripts, etc. is one of the great things about SL. The questions about that particular study relate to what the measurements really mean and whether what they measured was the right thing to be measuring given what they wanted to know. And I’m far from the only person who had those questions. There was a lot of buzz about it in SL at the time. For example, see:

http://www.sylvienoel.ca/blog/?p=595
http://www.secondlifeherald.com/slh/2007/02/eavesdropping_i.html

And I saw some more comments about it just recently (maybe on the Educators' list?).

I don’t mean to suggest that just because a lot of people are saying something it must be true, but in this case I think there are a lot valid questions to be asked. For example, they did indeed find that female / female pairs had less distance between them than male / male pairs. There’s no disputing that, but why? Their hypothesis is that “social behaviour and norms in virtual environments are comparable to those in the physical world” but there’s an obvious alternative possibility that would explain the exact same data: male avatars tend to be larger than female avatars and the distance that avatars stand apart from one another is related to their size (because larger avatars need to stand further apart to avoid bumping into each other). Similarly, they found that female / female dyads had more “eye contact”, but that also could be explained by a combination of variation in the propensity to use the settings that do that automatically (at one point I think that may have actually even been the default setting), and variation in mouse control.

In terms of measurement, as noted in the sites listed above, they didn’t include avatars of ambiguous gender, nor did they have any way to know whether the RL gender and the SL gender was the same. They used 12 feet as the limit for conversational distance since that is what has been found in the physical world to be “accepted social distance,” yet most SL builders scale things at a ratio of about 1.5 to 1 to take account of the fact that avatars are, on average, disproportionately large and clumsy compared to real humans and also unlike in RL, you “can “hear” chat that’s happening 20 meters away from where you are. They also didn’t have any way to account for where people were actually looking (since unlike RL, that may be somewhere entirely different from where the avatar was pointed) nor what they were actually doing at the time (IMing to someone somewhere else in SL, getting coffee, reading e-mail, talking on the phone…). They conclude that: “overall our findings support our hypothesis that our social interactions in online virtual environments, such as Second Life, are governed by the same social norms as social interactions in the physical world” but none of that stuff that I just described happens in the physical world and they didn’t measure its effect in the virtual world, so how do they know?

As it happens, I have been doing a lot of interviewing today, and one of the questions I asked an avatar I was interviewing (who was “gazing directly” at my avatar at the time) was whether he participates in other virtual worlds besides SL. His somewhat embarrassed reply was that he was playing WOW at the same time as he was answering my questions. I found that perfectly normal as that type of thing is really common in SL, and in fact while I was waiting for his replies to my questions, I was checking IMs and e-mails – all while our avatars were experiencing prolonged “eye gaze.” I don’t know about you, but I’ve never had someone play a digital game at the same time as I’ve been interviewing them in RL, nor have I ever checked my e-mails while waiting for a response to a question in a RL interview, so we were definitely not enacting RL social norms as I have experienced them.

I don’t mean to be too dismissive of that particular article. I actually do agree with the general idea that there are a lot of parallels between RL and SL. One of the reasons that I mentioned it is the reaction it created. I think that shows something fundamental and important about SL research. Had something like this happened ten years ago, or even now with data collected from undergraduates participating in some sort of simulation in RL, the types of things I’ve mentioned might be batted back and forth between co-authors, between authors and reviewers, or between presenters and an audience at an academic conference. But with social media in general and virtual worlds in particular, the subjects don’t quietly vanish once the experiment is over. They see the results and react to them. If they perceive them to lack face validity that could have serious consequences for people (even if they’re not the same researchers) trying to encourage participation in future research. That concerns me, and I think it’s something all of us who care about the long term viability of SL research need to be thinking about.

[Tom Novak]

The hypothesis that “social behaviour and norms in virtual environments are comparable to those in the physical world” is actually not a very controversial one at all. The media equation/media effects literature over the past 15+ years has provided considerable evidence that people tend to react to computer interfaces in many of the same basic ways that they react to "real people." This is actually pretty well established, and it should come as no great surprise that many of our intuitive repsonses to people in the physical world would carry over into virtual worlds.

The vast majority of previous studies on these media effects, however, have been controlled lab experiments. In the case of the current study, the authors are undertaking field research. Field research provides a more natural setting, but the tradeoff is lack of control over competing explanations - and little control over how individuals in the study behave. So, while the conversational context in the study was natural and not forced, it is realitically impossible in an observational field study to control for many the types of objections you note.

Looking over the various posts you provide links to, I find three general categories of comments:

1) Useful suggestions for how this research can be extended and clarified. For example, how does the RL vs. SL gender of the two parties affect interpersonal distance and eye-gaze angle? How does physical size affect results? What are the boundary conditons where virtual behavior starts to diverge from real world parallels?

2) Threats to the validity of the results. For example, women may choose different default settings than men.

3) Reasons to simply discount the results of the study. For example, reactions that this can't be true because "Avatars can't see. They don't have eyes. They don't have a gaze." Or observations from personal experience, like "I have frequently been standing next to someone while conversing out loud with someone twenty meters away."

I believe the first two categories of comments are extremely valuable, and that the review process would have benefited from peer reviewers familar with Second Life who could have raised these issues. It would have been good for the authors to have recognized some of these limitations in their discussion section. But I would also put responsibility on the review process - it is the job of the reviewers and editor to make sure that all bases have been covered.

This suggests to me that it is be extremely valuable for academic researchers working in this area to post working paper versions of their papers on the Web, and encourage the Second Life community to read and comment.

As you read and comment, however, you do have to recognize that once single study cannot answer all the world's questions. You have to ask, is there basic merit to the ideas presented? If the threats to validity are so enormous that results have to be discounted, then the paper should not be published. But I don't believe that is the case in the paper under question. Again, these media effects are not a new idea, and we have seen results like this in many other contexts. There is something going on here. Sure, there are many ways the results can be extended and clarified, and there are some threats to validity. But, but what concerns me is so many people who have commented want to throw out the baby with the bathwater.

Last, I would certainly hope that what you say about the public reaction to research creating "serious consequences for people...trying to encourage participation in future research" isn't true. The process of academic research is a wonderful system for generating new knowledge. Corporate researchers don't release to the general public the full details of how they gather their data, or the full results of their research. Academic researchers do - for every single piece of research they publish - and so their research can come under much greater strutiny. It would be a real shame if negative comments resulting from this transparency led some to advocate not participating in research studies.

[Tom Novak]

One further comment, to provide a historical context. You note:

"Had something like this happened ten years ago, or even now with data collected from undergraduates participating in some sort of simulation in RL, the types of things I’ve mentioned might be batted back and forth between co-authors, between authors and reviewers, or between presenters and an audience at an academic conference. But with social media in general and virtual worlds in particular, the subjects don’t quietly vanish once the experiment is over."

Actually, this sort of open debate *DID* happen 10+ years ago. For example, consider the "Cyberporn Debate" from 1995 that I was involved in:
http://sloan.ucr.edu/1995/07/31/cyberporn-debate/

Social networking was not developed in the mid 1990's, but people made the most of available technology to achieve the same effect of an open Web-based critique. Of course, it is much easier today for anyone to comment, and the speed at which ideas are exchanged is much greater.

All in all, these sorts of public exchanges are a good thing for the research process, and I can't help think that these types of exchanges encourage rather than discourage people from participating in research.

[Mary Ellen Gordon]

In my experience, people in SL get very frustrated when they feel that people don't "get" it or them. That seemed to be at the heart of the reaction to that particular study, and it's that feeling that I think could make people hesitant about participating in future research (or possibly even worse, figure they don't need to do so in an honest way because it doesn't really matter anyway). So in that sense, it's exactly the "throwing the baby out with the bathwater" reaction that I fear too. And it's a real threat. For example, there have been instances of people using "research" as a way to collect people's e-mail addresses for use in Spam. This got around in SL, and led some people to be hesitant to participate in research and others to participate, but by giving false information (and encouraging others to do the same).

I agree that all research has limitations, but I think in this case the question is whether things have been abstracted to such a degree they are no longer really meaningful. The fact that people facing each other may not actually be looking at each other in SL is really fundamentally important if what you're trying to understand is social norms and a key part of the social norm of looking at the person you're talking to in RL is to signal that you are paying attention to them. Everyone who has participated in SL for any length of time knows it is not a reliable signal of that there. Similarly, in RL you could pull out a handheld device and IM someone else while someone is talking to you in person, but the speaker would see you doing it and so the current social norm (who knows, the way things are going this may change) is that people generally don't do that. In SL it happens all of the time.

I agree that it would be advantageous to try to bring issues like this out as part of the review process (I'm all for anything that brings real world relevance into the review process :)). In this case, I think the results would have been much more credible if pre- or post-tests were used to see 1) how many people were actually conversing with the other people in their dyads (as opposed to those who just happened to be standing within the same 12 foot radius or conversing with someone beyond that radius), 2) how good "eye gaze" and IPD actually are as proxies for attentiveness to the speaker, and 3) Whether there is a correlation between distance between avatars and perceived comfort or discomfort. We may disagree about what we think the answers to those questions would have been, but as it is we're all just left to speculate.

[Tom Novak]

All good suggestions, Mary Ellen :) I'm actually quite interested in exploring some of this in a controlled setting. I think we are lucky that people participate in academic research in Second Life at all - given the horror that is Hippie Pay!

[Mary Ellen Gordon]

It's great that you're planning to look into some of those things further Tom. I think there's a lot of interesting work that could be done about attention, attentiveness and communication in SL.

Have you read Exodus to the Virtual World (Linda actually mentioned it previously in this blog)? The basic thesis of that is that people's experiences in digital games may shape their expectations in RL. While that is mostly about game-based worlds, it would be interesting to see if the same thing applies to communication patterns in more social worlds like SL. It certainly seems plausible.

I'll look forward to hearing about what you discover!

[Linda Zimmer]

Mary Ellen, I see every day how experiences in digital games - or virtual worlds - are already starting to shape expectations. I was in a conversation today in SL and someone jokingly referred to how nice it would be at a party to be able to click and get a profile of that person from across the room.

Someone else piped up and said, they sort of have that - via their iPhone (presumabley accessing the web to get information on a person). Of course this may happen organically anyway given technology, but environments like SL make it a "recognizable" application. There are companies beginning to enable proximity tagging for a similar purpose using RFID and Bluetooth.

These experiences "in world" will drive soical norms to some degree eventually.

Tom, I hope you keep us informed on your controlled studies as these environments are unique and unless you understand the environment, it is impossible to interpret findings in a meaningful way - we can use more of that understanding in the available research.

[Tom Novak]

The idea of experience in digital games/SL shaping RL experience bears resemblance to another idea - "online influences offline." Marketers increasingly recognize that consumers' experience with Web shopping impacts their expectations and perceptions of physical world stores. For example, you expect your local Barnes and Nobel physical store to be searchable, and you expect to find answers to your product questions at the big box electronics store. When this doesn't happen in the physical world (because it does nicely happen online), consumers can become upset. Imagine how much more virtual worlds can shape expectations...

[Emiel van Wegen]

A very interesting interview indeed. In my work I come across of many clients who ask me if any research results in SL are going to "be representative" , in other words: how accurate their research results will be. The reality is though that it’s really the wrong question to ask. The SL population does not need to be representative of the entire population for research in SL to be viable, the only thing that is really important is that we are able to "draw a representative sample" from all who are in SL.

The question clients really want an answer to is whether any biases in the SL population, or more accurately, any biases in the sample are actually going to affect the accuracy of the answers that you get from the study.

The critical thing to remember is that we don’t undertake research in SL using all the users of SL at large, we undertake research among a sample and this sample will be representative neither of the SL population nor of the general public.

Samples are made up only of people who are willing to participate in research. As a result they tend to be somewhat attitudinally different to the population at large and also tend to have different behaviours.

So when doing research in SL the questions we should answer first are:
Is the SL user penetration sufficient to make an SL methodology viable for GP work? (and not necessarily: is the sample size larger enought).

- Is the target market we are wanting to research represented in SL?

- Results we will collect in SL will be comparable with the results they’d get using other (offline) universes?

- So what it all comes down to, is the choice of an SL methodology the right one for their project?

[Suya]

Virtual world doesn't mean SL. There are better virtual worlds than SL. Check this list out :
http://www.oonania.com/social_network/virtual_world.html