INTRODUCTION

For many years, we at EPIC had to argue for the value of ethnography and of user research, more generally inside our organizations. Now that we have won a seat at the table, we are able to create stronger networks internally within our own organizations and externally, with partners outside. Recently there has been a push for ‘scaling up’ research teams within organizations. We have learned from this that it is no easy feat to go from a research team of two to thirteen (Clancy 2018); or to adapt the role of UX research as the team scales (Primadani 2019); or to prove inside organizations the value of research, while implementing processes of working cross-functionally, and to manage the actual scaling process simultaneously (Chokshi 2019).

In this paper we make the argument for the benefits of scaling out. We define ‘scaling out’ as reaching out beyond one’s organization to bring in external partners to accomplish UX research, cover more ground, to bring in more specialties and ultimately rely on collective intelligence to get work done. Scaling out is an inclusive approach that can produce meaningful results due to presence of diverse voices, including among the UX researchers working on the project. Today with more social networking and digital collaboration tools than ever before, as we describe in this Catalyst, scaling out has become manageable and efficient. We also consider power imbalances in UX research, particularly in the roles of client-supplier, researcher-participant and designer-user, and whether it is possible to scale out to such a point that we shift them. Can we as UX researchers, go from evangelizing user research and amplifying the voice of the user to collaborating continually with users – with people. We call upon the EPIC community to consider what it means to be on one side of the researcher-participant dynamic and ways to shift this while working in the realities of market economies and for-profit business sectors.

We believe this conversation on scaling out is more timely than ever in a COVID-19 world. The first half of this Catalyst tackles the practical side of today’s restrictions. As the conditions which make possible in-person research are tenuous or in flux (physical space restrictions such as lockdown, curfew, masks), we must rely more than ever on distributed local researchers, participants, and digital collaboration tools. The EPIC community will already be familiar with some of these tools and their advantages.

In the second half of the Catalyst, we take the concept of ‘scaling out’ to its logical conclusion. We speculate on a form of distributed UX research where our global partners are not local professionals, but rather “citizen researchers,” or even users themselves. We imagine scenarios where user research is conducted by crowdsourcing and collective intelligence, similar to models like Wikipedia and some COVID-19 response projects (OECD.org). This is not a new phenomenon but we believe it is more timely than ever.

BACKGROUND

EPIC practitioners have written about the obstacles and opportunities of globally distributed teams and partnership through ethnography. From the early days of EPIC, Mack and Mehta (2005) describe, “as more and more corporate ethnographic work is crossing international borders, we are increasingly collaborating with teams that are spread across the globe.” Churchill and Whalen (2005) emphasize the methodological challenges of geographically distributed projects. These include the extra effort required to build and maintain relations and empathy; widely varying experiences of ethnographic methods, local language and culture; and conflicting responsibilities and lines of accountability. At times, researchers train non-researchers to help conduct research. Di Leone and Edwards (2010) point to four key needs for knowledge sharing in collaborative ethnographic research. Brannen, Moore, and Mughan (2013) describe a project in which they acted as outside experts in collaboration with a multicultural project team of nine managers within an organization. The managers were trained in ethnographic techniques, and they conducted the data collection and participated fully in analysis. The consultants acted as advisors, trainers and coaches to the project team at every stage of the ethnographic process. Kearon and Earls (2010) describe a project that employed participants to conduct ethnography themselves where the initial batch of results were lackluster. They urge careful framing and training in order to obtain useful results.

As remote research and collaboration tools advance, so do the conversations at EPIC. Gorkovenko et al.’s paper (2019) describes a project in which ethnographers and participants were able to engage remotely in contextual inquiry around a product with the aid of sensing technology. Golias (2017) discusses how ethnography and remote usability testing can enhance one another.

Today, as travel restrictions loom due to COVID-19, these conversations take on a new sense of urgency. Henshall (2020) describes the benefits of remote research including participants being comfortable in their home environment, feeling safer sharing their point of view in a private space, reaching a larger pool of participants, scheduling flexibility, and including more stakeholders via remote observation. Drawbacks being that it is difficult to include unconnected households, people less comfortable with digital tools who then require extra support to participate, finding a private spot in a crowded household, and framing being limited. The researcher’s ability to zoom into details or zoom out for a broader view is not always possible. Evans (2020) echoes many of these sentiments when describing her own remote ethnographic research, relying heavily on diary study tools, online surveys, and participant-made videos. She offers solutions to overcome obstacles of remote research, including finding the least data heavy video conferencing tools, becoming more prescriptive in questions for participant-made videos, and balancing these data with other sources of insights. In a recapitulation of an EPIC panel from May 2020, Collier Jennings and Denny (2020) paint a picture of how ethnography evolves and how research today weaves together many different vantage points. If one view is eliminated or limited (such as, conducting in-person research) how can we use other available data points to provide a holistic picture? They also describe participant- and community-led engagement and co-creating insights.

SCALING OUT: A CASE STUDY IN DISTRIBUTED RESEARCH

Our Paris-based boutique agency, MindSpark, relies on scaling out our resources to accomplish projects. We offer UX and market research on a global scale to meet our clients’ research objectives. By scaling out we mean that we work with suppliers and clients to design and execute research by bringing in stakeholders and partners with specialized skillsets, expert and/or localized knowledge, as they are needed. Scaling out allows an organization broad reach and flexibility by relying on local experts globally.

For example we, the authors, worked on a project for a large tech company for which we conducted a user experience research study in five markets: South Korea, Bangladesh, Chile, Kenya, and Thailand. The company approached us to design the research, to collect data, and to deliver findings with our local partners. Before the client came to us, it would have been difficult to anticipate a need for resources in these markets, both internally for the client or for us as a research agency. The client did not have researchers on the ground in those countries, nor the network to contract directly with local researchers. We did not have in-house researchers on the ground full-time in those markets.

What we did have to offer was an extensive, trusted network of suppliers with whom we collaborate and could quickly enlist. When conducting user research on a global product, it often becomes necessary to “pick up” partners when a research question arises in a corner of the world little known to the researcher. We first piloted the research in Korea and the core team including two MindSpark researchers and four clients traveled to Seoul to observe and participate in-person alongside the local team, consisting of one moderator, one project manager, and a simultaneous translator. Then we, from MindSpark with the clients, worked remotely with local teams to execute the other four markets concurrently. By letting go of total ownership of this project and relying on the expertise of local researchers, we were able to gain insight into how people in those markets would want to interact with our clients’ product. We were also able to save time by running the studies concurrently.

When we scale out at MindSpark, we intentionally keep the research design flexible enough to allow local partners to tailor the methodology to their localized context. In the same study, the local Kenya team recommended conducting interviews in-person (pre-COVID-19) at local offices or cafés. In Chile, internet connection is strong enough it was deemed most appropriate to conduct remote interviews to capture a more geographically dispersed audience. The local researchers were able to provide much more cultural context than as outsiders we would have been able to glean on our own. Themes around public/private spaces, security/protection, sexuality and violence (all related to the product) emerged in conversations with the local researchers.

We often collaborate with researchers with special skillsets. In this project, we were able to work with moderators who has specific experience in the topic matter of the study and were able to provide additional insight and background knowledge for the final reports. In other instances, we might rely on a quant guru to validate or supplement the qualitative pieces we would be working on. We collaborate with colleagues specialized in visual design, UI and UX design, creative agencies who might be delivering assets that we could then test through research, inspiring creative production or testing it afterwards.

TOOLS OF THE TRADE

There are a few basic elements to MindSpark’s model of scaling out. First, we establish a core team, usually (at the very least) one researcher, one project manager, and one client point person who will see a project through for a certain period of time, usually from start to finish. This gives consistency and rigor to the work. Second, as mentioned above, we gather a network of partners with specialized skillsets or local knowledge. The core team acts as a hub that gathers and transmits information to that network. Third, we create extra layers of communication and documentation. This can include phone or video calls to review objectives and to introduce all the main players of the project to each other. This can mean separate project messaging channels (on Slack or WhatsApp, for example). It can take the form of shared calendars, google documents, collaboration boards (on Notion, Mural, Miro etc.) It means documenting objectives from beginning stages to end deliverables. When working remotely or in distributed teams, extra emphasis needs to be placed on careful communication in order to maintain shared goals and contribute meaningfully to the same output. Summarizing advances or changes in the project in emails or shared digital document spaces (Dropbox, Google docs) is crucial. To this end, templates are extremely helpful. Templates help ensure logic and consistency to note-taking, findings, and final reporting.

Ongoing conversations can occur around language choice and cultural context, particularly when testing copy or content. Cross-cultural conversations can be around screener crafting and recruitment. Perhaps the target participant is not representative or might not even exist in a particular market. Legal limitations differ across regions that will impact a research study. Even budgeting time per interview can be an issue. A user interview that might take 60-minutes in the US could run closer to 75 minutes in a different market. We usually require at least 30-45 minute buffers between remote interviews in markets where bandwidth is unreliable or where people tend to share computers and other devices and they might not be familiar with troubleshooting those particular devices.

BENEFITS OF SCALING OUT

A first benefit of scaling out is the ability to incorporate deep local knowledge into the research in order to produce a hybrid insider-outside view. By partnering with local teams, the researcher gets to come in as an outsider with a naive understanding of local conditions. The local moderator has the language and cultural background to execute the study and to assist greatly with analysis and findings generation.

By partnering, the researcher can become the hub, with a higher systems-level view. This gives us the ability to cut across organizational silos, across different markets, and across different populations of users. In another project, we worked with stakeholders from three different departments across two organizations, connected by a particular user journey we were charged with mapping. By using an ethnographic approach to study the whole system and to present that system back to stakeholders, we were able to help the organizations make strategic decisions to improve their internal processes and, ultimately, the user experience. Our advice could be considered as having less bias, due to not being part of the structure and their system of bonuses and promotions.

Another benefit is that it is less risky for us to try new approaches as outsiders. We can break the mold and experiment in ways that might be more difficult to do in-house or without a partner network. We have the advantage of multiple view points grounded in local contexts. They will be more relevant, diverse, and simply having more brains on a project with new ideas can help challenge our preconceived ideas of how a project should run.

Scaling out also has the benefit of budget precision, executing research and hiring researchers when and where they are needed, bringing in necessary skillsets or expertise in a certain geography, language, subject to answer particular questions. It has the advantage of speed, particularly when running multiple market studies concurrently. Arguably, the outcomes will be of higher quality when produced by a series of collaborators rather than a sole researcher attempting to bring all these pieces of data together alone. The quality will be enriched due to the inherent diverse nature of on the ground voices, lived experiences, and styles of approach. In the next session, we speculate on how deep this democratization of research could go in the future.

CONSIDERING COLLECTIVE INTELLIGENCE