Services Australia, University of Queensland.
[/s2If] [s2If current_user_can(access_s2member_level1)] [/s2If]
With user research becoming more common within organisations, there is an emerging issue of meeting demand whilst also developing the craft of research. A new profession is emerging in response – research operations. This paper will describe the current state of publicly available frameworks for research operations. These tend to deal with one aspect of scale – the people who are doing the research, not how they do the research, when, or what we do with the research. Two frameworks will be combined to create a matrix that provides the tools to identify an investment strategy for research within the context of an organisation and their strategic goals. This matrix provides a significant contribution to the field by making it possible to be strategic and proactive about developing research practices in the context of individual organisations, how and why they do research, and to better manage the tension between scale and craft.
Keywords: ResearchOps, UX, Strategy
Article citation: 2020 EPIC Proceedings pp 203–217, ISSN 1559-8918, https://www.epicpeople.org/epic
Please sign in OR create a free account to access our library—the leading collection of peer-reviewed work on ethnographic practice. To access video, Become an EPIC Member.
INTRODUCTION
Qualitative research as an embedded practice in industry and in government has been emergent since at least the 1960’si, and has grown to the point of being commonplace in the world today. Modern design is largely attributed to the collision of the arts and crafts movement with the machine age.ii Alongside this emerging popularity, the practice of modern design has matured, and our understanding of art and design as an embodied experience, one worthy of replicating in applied ways, has also matured. Seen in this light, it comes as little surprise therefore, that in recent years, the attention economyiii has raised the profile of (and pressure on) human researchers and research outcomes even further, as companies attempt to squeeze out every last minute in a person’s day that can be spent on their device, in a platform, and/or watching ads. It is a bleak picture to paint, but the opposite story is there too – through human centredness/system centredness, researchers have a role in enabling industry and government to have a meaningful impact on people’s lives. Our current state with regard to the health of the planet and the people therein may cause many to want to engage in qualitative research as a part of their design work in order to have the best possible chance of effecting meaningful, ethical and human centred change. That means it may seem that everyone, from the smallest to the largest company, are employing researchers to do more and more research in less and less time.
The possibility of effecting that change is incredibly exciting, intoxicating even. In many ways, the chance to do some real, lasting change work has never been more present. The profession of applied qualitative research (commonly known as user research, as it will be referred to throughout the rest of this paper, but also including design research, UX, CX – largely a broad umbrella of qualitative, human led, conceptually post-modern research) has developed significantly in the past few decades. Alongside traditional ethnographic and anthropological or human factors research methodologies, or other disciplines such as psychology, researchers can be trained in systems thinking, and in the practice of co-design.iv It is possible to see that the field is becoming more established.
The reality of having teams of any size – whether 1, 100, or 1000v, is that the demand for research far outstrips anyone’s ability to meet that demand. While it might be traditional for a research team of one, or 5 even, to individually be running their own processes and procedures, their own contracts, panels and ways of working when it comes to research data management and sharing, it can become a huge time impost, leading to duplication, unintended replication and burnout.
The ResearchOps Community is a volunteer run online community of over 6000 (to date) individuals from 62 countries working in the field of user research and research operations, coming together with a common goal of giving shape to, and validating research operations as a profession. The common challenge facing the members of the community is doing research at scale.
As work has progressed on understanding what research operations is, so too has our understanding of how to manage the tension of delivering research at scale, whilst also maintaining rigour in research. This paper will describe the current state of frameworks (that are publicly available) for research operations, which, to date, tend to have been maps for research and research operations. Following this, the paper will bring together two frameworks that, once combined, allow one to see the terrain of research in individual contexts. This blending of frameworks, known as the Pace Layers Matrix is the result of observation and experience from 3 global research projects undertaken by the ResearchOps Communityvi (one on what research operations is, one on building a research skills framework, and the most recent on research repositories) and the author’s own work in understanding the research outcomes from those projects. Having the tools to identify the terrain of one’s own research practice in context provides a significant contribution to the field by making it possible to be more strategic and proactive about developing research practices in context and better manage the tension between scale and craft.
The Problem with Scale
By now, the industry is well and truly aware that there is a problem with managing the demand for user research. The issue is a seemingly simple one – qualitative research takes time, lots of it, and this does not scale well. Demand for research grows, and the expectation that good research can be done in months moves to weeks, and sometimes even days. At what point fast research becomes poor quality research is what is at issue. At what point the profession suffers from poor quality outcomes from overstretched, under-resourced or untrained researchers is an ever-present burden when the topic of scale emerges.vii
There are myriad ways to deal with demand – adding more and more researchers, creating hub and spoke models to have core researchers at the hub and ‘people who do research’ (PWDRs, a phrase coined by Kate Towseyviii) operating in small teams, or having a core group of researchers embedded individually across the organisation but reporting to a central research leader. But each of these models only deals with one aspect of scale – the people who are doing the research, not how they do the research, when, or what we do with the research.
Research operations has emerged from this gap – a field dedicated to:
“the people, mechanisms, and strategies that set user research in motion. It provides the roles, tools and processes needed to support researchers in delivering and scaling the impact of the craft across an organisation.”ix
Within the field of research operations (also known as ReOps, or ResearchOps), there are a group of sub-fields, all addressing slightly different issues to do with how we create the right environment for research to happen. They include (non-exhaustively):
- Making better use of existing research through the creation of a research library or repository, though these often fail to achieve the results that are hoped for.
- Research operations playbooks or ‘centres of excellence’ are one of the first ways one sees operations leaders attempting to address the ‘how’ research happens at scale. This also attempts to address the additional issue of the tension caused when trying to do a lot of research in a short time – creating efficiency, and also enabling others to do research, even if not fully trained in doing research.
- Systematising and streamlining recruitment
- Centralising budgets and managing tools, platforms and contracts centrally within large organisations
All of these responses to scale create ripples that are felt across the organisation and the broader user research industry. Indeed, the concept of the democratization of research is a hot topic, occupying whole streams at research conferences (see for example: Advancing Research 2020x) and the topic of debate in blog posts and papers.xi Interestingly, the democratization of research has long been a topic of debate in academic circles also, but it is framed instead as a feminist act, or an act of ‘research justice’,xii enabling research to be decolonised. This is not two separate disciplines using the same terms for different ends. Rather, it is a different lens on the same issue – the practice of extending research spaces to people outside the role of research. From researched to researcher, from consumer of research to doer of research. Kara states that “The term ‘democratizing research’ covers a range of emancipatory approaches to research such as activist research, feminist research, decolonizing methodologies, community-based research and participatory research”.xiii User research, being embedded within design and design principles such as co-design, co-production and othersxiv is a practice of research in, of, and sometimes with, community, and is often participatory – the tension therefore, is the same.
Complex Systems and Frameworks: Tasks vs Strategy
To understand the current state of ‘research at scale’, it is worth acknowledging the myriad frameworks that have arisen in the wake of the emergence of research operations. Given that the ResearchOps community (that really is a catalyst currently for the emergence of the profession and the development of frameworks for understanding what ResearchOps is) started with the ‘what’ of research operations, early frameworks, such as the ResearchOps Community’s ‘What is ResearchOps’xv and the Nielsen Norman Group’s ‘ResearchOps 101’xvi have focused on dividing up the tasks or things that need to be done in order for research to happen effectively. It is possible to see however, that tasks are only part of the way research happens. How it happens, with whom, and most importantly for this paper, in what contexts, are all essential in effectively delivering research at scale.
Towards a Strategy for Scale: PESTLE Models
Towards the end of 2018, a group on the board of the ResearchOps Community (Emma Boulton, Holly Cole, Tomomi Sasaki and myself) realised that the taxonomy, or the conceptual framework we’d applied to understand the data from the ‘What is ResearchOps’ project could be used to understand the relationship between research and operations. Emma Boulton took this forward with the 8 Pillars framework.xvii This model (Figure 1 below) can be seen as a typical PESTLE strategy model. The PESTLE model arose from the work of Professor Francis Aguilar following his book, Scanning the Business Environment in 1967.xviii It is a framework for understanding the political, economic, socio-cultural, technological, legal and environmental factors that are involved in managing business, with the idea being that if one is aware of the forces impacting the business, then it is possible to create a strategy for optimizing opportunities and mitigating risk.
Figure 1: 8 Pillars of User Research
In the same way, the 8 Pillars, with the focus on environment (in the PESTLE model: environment), scope (political), people (socio-cultural), organisational context (economic and political), recruitment and admin, data and knowledge management, governance (legal), and tools and infrastructure (technological) can be used as a way to generate an understanding of the factors and forces at play when research happens.
Briefly stated, the 8 Pillars, as they pertain to research, are:
- Environment: Why does research happen? Who engages with what I do?
- Scope: The nuts and bolts. Methods, processes. How and when does research happen?
- People: Research maybe done by designers or product managers. Can we create a community of practice to support and mature the craft? What does a career path look like?
- Organisational context: What is the maturity level of the organisation I work in? What are the external constraints that affect me? Things such as budget, resources, time, space.
- Recruitment and admin: How do I manage all the logistics and admin for research projects and participants?
- Data and knowledge management: Often valuable insights are lost over time as teams grow and change. How do we ensure the same studies aren’t repeated? What happens to the research findings, data and insights?
- Governance: As a researcher what are the legal and ethical considerations that affect my work?
- Tools and infrastructure: What tools and infrastructure do I need to help me with my research?