Vignette I. Make the Unseen Visible, but Losing it from Sight: The Woeful Pie Chart

After having expressed the wish to bring her anthropological training to bear on her work at SAP, Natalie was given the opportunity to conduct a research study as a ‘proof-of-concept’ for what might be possible. That study has come to mark the beginning of the User Experience function in SAP’s Business Operations group. At the time, the operations function in the U.S. was focused almost exclusively on the sales line of business. The team’s charter was to increase the productivity of sales people, as measured by (among other things) license sales revenue per sales representative. It is therefore not surprising that this first research project involved shadowing revenue-generating employees. The research was justified to an executive team on the basis of the opportunities it brought to understanding sales activities, specifically looking for opportunities to increase their productivity.

At that moment, Natalie was the sole researcher who handled almost every aspect of the study– recruiting, scheduling, capturing field notes, data analysis, and the reporting of results. The final deliverables included a presentation that was given to the Senior Vice President (SVP) of Business Operations (the project sponsor) with a list of possible action items, including business scenarios illustrated with quotes from study participants, and a series of recommendations. This type of work is perhaps the standard of ethnographic practice at the service of business strategy in which the “unseen” of work practices is reconstructed and made accessible to decision makers who might be unaware or distant from it. Messaging such findings involves the creation of reports, diagrams and other artifacts that attempt to serve as boundary objects (Star & Griesemer, 1989) between recipients and researchers.

In this case, the presentation materials produced included a slide that attempted to visualize the frequency with which certain key activities occurred. It was a pie chart based on the coded data, intended to provide visual impact and designed as an anchor point for discussion with executives who might not tolerate the narrative detail in the findings. When the study results were presented to the Senior Vice President of Business Operations, this pie chart was used in addition to informal, verbal findings; the complete presentation materials were delivered later to an entire management team.

At the time, this pie chart in particular appeared to be extremely effective in stimulating dialog with the sponsor and others about the complex stories and data behind it. In fact, it effectively enabled messaging of complex findings all the way to the executive level, a success rarely experienced for this kind of work. Without it, the research and its outcomes may have not had the same lasting success. At this point, it appears to be the only artifact from the research that is still circulating. To the best of our knowledge, none of the detailed business scenarios (or even the sales people screen shots) have been distributed further, despite those being, from our point of view, the most interesting and valuable aspects of that research. We have come to call it the “woeful” pie chart, the one which has been used and re-used the most, and in many cases, unfortunately, misused and surely misinterpreted. This might be an inescapable fact of ethnographic practice. Or perhaps the artifact itself, with its simplified and attractive appearance, affords the twisting and positioning to suit the needs of the speaker.

As professional practitioners and as members of our particular organization, we are still learning and re-learning that it is the contextual interpretation and ongoing analysis that makes field data useful—the nuances are not made visible except in situated conversations. Partly in response to this, we have made some changes in how we handle our reports today. For example, we insist on providing readouts before we distribute the soft copy of a report, and we provide private readouts to help interpret or expand on key topics. We hope that this approach will help our audience understand the richness and complexity of the findings, so that they will come back to us again and again, rather than assuming all the data they need is in the final deliverables. What we have come to learn from this experience illustrates the subtle ways in which we adapt our professional practices to suit the contexts in which we operate.

There had been a few other studies conducted during a similar time frame, carried out mostly by outside vendors. The first had yielded very general findings, and had not provided any significant new insights that could be used to drive productivity improvements. The second study had been conducted by a usability testing firm, and its results had limited value to the operations management team because the outcomes were narrowly focused largely on ways to improve the intranet. At the time, the research Natalie conducted had succeeded in looking for ‘white space’ (opportunities to improve productivity that might not have been uncovered otherwise) and that the research approach and final deliverables permitted the management team to more deeply perceive the problem areas, the frequency with which they were occurring, and to begin to understand the real impact from a salesperson’s point of view. The research also made visible problem areas that were known but were previously not well understood. The detailed findings and recommendations permitted a level of visibility on a core process that had previously not existed, and specifically showed the impact of those process breakdowns on sales productivity.

Overall, this initial work was perceived as innovative and adding value, and it contributed to the increased understanding of the management team of how anthropology and ethnographic practices could be blended with work in business operations. This initial success began to make ethnographic research methods visible within the organization, and opened the door for future engagements. However, as a junior member of the operations team, Natalie had very little opportunity to drive uptake of the findings, and almost no visibility into what was done to address the issues uncovered. Lack of direct access to members of the executive team, lack of influence in general, and lack of resources severely limited what could be done to extend and act on the findings. At least in part, some of these challenges related from “losing sight” of research findings are derived from the position that ethnographers might occupy in the organizational landscape, and the visibility and access that such position might afford them. At the time of our study, we didn’t have enough visibility and access to the corporate strategy and direction, which in turn made it challenging for us to message the findings in ways that would be compelling to senior management. Several more years and projects had to pass before a team dedicated to similar work was constituted and our work better positioned in a way that provided us visibility and the means to manage uptake.

In addition to these challenges, we also face the problem of being disconnected from the results of our work. We know that our findings have been used to build a number of business cases, but for the most part we have learned about them after the fact. Not too long ago, we learned that the COO was speaking with enthusiasm about a pie chart that showed where sales representatives were spending their time. A member of the leadership had to explain to the COO that the person who had done that research worked in their group. On one hand, it is fantastic to see that the research has had such lasting value; because so little has changed since for the salespeople, so most of the findings remain quite valid. One of the interesting things here is that the company has changed, sponsors and stakeholders have changed, but much of the day to day work that was originally the subject of study has not changed all that much in the intervening years. What is bothersome is that the team is not being recognized or acknowledged. As individual practitioners gain recognition and teams dedicated to similar practices emerge, it is common that resources need to be justified on a regular basis; having our work mis-interpreted (or not getting recognized for our work) can present a significant long-term risk for resource justification. Therefore, in such situations one has to be even more careful to ensure that proper work recognition is assured.

However, at that time, the entire concept of User Experience was still being proven. As such, the team’s primary charter was in technical realms, for example bridging between the business and IT or standard development. These factors influenced the way we thought of the solution space and its presentation – i.e. many of the recommendations were of technical nature and presented in that way. Some of the findings were in fact used to design and implement a small internal software application to support important but previously unidentified concerns of sales personnel. However, without significant resources at our disposal, it was extremely challenging to demonstrate the ways in which technical recommendations could have larger business impact. Perhaps most importantly, having a sole researcher on the project made it extremely difficult to manage the presentation of outcomes in ways that would be simple and compelling enough to engage an executive audience and sustain the visibility of the work. In hindsight, it is fortunate that our work was able to have any impact at all, even if in some cases such impact was not completely aligned with our intentions as researchers.

Vignette II. Projecting retrospective inquiry as relevant to the present and to a set of envisioned futures

Despite the troubled visibility of some of the artifacts produced from the initial study, it cannot be denied that eventually that research and other changes played a significant role in making our expertise visible and opening opportunities for further ethnographic work. A few years later, the regional Chief Information Officer came to us to find out what further research, if any, had been done on sales personnel. His interest was around the ways that the existing Customer Relationship Management (CRM) software was used by sales personnel to keep track of activities with their customers and prospects, communicate projected revenue by quarter, etc. Although we had conducted numerous user research studies within the sales and marketing organization (a few of them using ethnographic methods) none of them directly answered his questions. However, exploring the relevance of prior insights in new situations has become a way of opening up new opportunities for organizational contributions and, naturally, we were not going to miss an opportunity to present our work to the CIO! As a result, research findings from multiple prior studies were used to prepare a summary of what had been learned about the CRM implementation. In a personal meeting with the CIO, a general overview presentation of all the research projects was provided and then, through the course of the discussions, specific topic areas and supporting materials were reviewed based on what appeared most relevant for the questions at hand. While the overview presentation provided the framework for discussion, however, the majority of the data and the rich stories were exchanged during that meeting were anecdotal, drawn from memory and reconstructed based on the questions the executive was asking.

As a result of this “retrospective” presentation and the dialog that ensued, we had a chance to learn about a new program intended to bring improvements to the internal implementation of CRM. As it turns out, many of the areas that had been identified in the original ethnographic research on salespeople continued to be a challenge several years later. Most importantly, through our retrospective review, we had succeeded in making the sales point-of-view visible for the CIO and his project team. Inclusion of the users’ perspective at the outset of the project would be critical to ensure uptake on the improvements that were planned for the system. Some of our prior research, for instance, had shown that the sales people actually spent very little time online in front of a computer; they were hyper-connected, but it was largely via a Blackberry or mobile phone, and not using a computer browser. Even when sitting at their desks (which were equipped with land lines), they more often than not opted to use their cell phones. Of the time they spent in front of their computer, most of it was spent looking at market news and trends through a personalized service like Yahoo! Finance, and only a very short amount of time each week was spent in the CRM application itself. One insight we were able to bring to the conversation at the time was that the sales representative perceived the tool as a vehicle for management reporting, and therefore only maintained the fields that they knew would end up on an executive report, or that impacted how and how much they were being paid for the software licenses they sold.

Using historical information, we were able to present the system user’s point of view in a way that was unique inside SAP. Prospectively, we were able to position not only our expertise but the ethnographic approach as strategic and useful. After years of practice talking about the findings, we were finally confident that we were both sought for and able to speak to that point of view in a way that was compelling at the executive level. It was more than two years after that initial study, but it was the first time that we had been approached by a member of the executive team to provide insights on users at the outset of the project. Even more importantly, the research was now being used to enable strategy, prioritization, and funding decisions, and not simply system usability.