Free Article: Please sign in or create a free account to access the leading collection of peer-reviewed work on ethnographic practice.
DOMAIN EXPERTISE AND HIRING PRACTICES
When we considered problem formation it was best to examine expertise as it was being recruited into the organization that had the potential to shape such problems. Our recruitment process had cognitive, technical, behavioral, and problem-solving dimensions. All the tests on potential new hires were offered in a ‘one-on-one’ context with a third person taking notes. Criteria for ranking candidates was subjective and based on who conducted the test and on the testing criteria although the criteria itself was viewed as objective with the same problems and same questions asked of each candidate. Additionally, a percentage of candidate testing was partially conducted over the phone or Skype.
We recruited and tested for what we called T-shaped skills. We were hiring for depth of skills, which represented the vertical part of the T and ability to collaborate in cross-functional teams, which represented horizontal part of the T. Data scientists/engineers were all supposed to possess T-shaped skills.
Nevertheless, the overall hiring process ran against this T-shaped approach often identifying skills only useful for coding, and setting aside other skills as consultative, domain specific or non-T-shaped. This blind spot in our hiring was rarely acknowledged although we urgently needed skills focused on disease classification, high mortality cancer and a complex healthcare system, all outside of a typical coding experience and seen as “other” skills. These other skills were not T-shaped but rather singular, particular, walled-in domain knowledge. Broadly, we needed T-shaped radiologists, business leaders, nurses, patient advocates, IT managers and operational managers, we needed a T-shaped organization. Instead we hired data scientists. What this did to us and to our organization was to miss and not encourage people in roles who could translate forms of clinical and technical knowledge across product, business and algorithm development, we missed a translational T-shaped organizational culture.
How did this “T-shaped” hiring process that missed good cross-domain knowledge got expressed in the process of algorithm development?
Domain Knowledge: Universality and Particularity
We arrived at a company retreat by a northern California lakeside surrounded by redwood and pine forests. Swimming pool and cluster of cottages were set among a small gravel parking lot and hiking trails that ascended the hillsides. The retreat was called in somewhat a haste and before we knew, we were all gathered together to discover improved teamwork and sharing of ideas. All our activities were constructed to give everyone a voice, to ask questions and make suggestions on how we could improve the company. But a strange pattern emerged. As machine learning engineers and data scientists held more and more air time, business development, operations, a few clinical consultants had less and less time to make suggestions. Throughout the retreat the problem increased with less and less time allotted for cross-functional expertise outside of coding to offer opinions. The emerging organization was getting locked into its own limited self-perception.
On the last day of the retreat with only 20 mins left a colleague raised his hand and questioned why so few suggestions were asked or offered outside of engineering. The room got quiet. The CEO said “that’s not true, everyone has been included.” Then my colleague went further. I will call him Paul.
“You’ve been soliciting suggestions from everyone in your mind, but in this room it has been with only maybe half the company.”
The CEO expressed disbelief: “no one’s been excluded!” And then, “it’s everyone’s responsibility to speak up.”
The actuality was different. When someone spoke up about problems or solutions outside of coding, the Q&A would move quickly on to the next person. The function of not having enough time to be democratic during a company retreat was not the point. It was a function of certain forms of knowledge that were recognized and other forms of knowledge left unrecognized or invisible. A group company retreat was in fact a fractured retreat of what one colleague said “ideas to be considered” and “ideas that were awesome,” and “creative” and “productive”. In this context the ideas that received the greatest attention concerned a narrow focus on machine learning algorithms, medicine as strictly a data problem at scale, machine learning expertise as universal knowledge, and team development and cohesiveness around these issues.
What was relegated to the “water cooler” conversations were ideas regarding lack of product direction, building and recruiting a clinical team with cross-functional responsibilities, leadership issues, a diverse organization at scale, work/life balance and low team morale.
Let us briefly unpack these ideas as they were expressed. The “awesome” ideas regarded machine learning as scalable, central to organizational development, around which all other forms of knowledge revolved. Outside this sphere clinical and business knowledge were consultative or being driven by algorithm development. Machine learning knowledge was the epistemological glue that bounded all other operational forms of technical and organizational mastery.
The second-tier ideas revolved around the organization and its people with their knowledge and experience. Disregarded clinical, cultural, retrospective feedback on mistakes and product direction could in fact act as a key to the emotional and professional bonds that reached inside and outside work-life balance. We believe the seeds of innovation were contained here, in this second tier of ideas that could not be well heard. The organization went on retreats and participated in team exercises but their contributions were not well captured nor well understood. The idea of what was universal and what was particular (domain-specific) knowledge were typically translated into concepts that were framed as “useful” or “not useful.”
Read or Experience
The following conversation between a team member and a radiologist, and later the reporting of their conversation to the team may serve to expand on this universalizing-particularizing tension in our machine learning start-up.
“How do you see that [spiculation]?” – a colleague who we will call Tim.5
“I compare it [nodule] to others I’ve seen” – the radiologist who we will call Don.
“It’s what you have seen or what you know from what you’ve read?”
“Years [of] experience” – Don explains.
In our team meeting comprised of product and engineering this conversation was translated by Tim as a certain kind of data-driven opportunity for algorithm development.
“You guys, we can detect nodules by edge characteristics. Radiologists have diagnostics in their head and I think we can get the same diagnostics from imaging.” Tim’s comment became clearer and got reinforced in the following.
“So, we hire radiologists to annotate and we build, annotate and build, right” – says the CEO.
“Not that simple but that’s the idea, we bring in domain experts.”
“We plug them in or we hire them full time?
“I’m not sure we’ll know what to do with them full time, the best would be consultants.” Tim speculates.
Domain expertise has been established here as a contributory source of expertise. It provided a certain kind of fuel to algorithm development but was not valued as central to such development. It could be read and learned without having to be experienced. Radiology expertise was reduced to an “industrial designer: everything involved in a situation, any kind of material, must be calibrated for its contribution as evidence” stated anthropologist Marilyn Strathern (2004). The radiologist as a domain expert was seen as delivering a snap shot of knowledge and yet providing knowledge gained over years of experience of disease identification and classification. Except, that experience was disregarded as “useful.” This artifact had to be used as evidence that could be codified into algorithm development. Such expertise was not seen as evolving and aspirational, it was seen “useful” only as evidence for creating clinical labels for algorithms. The team’s excitement of using radiologists as consultants and not knowing what “to do with them full time” had reduced radiologists’ value into a kind of automating piece work.
Contemporary radiology was often seen as a medical science of piece work. The radiologist has been well acquainted with holding a consultative role across medical disciplines with the widespread use of high-speed broadband and three decades of teleradiology. Medical images travel instantly, as radiologist interpretation travels remotely to hospitals and clinics, new imaging technologies continue to develop. Radiologists have been participants in a somewhat similar experience of reducing knowledge and their own knowledge to a commodity. Speed, distribution of radiology report and harsh efficiency have long attended upon radiologists. At first sight, it did seem to be a perfect collusion between algorithm development and teleradiology practice. However, the piece-working of radiology practice as a discipline into teleradiology practice has been different from its use as domain knowledge in a machine learning start up. The difference was radiologist’s autonomy and agency to practice their craft that has always been held in high regard and to develop and perfect their skills has been an ongoing commitment. In an algorithm development context, the opportunity to perfect radiologist skills went one direction, toward algorithm building and scale, radiologist-consultants were plugged in and the race to gain new diagnostic insight has been transformed into a race to produce diagnostic evidence useful for algorithm iteration.
One of radiologists’ key concerns at the turn of the 20th century was how to enable physicians to quickly reason with diagnostic insight while accompanying patients at bedside. One of the key medical concerns today has a twist: how radiologist’s insight is to collaborate with/aid machine intelligence to make life saving diagnostic judgments across billions of data points in an instant.