Black Box AIs and The Evolution of Understanding

The evolution of understanding framework (Figure 1 below), which I first introduced in The Field Study Handbook, breaks down the stages of understanding practitioners typically go through from initial hypothesis (or hunch) through to attaining wisdom.

 

FIGURE 1. The evolution of understanding

 

People often use the term “data” interchangeably with “information”, “knowledge”, and  “insight” so to make the distinctions clearer here I will refer to lowercase “data” as atomistic units—such as a raw, unprocessed data point recorded from an app, the first pass of the interview transcript, an unedited video recording or photo. Alternatively, uppercase “Data” can be used to encompass other stages in this evolution of understanding that have gone through some form of sensemaking process. 

It’s worth noting two things:

1. Skipping steps in the evolution of understanding fundamentally undermines our ability to ascertain the veracity of Data, and therefore our ability to effectively apply “insight”, and,
2. “wisdom” is generated weeks or even years later from the failed and successful application of insight. Failure is critical to wisdom generation, because it starts to put boundaries around the limitations of how insight might be applied.

We can adapt this framework to consider how generative AI tools such as ChatGPT affect the evolution of our understanding (Figure 2, below). In this example, a reliance on Generative AI output enables a person to either condense steps from weeks to minutes, or to skip some steps entirely. However, the black-box nature of the underlying AI models and their training data, means that it is impossible to gauge the veracity of outputs by today's standards, or to mitigate biases inherent to the underlying data to train the AI.

 

FIGURE 2. The Evolution of Understanding with heavy reliance on Generative AIs

 

Practitioners invested in a systematic approach to understanding a particular topic will (also) likely adopt generative tools to articulate an AI informed hypothesis, but for the layperson wanting a quick answer the generative AI output will often suffice. The cost of reliance on unchecked outputs should be obvious:

• a perception of having attained “knowledge” without being able to effectively gauge its limitations and biases;
• a superficial sense of what is insight;
• a growing inability to generate wisdom, because of limitations in understanding the context that it draws on.

 

My sense is that we'll see AI employed in three ways in organisational sensemaking:

1. As an accelerant, condensing certain existing practices from weeks to potentially minutes once systems are up and running;
2. As an extender: enabling Data to travel further in an organisation and be utilised in different, valued ways, and which also influences what kinds of Data is collected e.g. to train more representative/less biased AI models (I'll provide examples in a future article).
3. Generate novel practices of sensemaking beyond current practices. Adoption/rejection of these new practices will marked by a generational shift inside organisations, by my estimate within three years in incumbent corporations, providing plenty of short- and medium- term scope for agency service offerings (longer term, organisations will want to bring these skills in-house).

There are many brilliant researchers working on and drawing attention to related issues such as biases in training data and transparency. As a starting point I recommend reading up on research from Kate Crawford and Timnit Gebru and others.

This is a fast-moving space with frequent launches and updates to generative tools, including some that provide better context to original sources, making the edges of the black box somewhat less opaque, but at its core, a reliance on generative tools currently trades off convenience with our ability to effectively apply insight.

Footnotes

Any consultancy that provides sensemaking services goes through this evolution of understanding, although some are deliberately opaque about their "proprietary" process.

The second order effect of these shifts will be marked by lower quality benchmarks, as societal norms shift and seep into organisations.

The cyclical nature of consulting services is apparent, although I expect future cycles to be shorter in nature.

I'm currently conducting research for my new book about human and organisational sensemaking—two years thus far and five or so to go—and sharing snippets along the way. For more articles like this, join 8k subscribers drawn from many of he world's most interesting organisations on our monthly Radar mailing list.

—Jan