Bloom's Taxonomy & AI

Bloom’s Taxonomy classifies levels of thinking — Evaluation is the highest form—AI achieving human-level intelligence will require an ability to evaluate.

VIEW AS PDF

BLOOM’S TAXONOMY & AI

Bloom’s taxonomy was tacked to the wall of my second grade SCOPE class. The class was held in a dingy portable in the school parking lot. This colorful image was more interesting than the brick wall out the window and has stuck with me for years.

Moving up the pyramid shows higher orders of thinking. This structure also maps well to careers, where coveted jobs are often focused on evaluation while analysts perform lower orders of work to inform decision makers.

Bloom’s AI

This taxonomy is ideal for understanding AI’s problems and potential. Moving up the pyramid takes effort and training in how to think. To date, most AI improvements have been driven by simply providing more data, broadening the AI’s knowledge base, as opposed to enhancing the intelligence of the AI itself.

The Bottom of Pyramid

Today, AI is superhuman in the bottom of the pyramid, with nearly perfect knowledge and evolved comprehension of its training materials. For instance, ChatGPT has almost infinite depth when asked for a recipe (knowledge/comprehension) or when provided some ingredients and asked to construct a dish (application).

AI is also making progress into the middle of the pyramid. ChatGPT may be able to explain why a recipe includes specific ingredients and how these ingredients ultimately combine to enhance flavor, a successful analysis of a recipe. At this level though, a trained chef may be able to identify flaws in ChatGPT’s logic.

Certainly in the investment realm, I have yet to see a well constructed analysis of a stock and why it would be a good investment produced by AI. AI’s results in this task may look good, but upon deeper inspection typically fail to be compelling.

Whether or not AI can achieve higher levels of thinking remains an open question, particularly when the large language models have been trained on the internet – ultimately an “average” of the intelligence of humans.

Synthesis & Evaluation

Creativity lives at the top of the pyramid. Combining ideas from different fields in unique ways results in true, original creation. Evaluating these ideas as to whether they are groundbreaking or mundane is the apex of thinking.

Asking ChatGPT to recommend a recipe for dinner is far different than asking for original recipes that combine to create a unique menu for a restaurant opening. Even if AI succeeded in this task, could it continue to modify the menu seasonally with equal originality?

Higher orders of thinking are required for successful execution in high stake environments. It is this thinking that creates value—a new experience for guests or a differentiated view on a underappreciated stock. The value AI will ultimately create will depend on how well it can think.

A Conscious Decision

Before we turn over the power of evaluation to AI we will need to understand its thinking. Successful evaluation in the past is not enough; we must know the reason why the AI made a decision.

Trust requires both a successful track record as well as a clear articulation of the reasons why. This articulation is a challenge for AI. AI models are ultimately “black box” in the way they are trained—we cannot see inside to understand the logic. Worse, many of today’s AI issues are revealed when pressing an LLM in conversation. Repeatedly asking “why” or for data support often results in AI hallucination as the model struggles against revealing its “thinking.”

Coin Flip Callers & AI

Even if we see examples of successful AI evaluation – say an AI operated portfolio that outperforms for 3 or 5 years, can we believe it? This is a question that has faced human active managers for a century. Is Warren Buffett skilled or just the luckiest active manager to have ever lived?

Active managers were previously compared to coin flip guessers—if all Americans entered a coin flip contest many would inevitably guess 10 flips in a row correctly. Despite this amazing track record, they should not be trusted to guess the next flip.

Warren Buffett addressed this issue in his article about “Superinvestors.” He concedes that it is a statistical possibility that he is lucky, but believes it is a statistical impossibility for so many of Benjamin Graham’s students, all value investors, to have performed so well.

How will we be able to tell that a particular AI is truly skilled vs. just one of a trillion AI’s that took on the challenge and ultimately won? It will takes decades of data in the public markets to even identify AI skill, then the investigation will begin on how they were trained to understand if they were just lucky or if a similar philosophy was developed by the successful AI managers.

Evaluation Squared

Operating at the highest level of Bloom’s Taxonomy is challenging. When humans must evaluate options to make decisions we know we may be wrong. Allocators have the incredible challenge of evaluating stock evaluators—no easy feat!

Similarly, evaluating how well AI will perform at evaluation, or any advanced level of thinking, will pose a challenge for years to come.