Josh Pitzalis

AI is better at some tasks than others

The Jagged Frontier of Al Capabilities

“Multiple studies have identified what we refer to as the ‘Jagged Frontier’ – because AI is excellent at some tasks that seem hard to humans, and bad at some tasks that seem easy, it is hard to know what it is good at in advance. So, the only way to understand what AI can do is to use it and see what happens. Only with experience can you understand the shape of the frontier and learn to avoid relying on AI in cases where it does not operate well. Though the frontier of what AIs can do is constantly expanding.”

This is an excerpt from an Ethan Mollick post looking at where AI is heading in 2024. One of the studies he’s referring to found consultants who used GPT-4 completed tasks 25.1% faster and produced 40% higher quality results than consultants who didn’t use GPT-4.

What was interesting was that no one knew the best way to use GPT-4. There was no instruction manual. It was great at some tasks, and on others it failed completely.

For example, AI was great at writing fancy sonnets, but ask it to write a 50 word poem and it struggled. Language models conceptualize the world in tokens, rather than words, so it would consistently produces poems of more or less than 50 words.  Similarly, tasks like idea generation were easy but other tasks that should be easy (like basic math) were challenging.

This is what Mollick calls the “Jagged Frontier” of AI.

“Imagine a fortress wall, with some towers and battlements jutting out into the countryside, while others fold back towards the center of the castle. That wall is the capability of AI, and the further from the center, the harder the task. Everything inside the wall can be done by the AI, everything outside is hard for the AI to do. The problem is that the wall is invisible, so some tasks that might logically seem to be the same distance away from the center, and therefore equally difficult – say, writing a sonnet and an exactly 50 word poem – are actually on different sides of the wall.”

Discover more from Josh Pitzalis

Subscribe to get the latest posts sent to your email.

Josh

2 responses to “AI is better at some tasks than others”

  1. Great post. This isn’t directly relevant to your point, but I find it frustrating that most popular examples of AI use are of art. Any assessment of the effectiveness of art is subjective so on what parameters besides the fact that LLMs can get it done do we assess its ability to write a sonnet or a poem. We don’t need AI to write poems and sonnets, we need it to do the other more mundane work so we have the space to work on the poems.