Assessing the Abilities of AI Systems

---

This section provides a framework and background information for assessing the abilities of current AI technologies in performing human tasks. This section also explore the foreseeable improvements based on current research activities.

AI abilities and capabilities are commonly measured on academic ‘benchmarks’ specifically designed for AI systems. Most of them are adapted from tests applied to humans. Examples of common benchmarks include:

• MMLU - Evaluates an AI's ability to understand multiple languages and perform a wide range of natural language understanding tasks like translation, question answering, sentiment analysis that are important for customer service roles.
• SQuAD (Stanford Question Answering Dataset) - This is one of the largest and most widely used question answering datasets. The ability to understand text and accurately answer questions is important for many roles.
• GLUE (General Language Understanding Evaluation) - GLUE contains a diverse range of language tasks and has become a standard evaluation for natural language understanding capabilities. Several capabilities it measures like named entity recognition and sentiment analysis are applicable to many professions.
• ANLI benchmark (Tests multi-step logical reasoning by introducing contradictory information).

In addition, advanced AI models, notably LLMs are now commonly tested on professional exams designed for humans such as bar exams, Advanced Placement in environmental science, History, psychology, economics, math, physics, etc.