What is AI reasoning?

AI, particularly large language models (LLMs), can perform tasks previously thought to need human-level intelligence. One capability that makes LLMs useful is their ability to "reason." But what does this mean for a machine, and how do we know if it's doing it well? This article will explore the idea of reasoning in the context of LLMs, touch on how we can evaluate it, and provide some simple examples to make the concepts clear.

Reasoning Defined for LLMs

For a human, reasoning involves using logic and prior knowledge to arrive at a conclusion. It is the process of thinking about things in a sensible way. In the case of LLMs, reasoning isn't identical to human thought, but it does involve a process of analyzing information and generating responses that seem logical based on the data the model was trained on. LLMs can combine different pieces of information, infer relationships, and solve problems that require multiple steps, although it does not have a consciousness. When a model does this successfully, we say that it is reasoning, or that it is showing "reasoning" behavior.

Essentially, it involves the model processing the input, activating relevant information it has learned in its training process, and then following a process of applying that knowledge to produce an appropriate output. It can follow logical flows. The output should not be random, it has to make sense. It tries to connect input with output in a way that a human can follow. It is not a perfect system and sometimes makes mistakes.

Evaluating Reasoning in LLMs

Evaluating reasoning capabilities in LLMs isn't easy because, unlike direct tasks such as translation or summarization, there isn't one single "correct" answer. Often, we are looking for a process of thought that makes sense, and not just an answer that is factually correct. Here are some methods that people use:

One method is to look at complex problem-solving tasks. We can provide a model with questions that have multiple steps that require careful thinking. For example, we can give a math question that involves several steps of calculations, or give a logical puzzle. The question needs several steps to solve. Then we check not only the final answer, but also see the process of solving that problem. If a model can show a correct method, it can be counted as a sign that it can "reason".

Another method is to examine if a model can make correct inferences. Given some factual statements, can a model correctly conclude something based on this? For example, if we say "All cats are mammals. Fluffy is a cat," can the model correctly conclude "Fluffy is a mammal"? The ability to make these kinds of inferences shows that it has a reasoning capability. We can test this with more complex examples.

Also, we can test with questions that need "common sense," where we try to understand if the model can understand general rules of the world, like "If it is raining, you need an umbrella." We can ask a question about an unusual situation to see if it can generate an appropriate answer. The model needs some sense of how the world works and use that in its process.

Finally, we check if a model can produce a sensible step-by-step explanation for its solution. Just outputting a correct answer is not enough. To really confirm that it "reasons" through a problem, we expect it to show its process. This might involve generating step-by-step explanations in text or code or showing the logic behind its choice of answer. This shows more clearly what is going on "inside" the model.

Simple Examples

Let's use some examples to show this.

Math Problem:

• Question: "If a train travels at 100 km per hour for 3 hours, and then at 80 km per hour for 2 hours, how far has it traveled in total?"
• A model that shows reasoning will not only provide the final answer (460 km), but it will also provide the steps, such as: "First it travel 100 * 3 = 300 km. Then, it travels 80 * 2 = 160 km. And total is 300 + 160 = 460 km."
• A model that only provides the number can not show that it reasons through the problem.

Inference:

• Statements: "All birds can fly. Penguins are birds."
• Question: "Can penguins fly?"
• A model that shows reasoning will answer "no" because of the "all" statement in the premise is false. This question tests the model's ability to use the rules of logic to draw conclusions.

Common Sense:

• Question: "What should you do if you are feeling very cold and have no heater?"
• A model that shows reasoning will not say, "You should put on a coat or wrap yourself in blankets". A model that doesn't have common sense might suggest going to an outdoor pool. A right model uses information about the world to produce an output.

Explanation:

• Question: "Why does an apple fall down when you drop it instead of floating up?"
• A good model should be able to provide explanation that indicates it understands the concept of gravity. Such as " Because of the force of gravity, it pulls things to earth". Without explanation, the model might have just memorized the fact, but it is not reasoning about gravity.

These examples illustrate that, although LLMs don't "think" in the same way as humans, they can perform tasks that require a process of using knowledge, logic and steps to come up with an answer. Evaluating the model's process, not just its final result, is how we can measure this reasoning.