In-context Learning with Errors
Authors: Jianhao Huang, Yucheng Wang, Qihao Luo
Abstract
In-context learning (ICL) is a paradigm for large language models (LLMs) to learn to perform a new task without updating its parameters. This study investigates the capabilities of LLMs in in-context learning with errors, focusing on mathematical reasoning. Using the GSM8K and MATH datasets and a variety of prompt modifications, we explore how LLMs manage errors and modifications in demonstration steps. Our findings suggest that certain types of errors and reordering of demonstration components have minimal impact on the solve rates, thereby providing insights into the adaptability and error tolerance of in-context learning models. This research extends our understanding of the flexibility of transformers in maintaining performance despite modifications, positioning it as a step forward in the practical application of in-context learning methods in AI.