Submodular Context Partitioning and Compression for In-Context Learning

Research assistant. With Prof. Shengjie Wang and Prof. Tianyi Zhou.

arXiv

Overview

In-context learning (ICL) enables efficient few-shot learning without training, but suffers from quadratic input complexity of transformers. We propose Sub-CP, a block-aware context selection framework that leverages submodular objectives to control block diversity.

Contributions

• Studied Sub-CP, a block-aware context selection framework for in-context learning that leverages submodular objectives to balance global diversity and local coherence across demonstration context blocks.
• Implemented ensemble-based evaluations under the DENSE ICL framework using Mistral-7B, benchmarking Sub-CP against uniform and random partition strategies across five standard NLP datasets.
• Achieved up to +7.1% average F1 score over baseline partition methods and +15.4% F1 score on TREC.