[ACL 2025 (Main)] RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios

Ruiwen Zhou, Wenyue Hua, Liangming Pan, Sitao Cheng, Xiaobao Wu, En Yu, William Yang Wang

• We create the RuleArena benchmark for LLM rule-guided reasoning, with a comprehensive set of 95 policies/rules drawn from real-world scenarios and 816 datapoints of different difficulty levels for LLM evaluation. We further propose a suite of evaluation metrics for both rule selection and rule application, providing fine-grained insights into LLM performances.
• Existing state-of-the-art LLMs, such as GPT-4o and Claude-3.5 Sonnet, mostly fail on complex rule-guided reasoning tasks from RuleArena. By examining common failure cases and identifying difficult rule types, we uncover several systematic issues that limit LLMs’ rule-guided reasoning capabilities.

[ACL 2025 (Main)] AntiLeak-Bench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge

Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Shuai Zhao, Anh Tuan Luu, William Yang Wang

• We create the AntiLeak-Bench framework, which collects samples with explicitly new knowledge absent from LLMs’ training sets, and thus ensures strictly contamination-free evaluation. This building workflow is also automated for maintainance and update without human labor.
• Experiments show that pre-cutoff samples come with data contamination, which inflates LLMs’ performance, while contamination-free post-cutoff samples collected by our AntiLeak-Bench are more challenging and can more accurately assess LLMs.

[SIGIR 2024] TRAD: Enhancing LLM Agents with Step-Wise Thought Retrieval and Aligned Decision

Ruiwen Zhou, Yingxuan Yang, Muning Wen, Ying Wen, Wenhao Wang, Chunling Xi, Guoqiang Xu, Yong Yu, Weinan Zhang

• We propose a novel retrieval-augmented LLM agent TRAD. It achieves step-wise relevant demonstration selection via thought retrieval (TR), and applies an aligned decision (AD) module to augment LLM with temporal neighboring steps and their corresponding order information of retrieved demonstrations during action prediction.
• TRAD significantly outperforms existing SoTA LLM agents such as Synapse and ReAct on common LLM agent benchmarks like ALFWorld (householding) and Mind2Web (web navigation).

[arXiv preprint] Is Risk-Sensitive Reinforcement Learning Properly Resolved?

Ruiwen Zhou, Minghuan Liu, Kan Ren, Xufang Luo, Weinan Zhang, Dongsheng Li

• We reveal the improper optimization in existing distributional risk-sensitive RL (RSRL) algorithms, showing that maximizing distortion risk measure at every step in a dynamic programming style leads to divergence.
• To address this issue, we propose a novel RSRL algorithm Trajectory Q-Learning (TQL). By incorporating history return distribution, TQL achieves theoretical optimality and outperforms existing methods in RSRL tasks.

[NeurIPS 2022] Learning Enhanced Representations for Tabular Data via Neighborhood Propagation

Kounianhua Du, Weinan Zhang, Ruiwen Zhou, Yangkun Wang, Xilong Zhao, Jiarui Jin, Quan Gan, Zheng Zhang, David Wipf

• We propose a GNN-based tabular prediction model PET. It considers both row-wise (across sample) and column-wise (across feature) feature interaction by transforming each target sample with retrieved relevant sample into a hypergraph and conducting message passing on it.
• On a wide range of tabular prediction tasks like CTR prediction and top-n recommendation, PET surpasses SoTA methods by a large margin.