Research

Teaching AI to Ask
Before It Answers

Our research measures and improves a fundamental capability missing from current AI systems: recognizing gaps in available information and asking targeted questions to fill them.

Experiment 1

The Question-Asking Experiment

We gave AI models professional tasks with critical information deliberately hidden. Models that could ask clarifying questions through an Oracle (simulating a knowledgeable colleague) dramatically outperformed those forced to guess.

Setup

Information Masking: Key details systematically removed from 220+ DOLOMITES tasks across 30 professional domains

Oracle Interaction: Multi-turn dialogue where models ask questions to recover hidden information before answering

Evaluation: 3-judge LLM panel scoring rubric adherence on final output quality

Results

Without questions (single-turn)0.295

With questions (multi-turn)0.499

+69% improvement

30 matched tasks, best attempt per task, Qwen 3.5 9B

Experiment 2

Blocker Detection Benchmark

Using HIL-Bench, we evaluated how well models identify task blockers in SQL-based professional scenarios. We measured whether models ask about the right missing information using precision, recall, and F1 scores.

DeepSeek V3OSSF1 = 0.764

Llama 3.3 70BOSSF1 = 0.674

Qwen 3.5 3BOSSF1 = 0.642

DeepSeek Pro V4OSSF1 = 0.558

Qwen 3.8 8BOSSF1 = 0.402

Key Metrics

Total sessions evaluated

0.612

Overall ask F1 score

54%

Sessions with good detection

2.2%

Zero F1 rate (complete failure)

Failure Patterns

Question spam (low precision)13%

Low recall (missed blockers)7%

Total failure (F1=0)2%

Experiment 3

Cross-Model Task Completion

We evaluated 11 models across 5 professional task types using rubric-based scoring. Each task was graded on specific criteria (69–83 rubric items per task) by an LLM judge panel.

Model	Tier	Avg Hit Rate
Qwen 3.8 8B	OSS-frontier	90.7%
DeepSeek V3	OSS-base	77.5%
DeepSeek Pro V4	OSS-frontier	77.0%
Claude Sonnet	Commercial	74.8%
GPT-4o	Commercial	73.5%
Gemini Flash	Commercial	72.3%
Llama 3.3 70B	OSS-base	68.3%
Gemma 4	OSS-mid	52.2%

File Format Compliance

Generating correct output formats (docx, xlsx, etc.)

Numerical Hallucination

Fabricating numbers in financial calculations instead of flagging missing data

Exact Data Extraction

Extracting precise values from reference files without paraphrasing or guessing

Citation Attribution

Correctly sourcing and attributing information to references

Subtle Instruction Following

Detecting and following implicit requirements in task descriptions

Findings

Counter-Intuitive Results

Bigger Isn't Always Better

Qwen 3.5 3B (F1: 0.642) outperforms the larger Qwen 3.8 8B (F1: 0.402) at blocker detection. Larger models tend to be more “confident” and less likely to identify gaps in their knowledge.

Question Spam Is Real

13% of sessions showed “question spam” — models asking many questions with low precision. Knowing what to ask matters more than asking a lot.

Naive SFT Degrades Performance

Simply fine-tuning models on question-asking transcripts doesn't work. Models learn surface patterns (always ask) without learning discrimination. RL-based approaches show more promise.

Task Completion ≠ Question Quality

High rubric hit rates on task output (Qwen 3.8: 90.7%) don't correlate with good question-asking. Models that complete tasks well may still guess rather than ask.

Reasoning Tokens Help

Models with chain-of-thought reasoning (DeepSeek, Qwen 3.8) show higher rubric hit rates. Extended thinking time correlates with better professional task completion.

Domain Transfer Works

Question-asking improvements from one professional domain (e.g., law) transfer to others (e.g., medicine). The skill is generalizable, not domain-specific.

Methodology

Evaluation Infrastructure

GDPval-AA v2

Professional task evaluation framework

220+ tasks across 44 occupations and 9 industries
E2B sandboxed execution environment
Up to 250 interaction turns per session
3-judge LLM panel for rubric scoring
Bradley-Terry MLE Elo rating system
Human baseline Elo = 1000

HIL-Bench

Blocker detection evaluation pipeline

SQL-based professional task scenarios
Ground-truth blocker annotations per task
Precision, recall, F1 scoring for question quality
Pattern classification (spam, partial, good)
Multi-model comparison framework
Automated session analysis pipeline

Information Masking Protocol

Controlled knowledge gap creation

Systematic removal of key information from task descriptions
Multiple masking levels (light, medium, heavy)
Domain-expert validated masking decisions
Oracle system with ground-truth answers for multi-turn recovery

Training Pipeline

From evaluation to improvement

SFT on curated question-asking trajectories (negative result: degrades quality)
RL with question quality rewards (positive: improves discrimination)
LoRA fine-tuning for efficient adaptation
Cross-domain transfer evaluation to verify generalization

Explore the Code

All benchmarks, pipelines, and training code are open source. Run the experiments yourself.

View on GitHub Get in Touch