The original setting (gpt-4o-2024-11-20 as the judge, greedy decoding with temperature 0 for the evaluated model, default mode only) no longer reflects how current models are released and used: vendors ship explicit thinking / non-thinking modes and recommended sampling parameters, and nobody deploys these models at temperature 0. To evaluate models the way they are actually used, we updated the setting as follows:

• Judge model: gpt-5.6-luna with temperature 1, reasoning_effort=medium, max_completion_tokens=8192. Every llm constraint is judged by it; code constraints run the annotated check function.
• Modes: models that support both thinking and non-thinking are evaluated in both modes and reported as separate rows (chat_template_kwargs.enable_thinking or the vendor's own switch).
• Decoding: the vendor-recommended sampling parameters of each model and mode (temperature, top_p, top_k, min_p, presence / repetition penalty). When the vendor gives none: thinking T=1.0, top_p=0.95; non-thinking T=0.0, top_p=0.95.
• Output budget: 32768 tokens in thinking mode, 16384 in non-thinking mode, trimmed automatically when prompt + budget exceeds the model's context window.
• Runs: 3 independent samples per (model, mode); we report mean ± sample std.
• Empty or failed responses are scored as an empty response (every constraint fails); nothing is dropped.

The exact parameters of every evaluated model are the committed configs/<model>.yaml files; the judge parameters are in judge_configs/gpt-5.6-luna.yaml.

For each instruction, we annotate the associated constraints and corresponding evaluation metrics, including code-based evaluation, LLM-based evaluation, and hybrid code-LLM evaluation. The evaluation pipeline is config-driven: run.sh starts vLLM for each local model, runs all (mode × run) predictions, and judges them on the API. See the GitHub repository for the full documentation.

How to evaluate

1. Install. The dataset (data/eval.json, 707 instructions) is included; data/smoke/eval.json holds 5 instructions for a quick test.

   git clone https://github.com/THU-KEG/AgentIF.git && cd AgentIF
   pip install -r requirements.txt            # openai, pyyaml, requests, tqdm, ...
   pip install vllm                           # only for locally served models (vLLM >= 0.26 for the models in configs/)

2. Configure. Put the judge key (and optionally a base URL and the local weights directory) in .env, then pick or add a model config in configs/.

   cp .env.example .env     # OPENAI_API_KEY (judge), optional OPENAI_BASE_URL, MODELS_DIR (local weights)

   • Local model (vLLM) — copy one of the shipped configs (e.g. configs/Qwen3.5-4B.yaml). Required keys: model_path, max_model_len, modes; vllm_args are passed to vllm serve verbatim.
   • API model — copy configs/api-model.example.yaml, set api.base_url / api.model, and put the key in the environment variable named by api.api_key_env.
   • Each entry under modes is evaluated separately with its own sampling parameters (enable_thinking: true/false, or extra_body for vendor-specific switches). Single-mode models list one mode only.
   Global behaviour (GPU pool, number of repeats, default judge, concurrency) lives in configs/global.yaml; judge endpoints and parameters in judge_configs/. To reproduce the leaderboard, use the shipped configs and the gpt-5.6-luna judge.
3. Run.

   bash run.sh --list                                        # show configs
   bash run.sh --models Qwen3.5-4B --repeats 1 \
       --input-dir data/smoke --results-root results_smoke     # smoke test
   bash run.sh --models Qwen3.5-4B                           # full benchmark for one model
   nohup bash run.sh > logs/run.out 2>&1 &                   # every enabled config
   bash run.sh --stage prediction                            # predictions only
   bash run.sh --stage evaluation                            # judge existing predictions (no GPU)
   bash run.sh --judge gpt-4o-2024-11-20                     # another judge (judge_configs/<name>.yaml)

   Finished jobs are detected by their accuracy.json and interrupted judge jobs resume from results.partial.jsonl, so re-running the same command continues where it stopped.
4. Results. Per-run predictions and scores are written under results/; results/summary.md reports mean ± std over runs for every (model, mode, judge).

   results/
     prediction/<model>__<mode>/run<N>/eval.json          # model responses + run_config.json
     scores/<model>__<mode>/run<N>_<judge>/eval/
         results.json                                      # per-constraint scores, judge outputs
         accuracy.json                                     # CSR, ISR, breakdowns by presentation / constraint type
     summary.md / summary.json                             # mean ± std over runs

Data Format

Each data instance in AgentIF is structured as follows:

{
  "input": [
    { "role": "system", "content": "..." },
    { "role": "user",   "content": "..." }
  ],
  "constraints": [
    {
      "id": 0,
      "desc": "...",                // Constraint description
      "other_info": {               // Auxiliary information for evaluation
        "...": "..."
      },
      "dimension": "...",           // Constraint Presentation Type
      "type": "...",                // Constraint Type
      "is_meta": false,             // Whether it is a meta-constraint
      "evaluation": [               // Evaluation Method
        {
          "type": "llm",            // LLM-based evaluation
          "required_keys": ["response"],
          "exec": "..."             // Evaluation prompt for LLM
        },
        {
          "type": "code",           // Code-based evaluation
          "exec": "..."             // Executable code snippet
        }
      ]
    }
  ]
}

@article{qi2026agentif,
  title={AGENTIF: Benchmarking large language models instruction following ability in agentic scenarios},
  author={Qi, Yunjia and Peng, Hao and Wang, Xiaozhi and Xin, Amy and Liu, Youfeng and Xu, Bin and Hou, Lei and Li, Juanzi},
  journal={Advances in Neural Information Processing Systems},
  volume={38},
  year={2026}
}

The dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) License.

The code in the AgentIF repository is released under the Apache License 2.0.