{
    "unconditional": [
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称、公司简称或公司代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_legal_document_company_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "关联公司名称，形如xxx有限公司",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register_name', 'takes_inputs': {'identifier': {'description': '统一社会信用代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_info', 'takes_inputs': {'identifier': {'description': '公司名称、公司简称或公司代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_legal_document_company_list', 'takes_inputs': {'identifier': {'description': '关联公司名称，形如xxx有限公司', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "每次使用预设函数后都应该将结果打印出来",
            "other_info": {
                "from": "system_para_1",
                "available_functions": [
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称、公司简称或公司代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_legal_document_company_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "关联公司名称，形如xxx有限公司",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ]
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查下面文本中的每个函数调用表达式，判断其下一个执行语句是否为print。直接回答YES或NO，不要任何其他内容。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "您可以在代码中使用import，但只能从以下模块列表中导入",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称、公司简称或公司代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_info', 'takes_inputs': {'identifier': {'description': '公司名称、公司简称或公司代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info_list', 'takes_inputs': {'identifier': {'description': '母公司的名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register_name', 'takes_inputs': {'identifier': {'description': '统一社会信用代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "您可以在代码中使用import，但只能从以下模块列表中导入",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 0,
            "desc": "您可以使用一系列functions。",
            "other_info": {
                "function": [
                    "get_sub_company_info",
                    "get_company_info",
                    "get_company_register",
                    "get_sub_company_info_list",
                    "get_company_register_name",
                    "finish"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "请帮我提取出下面这段话中**当前步**使用的函数名称，不关注未来要做的事。\n请仅回答函数名称，并以list形式直接返回给我，不要回答其他的内容。\n\n{response}\n"
                },
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    import re\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    try:\n        tools = ['get_sub_company_info', 'get_company_info', 'get_company_register', 'get_sub_company_info_list', 'get_company_register_name', 'finish']\n        functions = extract_tool(response)\n        if isinstance(functions, list):\n            for item in functions:\n                if item not in tools:\n                    return False\n        return True\n    except Exception as e:\n        print(\"error in resource checker:\", e)\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 1,
            "desc": "注意不要在一次thought中连续使用多个functions。",
            "other_info": {
                "function": [
                    "get_sub_company_info",
                    "get_company_info",
                    "get_company_register",
                    "get_sub_company_info_list",
                    "get_company_register_name",
                    "finish"
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "请帮我提取出下面这段话中**当前步**使用的函数名称，不关注未来要做的事。\n请仅回答函数名称，并以list形式直接返回给我，不要回答其他的内容。\n\n{response}\n"
                },
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    import re\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    try:\n        tools = ['get_sub_company_info', 'get_company_info', 'get_company_register', 'get_sub_company_info_list', 'get_company_register_name', 'finish']\n        functions = extract_tool(response)\n        if len(functions) <= 1:\n            return True\n        return False\n    except Exception as e:\n        print(\"error in resource checker count:\", e)\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 4,
            "desc": "除了第一轮外，对上一轮的<observation></observation>进行总结。",
            "other_info": {
                "from": "system_para_2"
            },
            "dimension": "unconditional",
            "type": "semantic",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "response是否在一开始就总结了上一轮的<observation></observation>？直接回答 YES 或 NO。\n\nresponse: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 4,
            "desc": "除了第一轮外，对上一轮的<observation></observation>进行总结。",
            "other_info": {
                "from": "system_para_2"
            },
            "dimension": "unconditional",
            "type": "semantic",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "response是否在一开始就总结了上一轮的<observation></observation>？直接回答 YES 或 NO。\n\nresponse: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称、公司简称或公司代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_court_code",
                        "takes_inputs": {
                            "identifier": {
                                "description": "法院名称或者法院代字",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'get_company_register_name', 'takes_inputs': {'identifier': {'description': '统一社会信用代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_info', 'takes_inputs': {'identifier': {'description': '公司名称、公司简称或公司代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_court_code', 'takes_inputs': {'identifier': {'description': '法院名称或者法院代字', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "不要将任何新变量命名为与functions相同的名称",
            "other_info": {
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "response中新变量命名是否与functions的名称不同，请直接回答YES或NO。Function_list: ['get_company_register_name', 'get_company_register', 'get_sub_company_info', 'get_company_info', 'get_court_code']\n\nresponse: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 4,
            "desc": "除了第一轮外，对上一轮的<observation></observation>进行总结。",
            "other_info": {
                "from": "system_para_2"
            },
            "dimension": "unconditional",
            "type": "semantic",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "response是否在一开始就总结了上一轮的<observation></observation>？直接回答 YES 或 NO。\n\nresponse: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "您可以在代码中使用import，但只能从以下模块列表中导入",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称、公司简称或公司代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'get_company_info', 'takes_inputs': {'identifier': {'description': '公司名称、公司简称或公司代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register_name', 'takes_inputs': {'identifier': {'description': '统一社会信用代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info_list', 'takes_inputs': {'identifier': {'description': '母公司的名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "您可以在代码中使用import，但只能从以下模块列表中导入",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_address_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司地址",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'get_company_register_name', 'takes_inputs': {'identifier': {'description': '统一社会信用代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_address_info', 'takes_inputs': {'identifier': {'description': '公司地址', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info_list', 'takes_inputs': {'identifier': {'description': '母公司的名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "每次使用预设函数后都应该将结果打印出来",
            "other_info": {
                "from": "system_para_1",
                "available_functions": [
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_address_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司地址",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ]
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查下面文本中的每个函数调用表达式，判断其下一个执行语句是否为print。直接回答YES或NO，不要任何其他内容。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "您可以在代码中使用import，但只能从以下模块列表中导入",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称、公司简称或公司代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'get_company_info', 'takes_inputs': {'identifier': {'description': '公司名称、公司简称或公司代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info_list', 'takes_inputs': {'identifier': {'description': '母公司的名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register_name', 'takes_inputs': {'identifier': {'description': '统一社会信用代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "您可以在代码中使用import，但只能从以下模块列表中导入",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 0,
            "desc": "您可以使用一系列functions。",
            "other_info": {
                "function": [
                    "get_company_info",
                    "get_company_register_name",
                    "get_sub_company_info",
                    "get_sub_company_info_list",
                    "get_company_register",
                    "finish"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "请帮我提取出下面这段话中**当前步**使用的函数名称，不关注未来要做的事。\n请仅回答函数名称，并以list形式直接返回给我，不要回答其他的内容。\n\n{response}\n"
                },
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    import re\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    try:\n        tools = ['get_company_info', 'get_company_register_name', 'get_sub_company_info', 'get_sub_company_info_list', 'get_company_register', 'finish']\n        functions = extract_tool(response)\n        if isinstance(functions, list):\n            for item in functions:\n                if item not in tools:\n                    return False\n        return True\n    except Exception as e:\n        print(\"error in resource checker:\", e)\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 1,
            "desc": "注意不要在一次thought中连续使用多个functions。",
            "other_info": {
                "function": [
                    "get_company_info",
                    "get_company_register_name",
                    "get_sub_company_info",
                    "get_sub_company_info_list",
                    "get_company_register",
                    "finish"
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "请帮我提取出下面这段话中**当前步**使用的函数名称，不关注未来要做的事。\n请仅回答函数名称，并以list形式直接返回给我，不要回答其他的内容。\n\n{response}\n"
                },
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    import re\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    try:\n        tools = ['get_company_info', 'get_company_register_name', 'get_sub_company_info', 'get_sub_company_info_list', 'get_company_register', 'finish']\n        functions = extract_tool(response)\n        if len(functions) <= 1:\n            return True\n        return False\n    except Exception as e:\n        print(\"error in resource checker count:\", e)\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 4,
            "desc": "除了第一轮外，对上一轮的<observation></observation>进行总结。",
            "other_info": {
                "from": "system_para_2"
            },
            "dimension": "unconditional",
            "type": "semantic",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "response是否在一开始就总结了上一轮的<observation></observation>？直接回答 YES 或 NO。\n\nresponse: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称、公司简称或公司代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_finalized_case_company_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "终本公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'get_sub_company_info_list', 'takes_inputs': {'identifier': {'description': '母公司的名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_info', 'takes_inputs': {'identifier': {'description': '公司名称、公司简称或公司代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_finalized_case_company_list', 'takes_inputs': {'identifier': {'description': '终本公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "您可以在代码中使用import，但只能从以下模块列表中导入",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称、公司简称或公司代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_legal_document_company_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "关联公司名称，形如xxx有限公司",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info_list', 'takes_inputs': {'identifier': {'description': '母公司的名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_info', 'takes_inputs': {'identifier': {'description': '公司名称、公司简称或公司代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_legal_document_company_list', 'takes_inputs': {'identifier': {'description': '关联公司名称，形如xxx有限公司', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "每次使用预设函数后都应该将结果打印出来",
            "other_info": {
                "from": "system_para_1",
                "available_functions": [
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称、公司简称或公司代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_legal_document_company_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "关联公司名称，形如xxx有限公司",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ]
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查下面文本中的每个函数调用表达式，判断其下一个执行语句是否为print。直接回答YES或NO，不要任何其他内容。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 0,
            "desc": "您可以使用一系列functions。",
            "other_info": {
                "function": [
                    "get_sub_company_info",
                    "get_sub_company_info_list",
                    "get_company_info",
                    "get_address_info",
                    "get_company_register",
                    "finish"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "请帮我提取出下面这段话中**当前步**使用的函数名称，不关注未来要做的事。\n请仅回答函数名称，并以list形式直接返回给我，不要回答其他的内容。\n\n{response}\n"
                },
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    import re\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    try:\n        tools = ['get_sub_company_info', 'get_sub_company_info_list', 'get_company_info', 'get_address_info', 'get_company_register', 'finish']\n        functions = extract_tool(response)\n        if isinstance(functions, list):\n            for item in functions:\n                if item not in tools:\n                    return False\n        return True\n    except Exception as e:\n        print(\"error in resource checker:\", e)\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 1,
            "desc": "注意不要在一次thought中连续使用多个functions。",
            "other_info": {
                "function": [
                    "get_sub_company_info",
                    "get_sub_company_info_list",
                    "get_company_info",
                    "get_address_info",
                    "get_company_register",
                    "finish"
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "请帮我提取出下面这段话中**当前步**使用的函数名称，不关注未来要做的事。\n请仅回答函数名称，并以list形式直接返回给我，不要回答其他的内容。\n\n{response}\n"
                },
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    import re\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    try:\n        tools = ['get_sub_company_info', 'get_sub_company_info_list', 'get_company_info', 'get_address_info', 'get_company_register', 'finish']\n        functions = extract_tool(response)\n        if len(functions) <= 1:\n            return True\n        return False\n    except Exception as e:\n        print(\"error in resource checker count:\", e)\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 4,
            "desc": "除了第一轮外，对上一轮的<observation></observation>进行总结。",
            "other_info": {
                "from": "system_para_2"
            },
            "dimension": "unconditional",
            "type": "semantic",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "response是否在一开始就总结了上一轮的<observation></observation>？直接回答 YES 或 NO。\n\nresponse: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 4,
            "desc": "除了第一轮外，对上一轮的<observation></observation>进行总结。",
            "other_info": {
                "from": "system_para_2"
            },
            "dimension": "unconditional",
            "type": "semantic",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "response是否在一开始就总结了上一轮的<observation></observation>？直接回答 YES 或 NO。\n\nresponse: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称、公司简称或公司代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'get_company_register_name', 'takes_inputs': {'identifier': {'description': '统一社会信用代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_info', 'takes_inputs': {'identifier': {'description': '公司名称、公司简称或公司代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info_list', 'takes_inputs': {'identifier': {'description': '母公司的名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "您可以在代码中使用import，但只能从以下模块列表中导入",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称、公司简称或公司代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_finalized_case_company_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "终本公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'get_company_register_name', 'takes_inputs': {'identifier': {'description': '统一社会信用代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_info', 'takes_inputs': {'identifier': {'description': '公司名称、公司简称或公司代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_finalized_case_company_list', 'takes_inputs': {'identifier': {'description': '终本公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "您可以在代码中使用import，但只能从以下模块列表中导入",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 0,
            "desc": "您可以使用一系列functions。",
            "other_info": {
                "function": [
                    "get_company_register_name",
                    "get_company_register",
                    "get_company_info",
                    "get_sub_company_info",
                    "get_sub_company_info_list",
                    "finish"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "请帮我提取出下面这段话中**当前步**使用的函数名称，不关注未来要做的事。\n请仅回答函数名称，并以list形式直接返回给我，不要回答其他的内容。\n\n{response}\n"
                },
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    import re\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    try:\n        tools = ['get_company_register_name', 'get_company_register', 'get_company_info', 'get_sub_company_info', 'get_sub_company_info_list', 'finish']\n        functions = extract_tool(response)\n        if isinstance(functions, list):\n            for item in functions:\n                if item not in tools:\n                    return False\n        return True\n    except Exception as e:\n        print(\"error in resource checker:\", e)\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 1,
            "desc": "注意不要在一次thought中连续使用多个functions。",
            "other_info": {
                "function": [
                    "get_company_register_name",
                    "get_company_register",
                    "get_company_info",
                    "get_sub_company_info",
                    "get_sub_company_info_list",
                    "finish"
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "请帮我提取出下面这段话中**当前步**使用的函数名称，不关注未来要做的事。\n请仅回答函数名称，并以list形式直接返回给我，不要回答其他的内容。\n\n{response}\n"
                },
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    import re\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    try:\n        tools = ['get_company_register_name', 'get_company_register', 'get_company_info', 'get_sub_company_info', 'get_sub_company_info_list', 'finish']\n        functions = extract_tool(response)\n        if len(functions) <= 1:\n            return True\n        return False\n    except Exception as e:\n        print(\"error in resource checker count:\", e)\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 4,
            "desc": "除了第一轮外，对上一轮的<observation></observation>进行总结。",
            "other_info": {
                "from": "system_para_2"
            },
            "dimension": "unconditional",
            "type": "semantic",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "response是否在一开始就总结了上一轮的<observation></observation>？直接回答 YES 或 NO。\n\nresponse: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sum",
                        "takes_inputs": {
                            "nums": {
                                "description": "需要求和的list",
                                "type": "List[int] | List[float] | List[str]",
                                "required": false
                            }
                        }
                    },
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'get_sub_company_info_list', 'takes_inputs': {'identifier': {'description': '母公司的名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sum', 'takes_inputs': {'nums': {'description': '需要求和的list', 'type': 'List[int] | List[float] | List[str]', 'required': False}}}, {'name': 'get_company_register_name', 'takes_inputs': {'identifier': {'description': '统一社会信用代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 4,
            "desc": "注意不要在同一代码块中连续调用多个functions",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sum",
                        "takes_inputs": {
                            "nums": {
                                "description": "需要求和的list",
                                "type": "List[int] | List[float] | List[str]",
                                "required": false
                            }
                        }
                    },
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n\n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    available_functions = [{'name': 'get_sub_company_info_list', 'takes_inputs': {'identifier': {'description': '母公司的名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sum', 'takes_inputs': {'nums': {'description': '需要求和的list', 'type': 'List[int] | List[float] | List[str]', 'required': False}}}, {'name': 'get_company_register_name', 'takes_inputs': {'identifier': {'description': '统一社会信用代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    count = 0\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == True:\n            count += 1\n    if count <= 1:\n        return True\n    else:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "每次使用预设函数后都应该将结果打印出来",
            "other_info": {
                "from": "system_para_1",
                "available_functions": [
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sum",
                        "takes_inputs": {
                            "nums": {
                                "description": "需要求和的list",
                                "type": "List[int] | List[float] | List[str]",
                                "required": false
                            }
                        }
                    },
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ]
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查下面文本中的每个函数调用表达式，判断其下一个执行语句是否为print。直接回答YES或NO，不要任何其他内容。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "您可以在代码中使用import，但只能从以下模块列表中导入",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 8,
            "desc": "不要使用'if \"xx\" in comp[\"公司名称\"] or comp[\"公司名称\"] or comp[\"公司名称\"]'判断公司所在的省份,很多公司的省份并没有在公司名称，企业地址，注册地址中体现。",
            "other_info": {
                "available_packages": [
                    "get_sub_company_info_list",
                    "get_sub_company_info",
                    "get_company_register",
                    "get_sum",
                    "get_company_register_name"
                ],
                "from": "system_para_5"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "response中是否 **不存在** 'if \"xx\" in comp[\"公司名称\"] or comp[\"公司名称\"] or comp[\"公司名称\"]' 的表达式，直接回答 YES 或 NO。\nresponse: {response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "convert_amount",
                        "takes_inputs": {
                            "num_str": {
                                "description": "需要转化的带有'千','万','亿','元'单位的数字字符串",
                                "type": "str",
                                "required": false
                            },
                            "money_unit": {
                                "description": "需要转化的单位，如'千','万','亿','元'等",
                                "type": "str",
                                "required": false
                            }
                        }
                    },
                    {
                        "name": "get_sum",
                        "takes_inputs": {
                            "nums": {
                                "description": "需要求和的list",
                                "type": "List[int] | List[float] | List[str]",
                                "required": false
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'convert_amount', 'takes_inputs': {'num_str': {'description': \"需要转化的带有'千','万','亿','元'单位的数字字符串\", 'type': 'str', 'required': False}, 'money_unit': {'description': \"需要转化的单位，如'千','万','亿','元'等\", 'type': 'str', 'required': False}}}, {'name': 'get_sum', 'takes_inputs': {'nums': {'description': '需要求和的list', 'type': 'List[int] | List[float] | List[str]', 'required': False}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register_name', 'takes_inputs': {'identifier': {'description': '统一社会信用代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "您可以在代码中使用import，但只能从以下模块列表中导入",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "您可以在thought中使用python packages,但只能从以下packages列表中选择：",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "convert_amount",
                        "takes_inputs": {
                            "num_str": {
                                "description": "需要转化的带有'千','万','亿','元'单位的数字字符串",
                                "type": "str",
                                "required": false
                            },
                            "money_unit": {
                                "description": "需要转化的单位，如'千','万','亿','元'等",
                                "type": "str",
                                "required": false
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sum",
                        "takes_inputs": {
                            "nums": {
                                "description": "需要求和的list",
                                "type": "List[int] | List[float] | List[str]",
                                "required": false
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'convert_amount', 'takes_inputs': {'num_str': {'description': \"需要转化的带有'千','万','亿','元'单位的数字字符串\", 'type': 'str', 'required': False}, 'money_unit': {'description': \"需要转化的单位，如'千','万','亿','元'等\", 'type': 'str', 'required': False}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sum', 'takes_inputs': {'nums': {'description': '需要求和的list', 'type': 'List[int] | List[float] | List[str]', 'required': False}}}, {'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register_name', 'takes_inputs': {'identifier': {'description': '统一社会信用代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "每次使用预设函数后都应该将结果打印出来",
            "other_info": {
                "from": "system_para_1",
                "available_functions": [
                    {
                        "name": "convert_amount",
                        "takes_inputs": {
                            "num_str": {
                                "description": "需要转化的带有'千','万','亿','元'单位的数字字符串",
                                "type": "str",
                                "required": false
                            },
                            "money_unit": {
                                "description": "需要转化的单位，如'千','万','亿','元'等",
                                "type": "str",
                                "required": false
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sum",
                        "takes_inputs": {
                            "nums": {
                                "description": "需要求和的list",
                                "type": "List[int] | List[float] | List[str]",
                                "required": false
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ]
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查下面文本中的每个函数调用表达式，判断其下一个执行语句是否为print。直接回答YES或NO，不要任何其他内容。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "您可以在代码中使用import，但只能从以下模块列表中导入",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "</thought> 结束",
            "other_info": {
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</thought>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_address_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司地址",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'get_address_info', 'takes_inputs': {'identifier': {'description': '公司地址', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register_name', 'takes_inputs': {'identifier': {'description': '统一社会信用代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info_list', 'takes_inputs': {'identifier': {'description': '母公司的名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "每次使用预设函数后都应该将结果打印出来",
            "other_info": {
                "from": "system_para_1",
                "available_functions": [
                    {
                        "name": "get_address_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司地址",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register_name",
                        "takes_inputs": {
                            "identifier": {
                                "description": "统一社会信用代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ]
            },
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查下面文本中的每个函数调用表达式，判断其下一个执行语句是否为print。直接回答YES或NO，不要任何其他内容。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 6,
            "desc": "立即使用</code>结束本次编码",
            "other_info": {},
            "dimension": "unconditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "required_keys": [
                        "response"
                    ],
                    "exec": "检查文本中每个 print 语句后是否存在 </code>（允许中间有换行或空格）。直接回答 YES 或 NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 8,
            "desc": "您可以在代码中使用import，但只能从以下模块列表中导入",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ],
                "from": "system_para_1"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有的调用package，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"package1\", \"package2\", ...]:\n\nHere is model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "\nimport ast\n\ndef check_following(response: str) -> bool:\n    available_packages = ['math', 'time', 'stat', 'queue', 'itertools', 'statistics', 'random', 'collections', 'unicodedata']\n    try:\n        items = ast.literal_eval(response)\n        return isinstance(items, list) and all(item in available_packages for item in items)\n    except:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 3,
            "desc": "始终为functions使用正确的参数",
            "other_info": {
                "available_functions": [
                    {
                        "name": "get_sub_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "被投资的公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_sub_company_info_list",
                        "takes_inputs": {
                            "identifier": {
                                "description": "母公司的名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称、公司简称或公司代码",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_address_info",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司地址",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    },
                    {
                        "name": "get_company_register",
                        "takes_inputs": {
                            "identifier": {
                                "description": "公司名称",
                                "type": "str",
                                "required": "True"
                            },
                            "columns": {
                                "description": "需要返回的列名列表",
                                "type": "list",
                                "required": "False"
                            }
                        }
                    }
                ],
                "from": "system_para_3"
            },
            "dimension": "unconditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "提取下面文本中的所有函数调用表达式，直接返回list格式结果，不要任何解释、前缀或赋值语句。\n输出格式要求：[\"function(arg1=value1)\", \"function2(arg2=value2)\", ...]\n\n示例1：\n输入文本:\nresult = search(query=\"test\")\n\n输出函数列表:\n[\"search(query=\"test\")\"]\n\n示例2:\n输入文本:\ndata = get_data(id=123)\nprocessed = process(data)\nprint(\"done\")\n\n输出函数列表:\n[\"get_data(id=123)\", \"process(data)\"]\n\n现在请处理以下文本：\n{response}\n\n输出函数列表:"
                },
                {
                    "type": "code",
                    "exec": "\nimport re\nimport ast\n\ndef check_following(response):\n\n\n    def extract_tool(text):\n        pattern = r'\\[([^]]+)\\]'\n        matches = re.findall(pattern, text)\n        if not matches:\n            return []\n        try:\n            tools = eval(f\"[{matches[-1]}]\")\n            return tools\n        except Exception as e:\n            # print(\"error in extract_tool:\", e)\n            return []\n\n    def validate_function_call(call_str, func_def):\n        \"\"\"\n        验证函数调用是否符合函数定义\n        \n        :param call_str: 函数调用字符串，如 'search(query=\"test\", recency_days=7)'\n        :param func_def: 函数定义字典，包含name和takes_inputs\n        :return: (is_valid, error_msg) 元组，表示是否有效及错误信息\n        \"\"\"\n        # 1. 提取函数名和参数部分\n        try:\n            # 使用AST安全解析函数调用\n            module = ast.parse(call_str.strip())\n            if not isinstance(module, ast.Module) or len(module.body) != 1:\n                return False, \"Invalid statement\"\n            \n            call_node = module.body[0]\n            if not isinstance(call_node, ast.Expr) or not isinstance(call_node.value, ast.Call):\n                return False, \"Not a function call\"\n            \n            # 获取函数名\n            called_func_name = None\n            if isinstance(call_node.value.func, ast.Name):\n                called_func_name = call_node.value.func.id\n            \n            # 2. 检查函数名是否匹配\n            if called_func_name != func_def['name']:\n                return False, f\"Function name mismatch. Expected '{func_def['name']}', got '{called_func_name}'\"\n            \n            # 3. 提取调用参数\n            provided_args = {}\n            for kw in call_node.value.keywords:\n                arg_name = kw.arg\n                # 获取参数值（处理基本类型的字面量）\n                if isinstance(kw.value, ast.Constant):\n                    arg_value = kw.value.value\n                elif isinstance(kw.value, ast.Num):  # Python < 3.8\n                    arg_value = kw.value.n\n                elif isinstance(kw.value, ast.Str):  # Python < 3.8\n                    arg_value = kw.value.s\n                elif isinstance(kw.value, ast.NameConstant):  # Python < 3.8\n                    arg_value = kw.value.value\n                else:\n                    arg_value = None  # 复杂表达式暂不处理\n                \n                provided_args[arg_name] = arg_value\n            \n            # 4. 验证参数\n            expected_args = func_def['takes_inputs']\n            errors = []\n            \n            # 检查必填参数\n            for arg_name, arg_def in expected_args.items():\n                if arg_def['required'] == 'True' and arg_name not in provided_args:\n                    errors.append(f\"Missing required argument: '{arg_name}'\")\n            \n            # 检查未知参数\n            for arg_name in provided_args:\n                if arg_name not in expected_args:\n                    errors.append(f\"Unexpected argument: '{arg_name}'\")\n            \n            # 检查参数类型\n            for arg_name, arg_value in provided_args.items():\n                if arg_name in expected_args:\n                    expected_type = expected_args[arg_name]['type']\n                    actual_type = type(arg_value).__name__\n                    \n                    # 简单类型检查\n                    if expected_type == 'int' and not isinstance(arg_value, int):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n                    elif expected_type == 'str' and not isinstance(arg_value, str):\n                        errors.append(f\"Argument '{arg_name}' should be {expected_type}, got {actual_type}\")\n            \n            if errors:\n                return False, \" | \".join(errors)\n            \n            return True, \"Valid call\"\n        \n        except Exception as e:\n            return False, f\"Parsing error: {str(e)}\"\n\n    available_functions = [{'name': 'get_sub_company_info', 'takes_inputs': {'identifier': {'description': '被投资的公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_sub_company_info_list', 'takes_inputs': {'identifier': {'description': '母公司的名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_info', 'takes_inputs': {'identifier': {'description': '公司名称、公司简称或公司代码', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_address_info', 'takes_inputs': {'identifier': {'description': '公司地址', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}, {'name': 'get_company_register', 'takes_inputs': {'identifier': {'description': '公司名称', 'type': 'str', 'required': 'True'}, 'columns': {'description': '需要返回的列名列表', 'type': 'list', 'required': 'False'}}}]\n    test_calls =  extract_tool(response)\n    # print(test_calls)\n    for call in test_calls:\n        Flag = False\n        for function_def in available_functions:\n            is_valid, msg = validate_function_call(call, function_def)\n            # print(function_def[\"name\"], msg)\n            if is_valid == True:\n                Flag = True\n                break\n        if Flag == False:\n            return False\n    return True\n"
                }
            ],
            "score": false
        }
    ],
    "example_driven": [
        {
            "id": 0,
            "desc": "开始 <code>",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.lstrip().startswith(\"<code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 1,
            "desc": "</code> 结束",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 0,
            "desc": "开始 <code>",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.lstrip().startswith(\"<code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 1,
            "desc": "</code> 结束",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 0,
            "desc": "开始 <code>",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.lstrip().startswith(\"<code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 1,
            "desc": "</code> 结束",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 0,
            "desc": "开始 <code>",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.lstrip().startswith(\"<code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 1,
            "desc": "</code> 结束",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 7,
            "desc": "<code>- # 写注释",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查文本是否以<code>后接注释（包括单行或多行注释）开头，或无<code>时直接以注释开头。注释可以是#、'''或\"\"\"形式。直接回答YES或NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 0,
            "desc": "开始 <code>",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.lstrip().startswith(\"<code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 1,
            "desc": "</code> 结束",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 0,
            "desc": "开始 <code>",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.lstrip().startswith(\"<code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 7,
            "desc": "<code>- # 写注释",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查文本是否以<code>后接注释（包括单行或多行注释）开头，或无<code>时直接以注释开头。注释可以是#、'''或\"\"\"形式。直接回答YES或NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 0,
            "desc": "开始 <code>",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.lstrip().startswith(\"<code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 1,
            "desc": "</code> 结束",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 0,
            "desc": "开始 <code>",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.lstrip().startswith(\"<code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 0,
            "desc": "开始 <code>",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.lstrip().startswith(\"<code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 0,
            "desc": "开始 <code>",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.lstrip().startswith(\"<code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 1,
            "desc": "</code> 结束",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response):\n    return response.rstrip().endswith(\"</code>\")\n"
                }
            ],
            "score": false
        },
        {
            "id": 7,
            "desc": "<code>- # 写注释",
            "other_info": {
                "from": "system_para_6"
            },
            "dimension": "example_driven",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查文本是否以<code>后接注释（包括单行或多行注释）开头，或无<code>时直接以注释开头。注释可以是#、'''或\"\"\"形式。直接回答YES或NO。\n文本：{response}\n回答："
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use double quotes to enclose sub-questions and functions",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use double quotes to enclose all sub-questions and functions? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "For function leaf nodes, do not write nested functions such as Filter(Search(...))",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If function leaf nodes, do not write nested functions such as Filter(Search(...)).",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Are there function leaf nodes in the question decomposition tree? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "For all function leaf nodes in the model response, did the model avoid from writing any nested functions such as Filter(Search(...))? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "The tree starts with the original complex question as the root node, and each non-root node is a sub-question of its parent.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "In the model response, does the JSON question decomposition tree start with the original complex question as the root node, and each non-root node is a sub-question of its parent? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "Continue decomposing until a sub-question cannot be further decomposed and could either be: (1) directly answered by calling one of the three atomic functions Search(), Relate(), Filter(), or (2) directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, counting, etc.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "In the model response, does the model continue question decomposition until each sub-question cannot be further decomposed and could either be: (1) directly answered by calling one of the three atomic functions Search(), Relate(), Filter(), or (2) directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (1), write this sub-question with its corresponding function call as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (1), write this sub-question with its corresponding function call as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (1), which can be 'directly answered by calling one of the three atomic functions Search(), Relate(), Filter()'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (1) written with their corresponding function calls as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use double quotes to enclose sub-questions and functions",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use double quotes to enclose all sub-questions and functions? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "For [END] leaf questions, format your question as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of the previously answered questions required to answer this [END] question.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If [END] leaf questions, format your question as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of the previously answered questions required to answer this [END] question..",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Are there [END] leaf questions in the tree? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Are all [END] leaf questions formatted as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of previously answered questions required to answer this [END] question? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "For [END] leaf questions, format your question as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of the previously answered questions required to answer this [END] question.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If [END] leaf questions, format your question as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of the previously answered questions required to answer this [END] question..",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Are there [END] leaf questions in the tree? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Are all [END] leaf questions formatted as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of previously answered questions required to answer this [END] question? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "For [END] leaf questions, format your question as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of the previously answered questions required to answer this [END] question.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If [END] leaf questions, format your question as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of the previously answered questions required to answer this [END] question..",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Are there [END] leaf questions in the tree? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Are all [END] leaf questions formatted as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of previously answered questions required to answer this [END] question? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "For [END] leaf questions, format your question as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of the previously answered questions required to answer this [END] question.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If [END] leaf questions, format your question as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of the previously answered questions required to answer this [END] question..",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Are there [END] leaf questions in the tree? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Are all [END] leaf questions formatted as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of previously answered questions required to answer this [END] question? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "The tree starts with the original complex question as the root node, and each non-root node is a sub-question of its parent.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "In the model response, does the JSON question decomposition tree start with the original complex question as the root node, and each non-root node is a sub-question of its parent? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "For [END] leaf questions, format your question as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of the previously answered questions required to answer this [END] question.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If [END] leaf questions, format your question as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of the previously answered questions required to answer this [END] question..",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Are there [END] leaf questions in the tree? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Are all [END] leaf questions formatted as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of previously answered questions required to answer this [END] question? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "use escape quotes \"\" to enclose work titles and function parameters",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Did the model use escape quotes \"\" to enclose all work titles and function parameters? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "In case (2), write this sub-question with an [END] mark as a leaf node.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If case (2), write this sub-question with an [END] mark as a leaf node.",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does there exist sub-questions that satisfy case (2), which can be 'directly answered by analyzing the answers of at least two previously answered questions, such as comparing, judging, intersecting, or counting'? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "In the model response, are all sub-questions that satisfy case (2) written with an [END] mark as leaf nodes? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "For [END] leaf questions, format your question as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of the previously answered questions required to answer this [END] question.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If [END] leaf questions, format your question as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of the previously answered questions required to answer this [END] question..",
                "complete_instruction_para": []
            },
            "dimension": "example_driven",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Are there [END] leaf questions in the tree? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Are all [END] leaf questions formatted as 'Given answers of [q_idx_1] and [q_idx_2], ...', where [q_idx_1] and [q_idx_2] are question indices of previously answered questions required to answer this [END] question? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        }
    ],
    "conditional": [
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "根据获取的信息，金宏气体股份有限公司的公司代码是688106，法人代表是金向华，董秘是陈莹。"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 根据获取的信息，金宏气体股份有限公司的公司代码是688106，法人代表是金向华，董秘是陈莹。\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "使用 get_company_info 函数获取金宏气体股份有限公司的代码、法人代表以及董秘的详细信息"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 使用 get_company_info 函数获取金宏气体股份有限公司的代码、法人代表以及董秘的详细信息\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "我将使用这些信息来回答用户的问题"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 我将使用这些信息来回答用户的问题\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 9,
            "desc": "finish函数将在用于返回用户的final_goal的结果时结束推理流程",
            "other_info": {},
            "dimension": "conditional",
            "type": "formatting",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\ndef check_following(response: str) -> bool:\n    if \"finish()\" in response:\n        return True\n    else:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "我需要使用get_sub_company_info_list来获取上海现代制药股份有限公司的子公司信息"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 我需要使用get_sub_company_info_list来获取上海现代制药股份有限公司的子公司信息\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "使用get_company_info函数来查询上海家化的全称"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 使用get_company_info函数来查询上海家化的全称\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "筛选出参股比例为'100.0'且投资金额超过12.5亿元的子公司信息"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 筛选出参股比例为'100.0'且投资金额超过12.5亿元的子公司信息\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "我需要使用get_company_register来通过公司名称查找公司的注册信息"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 我需要使用get_company_register来通过公司名称查找公司的注册信息\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "使用get_company_register来通过公司名称查找公司的注册信息"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 使用get_company_register来通过公司名称查找公司的注册信息\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "我需要获取金诚信矿业管理股份有限公司投资的子公司信息。"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 我需要获取金诚信矿业管理股份有限公司投资的子公司信息。\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "将1800万元转换为亿元，并精确到小数点后两位"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 将1800万元转换为亿元，并精确到小数点后两位\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "将9000万元转换为亿元，并保留两位小数。"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 将9000万元转换为亿元，并保留两位小数。\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 5,
            "desc": "必须使用get_lawfirm_info查看律师事务所的负责人，不要直接使用案件的\"被告律师事务所\"",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "conditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "response是否提及律师事务所。直接回答 YES 或 NO。\n\nresponse: {response}"
                },
                {
                    "type": "code",
                    "exec": "\ndef check_following(response: str) -> bool:\n    if \"get_lawfirm_info\" in response:\n        return True\n    else:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "根据查询结果，唐山港的全称是“唐山港集团股份有限公司”。"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 根据查询结果，唐山港的全称是“唐山港集团股份有限公司”。\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "根据查询结果，安徽中能电源有限公司的母公司是天能电池集团股份有限公司。"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 根据查询结果，安徽中能电源有限公司的母公司是天能电池集团股份有限公司。\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "使用get_company_info来获取甘肃陇神戎发药业股份有限公司的公司全称"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 使用get_company_info来获取甘肃陇神戎发药业股份有限公司的公司全称\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "根据查询结果，上海家化的全称是“上海家化联合股份有限公司”。"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 根据查询结果，上海家化的全称是“上海家化联合股份有限公司”。\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 5,
            "desc": "必须使用get_lawfirm_info查看律师事务所的负责人，不要直接使用案件的\"被告律师事务所\"",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "conditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "response是否提及律师事务所。直接回答 YES 或 NO。\n\nresponse: {response}"
                },
                {
                    "type": "code",
                    "exec": "\ndef check_following(response: str) -> bool:\n    if \"get_lawfirm_info\" in response:\n        return True\n    else:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 5,
            "desc": "必须使用get_lawfirm_info查看律师事务所的负责人，不要直接使用案件的\"被告律师事务所\"",
            "other_info": {
                "available_packages": [
                    "math",
                    "time",
                    "stat",
                    "queue",
                    "itertools",
                    "statistics",
                    "random",
                    "collections",
                    "unicodedata"
                ]
            },
            "dimension": "conditional",
            "type": "resource",
            "is_meta": false,
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "response是否提及律师事务所。直接回答 YES 或 NO。\n\nresponse: {response}"
                },
                {
                    "type": "code",
                    "exec": "\ndef check_following(response: str) -> bool:\n    if \"get_lawfirm_info\" in response:\n        return True\n    else:\n        return False\n"
                }
            ],
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "调用get_company_register找到该公司的'注册资本'"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 调用get_company_register找到该公司的'注册资本'\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "筛选出涉案金额大于10000元的案件"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 筛选出涉案金额大于10000元的案件\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "将投资金额1.31亿转换为亿元并保留2位小数"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 将投资金额1.31亿转换为亿元并保留2位小数\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "根据查询结果，案件编号为（2019）京0105民初9223号的原告是北京市桃李食品有限公司，被告是北京全时叁陆伍连锁便利店有限公司。"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 根据查询结果，案件编号为（2019）京0105民初9223号的原告是北京市桃李食品有限公司，被告是北京全时叁陆伍连锁便利店有限公司。\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "我将使用get_legal_document函数获取案件编号为（2019）京0105民初9223号的法律文书内容"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 我将使用get_legal_document函数获取案件编号为（2019）京0105民初9223号的法律文书内容\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "我需要使用get_sub_company_info来获取包钢天彩靖江科技有限公司的母公司信息。"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 我需要使用get_sub_company_info来获取包钢天彩靖江科技有限公司的母公司信息。\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "我需要使用get_legal_document来获取（2020）苏0492民初2530号案件的文档内容。"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 我需要使用get_legal_document来获取（2020）苏0492民初2530号案件的文档内容。\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "我需要使用get_legal_document_company_list函数获取东风悦达起亚汽车有限公司相关的所有案件信息"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 我需要使用get_legal_document_company_list函数获取东风悦达起亚汽车有限公司相关的所有案件信息\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "根据获取的信息，华灿光电股份有限公司的股票代码是300323。"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 根据获取的信息，华灿光电股份有限公司的股票代码是300323。\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "我需要使用get_company_register来通过公司名称查找公司的注册信息"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 我需要使用get_company_register来通过公司名称查找公司的注册信息\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 2,
            "desc": "在写代码时你需要综合考虑最终目标<final_goal></final_goal>和本轮的任务<task></task>。",
            "other_info": {
                "from": "system_para_1",
                "first_task": "筛选出投资金额超过12.5亿元的公司"
            },
            "is_meta": true,
            "type": "semantic",
            "dimension": "conditional",
            "dependency": [],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "检查 response 是否严格按 task 要求执行核心操作（允许 print 辅助输出），没有执行任何其他额外操作，直接回答 YES 或 NO。\n\ntask: 筛选出投资金额超过12.5亿元的公司\n\nresponse:{response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 6,
            "desc": "Plan for multiplayer infrastructure if applicable.",
            "other_info": {
                "from": "system_para_0",
                "type_explanation": "The constraint focuses on the content's logic and completeness by requiring consideration of multiplayer infrastructure, but only if it is applicable. This relates to ensuring meaningful and accurate content based on situational conditions (conditional logic). The emphasis is on the conceptual requirement rather than formatting or resource usage.",
                "meta_expalnation": "This constraint directly guides the content or behavior of the model by specifying a conditional requirement related to multiplayer infrastructure. It does not involve high-level rules for managing other constraints, such as selecting, prioritizing, or combining them.",
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "This constraint requires a semantic understanding of multiplayer infrastructure, including assessing design choices, networking models, and scalability. It cannot be verified through simple code but rather requires an LLM's broader expertise in game development concepts."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response include a clear and specific plan for multiplayer infrastructure, addressing components such as server architecture, networking protocols, and player synchronization? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "state": "modified",
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 4,
            "desc": "Add visual elements descriptions (charts, diagrams, infographics).",
            "other_info": {
                "from": "system_para_1",
                "type_explanation": "The constraint focuses on enhancing the meaningfulness and completeness of the output by including descriptions of visual elements such as charts, diagrams, and infographics. This aligns with the semantic category, as it emphasizes the need for content that conveys information effectively and meets specific requirements in terms of completeness.",
                "meta_expalnation": "The given constraint directly specifies an output requirement (adding visual element descriptions) rather than defining how to manage other constraints. It directly constrains the model’s content and format rather than operating as a high-level rule for handling multiple constraints.",
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The task requires understanding the visual elements' content and generating descriptive text that complements them, which involves semantic understanding and subjective assessment of relevance and accuracy, making it suited for LLM-based validation."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response explicitly include descriptions for visual elements such as charts, diagrams, or infographics? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "state": "success",
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 3,
            "desc": "If using LaTeX, use double $$ as a delimiter instead of single $.",
            "other_info": {
                "from": "system_para_0",
                "type_explanation": "This constraint controls the structure and presentation format of the output, specifically the syntax norms of LaTeX by requiring double $$ delimiters rather than single $. This aligns directly with formatting requirements.",
                "meta_expalnation": "The given constraint directly governs the output format in terms of how LaTeX is delimited. It specifies a content-related output rule ('use double $$ instead of single $') based on a condition ('if using LaTeX'). Therefore, it is not a high-level rule managing other constraints, and does not qualify as a Meta Constraint.",
                "evaluation_type": {
                    "constraint_type": "code",
                    "explanation": "The constraint pertains to a specific formatting rule (using double $$ instead of single $), which can be validated through simple pattern matching and logic in code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "code",
                    "exec": "\nimport re\n\ndef check_following(response: str) -> bool:\n    if re.search(r'(?<!\\$)\\$(?!\\$)', response):\n        return False\n    count = response.count('$$')\n    if count % 2 != 0:\n        return False\n    return True\n"
                }
            ],
            "state": "modified",
            "score": false
        },
        {
            "id": 4,
            "desc": "Add visual elements descriptions (charts, diagrams, infographics).",
            "other_info": {
                "from": "system_para_1",
                "type_explanation": "The constraint focuses on enhancing the meaningfulness and completeness of the output by including descriptions of visual elements such as charts, diagrams, and infographics. This aligns with the semantic category, as it emphasizes the need for content that conveys information effectively and meets specific requirements in terms of completeness.",
                "meta_expalnation": "The given constraint directly specifies an output requirement (adding visual element descriptions) rather than defining how to manage other constraints. It directly constrains the model’s content and format rather than operating as a high-level rule for handling multiple constraints.",
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The task requires understanding the visual elements' content and generating descriptive text that complements them, which involves semantic understanding and subjective assessment of relevance and accuracy, making it suited for LLM-based validation."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response explicitly include descriptions for visual elements such as charts, diagrams, or infographics? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "state": "success",
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "When the question provides explicit entity knowledge, always write a descriptor for the Search() function based on the question's information.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If the question provides explicit entity knowledge, always write a descriptor for the Search() function based on the question's information.",
                "complete_instruction_para": []
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the question provide explicit entity knowledge? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Did the model write a descriptor parameter for the Search() function? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "When the question provides explicit entity knowledge, always write a descriptor for the Search() function based on the question's information.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If the question provides explicit entity knowledge, always write a descriptor for the Search() function based on the question's information.",
                "complete_instruction_para": []
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the question provide explicit entity knowledge? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Did the model write a descriptor parameter for the Search() function? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "When the question provides explicit entity knowledge, always write a descriptor for the Search() function based on the question's information.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If the question provides explicit entity knowledge, always write a descriptor for the Search() function based on the question's information.",
                "complete_instruction_para": []
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the question provide explicit entity knowledge? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Did the model write a descriptor parameter for the Search() function? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": -1,
            "desc": "When the question provides explicit entity knowledge, always write a descriptor for the Search() function based on the question's information.",
            "other_info": {
                "from": "system_para_0",
                "condition_desc": "If the question provides explicit entity knowledge, always write a descriptor for the Search() function based on the question's information.",
                "complete_instruction_para": []
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the question provide explicit entity knowledge? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Did the model write a descriptor parameter for the Search() function? Please answer YES/NO directly and do not enter anything else.\n\nHere is model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 26,
            "desc": "When using multiple SEARCH/REPLACE blocks, list them in the order they appear in the file.",
            "other_info": {
                "from": "system_para_14",
                "condition_desc": "If you are using multiple SEARCH/REPLACE blocks, then list them in the order in which they are positioned within the file. ",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm_assisted_code",
                    "explanation": "Validating this constraint requires extracting the sequence of SEARCH/REPLACE blocks from the input and checking their order against the file's content. Code can perform the validation, but it first needs the specific blocks to be identified and extracted, which may require LLM assistance."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Extract all SEARCH/REPLACE blocks from the response, preserving their order as they appear. Each block should include the full content between the delimiters <<<<<<< SEARCH and >>>>>>> REPLACE, including the SEARCH and REPLACE sections. Return the extracted content verbatim from the response. If multiple blocks are found, return them as a Python-style list of strings. If nothing is found, return None.\n\nHere is the model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "import re\n\ndef check_following(response):\n    search_replace_pattern = r\"<<<<<<< SEARCH\\n(.*?)\\n=======\\n(.*?)\\n>>>>>> REPLACE\"\n    blocks = re.findall(search_replace_pattern, response, re.DOTALL)\n    \n    if not blocks:\n        return False\n    \n    last_search_end = -1\n    for search, _ in blocks:\n        current_search_start = response.find(search)\n        if current_search_start == -1 or current_search_start < last_search_end:\n            return False\n        last_search_end = current_search_start + len(search)\n    \n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 71,
            "desc": "When the model calls new_task, comprehensively explain what has been accomplished in the current task and mention specific file names that are relevant.",
            "other_info": {
                "from": "system_para_23",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint requires semantic understanding to determine what has been accomplished in the current task and to identify relevant file names. This involves subjective assessment and contextual comprehension, which can only be performed by an LLM."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the model call the new_task? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the response comprehensively explain what has been accomplished in the current task, including specific file names that are relevant? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 72,
            "desc": "When the model calls new_task, the specific next steps or focus for the new task - mention specific file names that are relevant.",
            "other_info": {
                "from": "system_para_23",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether the 'specific next steps or focus' are adequately described and whether relevant file names are mentioned requires semantic understanding and subjective assessment of the content's clarity and relevance, which is best handled by an LLM."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the model call the new_task? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the model response explicitly mention the specific next steps or focus for the new task, including the relevant file names? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 74,
            "desc": "When the model calls new_task, provide a clear indication of how this new task relates to the overall workflow.",
            "other_info": {
                "from": "system_para_23",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining how a new task relates to the overall workflow requires semantic understanding of the task's purpose, context, and its connection to broader objectives. This involves subjective reasoning and cannot be directly validated by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the model call the new_task? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the response provide a clear indication of how the new task relates to the overall workflow? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 75,
            "desc": "When the model calls new_task, you should ensure the response is akin to a long handoff file, enough for a totally new developer to be able to pick up where you left off and know exactly what to do next and which files to look at.",
            "other_info": {
                "from": "system_para_23",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint requires semantic understanding and subjective assessment of whether the response is sufficiently detailed and clear for a new developer to pick up the task. This involves evaluating the quality, comprehensiveness, and relevance of the content, which cannot be directly validated by code or simple logic."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the model call the new_task? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the response provide sufficient detail and clarity, akin to a long handoff file, so that a totally new developer can pick up where the previous work left off, understand exactly what to do next, and know which files to look at? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 78,
            "desc": "if the user's task is to create a website, you may start by asking some clarifying questions, then present a detailed plan for how you will accomplish the task given the context, and perhaps engage in a back and forth to finalize the details before the user switches you to ACT MODE to implement the solution.",
            "other_info": {
                "from": "system_para_24",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint involves semantic understanding of the user's task, context, and the appropriateness of clarifying questions and detailed planning. This requires subjective assessment and open-ended reasoning, which can only be validated by an LLM."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model response begin by asking clarifying questions, then present a detailed plan for accomplishing the task given the context, and engage in a back-and-forth to finalize the details before switching to ACT MODE? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 105,
            "desc": "If you need multiple changes, you can stack multiple SEARCH/REPLACE blocks within a single replace_in_file call.",
            "other_info": {
                "from": "system_para_81",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "code",
                    "explanation": "The constraint specifies a structural rule about stacking multiple SEARCH/REPLACE blocks within a single replace_in_file call, which can be directly validated by checking the code for the presence of multiple SEARCH/REPLACE blocks within a single function call."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Do you need to make multiple changes? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "import re\n\ndef check_following(response: str) -> bool:\n    return bool(re.search(r\"replace_in_file.*SEARCH/REPLACE.*SEARCH/REPLACE\", response, re.DOTALL))\n"
                }
            ],
            "score": false
        },
        {
            "id": 106,
            "desc": "For major overhauls or initial file creation, rely on write_to_file.",
            "other_info": {
                "from": "system_para_81",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint 'Rely on write_to_file' requires semantic understanding to determine whether the context or situation justifies using 'write_to_file' over other methods. This involves subjective reasoning and cannot be directly validated by code or through structured content extraction."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "resource"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model response explicitly rely on the write_to_file method for major overhauls or initial file creation as specified in the constraint? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 114,
            "desc": "When starting in PLAN MODE, depending on the user's request, you may need to do some information gathering e.g. using read_file or search_files to get more context about the task.",
            "other_info": {
                "from": "system_para_88",
                "condition_desc": "If you are starting in PLAN MODE, then depending on the user's request, you may need to do some information gathering, for example, using read_file or search_files to obtain more context about the task. ",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint involves understanding the user's request and determining whether information gathering is necessary, which requires semantic interpretation and subjective assessment of the task's context and requirements."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response indicate that it performed the necessary information gathering, such as using read_file or search_files, to obtain more context about the task when required by the user's request? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 116,
            "desc": "You may return mermaid diagrams to visually display your understanding.",
            "other_info": {
                "from": "system_para_88",
                "condition_desc": "If it seems appropriate or beneficial, then you may return mermaid diagrams to visually display your understanding. ",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether a mermaid diagram visually displays understanding requires semantic assessment of the diagram's content and its alignment with the intended meaning or context, which is subjective and open-ended."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does it seem appropriate or beneficial to return mermaid diagrams to visually display your understanding? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the model's response include a mermaid diagram to visually display its understanding? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 120,
            "desc": "If at any point a mermaid diagram would make your plan clearer to help the user quickly see the structure, you are encouraged to include a Mermaid code block in the response.",
            "other_info": {
                "from": "system_para_88",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether a Mermaid code block is included requires semantic understanding of the response's content to identify if it is indeed a valid Mermaid code block and aligns with the intended purpose of clarifying the plan. This is subjective and cannot be directly validated by code alone."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the response include a Mermaid code block that visually represents the plan to make it clearer and help the user quickly see the structure? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 121,
            "desc": "If you use colors in your mermaid diagrams, be sure to use high contrast colors so the text is readable.",
            "other_info": {
                "from": "system_para_88",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether the colors in mermaid diagrams are 'high contrast' and ensure text readability requires subjective semantic assessment of visual design, which is not directly verifiable by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Are colors used in the mermaid diagrams? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the mermaid diagram use high contrast colors to ensure the text is readable? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 122,
            "desc": "Finally once it seems like you've reached a good plan, ask the user to switch you back to ACT MODE to implement the solution.",
            "other_info": {
                "from": "system_para_88",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether the response asks the user to switch back to ACT MODE requires semantic understanding of the phrasing and intent, which is subjective and open-ended. This cannot be directly validated by code or simple extraction."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does it seem like you've reached a good plan? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the model response explicitly ask the user to switch back to ACT MODE in order to implement the solution? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 124,
            "desc": "When the user initially gives you a task, a recursive list of all filepaths in the current working directory ('${cwd.toPosix()}') will be included in environment_details.",
            "other_info": {
                "from": "system_para_91",
                "condition_desc": "If the user initially provides you with a task, then a recursive list of all filepaths in the current working directory ('${cwd.toPosix()}') will be included in the environment_details. ",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "code",
                    "explanation": "The constraint involves generating a recursive list of filepaths in the current working directory, which can be directly validated using code by accessing the filesystem and comparing the output to the expected format."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "code",
                    "exec": "import os\n\ndef check_following(response: str) -> bool:\n    cwd = os.getcwd().replace(\"\\\\\", \"/\")\n    return f\"'{cwd}'\" in response or f'\"{cwd}\"' in response\n"
                }
            ],
            "score": false
        },
        {
            "id": 132,
            "desc": "When you need to execute a CLI command, you must provide a clear explanation of what the command does.",
            "other_info": {
                "from": "system_para_91",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether the explanation of a CLI command is clear requires semantic and subjective understanding, which can only be assessed by an LLM."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the response include a clear explanation of what the CLI command does? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 136,
            "desc": "If asked to make edits or improvements, you might analyze the file structure in the initial environment_details to get an overview of the project, then use list_code_definition_names to get further insight using source code definitions for files located in relevant directories, then read_file to examine the contents of relevant files, analyze the code and suggest improvements or make necessary edits, then use the replace_in_file tool to implement changes.",
            "other_info": {
                "from": "system_para_91",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint involves analyzing the code structure, understanding its relevance, suggesting improvements, and making edits, which requires semantic understanding and subjective assessment of the code's quality and purpose. This cannot be validated directly by code or through extraction alone."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model response demonstrate that it analyzed the file structure, used list_code_definition_names for further insight, examined relevant files with read_file, analyzed the code, suggested improvements or edits, and implemented changes using replace_in_file as specified in the constraint? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 137,
            "desc": "If supportsComputerUse, you can use the browser_action tool to interact with websites (including html files and locally running development servers) through a Puppeteer-controlled browser when you feel it is necessary in accomplishing the user's task.",
            "other_info": {
                "from": "system_para_91",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint involves determining whether the use of the browser_action tool is 'necessary' for accomplishing the user's task, which requires a semantic and subjective understanding of the task's requirements and context. This cannot be directly validated by code or through structured extraction."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "resource"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model response explicitly indicate that the browser_action tool was used through a Puppeteer-controlled browser to interact with websites when necessary for accomplishing the user's task? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 138,
            "desc": "If supportsComputerUse and asked to add a component to a react website, you might create the necessary files, use execute_command to run the site locally, then use browser_action to launch the browser, navigate to the local server, and verify the component renders & functions correctly before closing the browser.",
            "other_info": {
                "from": "system_para_91",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint involves verifying that a component 'renders and functions correctly,' which requires a semantic and subjective assessment of the component's behavior and appearance. This cannot be directly validated by code or by extracting specific content; it requires an LLM to interpret and assess the correctness of the rendering and functionality."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "resource"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the response include the creation of necessary files, the use of execute_command to run the site locally, the use of browser_action to launch the browser and navigate to the local server, and verification that the component renders and functions correctly before closing the browser? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 145,
            "desc": "Analyze the surrounding code when using the search_files tool to better understand the matches.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint requires semantic understanding of the surrounding code to assess how it relates to the matches, which involves subjective interpretation and contextual analysis that can only be performed by an LLM."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response demonstrate that it has analyzed the surrounding code when using the search_files tool to better understand the matches? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 147,
            "desc": "When creating a new project, organize all new files within a dedicated project directory unless the user specifies otherwise.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether files are organized within a dedicated project directory requires semantic understanding of the context and intent, especially to assess whether the user has specified otherwise. This involves subjective interpretation that cannot be directly validated by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Are all new files organized within a dedicated project directory unless the user has specified otherwise? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 149,
            "desc": "Use appropriate file paths when creating files, as the write_to_file tool will automatically create any necessary directories.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether 'appropriate file paths' are used requires a semantic understanding of what constitutes 'appropriate' in the given context, which is subjective and cannot be directly validated by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the response use appropriate file paths when creating files, ensuring that the write_to_file tool can automatically create any necessary directories? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 153,
            "desc": "When making changes to code, always consider the context in which the code is being used.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint requires semantic understanding of the code's context, which involves subjective and open-ended reasoning about how the code interacts with its environment or use case. This cannot be directly validated by code or extracted content alone."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response demonstrate consideration of the context in which the code is being used when making changes to the code? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 154,
            "desc": "When making changes to code, ensure that your changes are compatible with the existing codebase and that they follow the project's coding standards and best practices.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint involves assessing compatibility with an existing codebase and adherence to coding standards, which requires semantic understanding of the codebase, the changes, and the project's best practices. This is subjective and cannot be directly validated by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response ensure that the proposed code changes are compatible with the existing codebase and adhere to the project's coding standards and best practices? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 155,
            "desc": "When you want to modify a file, use the replace_in_file or write_to_file tool directly with the desired changes. You do not need to display the changes before using the tool.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint requires semantic understanding of whether the instruction to use specific tools (replace_in_file or write_to_file) is followed correctly, which involves assessing the appropriateness of the action in context. This is subjective and cannot be directly validated by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model response use the replace_in_file or write_to_file tool directly with the desired changes, without displaying the changes beforehand? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 158,
            "desc": "When you've completed your task, you must use the attempt_completion tool to present the result to the user.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint requires semantic understanding to assess whether the response appropriately uses the 'attempt_completion' tool to present the result to the user. This involves interpreting the intent and correctness of the action, which is subjective and cannot be directly validated by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "resource"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response explicitly state that the attempt_completion tool was used to present the result to the user? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 161,
            "desc": "Use the ask_followup_question tool only when you need additional details to complete a task.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether the ask_followup_question tool is needed requires semantic understanding of the task and the context, which is subjective and open-ended. This cannot be directly validated by code or by extracting specific content."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "resource"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model response use the ask_followup_question tool only when additional details are needed to complete the task? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 164,
            "desc": "When executing commands, if you don't see the expected output, assume the terminal executed the command successfully and proceed with the task.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint requires semantic understanding and subjective reasoning to assess whether the terminal output aligns with the expected behavior and whether the assumption of success is valid. This cannot be directly validated by code or by extracting specific content."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Is the expected output not explicitly shown? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the response assume the terminal executed the command successfully and proceed with the task? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 198,
            "desc": "When deciding if the parameter can be inferred, carefully consider all the context to see if it supports a specific value.",
            "other_info": {
                "from": "system_para_101",
                "condition_desc": "If you are deciding whether a parameter can be inferred, then carefully consider all of the context to determine if it supports a specific value.",
                "complete_instruction_para": []
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "When determining whether the parameter can be inferred, has the model carefully considered all the context to check if it supports a specific value? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 199,
            "desc": "If all of the required parameters are present or can be reasonably inferred, close the thinking tag and proceed with the tool use.",
            "other_info": {
                "from": "system_para_101",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Has the model closed the thinking tag and proceeded with the tool use when all of the required parameters are present or can be reasonably inferred? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 200,
            "desc": "If one of the values for a required parameter is missing, DO NOT invoke the tool and instead, ask the user to provide the missing parameters using the ask_followup_question tool.",
            "other_info": {
                "from": "system_para_101",
                "condition_desc": "",
                "complete_instruction_para": []
            },
            "dimension": "conditional",
            "is_meta": true,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Has the model closed the thinking tag and proceeded with the tool use when all of the required parameters are present or can be reasonably inferred? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 203,
            "desc": "You may provide a CLI command to showcase the result of your task (e.g., `open index.html` for web development tasks).",
            "other_info": {
                "from": "system_para_99",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "code",
                    "explanation": "The constraint involves providing a CLI command, which is a straightforward and objective task that can be directly validated by checking the format and correctness of the command syntax."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "code",
                    "exec": "import re\n\ndef check_following(response: str) -> bool:\n    return bool(re.search(r\"^Provide a CLI command to showcase the result of your task\\.$\", response.strip()))\n"
                }
            ],
            "score": false
        },
        {
            "id": 17,
            "desc": "When invoking the write_to_file , you must ensure that all parts of the file are included, even if certain portions remain unmodified.",
            "other_info": {
                "from": "system_para_13",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether all parts of the file, including unmodified ones, are included requires a semantic understanding of the file's intended content and context, which is subjective and cannot be directly validated by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the model call the write_to_file ? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the model response include the COMPLETE intended content of the file, without any truncation or omissions, ensuring that ALL parts of the file are included even if they haven't been modified? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 26,
            "desc": "When using multiple SEARCH/REPLACE blocks, list them in the order they appear in the file.",
            "other_info": {
                "from": "system_para_14",
                "condition_desc": "If you are using multiple SEARCH/REPLACE blocks, then list them in the order in which they are positioned within the file. ",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm_assisted_code",
                    "explanation": "Validating this constraint requires extracting the sequence of SEARCH/REPLACE blocks from the input and checking their order against the file's content. Code can perform the validation, but it first needs the specific blocks to be identified and extracted, which may require LLM assistance."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Extract all SEARCH/REPLACE blocks from the response, preserving their order as they appear. Each block should include the full content between the delimiters <<<<<<< SEARCH and >>>>>>> REPLACE, including the SEARCH and REPLACE sections. Return the extracted content verbatim from the response. If multiple blocks are found, return them as a Python-style list of strings. If nothing is found, return None.\n\nHere is the model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "import re\n\ndef check_following(response):\n    search_replace_pattern = r\"<<<<<<< SEARCH\\n(.*?)\\n=======\\n(.*?)\\n>>>>>> REPLACE\"\n    blocks = re.findall(search_replace_pattern, response, re.DOTALL)\n    \n    if not blocks:\n        return False\n    \n    last_search_end = -1\n    for search, _ in blocks:\n        current_search_start = response.find(search)\n        if current_search_start == -1 or current_search_start < last_search_end:\n            return False\n        last_search_end = current_search_start + len(search)\n    \n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 88,
            "desc": "If multiple actions are needed, use one tool at a time per message to accomplish the task iteratively, with each tool use being informed by the result of the previous tool use.",
            "other_info": {
                "from": "system_para_45",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint requires semantic understanding of whether the instruction adheres to the iterative use of tools, informed by the result of the previous tool use. This involves assessing the logical flow and adherence to the iterative process, which is subjective and cannot be directly validated by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": true,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response use only one tool at a time per message, with each tool use being informed by the result of the previous tool use? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 106,
            "desc": "For major overhauls or initial file creation, rely on write_to_file.",
            "other_info": {
                "from": "system_para_81",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint 'Rely on write_to_file' requires semantic understanding to determine whether the context or situation justifies using 'write_to_file' over other methods. This involves subjective reasoning and cannot be directly validated by code or through structured content extraction."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "resource"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model response explicitly rely on the write_to_file method for major overhauls or initial file creation as specified in the constraint? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 114,
            "desc": "When starting in PLAN MODE, depending on the user's request, you may need to do some information gathering e.g. using read_file or search_files to get more context about the task.",
            "other_info": {
                "from": "system_para_88",
                "condition_desc": "If you are starting in PLAN MODE, then depending on the user's request, you may need to do some information gathering, for example, using read_file or search_files to obtain more context about the task. ",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint involves understanding the user's request and determining whether information gathering is necessary, which requires semantic interpretation and subjective assessment of the task's context and requirements."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response indicate that it performed the necessary information gathering, such as using read_file or search_files, to obtain more context about the task when required by the user's request? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 119,
            "desc": "You might ask the user if they are pleased with this plan, or if they would like to make any changes.",
            "other_info": {
                "from": "system_para_88",
                "condition_desc": "You might ask the user if they are pleased with this plan, or if they would like to make any changes.",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether the response asks the user if they would like to make changes to the plan requires semantic understanding of the intent and phrasing, which is subjective and open-ended. This cannot be directly validated by code or simple extraction logic."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the response explicitly ask the user if they would like to make any changes to the plan? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 120,
            "desc": "If at any point a mermaid diagram would make your plan clearer to help the user quickly see the structure, you are encouraged to include a Mermaid code block in the response.",
            "other_info": {
                "from": "system_para_88",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether a Mermaid code block is included requires semantic understanding of the response's content to identify if it is indeed a valid Mermaid code block and aligns with the intended purpose of clarifying the plan. This is subjective and cannot be directly validated by code alone."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the response include a Mermaid code block that visually represents the plan to make it clearer and help the user quickly see the structure? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 121,
            "desc": "If you use colors in your mermaid diagrams, be sure to use high contrast colors so the text is readable.",
            "other_info": {
                "from": "system_para_88",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether the colors in mermaid diagrams are 'high contrast' and ensure text readability requires subjective semantic assessment of visual design, which is not directly verifiable by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Are colors used in the mermaid diagrams? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the mermaid diagram use high contrast colors to ensure the text is readable? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 122,
            "desc": "Finally once it seems like you've reached a good plan, ask the user to switch you back to ACT MODE to implement the solution.",
            "other_info": {
                "from": "system_para_88",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether the response asks the user to switch back to ACT MODE requires semantic understanding of the phrasing and intent, which is subjective and open-ended. This cannot be directly validated by code or simple extraction."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does it seem like you've reached a good plan? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the model response explicitly ask the user to switch back to ACT MODE in order to implement the solution? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 124,
            "desc": "When the user initially gives you a task, a recursive list of all filepaths in the current working directory ('${cwd.toPosix()}') will be included in environment_details.",
            "other_info": {
                "from": "system_para_91",
                "condition_desc": "If the user initially provides you with a task, then a recursive list of all filepaths in the current working directory ('${cwd.toPosix()}') will be included in the environment_details. ",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "code",
                    "explanation": "The constraint involves generating a recursive list of filepaths in the current working directory, which can be directly validated using code by accessing the filesystem and comparing the output to the expected format."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "code",
                    "exec": "import os\n\ndef check_following(response: str) -> bool:\n    cwd = os.getcwd().replace(\"\\\\\", \"/\")\n    return f\"'{cwd}'\" in response or f'\"{cwd}\"' in response\n"
                }
            ],
            "score": false
        },
        {
            "id": 132,
            "desc": "When you need to execute a CLI command, you must provide a clear explanation of what the command does.",
            "other_info": {
                "from": "system_para_91",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether the explanation of a CLI command is clear requires semantic and subjective understanding, which can only be assessed by an LLM."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the response include a clear explanation of what the CLI command does? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 136,
            "desc": "If asked to make edits or improvements, you might analyze the file structure in the initial environment_details to get an overview of the project, then use list_code_definition_names to get further insight using source code definitions for files located in relevant directories, then read_file to examine the contents of relevant files, analyze the code and suggest improvements or make necessary edits, then use the replace_in_file tool to implement changes.",
            "other_info": {
                "from": "system_para_91",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint involves analyzing the code structure, understanding its relevance, suggesting improvements, and making edits, which requires semantic understanding and subjective assessment of the code's quality and purpose. This cannot be validated directly by code or through extraction alone."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model response demonstrate that it analyzed the file structure, used list_code_definition_names for further insight, examined relevant files with read_file, analyzed the code, suggested improvements or edits, and implemented changes using replace_in_file as specified in the constraint? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 137,
            "desc": "If supportsComputerUse, you can use the browser_action tool to interact with websites (including html files and locally running development servers) through a Puppeteer-controlled browser when you feel it is necessary in accomplishing the user's task.",
            "other_info": {
                "from": "system_para_91",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint involves determining whether the use of the browser_action tool is 'necessary' for accomplishing the user's task, which requires a semantic and subjective understanding of the task's requirements and context. This cannot be directly validated by code or through structured extraction."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "resource"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model response explicitly indicate that the browser_action tool was used through a Puppeteer-controlled browser to interact with websites when necessary for accomplishing the user's task? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 145,
            "desc": "Analyze the surrounding code when using the search_files tool to better understand the matches.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint requires semantic understanding of the surrounding code to assess how it relates to the matches, which involves subjective interpretation and contextual analysis that can only be performed by an LLM."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response demonstrate that it has analyzed the surrounding code when using the search_files tool to better understand the matches? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 147,
            "desc": "When creating a new project, organize all new files within a dedicated project directory unless the user specifies otherwise.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether files are organized within a dedicated project directory requires semantic understanding of the context and intent, especially to assess whether the user has specified otherwise. This involves subjective interpretation that cannot be directly validated by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Are all new files organized within a dedicated project directory unless the user has specified otherwise? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 149,
            "desc": "Use appropriate file paths when creating files, as the write_to_file tool will automatically create any necessary directories.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether 'appropriate file paths' are used requires a semantic understanding of what constitutes 'appropriate' in the given context, which is subjective and cannot be directly validated by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the response use appropriate file paths when creating files, ensuring that the write_to_file tool can automatically create any necessary directories? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 151,
            "desc": "Be sure to consider the type of project (e.g., Python, JavaScript, web application) when determining the appropriate structure and files to include.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint requires semantic understanding of the type of project and its implications for structure and files, which involves subjective and open-ended reasoning that can only be assessed by an LLM."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model response explicitly consider the type of project (e.g., Python, JavaScript, web application) when determining the appropriate structure and files to include? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 154,
            "desc": "When making changes to code, ensure that your changes are compatible with the existing codebase and that they follow the project's coding standards and best practices.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint involves assessing compatibility with an existing codebase and adherence to coding standards, which requires semantic understanding of the codebase, the changes, and the project's best practices. This is subjective and cannot be directly validated by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response ensure that the proposed code changes are compatible with the existing codebase and adhere to the project's coding standards and best practices? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 158,
            "desc": "When you've completed your task, you must use the attempt_completion tool to present the result to the user.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint requires semantic understanding to assess whether the response appropriately uses the 'attempt_completion' tool to present the result to the user. This involves interpreting the intent and correctness of the action, which is subjective and cannot be directly validated by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "resource"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response explicitly state that the attempt_completion tool was used to present the result to the user? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 161,
            "desc": "Use the ask_followup_question tool only when you need additional details to complete a task.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether the ask_followup_question tool is needed requires semantic understanding of the task and the context, which is subjective and open-ended. This cannot be directly validated by code or by extracting specific content."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "resource"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model response use the ask_followup_question tool only when additional details are needed to complete the task? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 165,
            "desc": "If you absolutely need to see the actual terminal output, use the ask_followup_question tool to request the user to copy and paste it back to you.",
            "other_info": {
                "from": "system_para_94",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether the tool usage aligns with the semantic intent of requesting the user to copy and paste terminal output requires subjective understanding of the context and purpose, which is best assessed by an LLM."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "resource"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the response use the ask_followup_question tool to request the user to copy and paste the terminal output back? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 198,
            "desc": "When deciding if the parameter can be inferred, carefully consider all the context to see if it supports a specific value.",
            "other_info": {
                "from": "system_para_101",
                "condition_desc": "If you are deciding whether a parameter can be inferred, then carefully consider all of the context to determine if it supports a specific value.",
                "complete_instruction_para": []
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "When determining whether the parameter can be inferred, has the model carefully considered all the context to check if it supports a specific value? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 26,
            "desc": "When using multiple SEARCH/REPLACE blocks, list them in the order they appear in the file.",
            "other_info": {
                "from": "system_para_14",
                "condition_desc": "If you are using multiple SEARCH/REPLACE blocks, then list them in the order in which they are positioned within the file. ",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm_assisted_code",
                    "explanation": "Validating this constraint requires extracting the sequence of SEARCH/REPLACE blocks from the input and checking their order against the file's content. Code can perform the validation, but it first needs the specific blocks to be identified and extracted, which may require LLM assistance."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Extract all SEARCH/REPLACE blocks from the response, preserving their order as they appear. Each block should include the full content between the delimiters <<<<<<< SEARCH and >>>>>>> REPLACE, including the SEARCH and REPLACE sections. Return the extracted content verbatim from the response. If multiple blocks are found, return them as a Python-style list of strings. If nothing is found, return None.\n\nHere is the model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "import re\n\ndef check_following(response):\n    search_replace_pattern = r\"<<<<<<< SEARCH\\n(.*?)\\n=======\\n(.*?)\\n>>>>>> REPLACE\"\n    blocks = re.findall(search_replace_pattern, response, re.DOTALL)\n    \n    if not blocks:\n        return False\n    \n    last_search_end = -1\n    for search, _ in blocks:\n        current_search_start = response.find(search)\n        if current_search_start == -1 or current_search_start < last_search_end:\n            return False\n        last_search_end = current_search_start + len(search)\n    \n    return True\n"
                }
            ],
            "score": false
        },
        {
            "id": 55,
            "desc": "When the model calls browser_action,the `launch` action must always be the first action.",
            "other_info": {
                "from": "system_para_18",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "code",
                    "explanation": "The constraint specifies a simple logical rule that can be validated directly by checking whether the first action in the sequence is 'launch'. This does not require semantic understanding or content extraction."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the model call the execute_command? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "import re\n\ndef check_following(response: str) -> bool:\n    # Check if the first action is \"launch\"\n    match = re.search(r\"<action>(.*?)</action>\", response)\n    return match is not None and match.group(1).strip() == \"launch\"\n"
                }
            ],
            "score": false
        },
        {
            "id": 58,
            "desc": "When the model calls browser_action,the `close` action must always be the first action.",
            "other_info": {
                "from": "system_para_18",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "code",
                    "explanation": "This constraint can be validated directly by checking if the first action in the sequence is 'close'. It involves simple logic and does not require semantic understanding or content extraction."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the model call the execute_command? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "code",
                    "exec": "import re\n\ndef check_following(response: str) -> bool:\n    return bool(re.match(r\"(?s)^<browser_action>\\s*<action>close</action>\", response.strip()))\n"
                }
            ],
            "score": false
        },
        {
            "id": 71,
            "desc": "When the model calls new_task, comprehensively explain what has been accomplished in the current task and mention specific file names that are relevant.",
            "other_info": {
                "from": "system_para_23",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint requires semantic understanding to determine what has been accomplished in the current task and to identify relevant file names. This involves subjective assessment and contextual comprehension, which can only be performed by an LLM."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the model call the new_task? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the response comprehensively explain what has been accomplished in the current task, including specific file names that are relevant? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 72,
            "desc": "When the model calls new_task, the specific next steps or focus for the new task - mention specific file names that are relevant.",
            "other_info": {
                "from": "system_para_23",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether the 'specific next steps or focus' are adequately described and whether relevant file names are mentioned requires semantic understanding and subjective assessment of the content's clarity and relevance, which is best handled by an LLM."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the model call the new_task? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the model response explicitly mention the specific next steps or focus for the new task, including the relevant file names? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 74,
            "desc": "When the model calls new_task, provide a clear indication of how this new task relates to the overall workflow.",
            "other_info": {
                "from": "system_para_23",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining how a new task relates to the overall workflow requires semantic understanding of the task's purpose, context, and its connection to broader objectives. This involves subjective reasoning and cannot be directly validated by code."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the model call the new_task? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the response provide a clear indication of how the new task relates to the overall workflow? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 75,
            "desc": "When the model calls new_task, you should ensure the response is akin to a long handoff file, enough for a totally new developer to be able to pick up where you left off and know exactly what to do next and which files to look at.",
            "other_info": {
                "from": "system_para_23",
                "condition_desc": "",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint requires semantic understanding and subjective assessment of whether the response is sufficiently detailed and clear for a new developer to pick up the task. This involves evaluating the quality, comprehensiveness, and relevance of the content, which cannot be directly validated by code or simple logic."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm_conditional_check",
                    "exec": "Does the model call the new_task? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                },
                {
                    "type": "llm",
                    "exec": "Does the response provide sufficient detail and clarity, akin to a long handoff file, so that a totally new developer can pick up where the previous work left off, understand exactly what to do next, and know which files to look at? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 114,
            "desc": "When starting in PLAN MODE, depending on the user's request, you may need to do some information gathering e.g. using read_file or search_files to get more context about the task.",
            "other_info": {
                "from": "system_para_88",
                "condition_desc": "If you are starting in PLAN MODE, then depending on the user's request, you may need to do some information gathering, for example, using read_file or search_files to obtain more context about the task. ",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "The constraint involves understanding the user's request and determining whether information gathering is necessary, which requires semantic interpretation and subjective assessment of the task's context and requirements."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "formatting"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the model's response indicate that it performed the necessary information gathering, such as using read_file or search_files, to obtain more context about the task when required by the user's request? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        },
        {
            "id": 119,
            "desc": "You might ask the user if they are pleased with this plan, or if they would like to make any changes.",
            "other_info": {
                "from": "system_para_88",
                "condition_desc": "You might ask the user if they are pleased with this plan, or if they would like to make any changes.",
                "complete_instruction_para": [],
                "evaluation_type": {
                    "constraint_type": "llm",
                    "explanation": "Determining whether the response asks the user if they would like to make changes to the plan requires semantic understanding of the intent and phrasing, which is subjective and open-ended. This cannot be directly validated by code or simple extraction logic."
                },
                "evaluation_generation_success": true
            },
            "dimension": "conditional",
            "is_meta": false,
            "type": [
                "semantic"
            ],
            "evaluation": [
                {
                    "type": "llm",
                    "exec": "Does the response explicitly ask the user if they would like to make any changes to the plan? Please answer YES/NO directly and do not enter anything else.\n\nHere is the model response: {response}"
                }
            ],
            "llm_output": "NO",
            "score": false
        }
    ]
}