Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.
Wei-Lin Chiang 3f61c6e6fc | 3 months ago | |
---|---|---|
.. | ||
data | 9 months ago | |
README.md | 4 months ago | |
clean_judgment.py | 10 months ago | |
common.py | 3 months ago | |
compute_agreement.py | 10 months ago | |
download_mt_bench_pregenerated.py | 10 months ago | |
gen_api_answer.py | 4 months ago | |
gen_judgment.py | 8 months ago | |
gen_model_answer.py | 5 months ago | |
qa_browser.py | 3 months ago | |
show_result.py | 9 months ago |
| Paper | Leaderboard |
In this package, you can use MT-bench questions and prompts to evaluate your models with LLM-as-a-judge.
MT-bench is a set of challenging multi-turn open-ended questions for evaluating chat assistants.
To automate the evaluation process, we prompt strong LLMs like GPT-4 to act as judges and assess the quality of the models' responses.
git clone https://github.com/lm-sys/FastChat.git
cd FastChat
pip install -e ".[model_worker,llm_judge]"
We provide pre-generated model answers and judgments for some models.
You can view them at this demo.
To download the pre-generated data, use
python3 download_mt_bench_pregenerated.py
After downloading the data, you can view them locally by
python3 qa_browser.py --share
You can use this QA browser to view the answers generated by you later.
python gen_model_answer.py --model-path [MODEL-PATH] --model-id [MODEL-ID]
Arguments:
[MODEL-PATH]
is the path to the weights, which can be a local folder or a Hugging Face repo ID.[MODEL-ID]
is a name you give to the model.e.g.,
python gen_model_answer.py --model-path lmsys/vicuna-7b-v1.5 --model-id vicuna-7b-v1.5
The answers will be saved to data/mt_bench/model_answer/[MODEL-ID].jsonl
.
To make sure FastChat loads the correct prompt template, see the supported models and how to add a new model here.
You can also specify --num-gpus-per-model
for model parallelism (needed for large 65B models) and --num-gpus-total
to parallelize answer generation with multiple GPUs.
There are several options to use GPT-4 as a judge, such as pairwise winrate and single-answer grading.
In MT-bench, we recommend single-answer grading as the default mode.
This mode asks GPT-4 to grade and give a score to model's answer directly without pairwise comparison.
For each turn, GPT-4 will give a score on a scale of 10. We then compute the average score on all turns.
export OPENAI_API_KEY=XXXXXX # set the OpenAI API key
python gen_judgment.py --model-list [LIST-OF-MODEL-ID] --parallel [num-concurrent-api-call]
e.g.,
python gen_judgment.py --model-list vicuna-13b-v1.3 alpaca-13b llama-13b claude-v1 gpt-3.5-turbo gpt-4 --parallel 2
The judgments will be saved to data/mt_bench/model_judgment/gpt-4_single.jsonl
python show_result.py --model-list vicuna-13b-v1.3 alpaca-13b llama-13b claude-v1 gpt-3.5-turbo gpt-4
python show_result.py
Besides score-based single-answer grading, we also support two additional grading options based on win rates:
pariwise-baseline
: run pairwise comparison against a baseline model.pairwise-all
: run pairwise comparison between all model pairs on all questions.python gen_judgment.py --mode pairwise-baseline --model-list vicuna-13b-v1.3 alpaca-13b llama-13b --parallel 2
The judgments will be saved to data/mt_bench/model_judgment/gpt-4_pair.jsonl
python show_result.py --mode pairwise-baseline
Another option is to run pairwise comparisons on all possible pairs.
This could be more expensive when #models increases, but it gives you a more comprehensive information.
python gen_judgment.py --mode pairwise-all --model-list [LIST-OF-MODEL-ID] --parallel [num-concurrent-api-call]
python show_result.py --mode pairwise-all
python gen_api_answer.py --model [MODEL-NAME]
to generate GPT-3.5/4 and Claude's answers.You can use this colab notebook to plot the radar figure for MT-bench.
We can also use vLLM for answer generation, which can be faster for the models supported by vLLM.
python3 -m fastchat.serve.controller
python3 -m fastchat.serve.vllm_worker --model-path [MODEL-PATH]
python3 -m fastchat.serve.openai_api_server --host localhost --port 8000
[MODEL-PATH]
is the path to the weights, which can be a local folder or a Hugging Face repo ID.python gen_api_answer.py --model [MODEL-NAME] --openai-api-base http://localhost:8000/v1 --parallel 50
[MODEL-NAME]
is the name of the model from Step 1.--parallel
is the number of concurrent API calls to the vLLM worker.We released 3.3K human annotations for model responses generated by 6 models in response to 80 MT-bench questions. The dataset is available at lmsys/mt_bench_human_judgments.
This Colab notebook shows how to compute the agreement between humans and GPT-4 judge with the dataset. Our results show that humans and GPT-4 judge achieve over 80% agreement, the same level of agreement between humans.
Please cite the following paper if you find the code or datasets helpful.
@misc{zheng2023judging,
title={Judging LLM-as-a-judge with MT-Bench and Chatbot Arena},
author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric. P Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica},
year={2023},
eprint={2306.05685},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
No Description
Python Jupyter Notebook Shell Nginx other
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》