Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.
You can not select more than 25 topics
Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
alexisxy
1b4f8ce004
|
6 months ago | |
|---|---|---|
| .. | ||
| README.md | 6 months ago | |
README.md
WebArena Resources
[11/3/2023] Execution Traces from Our Experiments (v2)
The results on the release v2 can be found in this folder. It contains
- text-bison-001 + CoT + UA Hint
- GPT3.5-turbo-0613-16k + Direct + UA Hint
- GPT3.5-turbo-0613-16k + Direct
- GPT3.5-turbo-0613-16k + CoT + UA Hint
- GPT3.5-turbo-0613-16k + CoT
- GPT4-0613 + CoT
[8/7/2023] Execution Traces from Our Experiments (v1)
The results on the release v1 can be found in this folder. It contains
- GPT4-0613 + CoT
- GPT3.5-turbo-0613 + CoT
- GPT3.5-turbo-0613 + Direct
Once you unzip the file with unzip <file_name>.zip, you will see a list of render_*.html, a log file merge_log.txt recording whether an example failed or passed and a trace folder containing the playwright recording of the executions.
render_*.html
Each file render the execution trace of the correponding example with (1) the accessibility tree observations, (2) the raw prediction from the agent and (3) the parsed action. We also provide the correponding screenshot of each observation.
To extract specific information from the html, you could use the following code snippet:
from bs4 import BeautifulSoup
with open("render_<id>.html", 'r') as f:
content = f.read()
soup = BeautifulSoup(content, 'html.parser')
# get the observations
observations = soup.find_all("div", {"class": "state_obv"})
# urls
urls = soup.find_all("h3", {"class": "url"})
# get the raw predictions (e.g, let's think step-by-step ....)
raw_predictions = soup.find_all("div", {"class": "raw_parsed_prediction"})
# get the action object
actions = soup.find_all("div", {"class": "action_object"})
trace/*.zip
The zip files are generated automatically with playwright. You can view the concrete HTML, network traffic etc by playwright show-trace <example_idx>.zip. You will see something like this:
