An Agent Method Baseline for Spider 2.0-Snow based on Docker environment.

1. Install Docker. Follow the instructions in the Docker setup guide to install Docker on your machine.
2. Install conda environment.

git clone https://github.com/xlang-ai/Spider2.git
cd methods/spider-agent-snow

# Optional: Create a Conda environment for Spider 2.0
# conda create -n spider2 python=3.11
# conda activate spider2

# Install required dependencies
pip install -r requirements.txt

3. Configure credential: Follow this guideline to get your own Snowflake username and password in our snowflake database. You must update snowflake_credential.json.

4. Spider 2.0-Snow Setup

python spider_agent_setup_snow.py

5. Run agent

export OPENAI_API_KEY=your_openai_api_key
python run.py --model gpt-4o -s test1

Reorganize run results into a standard submission format, here we store the answer directly into the evaluation suite

python get_spider2snow_submission_data.py --experiment_suffix <The name of this experiment> --results_folder_name <Standard Submission Folders>
python get_spider2snow_submission_data.py --experiment_suffix gpt-4o-test1 --results_folder_name ../../spider2-snow/evaluation_suite/gpt-4o-test1

You can run evaluate.py in the evaluation suite folder of spider2-snow to get the evaluation results.

python evaluate.py --result_dir gpt-4o-test1 --mode exec_result