Skip to content

experiments

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

experiments

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
crosswords
crosswords
 
 
evaluator
evaluator
 
 
README.md
README.md
 
 
run_crosswords.py
run_crosswords.py
 
 
run_gaia.py
run_gaia.py
 
 
run_humaneval.py
run_humaneval.py
 
 
run_mmlu.py
run_mmlu.py
 
 

README.md

Run the following commands to reproduce our experiments in the paper

MMLU

Run the baseline:

PYTHONPATH=. python experiments/run_mmlu.py --mode=DirectAnswer

Run fully-connected swarm ablation:

PYTHONPATH=. python experiments/run_mmlu.py --num-truthful-agents=3 --mode=FullConnectedSwarm

Run randomly-connected swarm ablation:

PYTHONPATH=. python experiments/run_mmlu.py --num-truthful-agents=3 --mode=RandomSwarm

Run the main experiment with optimization and eventual evaluation:

PYTHONPATH=. python experiments/run_mmlu.py --num-truthful-agents=3 --mode=OptimizedSwarm

Mini Crosswords

Run the REINFORCE algorithm for edge optimization with three agents as described in the paper.

PYTHONPATH=. python experiments/run_crosswords.py

HumanEval

Run node optimization that improves the demonstration examples of each node.

PYTHONPATH=. python experiments/run_humaneval.py  --learn_demonstration True

GAIA

Run the general assistant tasks.

PYTHONPATH=. python experiments/run_gaia.py