Multitask learning (MTL) is the practice of training multiple models which solve different tasks at the same time. We aim to implement this approach in tactile properties reconstruction and segmentation as we believe knowing tactile properties of an object will improve segmentation capability, and probably more computer vision tasks.
This work has been done by Shai Yehezkel & Noa Shoham as project B in EE faculty, Technion.
Main project supervisor: Prof. Lihi Zelnik Manor, Supervisor: Anton Agafonov
You can see the project presentation and final report to review the work that has been done
Implement Multitask learning in MTAN to perform recognition of:
- Segmentation
- Depth
- Tactile Properties
And determine whether tactile is a constructive or destructive task w.r.t segmentation.
https://github.com/lorenmt/auto-lambda
https://github.com/matthewpurri/Teaching-Cameras-to-Feel, https://arxiv.org/pdf/2004.14487.pdf
Python == 3.8
Pytorch == 1.8.1
Libraries are written in the enviroment_tactile.yml file
To create dataset contains 1 blender scene (Room), with textures, use:
bpycv\create_dataset.py - update:
- textures path
- original scene path
- dataset output directory
- num_iteration (to change number of data generated))
- size of output scene (r.resolution_x, r.resolution_y)
All this related to the network - inside auto-lambda folder
cd auto-lambda
All experiments were written in PyTorch 1.7 and can be trained with different flags (hyper-parameters) when running each training script. We briefly introduce some important flags below.
| Flag Name | Usage | Comments |
|---|---|---|
network |
choose multi-task network: split, mtan |
both architectures are based on ResNet-50; only available in dense prediction tasks. default: split |
dataset |
choose dataset: room, nyuv2, cityscapes |
only available in dense prediction tasks. default: nyuv2 |
weight |
choose weighting-based method: equal, uncert, dwa, autol |
only autol will behave differently when set to different primary tasks. default: equal |
grad_method |
choose gradient-based method: graddrop, pcgrad, cagrad |
weight and grad_method can be applied together. default: none |
task |
choose primary tasks: seg, depth, tactile for Room,seg, depth, normal for NYUv2, seg, part_seg, disp for CityScapes, all: a combination of all standard 3 tasks, or any combination written in a list e.g ['seg','tactile'] |
only available in dense prediction tasks. default: all |
with_noise |
toggle on to add noise prediction task for training (to evaluate robustness in auxiliary learning setting) | only available in dense prediction tasks. default: store_true |
autol_init |
initialisation of Auto-Lambda, default 0.1 |
only available when applying Auto-Lambda |
autol_lr |
learning rate of Auto-Lambda, default 1e-4 for NYUv2 and 3e-5 for CityScapes |
only available when applying Auto-Lambda |
Training Auto-Lambda in Multi-task / Auxiliary Learning Mode:
python trainer_dense.py --dataset [room, nyuv2, cityscapes] --task [PRIMARY_TASK] --weight autol --gpu 0
python trainer_dense.py --dataset room --task all --weight autol --gpu 0
- Open
Tactile.ipynb - Update model path, dataset path, dataset name and train_tasks
- If you want to view the results on each task- Continue and run section as detailed per each task
- If you want to calculate full dataset accuracy- Go to "Evaluate Accuracy" section and run evaluate_dataset_accuracy
@article{liu2022auto_lambda, title={Auto-Lambda: Disentangling Dynamic Task Relationships}, author={Liu, Shikun and James, Stephen and Davison, Andrew J and Johns, Edward}, journal={Transactions on Machine Learning Research}, year={2022} }
