Could you share the training log of stage one?

[VijayLuo]

I am reimplementing the stage one of UniAD, but the result is very different from the metrics on the repo (AMOTA: 0.3491 vs 0.390).

Could you share your training log of stage one? Thanks!

2023-04-29 17:27:50,507 - mmdet - INFO - Epoch(val) [6][753] pts_bbox_NuScenes/car_AP_dist_0.5: 0.2318, pts_bbox_NuScenes/car_AP_dist_1.0: 0.5167, pts_bbox_NuScenes/car_AP_dist_2.0: 0.7178, pts_bbox_NuScenes/car_AP_dist_4.0: 0.8132, pts_bbox_NuScenes/car_trans_err: 0.4704, pts_bbox_NuScenes/car_scale_err: 0.1501, pts_bbox_NuScenes/car_orient_err: 0.0767, pts_bbox_NuScenes/car_vel_err: 0.3336, pts_bbox_NuScenes/car_attr_err: 0.2142, pts_bbox_NuScenes/mATE: 0.6905, pts_bbox_NuScenes/mASE: 0.2765, pts_bbox_NuScenes/mAOE: 0.3952, pts_bbox_NuScenes/mAVE: 0.3878, pts_bbox_NuScenes/mAAE: 0.2061, pts_bbox_NuScenes/truck_AP_dist_0.5: 0.0345, pts_bbox_NuScenes/truck_AP_dist_1.0: 0.1856, pts_bbox_NuScenes/truck_AP_dist_2.0: 0.4547, pts_bbox_NuScenes/truck_AP_dist_4.0: 0.5935, pts_bbox_NuScenes/truck_trans_err: 0.7482, pts_bbox_NuScenes/truck_scale_err: 0.2197, pts_bbox_NuScenes/truck_orient_err: 0.0908, pts_bbox_NuScenes/truck_vel_err: 0.3117, pts_bbox_NuScenes/truck_attr_err: 0.1936, pts_bbox_NuScenes/construction_vehicle_AP_dist_0.5: 0.0000, pts_bbox_NuScenes/construction_vehicle_AP_dist_1.0: 0.0197, pts_bbox_NuScenes/construction_vehicle_AP_dist_2.0: 0.1024, pts_bbox_NuScenes/construction_vehicle_AP_dist_4.0: 0.2860, pts_bbox_NuScenes/construction_vehicle_trans_err: 0.9201, pts_bbox_NuScenes/construction_vehicle_scale_err: 0.4812, pts_bbox_NuScenes/construction_vehicle_orient_err: 1.1397, pts_bbox_NuScenes/construction_vehicle_vel_err: 0.1425, pts_bbox_NuScenes/construction_vehicle_attr_err: 0.3970, pts_bbox_NuScenes/bus_AP_dist_0.5: 0.0418, pts_bbox_NuScenes/bus_AP_dist_1.0: 0.2882, pts_bbox_NuScenes/bus_AP_dist_2.0: 0.5570, pts_bbox_NuScenes/bus_AP_dist_4.0: 0.7124, pts_bbox_NuScenes/bus_trans_err: 0.7131, pts_bbox_NuScenes/bus_scale_err: 0.2101, pts_bbox_NuScenes/bus_orient_err: 0.0862, pts_bbox_NuScenes/bus_vel_err: 0.7570, pts_bbox_NuScenes/bus_attr_err: 0.2888, pts_bbox_NuScenes/trailer_AP_dist_0.5: 0.0000, pts_bbox_NuScenes/trailer_AP_dist_1.0: 0.0151, pts_bbox_NuScenes/trailer_AP_dist_2.0: 0.1401, pts_bbox_NuScenes/trailer_AP_dist_4.0: 0.4041, pts_bbox_NuScenes/trailer_trans_err: 1.0503, pts_bbox_NuScenes/trailer_scale_err: 0.2635, pts_bbox_NuScenes/trailer_orient_err: 0.4992, pts_bbox_NuScenes/trailer_vel_err: 0.3285, pts_bbox_NuScenes/trailer_attr_err: 0.0919, pts_bbox_NuScenes/barrier_AP_dist_0.5: 0.1771, pts_bbox_NuScenes/barrier_AP_dist_1.0: 0.4155, pts_bbox_NuScenes/barrier_AP_dist_2.0: 0.6018, pts_bbox_NuScenes/barrier_AP_dist_4.0: 0.7037, pts_bbox_NuScenes/barrier_trans_err: 0.5555, pts_bbox_NuScenes/barrier_scale_err: 0.2882, pts_bbox_NuScenes/barrier_orient_err: 0.1781, pts_bbox_NuScenes/barrier_vel_err: nan, pts_bbox_NuScenes/barrier_attr_err: nan, pts_bbox_NuScenes/motorcycle_AP_dist_0.5: 0.0846, pts_bbox_NuScenes/motorcycle_AP_dist_1.0: 0.2814, pts_bbox_NuScenes/motorcycle_AP_dist_2.0: 0.5137, pts_bbox_NuScenes/motorcycle_AP_dist_4.0: 0.6110, pts_bbox_NuScenes/motorcycle_trans_err: 0.6728, pts_bbox_NuScenes/motorcycle_scale_err: 0.2607, pts_bbox_NuScenes/motorcycle_orient_err: 0.4699, pts_bbox_NuScenes/motorcycle_vel_err: 0.5474, pts_bbox_NuScenes/motorcycle_attr_err: 0.2717, pts_bbox_NuScenes/bicycle_AP_dist_0.5: 0.1006, pts_bbox_NuScenes/bicycle_AP_dist_1.0: 0.3026, pts_bbox_NuScenes/bicycle_AP_dist_2.0: 0.4610, pts_bbox_NuScenes/bicycle_AP_dist_4.0: 0.5316, pts_bbox_NuScenes/bicycle_trans_err: 0.5727, pts_bbox_NuScenes/bicycle_scale_err: 0.2676, pts_bbox_NuScenes/bicycle_orient_err: 0.5571, pts_bbox_NuScenes/bicycle_vel_err: 0.2733, pts_bbox_NuScenes/bicycle_attr_err: 0.0177, pts_bbox_NuScenes/pedestrian_AP_dist_0.5: 0.0804, pts_bbox_NuScenes/pedestrian_AP_dist_1.0: 0.3098, pts_bbox_NuScenes/pedestrian_AP_dist_2.0: 0.5757, pts_bbox_NuScenes/pedestrian_AP_dist_4.0: 0.7174, pts_bbox_NuScenes/pedestrian_trans_err: 0.7118, pts_bbox_NuScenes/pedestrian_scale_err: 0.2931, pts_bbox_NuScenes/pedestrian_orient_err: 0.4587, pts_bbox_NuScenes/pedestrian_vel_err: 0.4087, pts_bbox_NuScenes/pedestrian_attr_err: 0.1734, pts_bbox_NuScenes/traffic_cone_AP_dist_0.5: 0.2482, pts_bbox_NuScenes/traffic_cone_AP_dist_1.0: 0.4972, pts_bbox_NuScenes/traffic_cone_AP_dist_2.0: 0.6581, pts_bbox_NuScenes/traffic_cone_AP_dist_4.0: 0.7315, pts_bbox_NuScenes/traffic_cone_trans_err: 0.4896, pts_bbox_NuScenes/traffic_cone_scale_err: 0.3307, pts_bbox_NuScenes/traffic_cone_orient_err: nan, pts_bbox_NuScenes/traffic_cone_vel_err: nan, pts_bbox_NuScenes/traffic_cone_attr_err: nan, pts_bbox_NuScenes/NDS: 0.4884, pts_bbox_NuScenes/mAP: 0.3679, pts_bbox_NuScenes/amota: 0.3491, pts_bbox_NuScenes/amotp: 1.3378, pts_bbox_NuScenes/recall: 0.4500, pts_bbox_NuScenes/motar: 0.6470, pts_bbox_NuScenes/gt: 14556.7143, pts_bbox_NuScenes/mota: 0.3015, pts_bbox_NuScenes/motp: 0.7156, pts_bbox_NuScenes/mt: 2043.0000, pts_bbox_NuScenes/ml: 2778.0000, pts_bbox_NuScenes/faf: 50.1770, pts_bbox_NuScenes/tp: 54523.0000, pts_bbox_NuScenes/fp: 14212.0000, pts_bbox_NuScenes/fn: 46560.0000, pts_bbox_NuScenes/ids: 814.0000, pts_bbox_NuScenes/frag: 774.0000, pts_bbox_NuScenes/tid: 1.6311, pts_bbox_NuScenes/lgd: 2.6311, drivable_iou: 0.6575, lanes_iou: 0.2736, divider_iou: 0.2135, crossing_iou: 0.0882, contour_iou: 0.2236, drivable_iou_mean: 0.6542, lanes_iou_mean: 0.2768, divider_iou_mean: 0.2124, crossing_iou_mean: 0.0472, contour_iou_mean: 0.2361

[YTEP-ZHI]

Hi @VijayLuo, thanks for your feedback.

• Did you load the pretrained bevformer weights correctly when training stage1?

  UniAD/projects/configs/stage1_track_map/base_track_map.py

  Line 567 in 059efe4

  load_from = "ckpts/bevformer_r101_dcn_24ep.pth"

• Could you please provide the full training log in this thread so I can track the problem down?

[VijayLuo]

Hi @VijayLuo, thanks for your feedback.

• Did you load the pretrained bevformer weights correctly when training stage1?

  UniAD/projects/configs/stage1_track_map/base_track_map.py

  Line 567 in 059efe4

  load_from = "ckpts/bevformer_r101_dcn_24ep.pth"

• Could you please provide the full training log in this thread so I can track the problem down?

Thanks for your answer.

1. I did load the pretrained bevformer weights and the config is not modified.
2. Here is the whole training log

20230429_052541.log

[YTEP-ZHI]

Hi, @VijayLuo. Thanks for reporting this issue on unable to reproduce the stage1 results. We have carefully investigated this problem and found that the performance disparity is led by two causes:
1) mistakenly adding the loss_past_traj
2) freezing the img_neck and BN in stage1 training
By removing loss_past_traj and unfreezing img_neck and BN in training, the reported results could be reproduced (AMOTA: 0.393).
Here is the full training log of the fixed stage1 model: https://github.com/OpenDriveLab/UniAD/releases/download/v1.0/uniad_reproduce_stage1_gpu16_train.log

[sunbin1357]

I find that loss_past_traj is removed by setting loss_past_traj_weight to 0.0 in config，track_instances.pred_past_trajs will still be obtained during training. It may be better to add a true/false in the code to determine whether to obtain track_instances.pred_past_trajs

[YTEP-ZHI]

@sunbin1357 Thanks for your suggestion. You're right. Setting loss_past_traj_weight to 0 is an easy and correct implementation, but it could be better structured to completely remove pred_past_trajs in training. We will consider implementing it.

[YTEP-ZHI]

I'm closing this issue as it's resolved, feel free to reopen it if needed @sunbin1357 @VijayLuo.

[jagdishbhanushali]

Hi @VijayLuo , I recently just got started with UniAD. I am using latest branch v2.0. I can run eval single gpu, but on dual gpu doesn't work. Same things is for training. Launching training with multiple GPU throws exception without any error. Can you please share log file where you used v2.0 ? Also I noticed Dockerfile haven't changed for new version, it would be really helpful if you could update the dockerfile.

Thanks,
Jagdish Bhanushali