|
- train config:
- {'device_target': 'GPU', 'device_id': 0, 'variance': [0.1, 0.2], 'clip': False, 'loc_weight': 2.0, 'class_weight': 1.0, 'landm_weight': 1.0, 'batch_size': 8, 'num_workers': 16, 'num_anchor': 29126, 'nnpu': 1, 'ngpu': 1, 'image_size': 840, 'match_thresh': 0.35, 'optim': 'sgd', 'momentum': 0.9, 'weight_decay': 0.0001, 'loss_scale': 1, 'seed': 1, 'epoch': 50, 'T_max': 40, 'eta_min': 0.0, 'decay1': 15, 'decay2': 30, 'lr_type': 'dynamic_lr', 'initial_lr': 0.04, 'warmup_epoch': -1, 'gamma': 0.1, 'ckpt_path': '/model/checkpoint/', 'keep_checkpoint_max': 5, 'resume_net': None, 'training_dataset': '/dataset/WIDER_train/label.txt', 'pretrain': True, 'pretrain_path': '/code/resnet50_ascend_v130_imagenet2012_official_cv_bs32_acc77.06.ckpt', 'val_model': '/model/checkpoint/*.ckpt', 'val_dataset_folder': '/dataset/WIDER_val/WIDER_val/', 'val_origin_size': False, 'val_confidence_threshold': 0.02, 'val_nms_threshold': 0.4, 'val_iou_threshold': 0.5, 'val_save_result': True, 'val_predict_save_folder': '/model/widerface_result', 'val_gt_dir': '/code/val_gt/'}
- dataset size is :
- 1609
- Load resnet50 from [/code/resnet50_ascend_v130_imagenet2012_official_cv_bs32_acc77.06.ckpt] done.
- ============== Starting Training ==============
- epoch: 1 step: 1609, loss is 9.31954574584961
- Train epoch time: 1300172.932 ms, per step time: 808.063 ms
- epoch: 2 step: 1609, loss is 7.262234687805176
- Train epoch time: 886118.879 ms, per step time: 550.726 ms
- epoch: 3 step: 1609, loss is 9.040359497070312
- Train epoch time: 883875.596 ms, per step time: 549.332 ms
- epoch: 4 step: 1609, loss is 7.527884483337402
- Train epoch time: 884407.077 ms, per step time: 549.663 ms
- epoch: 5 step: 1609, loss is 5.975747108459473
- Train epoch time: 882784.162 ms, per step time: 548.654 ms
- epoch: 6 step: 1609, loss is 8.086309432983398
- Train epoch time: 884558.975 ms, per step time: 549.757 ms
- epoch: 7 step: 1609, loss is 7.406343460083008
- Train epoch time: 883043.605 ms, per step time: 548.815 ms
- epoch: 8 step: 1609, loss is 5.974604606628418
- Train epoch time: 885067.998 ms, per step time: 550.073 ms
- epoch: 9 step: 1609, loss is 6.330732345581055
- Train epoch time: 883027.966 ms, per step time: 548.805 ms
- epoch: 10 step: 1609, loss is 4.969812393188477
- Train epoch time: 883279.517 ms, per step time: 548.962 ms
- epoch: 11 step: 1609, loss is 4.083825588226318
- Train epoch time: 883425.018 ms, per step time: 549.052 ms
- epoch: 12 step: 1609, loss is 6.852987766265869
- Train epoch time: 887205.702 ms, per step time: 551.402 ms
- epoch: 13 step: 1609, loss is 14.000219345092773
- Train epoch time: 885626.394 ms, per step time: 550.420 ms
- epoch: 14 step: 1609, loss is 8.389142036437988
- Train epoch time: 887002.637 ms, per step time: 551.276 ms
- epoch: 15 step: 1609, loss is 5.297081470489502
- Train epoch time: 884666.817 ms, per step time: 549.824 ms
- epoch: 16 step: 1609, loss is 10.88006591796875
- Train epoch time: 886402.530 ms, per step time: 550.903 ms
- epoch: 17 step: 1609, loss is 4.869266510009766
- Train epoch time: 881485.622 ms, per step time: 547.847 ms
- epoch: 18 step: 1609, loss is 3.6987085342407227
- Train epoch time: 883735.410 ms, per step time: 549.245 ms
- epoch: 19 step: 1609, loss is 2.136197090148926
- Train epoch time: 883369.623 ms, per step time: 549.018 ms
- epoch: 20 step: 1609, loss is 5.587201118469238
- Train epoch time: 882073.929 ms, per step time: 548.213 ms
- epoch: 21 step: 1609, loss is 4.590069770812988
- Train epoch time: 879817.536 ms, per step time: 546.810 ms
- epoch: 22 step: 1609, loss is 2.328248977661133
- Train epoch time: 882098.776 ms, per step time: 548.228 ms
- epoch: 23 step: 1609, loss is 6.505532264709473
- Train epoch time: 879810.987 ms, per step time: 546.806 ms
- epoch: 24 step: 1609, loss is 3.0247206687927246
- Train epoch time: 881878.829 ms, per step time: 548.091 ms
- epoch: 25 step: 1609, loss is 2.2475247383117676
- Train epoch time: 880099.068 ms, per step time: 546.985 ms
- epoch: 26 step: 1609, loss is 5.055092811584473
- Train epoch time: 882282.720 ms, per step time: 548.342 ms
- epoch: 27 step: 1609, loss is 5.142040252685547
- Train epoch time: 880341.802 ms, per step time: 547.136 ms
- epoch: 28 step: 1609, loss is 3.911944627761841
- Train epoch time: 882300.602 ms, per step time: 548.353 ms
- epoch: 29 step: 1609, loss is 2.5965747833251953
- Train epoch time: 879782.473 ms, per step time: 546.788 ms
- epoch: 30 step: 1609, loss is 3.820908546447754
- Train epoch time: 882139.129 ms, per step time: 548.253 ms
- epoch: 31 step: 1609, loss is 3.0365967750549316
- Train epoch time: 880065.976 ms, per step time: 546.965 ms
- epoch: 32 step: 1609, loss is 4.166215896606445
- Train epoch time: 882405.215 ms, per step time: 548.418 ms
- epoch: 33 step: 1609, loss is 5.320904731750488
- Train epoch time: 879489.339 ms, per step time: 546.606 ms
- epoch: 34 step: 1609, loss is 4.671609878540039
- Train epoch time: 881538.800 ms, per step time: 547.880 ms
- epoch: 35 step: 1609, loss is 2.41994571685791
- Train epoch time: 879439.434 ms, per step time: 546.575 ms
- epoch: 36 step: 1609, loss is 3.5501317977905273
- Train epoch time: 882038.074 ms, per step time: 548.190 ms
- epoch: 37 step: 1609, loss is 4.534431457519531
- Train epoch time: 879889.235 ms, per step time: 546.855 ms
- epoch: 38 step: 1609, loss is 3.7985620498657227
- Train epoch time: 882204.583 ms, per step time: 548.294 ms
- epoch: 39 step: 1609, loss is 2.5283396244049072
- Train epoch time: 880084.962 ms, per step time: 546.976 ms
- epoch: 40 step: 1609, loss is 2.2430105209350586
- Train epoch time: 881855.025 ms, per step time: 548.076 ms
- epoch: 41 step: 1609, loss is 4.9876790046691895
- Train epoch time: 880155.997 ms, per step time: 547.021 ms
- epoch: 42 step: 1609, loss is 1.7824301719665527
- Train epoch time: 881927.722 ms, per step time: 548.122 ms
- epoch: 43 step: 1609, loss is 3.187411308288574
- Train epoch time: 879753.337 ms, per step time: 546.770 ms
- epoch: 44 step: 1609, loss is 2.397024393081665
- Train epoch time: 881858.559 ms, per step time: 548.079 ms
- epoch: 45 step: 1609, loss is 2.1159958839416504
- Train epoch time: 880257.180 ms, per step time: 547.083 ms
- epoch: 46 step: 1609, loss is 3.3619298934936523
- Train epoch time: 881781.839 ms, per step time: 548.031 ms
- epoch: 47 step: 1609, loss is 1.901867389678955
- Train epoch time: 879730.837 ms, per step time: 546.756 ms
- epoch: 48 step: 1609, loss is 4.295269966125488
- Train epoch time: 881925.866 ms, per step time: 548.120 ms
- epoch: 49 step: 1609, loss is 2.0067696571350098
- Train epoch time: 879095.404 ms, per step time: 546.361 ms
- epoch: 50 step: 1609, loss is 1.477933406829834
- Train epoch time: 882005.139 ms, per step time: 548.170 ms
|