|
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:55217']
- I1121 19:24:15.140221 975 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1121 19:24:15.960604 975 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1121 19:24:15.964205 975 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/21 19:24:17] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- [11/21 19:24:17] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/21 19:24:18] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- [11/21 19:24:18] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- [11/21 19:24:19] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- [11/21 19:24:20] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- [11/21 19:24:21] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- [11/21 19:24:21] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- [11/21 19:24:21] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- [11/21 19:24:22] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- [11/21 19:24:22] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/21 19:24:22] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- [11/21 19:24:23] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- [11/21 19:24:23] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/21 19:24:23] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- [11/21 19:24:24] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- [11/21 19:24:26] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/21 19:24:27] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/best_model.pdparams
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:55905', '127.0.0.1:49069', '127.0.0.1:38575']
- I1121 19:24:49.578426 1424 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1121 19:24:50.340502 1424 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1121 19:24:50.347070 1424 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/21 19:24:51] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- [11/21 19:24:52] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/21 19:24:52] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- [11/21 19:24:52] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- [11/21 19:24:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- [11/21 19:24:55] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- [11/21 19:24:55] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- [11/21 19:24:55] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- [11/21 19:24:56] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- [11/21 19:24:56] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- [11/21 19:24:56] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/21 19:24:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- [11/21 19:24:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- [11/21 19:24:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/21 19:24:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- [11/21 19:24:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- [11/21 19:25:01] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/21 19:25:02] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/best_model.pdparams
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:57797', '127.0.0.1:36759']
- I1121 19:25:52.423684 1906 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1121 19:25:53.237172 1906 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1121 19:25:53.240975 1906 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/21 19:25:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- [11/21 19:25:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/21 19:25:55] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- [11/21 19:25:55] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- [11/21 19:25:56] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- [11/21 19:25:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- [11/21 19:25:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- [11/21 19:25:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- [11/21 19:25:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- [11/21 19:25:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- [11/21 19:25:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/21 19:25:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- [11/21 19:26:00] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- [11/21 19:26:00] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/21 19:26:00] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- [11/21 19:26:01] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- [11/21 19:26:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/21 19:26:05] ppdet.utils.checkpoint INFO: The shape [36] in pretrained weight yolo_head.yolo_output.0.bias is unmatched with the shape [54] in model yolo_head.yolo_output.0.bias. And the weight yolo_head.yolo_output.0.bias will not be loaded
- [11/21 19:26:05] ppdet.utils.checkpoint INFO: The shape [36, 1024, 1, 1] in pretrained weight yolo_head.yolo_output.0.weight is unmatched with the shape [54, 1024, 1, 1] in model yolo_head.yolo_output.0.weight. And the weight yolo_head.yolo_output.0.weight will not be loaded
- [11/21 19:26:05] ppdet.utils.checkpoint INFO: The shape [36] in pretrained weight yolo_head.yolo_output.1.bias is unmatched with the shape [54] in model yolo_head.yolo_output.1.bias. And the weight yolo_head.yolo_output.1.bias will not be loaded
- [11/21 19:26:05] ppdet.utils.checkpoint INFO: The shape [36, 512, 1, 1] in pretrained weight yolo_head.yolo_output.1.weight is unmatched with the shape [54, 512, 1, 1] in model yolo_head.yolo_output.1.weight. And the weight yolo_head.yolo_output.1.weight will not be loaded
- [11/21 19:26:05] ppdet.utils.checkpoint INFO: The shape [36] in pretrained weight yolo_head.yolo_output.2.bias is unmatched with the shape [54] in model yolo_head.yolo_output.2.bias. And the weight yolo_head.yolo_output.2.bias will not be loaded
- [11/21 19:26:05] ppdet.utils.checkpoint INFO: The shape [36, 256, 1, 1] in pretrained weight yolo_head.yolo_output.2.weight is unmatched with the shape [54, 256, 1, 1] in model yolo_head.yolo_output.2.weight. And the weight yolo_head.yolo_output.2.weight will not be loaded
- [11/21 19:26:05] ppdet.utils.checkpoint INFO: Finish loading model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/best_model.pdparams
- [11/21 19:26:07] ppdet.engine INFO: Epoch: [0] [ 0/1215] learning_rate: 0.000000 loss_xy: 6.232897 loss_wh: 4.408241 loss_obj: 10313.620117 loss_cls: 22.682142 loss: 10346.944336 eta: 1 day, 19:55:02 batch_cost: 2.1688 data_cost: 0.0007 ips: 3.6888 images/s
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:46513', '127.0.0.1:41197', '127.0.0.1:33351']
- I1121 19:27:04.088649 13194 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1121 19:27:04.982028 13194 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1121 19:27:04.986017 13194 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/21 19:27:06] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- [11/21 19:27:06] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/21 19:27:07] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- [11/21 19:27:07] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- [11/21 19:27:08] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- [11/21 19:27:09] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- [11/21 19:27:10] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- [11/21 19:27:10] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- [11/21 19:27:10] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- [11/21 19:27:11] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- [11/21 19:27:11] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/21 19:27:12] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- [11/21 19:27:12] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- [11/21 19:27:13] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/21 19:27:13] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- [11/21 19:27:13] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- [11/21 19:27:15] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/21 19:27:17] ppdet.utils.checkpoint INFO: The shape [36] in pretrained weight yolo_head.yolo_output.0.bias is unmatched with the shape [54] in model yolo_head.yolo_output.0.bias. And the weight yolo_head.yolo_output.0.bias will not be loaded
- [11/21 19:27:17] ppdet.utils.checkpoint INFO: The shape [36, 1024, 1, 1] in pretrained weight yolo_head.yolo_output.0.weight is unmatched with the shape [54, 1024, 1, 1] in model yolo_head.yolo_output.0.weight. And the weight yolo_head.yolo_output.0.weight will not be loaded
- [11/21 19:27:17] ppdet.utils.checkpoint INFO: The shape [36] in pretrained weight yolo_head.yolo_output.1.bias is unmatched with the shape [54] in model yolo_head.yolo_output.1.bias. And the weight yolo_head.yolo_output.1.bias will not be loaded
- [11/21 19:27:17] ppdet.utils.checkpoint INFO: The shape [36, 512, 1, 1] in pretrained weight yolo_head.yolo_output.1.weight is unmatched with the shape [54, 512, 1, 1] in model yolo_head.yolo_output.1.weight. And the weight yolo_head.yolo_output.1.weight will not be loaded
- [11/21 19:27:17] ppdet.utils.checkpoint INFO: The shape [36] in pretrained weight yolo_head.yolo_output.2.bias is unmatched with the shape [54] in model yolo_head.yolo_output.2.bias. And the weight yolo_head.yolo_output.2.bias will not be loaded
- [11/21 19:27:17] ppdet.utils.checkpoint INFO: The shape [36, 256, 1, 1] in pretrained weight yolo_head.yolo_output.2.weight is unmatched with the shape [54, 256, 1, 1] in model yolo_head.yolo_output.2.weight. And the weight yolo_head.yolo_output.2.weight will not be loaded
- [11/21 19:27:17] ppdet.utils.checkpoint INFO: Finish loading model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/best_model.pdparams
- [11/21 19:27:19] ppdet.engine INFO: Epoch: [0] [ 0/1215] learning_rate: 0.000000 loss_xy: 6.584970 loss_wh: 4.049393 loss_obj: 6042.750000 loss_cls: 22.139132 loss: 6075.523438 eta: 1 day, 20:10:59 batch_cost: 2.1819 data_cost: 0.0005 ips: 3.6665 images/s
- [11/21 19:28:31] ppdet.engine INFO: Epoch: [0] [ 200/1215] learning_rate: 0.000250 loss_xy: 3.507456 loss_wh: 1.644229 loss_obj: 8.646143 loss_cls: 10.590244 loss: 25.119581 eta: 7:20:38 batch_cost: 0.3546 data_cost: 0.0119 ips: 22.5622 images/s
- [11/21 19:29:45] ppdet.engine INFO: Epoch: [0] [ 400/1215] learning_rate: 0.000500 loss_xy: 3.347090 loss_wh: 1.026411 loss_obj: 5.709229 loss_cls: 5.289563 loss: 15.631941 eta: 7:24:22 batch_cost: 0.3719 data_cost: 0.0136 ips: 21.5125 images/s
- [11/21 19:30:59] ppdet.engine INFO: Epoch: [0] [ 600/1215] learning_rate: 0.000750 loss_xy: 3.183103 loss_wh: 0.994600 loss_obj: 5.344269 loss_cls: 3.980801 loss: 13.831856 eta: 7:22:50 batch_cost: 0.3670 data_cost: 0.0279 ips: 21.7993 images/s
- [11/21 19:32:12] ppdet.engine INFO: Epoch: [0] [ 800/1215] learning_rate: 0.001000 loss_xy: 3.121128 loss_wh: 0.968698 loss_obj: 5.024123 loss_cls: 3.367781 loss: 12.553358 eta: 7:20:23 batch_cost: 0.3634 data_cost: 0.0189 ips: 22.0115 images/s
- [11/21 19:33:25] ppdet.engine INFO: Epoch: [0] [1000/1215] learning_rate: 0.001250 loss_xy: 3.192889 loss_wh: 1.076779 loss_obj: 5.408927 loss_cls: 3.105726 loss: 12.979540 eta: 7:18:21 batch_cost: 0.3630 data_cost: 0.0241 ips: 22.0362 images/s
- [11/21 19:34:36] ppdet.engine INFO: Epoch: [0] [1200/1215] learning_rate: 0.001500 loss_xy: 3.093736 loss_wh: 1.006285 loss_obj: 5.035386 loss_cls: 2.748540 loss: 12.172326 eta: 7:15:27 batch_cost: 0.3574 data_cost: 0.0164 ips: 22.3864 images/s
- [11/21 19:34:43] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 19:34:43] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_2.xml, x1: 217.0, y1: 763.0, x2: 217.0, y2: 764.0.
- [11/21 19:34:44] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_8.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/21 19:34:44] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_881_3.xml, x1: 35.0, y1: 198.0, x2: 75.0, y2: 197.0.
- [11/21 19:34:44] ppdet.engine INFO: Eval iter: 0
- [11/21 19:34:48] ppdet.engine INFO: Eval iter: 100
- [11/21 19:34:52] ppdet.engine INFO: Eval iter: 200
- [11/21 19:34:56] ppdet.engine INFO: Eval iter: 300
- [11/21 19:35:00] ppdet.engine INFO: Eval iter: 400
- [11/21 19:35:04] ppdet.engine INFO: Eval iter: 500
- [11/21 19:35:07] ppdet.engine INFO: Eval iter: 600
- [11/21 19:35:11] ppdet.engine INFO: Eval iter: 700
- [11/21 19:35:15] ppdet.engine INFO: Eval iter: 800
- [11/21 19:35:19] ppdet.engine INFO: Eval iter: 900
- [11/21 19:35:23] ppdet.engine INFO: Eval iter: 1000
- [11/21 19:35:27] ppdet.engine INFO: Eval iter: 1100
- [11/21 19:35:31] ppdet.engine INFO: Eval iter: 1200
- [11/21 19:35:35] ppdet.engine INFO: Eval iter: 1300
- [11/21 19:35:39] ppdet.engine INFO: Eval iter: 1400
- [11/21 19:35:43] ppdet.engine INFO: Eval iter: 1500
- [11/21 19:35:47] ppdet.engine INFO: Eval iter: 1600
- [11/21 19:35:50] ppdet.engine INFO: Eval iter: 1700
- [11/21 19:35:54] ppdet.engine INFO: Eval iter: 1800
- [11/21 19:35:58] ppdet.engine INFO: Eval iter: 1900
- [11/21 19:36:02] ppdet.engine INFO: Eval iter: 2000
- [11/21 19:36:06] ppdet.engine INFO: Eval iter: 2100
- [11/21 19:36:10] ppdet.engine INFO: Eval iter: 2200
- [11/21 19:36:14] ppdet.engine INFO: Eval iter: 2300
- [11/21 19:36:18] ppdet.engine INFO: Eval iter: 2400
- [11/21 19:36:21] ppdet.engine INFO: Eval iter: 2500
- [11/21 19:36:25] ppdet.engine INFO: Eval iter: 2600
- [11/21 19:36:29] ppdet.engine INFO: Eval iter: 2700
- [11/21 19:36:33] ppdet.engine INFO: Eval iter: 2800
- [11/21 19:36:37] ppdet.engine INFO: Eval iter: 2900
- [11/21 19:36:41] ppdet.engine INFO: Eval iter: 3000
- [11/21 19:36:45] ppdet.engine INFO: Eval iter: 3100
- [11/21 19:36:49] ppdet.engine INFO: Eval iter: 3200
- [11/21 19:36:53] ppdet.engine INFO: Eval iter: 3300
- [11/21 19:36:56] ppdet.engine INFO: Eval iter: 3400
- [11/21 19:37:00] ppdet.engine INFO: Eval iter: 3500
- [11/21 19:37:04] ppdet.engine INFO: Eval iter: 3600
- [11/21 19:37:08] ppdet.engine INFO: Eval iter: 3700
- [11/21 19:37:12] ppdet.engine INFO: Eval iter: 3800
- [11/21 19:37:16] ppdet.engine INFO: Eval iter: 3900
- [11/21 19:37:19] ppdet.engine INFO: Eval iter: 4000
- [11/21 19:37:23] ppdet.engine INFO: Eval iter: 4100
- [11/21 19:37:27] ppdet.engine INFO: Eval iter: 4200
- [11/21 19:37:31] ppdet.engine INFO: Eval iter: 4300
- [11/21 19:37:32] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 19:37:32] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 41.40%
- [11/21 19:37:32] ppdet.engine INFO: Total sample number: 4322, averge FPS: 25.770799431658894
- [11/21 19:37:32] ppdet.engine INFO: Best test bbox ap is 0.414.
- [11/21 19:37:35] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 19:37:36] ppdet.engine INFO: Epoch: [1] [ 0/1215] learning_rate: 0.001519 loss_xy: 3.129848 loss_wh: 1.009411 loss_obj: 5.028245 loss_cls: 2.743263 loss: 11.999268 eta: 7:15:20 batch_cost: 0.3580 data_cost: 0.0199 ips: 22.3436 images/s
- [11/21 19:38:49] ppdet.engine INFO: Epoch: [1] [ 200/1215] learning_rate: 0.001769 loss_xy: 3.008589 loss_wh: 0.991282 loss_obj: 4.772413 loss_cls: 2.554336 loss: 11.688641 eta: 7:14:14 batch_cost: 0.3651 data_cost: 0.0215 ips: 21.9144 images/s
- [11/21 19:40:04] ppdet.engine INFO: Epoch: [1] [ 400/1215] learning_rate: 0.002019 loss_xy: 3.029603 loss_wh: 1.022494 loss_obj: 4.840027 loss_cls: 2.401364 loss: 11.315834 eta: 7:14:38 batch_cost: 0.3754 data_cost: 0.0326 ips: 21.3110 images/s
- [11/21 19:41:18] ppdet.engine INFO: Epoch: [1] [ 600/1215] learning_rate: 0.002269 loss_xy: 3.236775 loss_wh: 1.061211 loss_obj: 4.986590 loss_cls: 2.510568 loss: 11.965545 eta: 7:13:43 batch_cost: 0.3682 data_cost: 0.0230 ips: 21.7301 images/s
- [11/21 19:42:32] ppdet.engine INFO: Epoch: [1] [ 800/1215] learning_rate: 0.002519 loss_xy: 3.217122 loss_wh: 1.052514 loss_obj: 5.351213 loss_cls: 2.313849 loss: 12.190327 eta: 7:12:47 batch_cost: 0.3686 data_cost: 0.0249 ips: 21.7028 images/s
- [11/21 19:43:45] ppdet.engine INFO: Epoch: [1] [1000/1215] learning_rate: 0.002769 loss_xy: 3.300695 loss_wh: 1.073778 loss_obj: 5.153863 loss_cls: 2.363563 loss: 12.080223 eta: 7:11:19 batch_cost: 0.3640 data_cost: 0.0428 ips: 21.9755 images/s
- [11/21 19:44:59] ppdet.engine INFO: Epoch: [1] [1200/1215] learning_rate: 0.003019 loss_xy: 3.094875 loss_wh: 1.007618 loss_obj: 4.970106 loss_cls: 2.284946 loss: 11.659380 eta: 7:10:20 batch_cost: 0.3685 data_cost: 0.0191 ips: 21.7109 images/s
- [11/21 19:45:05] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 19:45:05] ppdet.engine INFO: Eval iter: 0
- [11/21 19:45:09] ppdet.engine INFO: Eval iter: 100
- [11/21 19:45:13] ppdet.engine INFO: Eval iter: 200
- [11/21 19:45:17] ppdet.engine INFO: Eval iter: 300
- [11/21 19:45:21] ppdet.engine INFO: Eval iter: 400
- [11/21 19:45:25] ppdet.engine INFO: Eval iter: 500
- [11/21 19:45:29] ppdet.engine INFO: Eval iter: 600
- [11/21 19:45:33] ppdet.engine INFO: Eval iter: 700
- [11/21 19:45:37] ppdet.engine INFO: Eval iter: 800
- [11/21 19:45:41] ppdet.engine INFO: Eval iter: 900
- [11/21 19:45:44] ppdet.engine INFO: Eval iter: 1000
- [11/21 19:45:48] ppdet.engine INFO: Eval iter: 1100
- [11/21 19:45:52] ppdet.engine INFO: Eval iter: 1200
- [11/21 19:45:56] ppdet.engine INFO: Eval iter: 1300
- [11/21 19:46:00] ppdet.engine INFO: Eval iter: 1400
- [11/21 19:46:05] ppdet.engine INFO: Eval iter: 1500
- [11/21 19:46:09] ppdet.engine INFO: Eval iter: 1600
- [11/21 19:46:13] ppdet.engine INFO: Eval iter: 1700
- [11/21 19:46:18] ppdet.engine INFO: Eval iter: 1800
- [11/21 19:46:22] ppdet.engine INFO: Eval iter: 1900
- [11/21 19:46:26] ppdet.engine INFO: Eval iter: 2000
- [11/21 19:46:30] ppdet.engine INFO: Eval iter: 2100
- [11/21 19:46:34] ppdet.engine INFO: Eval iter: 2200
- [11/21 19:46:38] ppdet.engine INFO: Eval iter: 2300
- [11/21 19:46:42] ppdet.engine INFO: Eval iter: 2400
- [11/21 19:46:46] ppdet.engine INFO: Eval iter: 2500
- [11/21 19:46:50] ppdet.engine INFO: Eval iter: 2600
- [11/21 19:46:54] ppdet.engine INFO: Eval iter: 2700
- [11/21 19:46:58] ppdet.engine INFO: Eval iter: 2800
- [11/21 19:47:02] ppdet.engine INFO: Eval iter: 2900
- [11/21 19:47:07] ppdet.engine INFO: Eval iter: 3000
- [11/21 19:47:11] ppdet.engine INFO: Eval iter: 3100
- [11/21 19:47:15] ppdet.engine INFO: Eval iter: 3200
- [11/21 19:47:19] ppdet.engine INFO: Eval iter: 3300
- [11/21 19:47:23] ppdet.engine INFO: Eval iter: 3400
- [11/21 19:47:27] ppdet.engine INFO: Eval iter: 3500
- [11/21 19:47:31] ppdet.engine INFO: Eval iter: 3600
- [11/21 19:47:35] ppdet.engine INFO: Eval iter: 3700
- [11/21 19:47:40] ppdet.engine INFO: Eval iter: 3800
- [11/21 19:47:44] ppdet.engine INFO: Eval iter: 3900
- [11/21 19:47:48] ppdet.engine INFO: Eval iter: 4000
- [11/21 19:47:52] ppdet.engine INFO: Eval iter: 4100
- [11/21 19:47:56] ppdet.engine INFO: Eval iter: 4200
- [11/21 19:48:00] ppdet.engine INFO: Eval iter: 4300
- [11/21 19:48:01] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 19:48:01] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 57.70%
- [11/21 19:48:01] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.56883931421993
- [11/21 19:48:01] ppdet.engine INFO: Best test bbox ap is 0.577.
- [11/21 19:48:05] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 19:48:05] ppdet.engine INFO: Epoch: [2] [ 0/1215] learning_rate: 0.003038 loss_xy: 3.163223 loss_wh: 1.007618 loss_obj: 4.982907 loss_cls: 2.316991 loss: 11.710783 eta: 7:10:04 batch_cost: 0.3674 data_cost: 0.0206 ips: 21.7754 images/s
- [11/21 19:49:18] ppdet.engine INFO: Epoch: [2] [ 200/1215] learning_rate: 0.003288 loss_xy: 3.294109 loss_wh: 1.051273 loss_obj: 5.401410 loss_cls: 2.293939 loss: 12.032770 eta: 7:08:25 batch_cost: 0.3614 data_cost: 0.0196 ips: 22.1345 images/s
- [11/21 19:50:31] ppdet.engine INFO: Epoch: [2] [ 400/1215] learning_rate: 0.003538 loss_xy: 3.296545 loss_wh: 1.039979 loss_obj: 5.199859 loss_cls: 2.194230 loss: 11.848125 eta: 7:07:03 batch_cost: 0.3639 data_cost: 0.0075 ips: 21.9822 images/s
- [11/21 19:51:43] ppdet.engine INFO: Epoch: [2] [ 600/1215] learning_rate: 0.003787 loss_xy: 3.206888 loss_wh: 1.002520 loss_obj: 5.050076 loss_cls: 2.167237 loss: 11.213085 eta: 7:05:08 batch_cost: 0.3566 data_cost: 0.0216 ips: 22.4359 images/s
- [11/21 19:52:54] ppdet.engine INFO: Epoch: [2] [ 800/1215] learning_rate: 0.004038 loss_xy: 3.020834 loss_wh: 0.931943 loss_obj: 4.782002 loss_cls: 2.097424 loss: 10.719288 eta: 7:03:15 batch_cost: 0.3560 data_cost: 0.0081 ips: 22.4734 images/s
- [11/21 19:54:08] ppdet.engine INFO: Epoch: [2] [1000/1215] learning_rate: 0.004288 loss_xy: 3.267519 loss_wh: 1.047458 loss_obj: 5.620172 loss_cls: 2.300112 loss: 12.389902 eta: 7:02:27 batch_cost: 0.3705 data_cost: 0.0279 ips: 21.5917 images/s
- [11/21 19:55:20] ppdet.engine INFO: Epoch: [2] [1200/1215] learning_rate: 0.004537 loss_xy: 3.159027 loss_wh: 0.975246 loss_obj: 5.301854 loss_cls: 2.273463 loss: 11.667622 eta: 7:00:50 batch_cost: 0.3586 data_cost: 0.0354 ips: 22.3110 images/s
- [11/21 19:55:27] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 19:55:27] ppdet.engine INFO: Eval iter: 0
- [11/21 19:55:31] ppdet.engine INFO: Eval iter: 100
- [11/21 19:55:35] ppdet.engine INFO: Eval iter: 200
- [11/21 19:55:39] ppdet.engine INFO: Eval iter: 300
- [11/21 19:55:43] ppdet.engine INFO: Eval iter: 400
- [11/21 19:55:47] ppdet.engine INFO: Eval iter: 500
- [11/21 19:55:51] ppdet.engine INFO: Eval iter: 600
- [11/21 19:55:55] ppdet.engine INFO: Eval iter: 700
- [11/21 19:55:59] ppdet.engine INFO: Eval iter: 800
- [11/21 19:56:04] ppdet.engine INFO: Eval iter: 900
- [11/21 19:56:08] ppdet.engine INFO: Eval iter: 1000
- [11/21 19:56:12] ppdet.engine INFO: Eval iter: 1100
- [11/21 19:56:16] ppdet.engine INFO: Eval iter: 1200
- [11/21 19:56:20] ppdet.engine INFO: Eval iter: 1300
- [11/21 19:56:24] ppdet.engine INFO: Eval iter: 1400
- [11/21 19:56:29] ppdet.engine INFO: Eval iter: 1500
- [11/21 19:56:33] ppdet.engine INFO: Eval iter: 1600
- [11/21 19:56:37] ppdet.engine INFO: Eval iter: 1700
- [11/21 19:56:41] ppdet.engine INFO: Eval iter: 1800
- [11/21 19:56:45] ppdet.engine INFO: Eval iter: 1900
- [11/21 19:56:49] ppdet.engine INFO: Eval iter: 2000
- [11/21 19:56:53] ppdet.engine INFO: Eval iter: 2100
- [11/21 19:56:57] ppdet.engine INFO: Eval iter: 2200
- [11/21 19:57:01] ppdet.engine INFO: Eval iter: 2300
- [11/21 19:57:05] ppdet.engine INFO: Eval iter: 2400
- [11/21 19:57:09] ppdet.engine INFO: Eval iter: 2500
- [11/21 19:57:13] ppdet.engine INFO: Eval iter: 2600
- [11/21 19:57:17] ppdet.engine INFO: Eval iter: 2700
- [11/21 19:57:21] ppdet.engine INFO: Eval iter: 2800
- [11/21 19:57:25] ppdet.engine INFO: Eval iter: 2900
- [11/21 19:57:29] ppdet.engine INFO: Eval iter: 3000
- [11/21 19:57:33] ppdet.engine INFO: Eval iter: 3100
- [11/21 19:57:37] ppdet.engine INFO: Eval iter: 3200
- [11/21 19:57:42] ppdet.engine INFO: Eval iter: 3300
- [11/21 19:57:46] ppdet.engine INFO: Eval iter: 3400
- [11/21 19:57:50] ppdet.engine INFO: Eval iter: 3500
- [11/21 19:57:54] ppdet.engine INFO: Eval iter: 3600
- [11/21 19:57:58] ppdet.engine INFO: Eval iter: 3700
- [11/21 19:58:02] ppdet.engine INFO: Eval iter: 3800
- [11/21 19:58:06] ppdet.engine INFO: Eval iter: 3900
- [11/21 19:58:10] ppdet.engine INFO: Eval iter: 4000
- [11/21 19:58:14] ppdet.engine INFO: Eval iter: 4100
- [11/21 19:58:18] ppdet.engine INFO: Eval iter: 4200
- [11/21 19:58:22] ppdet.engine INFO: Eval iter: 4300
- [11/21 19:58:23] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 19:58:23] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 57.39%
- [11/21 19:58:23] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.477707138671747
- [11/21 19:58:23] ppdet.engine INFO: Best test bbox ap is 0.577.
- [11/21 19:58:24] ppdet.engine INFO: Epoch: [3] [ 0/1215] learning_rate: 0.004556 loss_xy: 3.145322 loss_wh: 0.964718 loss_obj: 5.279423 loss_cls: 2.286693 loss: 11.644724 eta: 7:00:42 batch_cost: 0.3561 data_cost: 0.0278 ips: 22.4638 images/s
- [11/21 19:59:37] ppdet.engine INFO: Epoch: [3] [ 200/1215] learning_rate: 0.004806 loss_xy: 3.047824 loss_wh: 0.926055 loss_obj: 4.996692 loss_cls: 2.041310 loss: 11.065186 eta: 6:59:24 batch_cost: 0.3630 data_cost: 0.0129 ips: 22.0372 images/s
- [11/21 20:00:49] ppdet.engine INFO: Epoch: [3] [ 400/1215] learning_rate: 0.005000 loss_xy: 3.220617 loss_wh: 1.041860 loss_obj: 5.339504 loss_cls: 2.118215 loss: 11.895643 eta: 6:57:56 batch_cost: 0.3600 data_cost: 0.0236 ips: 22.2251 images/s
- [11/21 20:02:01] ppdet.engine INFO: Epoch: [3] [ 600/1215] learning_rate: 0.005000 loss_xy: 3.043465 loss_wh: 0.947022 loss_obj: 4.925890 loss_cls: 2.087257 loss: 11.111707 eta: 6:56:30 batch_cost: 0.3602 data_cost: 0.0200 ips: 22.2087 images/s
- [11/21 20:03:13] ppdet.engine INFO: Epoch: [3] [ 800/1215] learning_rate: 0.005000 loss_xy: 3.131300 loss_wh: 0.941334 loss_obj: 5.204431 loss_cls: 1.914465 loss: 11.422792 eta: 6:55:06 batch_cost: 0.3601 data_cost: 0.0071 ips: 22.2159 images/s
- [11/21 20:04:27] ppdet.engine INFO: Epoch: [3] [1000/1215] learning_rate: 0.005000 loss_xy: 3.209391 loss_wh: 0.968142 loss_obj: 5.062517 loss_cls: 2.035069 loss: 11.355882 eta: 6:54:00 batch_cost: 0.3664 data_cost: 0.0330 ips: 21.8333 images/s
- [11/21 20:05:40] ppdet.engine INFO: Epoch: [3] [1200/1215] learning_rate: 0.005000 loss_xy: 3.258312 loss_wh: 0.941631 loss_obj: 5.220723 loss_cls: 1.935663 loss: 11.582565 eta: 6:52:40 batch_cost: 0.3613 data_cost: 0.0165 ips: 22.1448 images/s
- [11/21 20:05:46] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 20:05:46] ppdet.engine INFO: Eval iter: 0
- [11/21 20:05:50] ppdet.engine INFO: Eval iter: 100
- [11/21 20:05:54] ppdet.engine INFO: Eval iter: 200
- [11/21 20:05:59] ppdet.engine INFO: Eval iter: 300
- [11/21 20:06:03] ppdet.engine INFO: Eval iter: 400
- [11/21 20:06:07] ppdet.engine INFO: Eval iter: 500
- [11/21 20:06:11] ppdet.engine INFO: Eval iter: 600
- [11/21 20:06:15] ppdet.engine INFO: Eval iter: 700
- [11/21 20:06:19] ppdet.engine INFO: Eval iter: 800
- [11/21 20:06:23] ppdet.engine INFO: Eval iter: 900
- [11/21 20:06:27] ppdet.engine INFO: Eval iter: 1000
- [11/21 20:06:31] ppdet.engine INFO: Eval iter: 1100
- [11/21 20:06:36] ppdet.engine INFO: Eval iter: 1200
- [11/21 20:06:40] ppdet.engine INFO: Eval iter: 1300
- [11/21 20:06:44] ppdet.engine INFO: Eval iter: 1400
- [11/21 20:06:48] ppdet.engine INFO: Eval iter: 1500
- [11/21 20:06:52] ppdet.engine INFO: Eval iter: 1600
- [11/21 20:06:56] ppdet.engine INFO: Eval iter: 1700
- [11/21 20:07:00] ppdet.engine INFO: Eval iter: 1800
- [11/21 20:07:04] ppdet.engine INFO: Eval iter: 1900
- [11/21 20:07:09] ppdet.engine INFO: Eval iter: 2000
- [11/21 20:07:13] ppdet.engine INFO: Eval iter: 2100
- [11/21 20:07:17] ppdet.engine INFO: Eval iter: 2200
- [11/21 20:07:21] ppdet.engine INFO: Eval iter: 2300
- [11/21 20:07:25] ppdet.engine INFO: Eval iter: 2400
- [11/21 20:07:30] ppdet.engine INFO: Eval iter: 2500
- [11/21 20:07:34] ppdet.engine INFO: Eval iter: 2600
- [11/21 20:07:38] ppdet.engine INFO: Eval iter: 2700
- [11/21 20:07:43] ppdet.engine INFO: Eval iter: 2800
- [11/21 20:07:47] ppdet.engine INFO: Eval iter: 2900
- [11/21 20:07:52] ppdet.engine INFO: Eval iter: 3000
- [11/21 20:07:56] ppdet.engine INFO: Eval iter: 3100
- [11/21 20:08:01] ppdet.engine INFO: Eval iter: 3200
- [11/21 20:08:06] ppdet.engine INFO: Eval iter: 3300
- [11/21 20:08:10] ppdet.engine INFO: Eval iter: 3400
- [11/21 20:08:14] ppdet.engine INFO: Eval iter: 3500
- [11/21 20:08:18] ppdet.engine INFO: Eval iter: 3600
- [11/21 20:08:22] ppdet.engine INFO: Eval iter: 3700
- [11/21 20:08:27] ppdet.engine INFO: Eval iter: 3800
- [11/21 20:08:31] ppdet.engine INFO: Eval iter: 3900
- [11/21 20:08:35] ppdet.engine INFO: Eval iter: 4000
- [11/21 20:08:39] ppdet.engine INFO: Eval iter: 4100
- [11/21 20:08:43] ppdet.engine INFO: Eval iter: 4200
- [11/21 20:08:47] ppdet.engine INFO: Eval iter: 4300
- [11/21 20:08:48] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 20:08:48] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 63.05%
- [11/21 20:08:48] ppdet.engine INFO: Total sample number: 4322, averge FPS: 23.764022410801985
- [11/21 20:08:48] ppdet.engine INFO: Best test bbox ap is 0.630.
- [11/21 20:08:52] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 20:08:52] ppdet.engine INFO: Epoch: [4] [ 0/1215] learning_rate: 0.005000 loss_xy: 3.280983 loss_wh: 0.946576 loss_obj: 5.242375 loss_cls: 1.925421 loss: 11.625092 eta: 6:52:39 batch_cost: 0.3646 data_cost: 0.0140 ips: 21.9415 images/s
- [11/21 20:10:04] ppdet.engine INFO: Epoch: [4] [ 200/1215] learning_rate: 0.005000 loss_xy: 3.146616 loss_wh: 0.932768 loss_obj: 5.131503 loss_cls: 1.853709 loss: 11.116609 eta: 6:51:09 batch_cost: 0.3574 data_cost: 0.0347 ips: 22.3815 images/s
- [11/21 20:11:18] ppdet.engine INFO: Epoch: [4] [ 400/1215] learning_rate: 0.005000 loss_xy: 2.945934 loss_wh: 0.871850 loss_obj: 4.636428 loss_cls: 1.755186 loss: 10.180370 eta: 6:50:17 batch_cost: 0.3717 data_cost: 0.0199 ips: 21.5245 images/s
- [11/21 20:12:30] ppdet.engine INFO: Epoch: [4] [ 600/1215] learning_rate: 0.005000 loss_xy: 3.144176 loss_wh: 0.951384 loss_obj: 5.188190 loss_cls: 1.893425 loss: 11.242443 eta: 6:48:47 batch_cost: 0.3571 data_cost: 0.0142 ips: 22.4047 images/s
- [11/21 20:13:44] ppdet.engine INFO: Epoch: [4] [ 800/1215] learning_rate: 0.005000 loss_xy: 3.023146 loss_wh: 0.904687 loss_obj: 4.928915 loss_cls: 1.745939 loss: 10.585772 eta: 6:47:44 batch_cost: 0.3676 data_cost: 0.0270 ips: 21.7631 images/s
- [11/21 20:14:57] ppdet.engine INFO: Epoch: [4] [1000/1215] learning_rate: 0.005000 loss_xy: 2.970725 loss_wh: 0.866222 loss_obj: 4.703340 loss_cls: 1.777916 loss: 10.466081 eta: 6:46:32 batch_cost: 0.3642 data_cost: 0.0071 ips: 21.9686 images/s
- [11/21 20:16:08] ppdet.engine INFO: Epoch: [4] [1200/1215] learning_rate: 0.005000 loss_xy: 3.272458 loss_wh: 0.933599 loss_obj: 5.003303 loss_cls: 1.904810 loss: 11.160997 eta: 6:44:55 batch_cost: 0.3529 data_cost: 0.0122 ips: 22.6712 images/s
- [11/21 20:16:14] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 20:16:14] ppdet.engine INFO: Eval iter: 0
- [11/21 20:16:18] ppdet.engine INFO: Eval iter: 100
- [11/21 20:16:22] ppdet.engine INFO: Eval iter: 200
- [11/21 20:16:26] ppdet.engine INFO: Eval iter: 300
- [11/21 20:16:30] ppdet.engine INFO: Eval iter: 400
- [11/21 20:16:34] ppdet.engine INFO: Eval iter: 500
- [11/21 20:16:38] ppdet.engine INFO: Eval iter: 600
- [11/21 20:16:42] ppdet.engine INFO: Eval iter: 700
- [11/21 20:16:46] ppdet.engine INFO: Eval iter: 800
- [11/21 20:16:50] ppdet.engine INFO: Eval iter: 900
- [11/21 20:16:54] ppdet.engine INFO: Eval iter: 1000
- [11/21 20:16:58] ppdet.engine INFO: Eval iter: 1100
- [11/21 20:17:02] ppdet.engine INFO: Eval iter: 1200
- [11/21 20:17:06] ppdet.engine INFO: Eval iter: 1300
- [11/21 20:17:10] ppdet.engine INFO: Eval iter: 1400
- [11/21 20:17:14] ppdet.engine INFO: Eval iter: 1500
- [11/21 20:17:18] ppdet.engine INFO: Eval iter: 1600
- [11/21 20:17:23] ppdet.engine INFO: Eval iter: 1700
- [11/21 20:17:27] ppdet.engine INFO: Eval iter: 1800
- [11/21 20:17:31] ppdet.engine INFO: Eval iter: 1900
- [11/21 20:17:35] ppdet.engine INFO: Eval iter: 2000
- [11/21 20:17:39] ppdet.engine INFO: Eval iter: 2100
- [11/21 20:17:43] ppdet.engine INFO: Eval iter: 2200
- [11/21 20:17:48] ppdet.engine INFO: Eval iter: 2300
- [11/21 20:17:52] ppdet.engine INFO: Eval iter: 2400
- [11/21 20:17:56] ppdet.engine INFO: Eval iter: 2500
- [11/21 20:18:00] ppdet.engine INFO: Eval iter: 2600
- [11/21 20:18:04] ppdet.engine INFO: Eval iter: 2700
- [11/21 20:18:08] ppdet.engine INFO: Eval iter: 2800
- [11/21 20:18:12] ppdet.engine INFO: Eval iter: 2900
- [11/21 20:18:16] ppdet.engine INFO: Eval iter: 3000
- [11/21 20:18:21] ppdet.engine INFO: Eval iter: 3100
- [11/21 20:18:25] ppdet.engine INFO: Eval iter: 3200
- [11/21 20:18:29] ppdet.engine INFO: Eval iter: 3300
- [11/21 20:18:33] ppdet.engine INFO: Eval iter: 3400
- [11/21 20:18:37] ppdet.engine INFO: Eval iter: 3500
- [11/21 20:18:41] ppdet.engine INFO: Eval iter: 3600
- [11/21 20:18:45] ppdet.engine INFO: Eval iter: 3700
- [11/21 20:18:49] ppdet.engine INFO: Eval iter: 3800
- [11/21 20:18:53] ppdet.engine INFO: Eval iter: 3900
- [11/21 20:18:57] ppdet.engine INFO: Eval iter: 4000
- [11/21 20:19:01] ppdet.engine INFO: Eval iter: 4100
- [11/21 20:19:05] ppdet.engine INFO: Eval iter: 4200
- [11/21 20:19:09] ppdet.engine INFO: Eval iter: 4300
- [11/21 20:19:10] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 20:19:10] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 64.68%
- [11/21 20:19:10] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.459351907375368
- [11/21 20:19:10] ppdet.engine INFO: Best test bbox ap is 0.647.
- [11/21 20:19:14] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 20:19:14] ppdet.engine INFO: Epoch: [5] [ 0/1215] learning_rate: 0.005000 loss_xy: 3.285943 loss_wh: 0.929083 loss_obj: 5.088337 loss_cls: 1.882272 loss: 11.214098 eta: 6:44:44 batch_cost: 0.3518 data_cost: 0.0149 ips: 22.7420 images/s
- [11/21 20:20:28] ppdet.engine INFO: Epoch: [5] [ 200/1215] learning_rate: 0.005000 loss_xy: 3.169443 loss_wh: 0.889203 loss_obj: 4.844648 loss_cls: 1.699094 loss: 10.758291 eta: 6:43:36 batch_cost: 0.3656 data_cost: 0.0140 ips: 21.8793 images/s
- [11/21 20:21:42] ppdet.engine INFO: Epoch: [5] [ 400/1215] learning_rate: 0.005000 loss_xy: 3.117121 loss_wh: 0.903281 loss_obj: 4.857103 loss_cls: 1.725675 loss: 10.575854 eta: 6:42:35 batch_cost: 0.3689 data_cost: 0.0020 ips: 21.6846 images/s
- [11/21 20:22:56] ppdet.engine INFO: Epoch: [5] [ 600/1215] learning_rate: 0.005000 loss_xy: 3.139892 loss_wh: 0.883015 loss_obj: 4.775256 loss_cls: 1.693913 loss: 10.631266 eta: 6:41:37 batch_cost: 0.3711 data_cost: 0.0080 ips: 21.5577 images/s
- [11/21 20:24:08] ppdet.engine INFO: Epoch: [5] [ 800/1215] learning_rate: 0.005000 loss_xy: 3.039191 loss_wh: 0.865632 loss_obj: 4.830009 loss_cls: 1.766668 loss: 10.627556 eta: 6:40:16 batch_cost: 0.3599 data_cost: 0.0064 ips: 22.2311 images/s
- [11/21 20:25:23] ppdet.engine INFO: Epoch: [5] [1000/1215] learning_rate: 0.005000 loss_xy: 3.096015 loss_wh: 0.885996 loss_obj: 5.037317 loss_cls: 1.740519 loss: 11.044550 eta: 6:39:19 batch_cost: 0.3724 data_cost: 0.0007 ips: 21.4849 images/s
- [11/21 20:26:38] ppdet.engine INFO: Epoch: [5] [1200/1215] learning_rate: 0.005000 loss_xy: 3.254282 loss_wh: 0.882240 loss_obj: 4.854972 loss_cls: 1.783167 loss: 10.913807 eta: 6:38:18 batch_cost: 0.3704 data_cost: 0.0201 ips: 21.5996 images/s
- [11/21 20:26:43] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 20:26:43] ppdet.engine INFO: Eval iter: 0
- [11/21 20:26:48] ppdet.engine INFO: Eval iter: 100
- [11/21 20:26:52] ppdet.engine INFO: Eval iter: 200
- [11/21 20:26:56] ppdet.engine INFO: Eval iter: 300
- [11/21 20:27:00] ppdet.engine INFO: Eval iter: 400
- [11/21 20:27:04] ppdet.engine INFO: Eval iter: 500
- [11/21 20:27:08] ppdet.engine INFO: Eval iter: 600
- [11/21 20:27:12] ppdet.engine INFO: Eval iter: 700
- [11/21 20:27:16] ppdet.engine INFO: Eval iter: 800
- [11/21 20:27:20] ppdet.engine INFO: Eval iter: 900
- [11/21 20:27:25] ppdet.engine INFO: Eval iter: 1000
- [11/21 20:27:29] ppdet.engine INFO: Eval iter: 1100
- [11/21 20:27:33] ppdet.engine INFO: Eval iter: 1200
- [11/21 20:27:37] ppdet.engine INFO: Eval iter: 1300
- [11/21 20:27:41] ppdet.engine INFO: Eval iter: 1400
- [11/21 20:27:45] ppdet.engine INFO: Eval iter: 1500
- [11/21 20:27:49] ppdet.engine INFO: Eval iter: 1600
- [11/21 20:27:53] ppdet.engine INFO: Eval iter: 1700
- [11/21 20:27:57] ppdet.engine INFO: Eval iter: 1800
- [11/21 20:28:01] ppdet.engine INFO: Eval iter: 1900
- [11/21 20:28:05] ppdet.engine INFO: Eval iter: 2000
- [11/21 20:28:10] ppdet.engine INFO: Eval iter: 2100
- [11/21 20:28:14] ppdet.engine INFO: Eval iter: 2200
- [11/21 20:28:18] ppdet.engine INFO: Eval iter: 2300
- [11/21 20:28:22] ppdet.engine INFO: Eval iter: 2400
- [11/21 20:28:26] ppdet.engine INFO: Eval iter: 2500
- [11/21 20:28:30] ppdet.engine INFO: Eval iter: 2600
- [11/21 20:28:35] ppdet.engine INFO: Eval iter: 2700
- [11/21 20:28:39] ppdet.engine INFO: Eval iter: 2800
- [11/21 20:28:43] ppdet.engine INFO: Eval iter: 2900
- [11/21 20:28:47] ppdet.engine INFO: Eval iter: 3000
- [11/21 20:28:51] ppdet.engine INFO: Eval iter: 3100
- [11/21 20:28:55] ppdet.engine INFO: Eval iter: 3200
- [11/21 20:28:59] ppdet.engine INFO: Eval iter: 3300
- [11/21 20:29:03] ppdet.engine INFO: Eval iter: 3400
- [11/21 20:29:07] ppdet.engine INFO: Eval iter: 3500
- [11/21 20:29:11] ppdet.engine INFO: Eval iter: 3600
- [11/21 20:29:15] ppdet.engine INFO: Eval iter: 3700
- [11/21 20:29:20] ppdet.engine INFO: Eval iter: 3800
- [11/21 20:29:24] ppdet.engine INFO: Eval iter: 3900
- [11/21 20:29:28] ppdet.engine INFO: Eval iter: 4000
- [11/21 20:29:32] ppdet.engine INFO: Eval iter: 4100
- [11/21 20:29:36] ppdet.engine INFO: Eval iter: 4200
- [11/21 20:29:41] ppdet.engine INFO: Eval iter: 4300
- [11/21 20:29:42] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 20:29:42] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 62.76%
- [11/21 20:29:42] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.24435702122981
- [11/21 20:29:42] ppdet.engine INFO: Best test bbox ap is 0.647.
- [11/21 20:29:42] ppdet.engine INFO: Epoch: [6] [ 0/1215] learning_rate: 0.005000 loss_xy: 3.251781 loss_wh: 0.878039 loss_obj: 4.893014 loss_cls: 1.810419 loss: 10.938446 eta: 6:38:08 batch_cost: 0.3654 data_cost: 0.0208 ips: 21.8967 images/s
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:51963', '127.0.0.1:56813', '127.0.0.1:44965']
- I1121 20:37:24.178506 60136 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1121 20:37:25.023816 60136 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1121 20:37:25.028494 60136 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/21 20:37:26] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- [11/21 20:37:26] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/21 20:37:27] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- [11/21 20:37:27] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- [11/21 20:37:28] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- [11/21 20:37:29] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- [11/21 20:37:30] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- [11/21 20:37:30] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- [11/21 20:37:30] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- [11/21 20:37:31] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- [11/21 20:37:31] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/21 20:37:32] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- [11/21 20:37:32] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- [11/21 20:37:32] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/21 20:37:33] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- [11/21 20:37:33] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- [11/21 20:37:35] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/21 20:37:37] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/5.pdparams
- [11/21 20:37:40] ppdet.engine INFO: Epoch: [6] [ 0/1215] learning_rate: 0.005000 loss_xy: 5.172583 loss_wh: 1.443541 loss_obj: 9.489234 loss_cls: 2.813885 loss: 18.919243 eta: 1 day, 18:01:10 batch_cost: 2.3056 data_cost: 0.0003 ips: 3.4698 images/s
- [11/21 20:38:52] ppdet.engine INFO: Epoch: [6] [ 200/1215] learning_rate: 0.005000 loss_xy: 3.049188 loss_wh: 0.833463 loss_obj: 4.790462 loss_cls: 1.663506 loss: 10.406212 eta: 6:43:53 batch_cost: 0.3608 data_cost: 0.0116 ips: 22.1720 images/s
- [11/21 20:40:08] ppdet.engine INFO: Epoch: [6] [ 400/1215] learning_rate: 0.005000 loss_xy: 3.017318 loss_wh: 0.801892 loss_obj: 4.673733 loss_cls: 1.751022 loss: 10.331031 eta: 6:46:00 batch_cost: 0.3767 data_cost: 0.0107 ips: 21.2394 images/s
- [11/21 20:41:22] ppdet.engine INFO: Epoch: [6] [ 600/1215] learning_rate: 0.005000 loss_xy: 3.046758 loss_wh: 0.815877 loss_obj: 4.862494 loss_cls: 1.651854 loss: 10.510056 eta: 6:42:51 batch_cost: 0.3683 data_cost: 0.0076 ips: 21.7208 images/s
- [11/21 20:42:36] ppdet.engine INFO: Epoch: [6] [ 800/1215] learning_rate: 0.005000 loss_xy: 3.037067 loss_wh: 0.814837 loss_obj: 4.375135 loss_cls: 1.611395 loss: 9.955120 eta: 6:41:19 batch_cost: 0.3707 data_cost: 0.0176 ips: 21.5803 images/s
- [11/21 20:43:50] ppdet.engine INFO: Epoch: [6] [1000/1215] learning_rate: 0.005000 loss_xy: 3.143566 loss_wh: 0.863554 loss_obj: 4.900222 loss_cls: 1.645585 loss: 10.951302 eta: 6:39:20 batch_cost: 0.3681 data_cost: 0.0188 ips: 21.7327 images/s
- [11/21 20:45:04] ppdet.engine INFO: Epoch: [6] [1200/1215] learning_rate: 0.005000 loss_xy: 3.065593 loss_wh: 0.838312 loss_obj: 4.854993 loss_cls: 1.584060 loss: 10.253614 eta: 6:37:24 batch_cost: 0.3669 data_cost: 0.0116 ips: 21.8026 images/s
- [11/21 20:45:10] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 20:45:11] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_2.xml, x1: 217.0, y1: 763.0, x2: 217.0, y2: 764.0.
- [11/21 20:45:11] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_8.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/21 20:45:11] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_881_3.xml, x1: 35.0, y1: 198.0, x2: 75.0, y2: 197.0.
- [11/21 20:45:12] ppdet.engine INFO: Eval iter: 0
- [11/21 20:45:16] ppdet.engine INFO: Eval iter: 100
- [11/21 20:45:20] ppdet.engine INFO: Eval iter: 200
- [11/21 20:45:24] ppdet.engine INFO: Eval iter: 300
- [11/21 20:45:28] ppdet.engine INFO: Eval iter: 400
- [11/21 20:45:32] ppdet.engine INFO: Eval iter: 500
- [11/21 20:45:36] ppdet.engine INFO: Eval iter: 600
- [11/21 20:45:41] ppdet.engine INFO: Eval iter: 700
- [11/21 20:45:45] ppdet.engine INFO: Eval iter: 800
- [11/21 20:45:49] ppdet.engine INFO: Eval iter: 900
- [11/21 20:45:53] ppdet.engine INFO: Eval iter: 1000
- [11/21 20:45:56] ppdet.engine INFO: Eval iter: 1100
- [11/21 20:46:00] ppdet.engine INFO: Eval iter: 1200
- [11/21 20:46:04] ppdet.engine INFO: Eval iter: 1300
- [11/21 20:46:08] ppdet.engine INFO: Eval iter: 1400
- [11/21 20:46:12] ppdet.engine INFO: Eval iter: 1500
- [11/21 20:46:16] ppdet.engine INFO: Eval iter: 1600
- [11/21 20:46:20] ppdet.engine INFO: Eval iter: 1700
- [11/21 20:46:24] ppdet.engine INFO: Eval iter: 1800
- [11/21 20:46:28] ppdet.engine INFO: Eval iter: 1900
- [11/21 20:46:32] ppdet.engine INFO: Eval iter: 2000
- [11/21 20:46:36] ppdet.engine INFO: Eval iter: 2100
- [11/21 20:46:40] ppdet.engine INFO: Eval iter: 2200
- [11/21 20:46:44] ppdet.engine INFO: Eval iter: 2300
- [11/21 20:46:48] ppdet.engine INFO: Eval iter: 2400
- [11/21 20:46:52] ppdet.engine INFO: Eval iter: 2500
- [11/21 20:46:56] ppdet.engine INFO: Eval iter: 2600
- [11/21 20:47:00] ppdet.engine INFO: Eval iter: 2700
- [11/21 20:47:04] ppdet.engine INFO: Eval iter: 2800
- [11/21 20:47:08] ppdet.engine INFO: Eval iter: 2900
- [11/21 20:47:11] ppdet.engine INFO: Eval iter: 3000
- [11/21 20:47:15] ppdet.engine INFO: Eval iter: 3100
- [11/21 20:47:19] ppdet.engine INFO: Eval iter: 3200
- [11/21 20:47:23] ppdet.engine INFO: Eval iter: 3300
- [11/21 20:47:27] ppdet.engine INFO: Eval iter: 3400
- [11/21 20:47:31] ppdet.engine INFO: Eval iter: 3500
- [11/21 20:47:35] ppdet.engine INFO: Eval iter: 3600
- [11/21 20:47:39] ppdet.engine INFO: Eval iter: 3700
- [11/21 20:47:43] ppdet.engine INFO: Eval iter: 3800
- [11/21 20:47:47] ppdet.engine INFO: Eval iter: 3900
- [11/21 20:47:51] ppdet.engine INFO: Eval iter: 4000
- [11/21 20:47:55] ppdet.engine INFO: Eval iter: 4100
- [11/21 20:47:59] ppdet.engine INFO: Eval iter: 4200
- [11/21 20:48:03] ppdet.engine INFO: Eval iter: 4300
- [11/21 20:48:04] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 20:48:04] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 64.54%
- [11/21 20:48:04] ppdet.engine INFO: Total sample number: 4322, averge FPS: 25.08974089908109
- [11/21 20:48:04] ppdet.engine INFO: Best test bbox ap is 0.645.
- [11/21 20:48:08] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 20:48:08] ppdet.engine INFO: Epoch: [7] [ 0/1215] learning_rate: 0.005000 loss_xy: 3.124048 loss_wh: 0.847830 loss_obj: 4.791386 loss_cls: 1.607728 loss: 10.188902 eta: 6:37:17 batch_cost: 0.3674 data_cost: 0.0138 ips: 21.7772 images/s
- [11/21 20:49:23] ppdet.engine INFO: Epoch: [7] [ 200/1215] learning_rate: 0.005000 loss_xy: 2.952436 loss_wh: 0.809239 loss_obj: 4.409677 loss_cls: 1.533153 loss: 9.687810 eta: 6:36:22 batch_cost: 0.3722 data_cost: 0.0157 ips: 21.4930 images/s
- [11/21 20:50:38] ppdet.engine INFO: Epoch: [7] [ 400/1215] learning_rate: 0.005000 loss_xy: 2.994928 loss_wh: 0.815472 loss_obj: 4.376843 loss_cls: 1.419490 loss: 9.952207 eta: 6:36:00 batch_cost: 0.3771 data_cost: 0.0195 ips: 21.2157 images/s
- [11/21 20:51:54] ppdet.engine INFO: Epoch: [7] [ 600/1215] learning_rate: 0.005000 loss_xy: 3.173035 loss_wh: 0.845112 loss_obj: 4.502929 loss_cls: 1.553364 loss: 10.493731 eta: 6:35:15 batch_cost: 0.3755 data_cost: 0.0242 ips: 21.3052 images/s
- [11/21 20:53:09] ppdet.engine INFO: Epoch: [7] [ 800/1215] learning_rate: 0.005000 loss_xy: 3.208590 loss_wh: 0.815607 loss_obj: 4.776333 loss_cls: 1.570000 loss: 10.657371 eta: 6:34:30 batch_cost: 0.3763 data_cost: 0.0247 ips: 21.2615 images/s
- [11/21 20:54:24] ppdet.engine INFO: Epoch: [7] [1000/1215] learning_rate: 0.005000 loss_xy: 3.277271 loss_wh: 0.848793 loss_obj: 4.856776 loss_cls: 1.571626 loss: 10.568837 eta: 6:33:03 batch_cost: 0.3702 data_cost: 0.0425 ips: 21.6128 images/s
- [11/21 20:55:39] ppdet.engine INFO: Epoch: [7] [1200/1215] learning_rate: 0.005000 loss_xy: 3.057781 loss_wh: 0.807545 loss_obj: 4.527632 loss_cls: 1.434904 loss: 9.984177 eta: 6:31:56 batch_cost: 0.3733 data_cost: 0.0160 ips: 21.4282 images/s
- [11/21 20:55:45] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 20:55:45] ppdet.engine INFO: Eval iter: 0
- [11/21 20:55:49] ppdet.engine INFO: Eval iter: 100
- [11/21 20:55:53] ppdet.engine INFO: Eval iter: 200
- [11/21 20:55:57] ppdet.engine INFO: Eval iter: 300
- [11/21 20:56:01] ppdet.engine INFO: Eval iter: 400
- [11/21 20:56:06] ppdet.engine INFO: Eval iter: 500
- [11/21 20:56:10] ppdet.engine INFO: Eval iter: 600
- [11/21 20:56:14] ppdet.engine INFO: Eval iter: 700
- [11/21 20:56:18] ppdet.engine INFO: Eval iter: 800
- [11/21 20:56:22] ppdet.engine INFO: Eval iter: 900
- [11/21 20:56:26] ppdet.engine INFO: Eval iter: 1000
- [11/21 20:56:30] ppdet.engine INFO: Eval iter: 1100
- [11/21 20:56:35] ppdet.engine INFO: Eval iter: 1200
- [11/21 20:56:39] ppdet.engine INFO: Eval iter: 1300
- [11/21 20:56:43] ppdet.engine INFO: Eval iter: 1400
- [11/21 20:56:47] ppdet.engine INFO: Eval iter: 1500
- [11/21 20:56:51] ppdet.engine INFO: Eval iter: 1600
- [11/21 20:56:55] ppdet.engine INFO: Eval iter: 1700
- [11/21 20:56:59] ppdet.engine INFO: Eval iter: 1800
- [11/21 20:57:03] ppdet.engine INFO: Eval iter: 1900
- [11/21 20:57:07] ppdet.engine INFO: Eval iter: 2000
- [11/21 20:57:12] ppdet.engine INFO: Eval iter: 2100
- [11/21 20:57:16] ppdet.engine INFO: Eval iter: 2200
- [11/21 20:57:20] ppdet.engine INFO: Eval iter: 2300
- [11/21 20:57:24] ppdet.engine INFO: Eval iter: 2400
- [11/21 20:57:29] ppdet.engine INFO: Eval iter: 2500
- [11/21 20:57:33] ppdet.engine INFO: Eval iter: 2600
- [11/21 20:57:37] ppdet.engine INFO: Eval iter: 2700
- [11/21 20:57:41] ppdet.engine INFO: Eval iter: 2800
- [11/21 20:57:45] ppdet.engine INFO: Eval iter: 2900
- [11/21 20:57:49] ppdet.engine INFO: Eval iter: 3000
- [11/21 20:57:53] ppdet.engine INFO: Eval iter: 3100
- [11/21 20:57:57] ppdet.engine INFO: Eval iter: 3200
- [11/21 20:58:02] ppdet.engine INFO: Eval iter: 3300
- [11/21 20:58:06] ppdet.engine INFO: Eval iter: 3400
- [11/21 20:58:10] ppdet.engine INFO: Eval iter: 3500
- [11/21 20:58:14] ppdet.engine INFO: Eval iter: 3600
- [11/21 20:58:18] ppdet.engine INFO: Eval iter: 3700
- [11/21 20:58:22] ppdet.engine INFO: Eval iter: 3800
- [11/21 20:58:26] ppdet.engine INFO: Eval iter: 3900
- [11/21 20:58:30] ppdet.engine INFO: Eval iter: 4000
- [11/21 20:58:35] ppdet.engine INFO: Eval iter: 4100
- [11/21 20:58:39] ppdet.engine INFO: Eval iter: 4200
- [11/21 20:58:43] ppdet.engine INFO: Eval iter: 4300
- [11/21 20:58:44] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 20:58:44] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 73.14%
- [11/21 20:58:44] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.140250588217615
- [11/21 20:58:44] ppdet.engine INFO: Best test bbox ap is 0.731.
- [11/21 20:58:48] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 20:58:48] ppdet.engine INFO: Epoch: [8] [ 0/1215] learning_rate: 0.005000 loss_xy: 3.094211 loss_wh: 0.811069 loss_obj: 4.527632 loss_cls: 1.448323 loss: 9.997572 eta: 6:31:42 batch_cost: 0.3731 data_cost: 0.0160 ips: 21.4438 images/s
- [11/21 21:00:01] ppdet.engine INFO: Epoch: [8] [ 200/1215] learning_rate: 0.005000 loss_xy: 3.237075 loss_wh: 0.863777 loss_obj: 4.883410 loss_cls: 1.554860 loss: 10.425458 eta: 6:29:41 batch_cost: 0.3621 data_cost: 0.0203 ips: 22.0919 images/s
- [11/21 21:01:15] ppdet.engine INFO: Epoch: [8] [ 400/1215] learning_rate: 0.005000 loss_xy: 3.249852 loss_wh: 0.884585 loss_obj: 4.597641 loss_cls: 1.452735 loss: 10.255089 eta: 6:28:07 batch_cost: 0.3669 data_cost: 0.0040 ips: 21.8048 images/s
- [11/21 21:02:26] ppdet.engine INFO: Epoch: [8] [ 600/1215] learning_rate: 0.005000 loss_xy: 3.185300 loss_wh: 0.847045 loss_obj: 4.355394 loss_cls: 1.487365 loss: 9.963678 eta: 6:25:52 batch_cost: 0.3561 data_cost: 0.0210 ips: 22.4653 images/s
- [11/21 21:03:38] ppdet.engine INFO: Epoch: [8] [ 800/1215] learning_rate: 0.005000 loss_xy: 2.974223 loss_wh: 0.793044 loss_obj: 4.226223 loss_cls: 1.441566 loss: 9.619604 eta: 6:23:48 batch_cost: 0.3570 data_cost: 0.0091 ips: 22.4098 images/s
- [11/21 21:04:52] ppdet.engine INFO: Epoch: [8] [1000/1215] learning_rate: 0.005000 loss_xy: 3.209449 loss_wh: 0.881137 loss_obj: 4.859863 loss_cls: 1.549961 loss: 10.768457 eta: 6:22:43 batch_cost: 0.3716 data_cost: 0.0170 ips: 21.5301 images/s
- [11/21 21:06:05] ppdet.engine INFO: Epoch: [8] [1200/1215] learning_rate: 0.005000 loss_xy: 3.078016 loss_wh: 0.849547 loss_obj: 4.390286 loss_cls: 1.666456 loss: 10.104526 eta: 6:21:02 batch_cost: 0.3616 data_cost: 0.0333 ips: 22.1225 images/s
- [11/21 21:06:11] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 21:06:11] ppdet.engine INFO: Eval iter: 0
- [11/21 21:06:16] ppdet.engine INFO: Eval iter: 100
- [11/21 21:06:20] ppdet.engine INFO: Eval iter: 200
- [11/21 21:06:24] ppdet.engine INFO: Eval iter: 300
- [11/21 21:06:28] ppdet.engine INFO: Eval iter: 400
- [11/21 21:06:32] ppdet.engine INFO: Eval iter: 500
- [11/21 21:06:36] ppdet.engine INFO: Eval iter: 600
- [11/21 21:06:40] ppdet.engine INFO: Eval iter: 700
- [11/21 21:06:44] ppdet.engine INFO: Eval iter: 800
- [11/21 21:06:49] ppdet.engine INFO: Eval iter: 900
- [11/21 21:06:53] ppdet.engine INFO: Eval iter: 1000
- [11/21 21:06:57] ppdet.engine INFO: Eval iter: 1100
- [11/21 21:07:01] ppdet.engine INFO: Eval iter: 1200
- [11/21 21:07:05] ppdet.engine INFO: Eval iter: 1300
- [11/21 21:07:09] ppdet.engine INFO: Eval iter: 1400
- [11/21 21:07:13] ppdet.engine INFO: Eval iter: 1500
- [11/21 21:07:18] ppdet.engine INFO: Eval iter: 1600
- [11/21 21:07:22] ppdet.engine INFO: Eval iter: 1700
- [11/21 21:07:26] ppdet.engine INFO: Eval iter: 1800
- [11/21 21:07:30] ppdet.engine INFO: Eval iter: 1900
- [11/21 21:07:34] ppdet.engine INFO: Eval iter: 2000
- [11/21 21:07:38] ppdet.engine INFO: Eval iter: 2100
- [11/21 21:07:43] ppdet.engine INFO: Eval iter: 2200
- [11/21 21:07:47] ppdet.engine INFO: Eval iter: 2300
- [11/21 21:07:51] ppdet.engine INFO: Eval iter: 2400
- [11/21 21:07:55] ppdet.engine INFO: Eval iter: 2500
- [11/21 21:07:59] ppdet.engine INFO: Eval iter: 2600
- [11/21 21:08:03] ppdet.engine INFO: Eval iter: 2700
- [11/21 21:08:08] ppdet.engine INFO: Eval iter: 2800
- [11/21 21:08:12] ppdet.engine INFO: Eval iter: 2900
- [11/21 21:08:16] ppdet.engine INFO: Eval iter: 3000
- [11/21 21:08:20] ppdet.engine INFO: Eval iter: 3100
- [11/21 21:08:24] ppdet.engine INFO: Eval iter: 3200
- [11/21 21:08:28] ppdet.engine INFO: Eval iter: 3300
- [11/21 21:08:32] ppdet.engine INFO: Eval iter: 3400
- [11/21 21:08:36] ppdet.engine INFO: Eval iter: 3500
- [11/21 21:08:41] ppdet.engine INFO: Eval iter: 3600
- [11/21 21:08:45] ppdet.engine INFO: Eval iter: 3700
- [11/21 21:08:49] ppdet.engine INFO: Eval iter: 3800
- [11/21 21:08:53] ppdet.engine INFO: Eval iter: 3900
- [11/21 21:08:57] ppdet.engine INFO: Eval iter: 4000
- [11/21 21:09:02] ppdet.engine INFO: Eval iter: 4100
- [11/21 21:09:06] ppdet.engine INFO: Eval iter: 4200
- [11/21 21:09:10] ppdet.engine INFO: Eval iter: 4300
- [11/21 21:09:11] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 21:09:11] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 70.38%
- [11/21 21:09:11] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.056198940035397
- [11/21 21:09:11] ppdet.engine INFO: Best test bbox ap is 0.731.
- [11/21 21:09:11] ppdet.engine INFO: Epoch: [9] [ 0/1215] learning_rate: 0.005000 loss_xy: 3.068559 loss_wh: 0.844330 loss_obj: 4.390286 loss_cls: 1.598435 loss: 9.932621 eta: 6:20:58 batch_cost: 0.3620 data_cost: 0.0277 ips: 22.1013 images/s
- [11/21 21:10:25] ppdet.engine INFO: Epoch: [9] [ 200/1215] learning_rate: 0.005000 loss_xy: 2.994976 loss_wh: 0.750517 loss_obj: 4.243375 loss_cls: 1.291535 loss: 9.520647 eta: 6:19:35 batch_cost: 0.3662 data_cost: 0.0190 ips: 21.8441 images/s
- [11/21 21:11:37] ppdet.engine INFO: Epoch: [9] [ 400/1215] learning_rate: 0.005000 loss_xy: 3.153412 loss_wh: 0.841728 loss_obj: 4.538942 loss_cls: 1.420639 loss: 10.126094 eta: 6:17:56 batch_cost: 0.3602 data_cost: 0.0287 ips: 22.2069 images/s
- [11/21 21:12:50] ppdet.engine INFO: Epoch: [9] [ 600/1215] learning_rate: 0.005000 loss_xy: 2.987780 loss_wh: 0.741688 loss_obj: 4.019764 loss_cls: 1.330453 loss: 9.367243 eta: 6:16:29 batch_cost: 0.3637 data_cost: 0.0198 ips: 21.9937 images/s
- [11/21 21:14:03] ppdet.engine INFO: Epoch: [9] [ 800/1215] learning_rate: 0.005000 loss_xy: 3.038010 loss_wh: 0.783942 loss_obj: 4.422535 loss_cls: 1.353974 loss: 9.683443 eta: 6:15:01 batch_cost: 0.3628 data_cost: 0.0039 ips: 22.0513 images/s
- [11/21 21:15:17] ppdet.engine INFO: Epoch: [9] [1000/1215] learning_rate: 0.005000 loss_xy: 3.191090 loss_wh: 0.846046 loss_obj: 4.206779 loss_cls: 1.379730 loss: 9.693302 eta: 6:13:46 batch_cost: 0.3674 data_cost: 0.0236 ips: 21.7761 images/s
- [11/21 21:16:30] ppdet.engine INFO: Epoch: [9] [1200/1215] learning_rate: 0.005000 loss_xy: 3.215653 loss_wh: 0.830798 loss_obj: 4.421699 loss_cls: 1.324503 loss: 9.929082 eta: 6:12:19 batch_cost: 0.3626 data_cost: 0.0190 ips: 22.0640 images/s
- [11/21 21:16:36] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 21:16:36] ppdet.engine INFO: Eval iter: 0
- [11/21 21:16:41] ppdet.engine INFO: Eval iter: 100
- [11/21 21:16:45] ppdet.engine INFO: Eval iter: 200
- [11/21 21:16:49] ppdet.engine INFO: Eval iter: 300
- [11/21 21:16:53] ppdet.engine INFO: Eval iter: 400
- [11/21 21:16:57] ppdet.engine INFO: Eval iter: 500
- [11/21 21:17:01] ppdet.engine INFO: Eval iter: 600
- [11/21 21:17:05] ppdet.engine INFO: Eval iter: 700
- [11/21 21:17:09] ppdet.engine INFO: Eval iter: 800
- [11/21 21:17:13] ppdet.engine INFO: Eval iter: 900
- [11/21 21:17:17] ppdet.engine INFO: Eval iter: 1000
- [11/21 21:17:21] ppdet.engine INFO: Eval iter: 1100
- [11/21 21:17:25] ppdet.engine INFO: Eval iter: 1200
- [11/21 21:17:29] ppdet.engine INFO: Eval iter: 1300
- [11/21 21:17:33] ppdet.engine INFO: Eval iter: 1400
- [11/21 21:17:37] ppdet.engine INFO: Eval iter: 1500
- [11/21 21:17:41] ppdet.engine INFO: Eval iter: 1600
- [11/21 21:17:45] ppdet.engine INFO: Eval iter: 1700
- [11/21 21:17:49] ppdet.engine INFO: Eval iter: 1800
- [11/21 21:17:53] ppdet.engine INFO: Eval iter: 1900
- [11/21 21:17:57] ppdet.engine INFO: Eval iter: 2000
- [11/21 21:18:02] ppdet.engine INFO: Eval iter: 2100
- [11/21 21:18:06] ppdet.engine INFO: Eval iter: 2200
- [11/21 21:18:10] ppdet.engine INFO: Eval iter: 2300
- [11/21 21:18:14] ppdet.engine INFO: Eval iter: 2400
- [11/21 21:18:18] ppdet.engine INFO: Eval iter: 2500
- [11/21 21:18:22] ppdet.engine INFO: Eval iter: 2600
- [11/21 21:18:26] ppdet.engine INFO: Eval iter: 2700
- [11/21 21:18:30] ppdet.engine INFO: Eval iter: 2800
- [11/21 21:18:34] ppdet.engine INFO: Eval iter: 2900
- [11/21 21:18:38] ppdet.engine INFO: Eval iter: 3000
- [11/21 21:18:42] ppdet.engine INFO: Eval iter: 3100
- [11/21 21:18:46] ppdet.engine INFO: Eval iter: 3200
- [11/21 21:18:50] ppdet.engine INFO: Eval iter: 3300
- [11/21 21:18:54] ppdet.engine INFO: Eval iter: 3400
- [11/21 21:18:59] ppdet.engine INFO: Eval iter: 3500
- [11/21 21:19:03] ppdet.engine INFO: Eval iter: 3600
- [11/21 21:19:07] ppdet.engine INFO: Eval iter: 3700
- [11/21 21:19:11] ppdet.engine INFO: Eval iter: 3800
- [11/21 21:19:15] ppdet.engine INFO: Eval iter: 3900
- [11/21 21:19:19] ppdet.engine INFO: Eval iter: 4000
- [11/21 21:19:23] ppdet.engine INFO: Eval iter: 4100
- [11/21 21:19:27] ppdet.engine INFO: Eval iter: 4200
- [11/21 21:19:31] ppdet.engine INFO: Eval iter: 4300
- [11/21 21:19:32] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 21:19:32] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 73.96%
- [11/21 21:19:32] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.662477913157474
- [11/21 21:19:32] ppdet.engine INFO: Best test bbox ap is 0.740.
- [11/21 21:19:35] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 21:19:36] ppdet.engine INFO: Epoch: [10] [ 0/1215] learning_rate: 0.005000 loss_xy: 3.231498 loss_wh: 0.840385 loss_obj: 4.482869 loss_cls: 1.316933 loss: 9.963638 eta: 6:12:17 batch_cost: 0.3648 data_cost: 0.0178 ips: 21.9326 images/s
- [11/21 21:20:48] ppdet.engine INFO: Epoch: [10] [ 200/1215] learning_rate: 0.005000 loss_xy: 3.105217 loss_wh: 0.804261 loss_obj: 4.320622 loss_cls: 1.302809 loss: 9.551403 eta: 6:10:42 batch_cost: 0.3589 data_cost: 0.0290 ips: 22.2923 images/s
- [11/21 21:22:02] ppdet.engine INFO: Epoch: [10] [ 400/1215] learning_rate: 0.005000 loss_xy: 2.898128 loss_wh: 0.768502 loss_obj: 3.944427 loss_cls: 1.170288 loss: 8.839090 eta: 6:09:36 batch_cost: 0.3704 data_cost: 0.0233 ips: 21.5978 images/s
- [11/21 21:23:13] ppdet.engine INFO: Epoch: [10] [ 600/1215] learning_rate: 0.005000 loss_xy: 3.110843 loss_wh: 0.811502 loss_obj: 4.404681 loss_cls: 1.367808 loss: 9.715208 eta: 6:07:45 batch_cost: 0.3504 data_cost: 0.0192 ips: 22.8342 images/s
- [11/21 21:24:26] ppdet.engine INFO: Epoch: [10] [ 800/1215] learning_rate: 0.005000 loss_xy: 2.976946 loss_wh: 0.816105 loss_obj: 4.132264 loss_cls: 1.298354 loss: 9.380390 eta: 6:06:32 batch_cost: 0.3671 data_cost: 0.0232 ips: 21.7929 images/s
- [11/21 21:25:39] ppdet.engine INFO: Epoch: [10] [1000/1215] learning_rate: 0.005000 loss_xy: 2.885143 loss_wh: 0.759521 loss_obj: 4.020351 loss_cls: 1.298985 loss: 9.133148 eta: 6:05:12 batch_cost: 0.3639 data_cost: 0.0141 ips: 21.9811 images/s
- [11/21 21:26:51] ppdet.engine INFO: Epoch: [10] [1200/1215] learning_rate: 0.005000 loss_xy: 3.231733 loss_wh: 0.842759 loss_obj: 4.434932 loss_cls: 1.356978 loss: 10.025740 eta: 6:03:43 batch_cost: 0.3587 data_cost: 0.0169 ips: 22.3018 images/s
- [11/21 21:26:58] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 21:26:58] ppdet.engine INFO: Eval iter: 0
- [11/21 21:27:02] ppdet.engine INFO: Eval iter: 100
- [11/21 21:27:06] ppdet.engine INFO: Eval iter: 200
- [11/21 21:27:10] ppdet.engine INFO: Eval iter: 300
- [11/21 21:27:14] ppdet.engine INFO: Eval iter: 400
- [11/21 21:27:18] ppdet.engine INFO: Eval iter: 500
- [11/21 21:27:22] ppdet.engine INFO: Eval iter: 600
- [11/21 21:27:26] ppdet.engine INFO: Eval iter: 700
- [11/21 21:27:30] ppdet.engine INFO: Eval iter: 800
- [11/21 21:27:34] ppdet.engine INFO: Eval iter: 900
- [11/21 21:27:38] ppdet.engine INFO: Eval iter: 1000
- [11/21 21:27:42] ppdet.engine INFO: Eval iter: 1100
- [11/21 21:27:46] ppdet.engine INFO: Eval iter: 1200
- [11/21 21:27:50] ppdet.engine INFO: Eval iter: 1300
- [11/21 21:27:54] ppdet.engine INFO: Eval iter: 1400
- [11/21 21:27:58] ppdet.engine INFO: Eval iter: 1500
- [11/21 21:28:02] ppdet.engine INFO: Eval iter: 1600
- [11/21 21:28:06] ppdet.engine INFO: Eval iter: 1700
- [11/21 21:28:10] ppdet.engine INFO: Eval iter: 1800
- [11/21 21:28:14] ppdet.engine INFO: Eval iter: 1900
- [11/21 21:28:18] ppdet.engine INFO: Eval iter: 2000
- [11/21 21:28:23] ppdet.engine INFO: Eval iter: 2100
- [11/21 21:28:27] ppdet.engine INFO: Eval iter: 2200
- [11/21 21:28:31] ppdet.engine INFO: Eval iter: 2300
- [11/21 21:28:35] ppdet.engine INFO: Eval iter: 2400
- [11/21 21:28:39] ppdet.engine INFO: Eval iter: 2500
- [11/21 21:28:43] ppdet.engine INFO: Eval iter: 2600
- [11/21 21:28:47] ppdet.engine INFO: Eval iter: 2700
- [11/21 21:28:51] ppdet.engine INFO: Eval iter: 2800
- [11/21 21:28:55] ppdet.engine INFO: Eval iter: 2900
- [11/21 21:28:59] ppdet.engine INFO: Eval iter: 3000
- [11/21 21:29:03] ppdet.engine INFO: Eval iter: 3100
- [11/21 21:29:07] ppdet.engine INFO: Eval iter: 3200
- [11/21 21:29:11] ppdet.engine INFO: Eval iter: 3300
- [11/21 21:29:15] ppdet.engine INFO: Eval iter: 3400
- [11/21 21:29:19] ppdet.engine INFO: Eval iter: 3500
- [11/21 21:29:23] ppdet.engine INFO: Eval iter: 3600
- [11/21 21:29:27] ppdet.engine INFO: Eval iter: 3700
- [11/21 21:29:30] ppdet.engine INFO: Eval iter: 3800
- [11/21 21:29:34] ppdet.engine INFO: Eval iter: 3900
- [11/21 21:29:38] ppdet.engine INFO: Eval iter: 4000
- [11/21 21:29:42] ppdet.engine INFO: Eval iter: 4100
- [11/21 21:29:46] ppdet.engine INFO: Eval iter: 4200
- [11/21 21:29:50] ppdet.engine INFO: Eval iter: 4300
- [11/21 21:29:51] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 21:29:51] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 73.12%
- [11/21 21:29:51] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.907222725748753
- [11/21 21:29:51] ppdet.engine INFO: Best test bbox ap is 0.740.
- [11/21 21:29:52] ppdet.engine INFO: Epoch: [11] [ 0/1215] learning_rate: 0.005000 loss_xy: 3.245780 loss_wh: 0.842759 loss_obj: 4.490942 loss_cls: 1.313776 loss: 10.025740 eta: 6:03:38 batch_cost: 0.3594 data_cost: 0.0202 ips: 22.2596 images/s
- [11/21 21:31:05] ppdet.engine INFO: Epoch: [11] [ 200/1215] learning_rate: 0.005000 loss_xy: 3.118668 loss_wh: 0.821081 loss_obj: 4.016738 loss_cls: 1.280963 loss: 9.455399 eta: 6:02:20 batch_cost: 0.3639 data_cost: 0.0515 ips: 21.9812 images/s
- [11/21 21:32:19] ppdet.engine INFO: Epoch: [11] [ 400/1215] learning_rate: 0.005000 loss_xy: 3.085739 loss_wh: 0.799854 loss_obj: 4.239019 loss_cls: 1.144755 loss: 9.496972 eta: 6:01:13 batch_cost: 0.3702 data_cost: 0.0215 ips: 21.6086 images/s
- [11/21 21:33:34] ppdet.engine INFO: Epoch: [11] [ 600/1215] learning_rate: 0.005000 loss_xy: 3.062665 loss_wh: 0.765819 loss_obj: 4.249835 loss_cls: 1.255022 loss: 9.526673 eta: 6:00:12 batch_cost: 0.3734 data_cost: 0.0190 ips: 21.4235 images/s
- [11/21 21:34:46] ppdet.engine INFO: Epoch: [11] [ 800/1215] learning_rate: 0.005000 loss_xy: 2.994057 loss_wh: 0.810020 loss_obj: 4.185465 loss_cls: 1.271169 loss: 9.386173 eta: 5:58:45 batch_cost: 0.3587 data_cost: 0.0175 ips: 22.3033 images/s
- [11/21 21:36:01] ppdet.engine INFO: Epoch: [11] [1000/1215] learning_rate: 0.005000 loss_xy: 3.086114 loss_wh: 0.824392 loss_obj: 4.247724 loss_cls: 1.259302 loss: 9.606213 eta: 5:57:44 batch_cost: 0.3738 data_cost: 0.0086 ips: 21.3993 images/s
- [11/21 21:37:16] ppdet.engine INFO: Epoch: [11] [1200/1215] learning_rate: 0.005000 loss_xy: 3.166054 loss_wh: 0.813043 loss_obj: 4.408890 loss_cls: 1.351862 loss: 9.758865 eta: 5:56:44 batch_cost: 0.3752 data_cost: 0.0284 ips: 21.3217 images/s
- [11/21 21:37:22] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 21:37:22] ppdet.engine INFO: Eval iter: 0
- [11/21 21:37:26] ppdet.engine INFO: Eval iter: 100
- [11/21 21:37:30] ppdet.engine INFO: Eval iter: 200
- [11/21 21:37:35] ppdet.engine INFO: Eval iter: 300
- [11/21 21:37:39] ppdet.engine INFO: Eval iter: 400
- [11/21 21:37:42] ppdet.engine INFO: Eval iter: 500
- [11/21 21:37:46] ppdet.engine INFO: Eval iter: 600
- [11/21 21:37:50] ppdet.engine INFO: Eval iter: 700
- [11/21 21:37:54] ppdet.engine INFO: Eval iter: 800
- [11/21 21:37:58] ppdet.engine INFO: Eval iter: 900
- [11/21 21:38:02] ppdet.engine INFO: Eval iter: 1000
- [11/21 21:38:06] ppdet.engine INFO: Eval iter: 1100
- [11/21 21:38:10] ppdet.engine INFO: Eval iter: 1200
- [11/21 21:38:14] ppdet.engine INFO: Eval iter: 1300
- [11/21 21:38:18] ppdet.engine INFO: Eval iter: 1400
- [11/21 21:38:22] ppdet.engine INFO: Eval iter: 1500
- [11/21 21:38:26] ppdet.engine INFO: Eval iter: 1600
- [11/21 21:38:29] ppdet.engine INFO: Eval iter: 1700
- [11/21 21:38:33] ppdet.engine INFO: Eval iter: 1800
- [11/21 21:38:37] ppdet.engine INFO: Eval iter: 1900
- [11/21 21:38:41] ppdet.engine INFO: Eval iter: 2000
- [11/21 21:38:45] ppdet.engine INFO: Eval iter: 2100
- [11/21 21:38:49] ppdet.engine INFO: Eval iter: 2200
- [11/21 21:38:53] ppdet.engine INFO: Eval iter: 2300
- [11/21 21:38:56] ppdet.engine INFO: Eval iter: 2400
- [11/21 21:39:00] ppdet.engine INFO: Eval iter: 2500
- [11/21 21:39:04] ppdet.engine INFO: Eval iter: 2600
- [11/21 21:39:08] ppdet.engine INFO: Eval iter: 2700
- [11/21 21:39:12] ppdet.engine INFO: Eval iter: 2800
- [11/21 21:39:16] ppdet.engine INFO: Eval iter: 2900
- [11/21 21:39:20] ppdet.engine INFO: Eval iter: 3000
- [11/21 21:39:24] ppdet.engine INFO: Eval iter: 3100
- [11/21 21:39:27] ppdet.engine INFO: Eval iter: 3200
- [11/21 21:39:31] ppdet.engine INFO: Eval iter: 3300
- [11/21 21:39:35] ppdet.engine INFO: Eval iter: 3400
- [11/21 21:39:39] ppdet.engine INFO: Eval iter: 3500
- [11/21 21:39:43] ppdet.engine INFO: Eval iter: 3600
- [11/21 21:39:47] ppdet.engine INFO: Eval iter: 3700
- [11/21 21:39:51] ppdet.engine INFO: Eval iter: 3800
- [11/21 21:39:55] ppdet.engine INFO: Eval iter: 3900
- [11/21 21:39:59] ppdet.engine INFO: Eval iter: 4000
- [11/21 21:40:03] ppdet.engine INFO: Eval iter: 4100
- [11/21 21:40:06] ppdet.engine INFO: Eval iter: 4200
- [11/21 21:40:10] ppdet.engine INFO: Eval iter: 4300
- [11/21 21:40:11] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 21:40:11] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 75.25%
- [11/21 21:40:11] ppdet.engine INFO: Total sample number: 4322, averge FPS: 25.579222864819975
- [11/21 21:40:11] ppdet.engine INFO: Best test bbox ap is 0.752.
- [11/21 21:40:15] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 21:40:15] ppdet.engine INFO: Epoch: [12] [ 0/1215] learning_rate: 0.005000 loss_xy: 3.157222 loss_wh: 0.789062 loss_obj: 4.408890 loss_cls: 1.384141 loss: 9.767687 eta: 5:56:35 batch_cost: 0.3700 data_cost: 0.0302 ips: 21.6199 images/s
- [11/21 21:41:29] ppdet.engine INFO: Epoch: [12] [ 200/1215] learning_rate: 0.005000 loss_xy: 3.097861 loss_wh: 0.788223 loss_obj: 4.566910 loss_cls: 1.348943 loss: 9.828667 eta: 5:55:26 batch_cost: 0.3694 data_cost: 0.0186 ips: 21.6546 images/s
- [11/21 21:42:45] ppdet.engine INFO: Epoch: [12] [ 400/1215] learning_rate: 0.005000 loss_xy: 3.211385 loss_wh: 0.828650 loss_obj: 4.710813 loss_cls: 1.465709 loss: 10.157497 eta: 5:54:29 batch_cost: 0.3779 data_cost: 0.0105 ips: 21.1705 images/s
- [11/21 21:44:00] ppdet.engine INFO: Epoch: [12] [ 600/1215] learning_rate: 0.005000 loss_xy: 3.272685 loss_wh: 0.851797 loss_obj: 4.798033 loss_cls: 1.350842 loss: 10.525963 eta: 5:53:21 batch_cost: 0.3711 data_cost: 0.0012 ips: 21.5557 images/s
- [11/21 21:45:12] ppdet.engine INFO: Epoch: [12] [ 800/1215] learning_rate: 0.005000 loss_xy: 3.123688 loss_wh: 0.816545 loss_obj: 4.390236 loss_cls: 1.465376 loss: 9.965006 eta: 5:52:00 batch_cost: 0.3618 data_cost: 0.0018 ips: 22.1108 images/s
- [11/21 21:46:27] ppdet.engine INFO: Epoch: [12] [1000/1215] learning_rate: 0.005000 loss_xy: 3.253678 loss_wh: 0.845330 loss_obj: 4.707134 loss_cls: 1.346149 loss: 10.361443 eta: 5:50:51 batch_cost: 0.3703 data_cost: 0.0125 ips: 21.6024 images/s
- [11/21 21:47:40] ppdet.engine INFO: Epoch: [12] [1200/1215] learning_rate: 0.005000 loss_xy: 3.164453 loss_wh: 0.782429 loss_obj: 4.407469 loss_cls: 1.392390 loss: 9.656019 eta: 5:49:30 batch_cost: 0.3622 data_cost: 0.0100 ips: 22.0896 images/s
- [11/21 21:47:46] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 21:47:46] ppdet.engine INFO: Eval iter: 0
- [11/21 21:47:50] ppdet.engine INFO: Eval iter: 100
- [11/21 21:47:54] ppdet.engine INFO: Eval iter: 200
- [11/21 21:47:58] ppdet.engine INFO: Eval iter: 300
- [11/21 21:48:03] ppdet.engine INFO: Eval iter: 400
- [11/21 21:48:07] ppdet.engine INFO: Eval iter: 500
- [11/21 21:48:11] ppdet.engine INFO: Eval iter: 600
- [11/21 21:48:15] ppdet.engine INFO: Eval iter: 700
- [11/21 21:48:19] ppdet.engine INFO: Eval iter: 800
- [11/21 21:48:23] ppdet.engine INFO: Eval iter: 900
- [11/21 21:48:27] ppdet.engine INFO: Eval iter: 1000
- [11/21 21:48:31] ppdet.engine INFO: Eval iter: 1100
- [11/21 21:48:35] ppdet.engine INFO: Eval iter: 1200
- [11/21 21:48:39] ppdet.engine INFO: Eval iter: 1300
- [11/21 21:48:43] ppdet.engine INFO: Eval iter: 1400
- [11/21 21:48:47] ppdet.engine INFO: Eval iter: 1500
- [11/21 21:48:51] ppdet.engine INFO: Eval iter: 1600
- [11/21 21:48:54] ppdet.engine INFO: Eval iter: 1700
- [11/21 21:48:58] ppdet.engine INFO: Eval iter: 1800
- [11/21 21:49:02] ppdet.engine INFO: Eval iter: 1900
- [11/21 21:49:06] ppdet.engine INFO: Eval iter: 2000
- [11/21 21:49:10] ppdet.engine INFO: Eval iter: 2100
- [11/21 21:49:14] ppdet.engine INFO: Eval iter: 2200
- [11/21 21:49:18] ppdet.engine INFO: Eval iter: 2300
- [11/21 21:49:22] ppdet.engine INFO: Eval iter: 2400
- [11/21 21:49:26] ppdet.engine INFO: Eval iter: 2500
- [11/21 21:49:30] ppdet.engine INFO: Eval iter: 2600
- [11/21 21:49:34] ppdet.engine INFO: Eval iter: 2700
- [11/21 21:49:38] ppdet.engine INFO: Eval iter: 2800
- [11/21 21:49:42] ppdet.engine INFO: Eval iter: 2900
- [11/21 21:49:45] ppdet.engine INFO: Eval iter: 3000
- [11/21 21:49:49] ppdet.engine INFO: Eval iter: 3100
- [11/21 21:49:53] ppdet.engine INFO: Eval iter: 3200
- [11/21 21:49:57] ppdet.engine INFO: Eval iter: 3300
- [11/21 21:50:01] ppdet.engine INFO: Eval iter: 3400
- [11/21 21:50:05] ppdet.engine INFO: Eval iter: 3500
- [11/21 21:50:09] ppdet.engine INFO: Eval iter: 3600
- [11/21 21:50:12] ppdet.engine INFO: Eval iter: 3700
- [11/21 21:50:16] ppdet.engine INFO: Eval iter: 3800
- [11/21 21:50:20] ppdet.engine INFO: Eval iter: 3900
- [11/21 21:50:24] ppdet.engine INFO: Eval iter: 4000
- [11/21 21:50:28] ppdet.engine INFO: Eval iter: 4100
- [11/21 21:50:32] ppdet.engine INFO: Eval iter: 4200
- [11/21 21:50:36] ppdet.engine INFO: Eval iter: 4300
- [11/21 21:50:37] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 21:50:37] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 74.76%
- [11/21 21:50:37] ppdet.engine INFO: Total sample number: 4322, averge FPS: 25.288666265036856
- [11/21 21:50:37] ppdet.engine INFO: Best test bbox ap is 0.752.
- [11/21 21:50:37] ppdet.engine INFO: Epoch: [13] [ 0/1215] learning_rate: 0.005000 loss_xy: 3.164453 loss_wh: 0.792955 loss_obj: 4.430921 loss_cls: 1.392390 loss: 9.600431 eta: 5:49:24 batch_cost: 0.3605 data_cost: 0.0099 ips: 22.1885 images/s
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:35853']
- I1121 21:51:30.995002 65214 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1121 21:51:31.821166 65214 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1121 21:51:31.826800 65214 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/21 21:51:33] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- [11/21 21:51:33] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/21 21:51:34] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- [11/21 21:51:34] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- [11/21 21:51:36] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- [11/21 21:51:36] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- [11/21 21:51:37] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- [11/21 21:51:37] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- [11/21 21:51:37] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- [11/21 21:51:38] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- [11/21 21:51:38] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/21 21:51:39] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- [11/21 21:51:39] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- [11/21 21:51:40] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/21 21:51:40] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- [11/21 21:51:41] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- [11/21 21:51:43] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/21 21:51:44] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/12.pdparams
- [11/21 21:51:47] ppdet.engine INFO: Epoch: [13] [ 0/1215] learning_rate: 0.005000 loss_xy: 5.165677 loss_wh: 1.290764 loss_obj: 9.465256 loss_cls: 2.573740 loss: 18.495438 eta: 1 day, 11:49:49 batch_cost: 2.2588 data_cost: 0.0005 ips: 3.5417 images/s
- [11/21 21:52:59] ppdet.engine INFO: Epoch: [13] [ 200/1215] learning_rate: 0.003500 loss_xy: 2.950628 loss_wh: 0.703547 loss_obj: 4.139517 loss_cls: 1.211634 loss: 9.219433 eta: 5:45:54 batch_cost: 0.3552 data_cost: 0.0122 ips: 22.5195 images/s
- [11/21 21:54:13] ppdet.engine INFO: Epoch: [13] [ 400/1215] learning_rate: 0.003500 loss_xy: 2.983870 loss_wh: 0.702615 loss_obj: 3.941890 loss_cls: 1.191891 loss: 9.034084 eta: 5:46:42 batch_cost: 0.3690 data_cost: 0.0087 ips: 21.6807 images/s
- [11/21 21:55:27] ppdet.engine INFO: Epoch: [13] [ 600/1215] learning_rate: 0.003500 loss_xy: 2.987986 loss_wh: 0.720156 loss_obj: 3.905967 loss_cls: 1.114148 loss: 8.757767 eta: 5:46:17 batch_cost: 0.3694 data_cost: 0.0196 ips: 21.6560 images/s
- [11/21 21:56:39] ppdet.engine INFO: Epoch: [13] [ 800/1215] learning_rate: 0.003500 loss_xy: 2.977535 loss_wh: 0.700097 loss_obj: 3.555540 loss_cls: 1.069162 loss: 8.424931 eta: 5:43:38 batch_cost: 0.3617 data_cost: 0.0388 ips: 22.1187 images/s
- [11/21 21:57:51] ppdet.engine INFO: Epoch: [13] [1000/1215] learning_rate: 0.003500 loss_xy: 3.063575 loss_wh: 0.727884 loss_obj: 4.002009 loss_cls: 1.093647 loss: 9.039768 eta: 5:40:59 batch_cost: 0.3585 data_cost: 0.0251 ips: 22.3158 images/s
- [11/21 21:59:03] ppdet.engine INFO: Epoch: [13] [1200/1215] learning_rate: 0.003500 loss_xy: 2.938062 loss_wh: 0.715292 loss_obj: 3.787024 loss_cls: 1.043932 loss: 8.627810 eta: 5:38:46 batch_cost: 0.3582 data_cost: 0.0331 ips: 22.3323 images/s
- [11/21 21:59:10] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 21:59:10] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_2.xml, x1: 217.0, y1: 763.0, x2: 217.0, y2: 764.0.
- [11/21 21:59:11] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_8.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/21 21:59:11] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_881_3.xml, x1: 35.0, y1: 198.0, x2: 75.0, y2: 197.0.
- [11/21 21:59:11] ppdet.engine INFO: Eval iter: 0
- [11/21 21:59:15] ppdet.engine INFO: Eval iter: 100
- [11/21 21:59:19] ppdet.engine INFO: Eval iter: 200
- [11/21 21:59:23] ppdet.engine INFO: Eval iter: 300
- [11/21 21:59:27] ppdet.engine INFO: Eval iter: 400
- [11/21 21:59:33] ppdet.engine INFO: Eval iter: 500
- [11/21 21:59:37] ppdet.engine INFO: Eval iter: 600
- [11/21 21:59:41] ppdet.engine INFO: Eval iter: 700
- [11/21 21:59:45] ppdet.engine INFO: Eval iter: 800
- [11/21 21:59:49] ppdet.engine INFO: Eval iter: 900
- [11/21 21:59:53] ppdet.engine INFO: Eval iter: 1000
- [11/21 21:59:57] ppdet.engine INFO: Eval iter: 1100
- [11/21 22:00:01] ppdet.engine INFO: Eval iter: 1200
- [11/21 22:00:05] ppdet.engine INFO: Eval iter: 1300
- [11/21 22:00:09] ppdet.engine INFO: Eval iter: 1400
- [11/21 22:00:13] ppdet.engine INFO: Eval iter: 1500
- [11/21 22:00:17] ppdet.engine INFO: Eval iter: 1600
- [11/21 22:00:21] ppdet.engine INFO: Eval iter: 1700
- [11/21 22:00:25] ppdet.engine INFO: Eval iter: 1800
- [11/21 22:00:29] ppdet.engine INFO: Eval iter: 1900
- [11/21 22:00:33] ppdet.engine INFO: Eval iter: 2000
- [11/21 22:00:37] ppdet.engine INFO: Eval iter: 2100
- [11/21 22:00:41] ppdet.engine INFO: Eval iter: 2200
- [11/21 22:00:45] ppdet.engine INFO: Eval iter: 2300
- [11/21 22:00:49] ppdet.engine INFO: Eval iter: 2400
- [11/21 22:00:53] ppdet.engine INFO: Eval iter: 2500
- [11/21 22:00:57] ppdet.engine INFO: Eval iter: 2600
- [11/21 22:01:01] ppdet.engine INFO: Eval iter: 2700
- [11/21 22:01:05] ppdet.engine INFO: Eval iter: 2800
- [11/21 22:01:09] ppdet.engine INFO: Eval iter: 2900
- [11/21 22:01:13] ppdet.engine INFO: Eval iter: 3000
- [11/21 22:01:18] ppdet.engine INFO: Eval iter: 3100
- [11/21 22:01:22] ppdet.engine INFO: Eval iter: 3200
- [11/21 22:01:26] ppdet.engine INFO: Eval iter: 3300
- [11/21 22:01:30] ppdet.engine INFO: Eval iter: 3400
- [11/21 22:01:34] ppdet.engine INFO: Eval iter: 3500
- [11/21 22:01:38] ppdet.engine INFO: Eval iter: 3600
- [11/21 22:01:42] ppdet.engine INFO: Eval iter: 3700
- [11/21 22:01:46] ppdet.engine INFO: Eval iter: 3800
- [11/21 22:01:50] ppdet.engine INFO: Eval iter: 3900
- [11/21 22:01:54] ppdet.engine INFO: Eval iter: 4000
- [11/21 22:01:58] ppdet.engine INFO: Eval iter: 4100
- [11/21 22:02:02] ppdet.engine INFO: Eval iter: 4200
- [11/21 22:02:06] ppdet.engine INFO: Eval iter: 4300
- [11/21 22:02:07] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/21 22:02:07] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 80.78%
- [11/21 22:02:07] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.521041754635508
- [11/21 22:02:07] ppdet.engine INFO: Best test bbox ap is 0.808.
- [11/21 22:02:11] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/21 22:02:11] ppdet.engine INFO: Epoch: [14] [ 0/1215] learning_rate: 0.003500 loss_xy: 3.000990 loss_wh: 0.715292 loss_obj: 3.746094 loss_cls: 1.037532 loss: 8.641678 eta: 5:38:42 batch_cost: 0.3596 data_cost: 0.0336 ips: 22.2446 images/s
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::BasicEngine::Execute()
- 1 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVariableWrapperMap const&, paddle::imperative::NameVariableWrapperMap const&, paddle::framework::AttributeMap const&, paddle::framework::AttributeMap const&)
- 2 std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<phi::GPUPlace, false, 0ul, paddle::operators::SyncBatchNormGradKernel<paddle::platform::CUDADeviceContext, float>, paddle::operators::SyncBatchNormGradKernel<paddle::platform::CUDADeviceContext, double>, paddle::operators::SyncBatchNormGradKernel<paddle::platform::CUDADeviceContext, phi::dtype::float16> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&)
- 3 paddle::operators::SyncBatchNormGradKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const
- 4 void paddle::operators::SyncBatchNormGradFunctor<paddle::platform::CUDADeviceContext, float>(paddle::framework::ExecutionContext const&, paddle::experimental::DataLayout, phi::DenseTensor const*, phi::DenseTensor const*, phi::DenseTensor*, phi::DenseTensor const*, phi::DenseTensor*, phi::DenseTensor*, phi::DenseTensor const*, phi::DenseTensor const*, double)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669039373 (unix time) try "date -d @1669039373" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e80000fe6e) received by PID 65214 (TID 0x7f06a0cfa700) from PID 65134 ***]
-
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:40421', '127.0.0.1:37959', '127.0.0.1:44743']
- I1122 12:32:10.502146 824 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1122 12:32:11.153532 824 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1122 12:32:11.156893 824 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/22 12:32:12] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- [11/22 12:32:12] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/22 12:32:13] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- [11/22 12:32:13] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- [11/22 12:32:15] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- [11/22 12:32:16] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- [11/22 12:32:16] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- [11/22 12:32:16] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- [11/22 12:32:16] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- [11/22 12:32:17] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- [11/22 12:32:17] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/22 12:32:18] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- [11/22 12:32:18] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- [11/22 12:32:18] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/22 12:32:19] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- [11/22 12:32:19] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- [11/22 12:32:21] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/22 12:32:23] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/best_model.pdparams
- [11/22 12:32:26] ppdet.engine INFO: Epoch: [14] [ 0/1215] learning_rate: 0.003500 loss_xy: 4.870430 loss_wh: 1.452444 loss_obj: 6.999523 loss_cls: 1.177761 loss: 14.500158 eta: 1 day, 15:42:28 batch_cost: 2.5577 data_cost: 0.0005 ips: 3.1278 images/s
- [11/22 12:33:37] ppdet.engine INFO: Epoch: [14] [ 200/1215] learning_rate: 0.003500 loss_xy: 2.910850 loss_wh: 0.667256 loss_obj: 3.176493 loss_cls: 0.779023 loss: 7.795774 eta: 5:38:26 batch_cost: 0.3537 data_cost: 0.0120 ips: 22.6198 images/s
- [11/22 12:34:52] ppdet.engine INFO: Epoch: [14] [ 400/1215] learning_rate: 0.003500 loss_xy: 2.940148 loss_wh: 0.686749 loss_obj: 3.059648 loss_cls: 0.781628 loss: 7.488962 eta: 5:41:49 batch_cost: 0.3746 data_cost: 0.0111 ips: 21.3565 images/s
- [11/22 12:36:06] ppdet.engine INFO: Epoch: [14] [ 600/1215] learning_rate: 0.003500 loss_xy: 2.943156 loss_wh: 0.668943 loss_obj: 3.031779 loss_cls: 0.747034 loss: 7.453434 eta: 5:39:22 batch_cost: 0.3656 data_cost: 0.0253 ips: 21.8789 images/s
- [11/22 12:37:19] ppdet.engine INFO: Epoch: [14] [ 800/1215] learning_rate: 0.003500 loss_xy: 2.875080 loss_wh: 0.689071 loss_obj: 2.876212 loss_cls: 0.753771 loss: 7.215681 eta: 5:37:00 batch_cost: 0.3633 data_cost: 0.0164 ips: 22.0209 images/s
- [11/22 12:38:32] ppdet.engine INFO: Epoch: [14] [1000/1215] learning_rate: 0.003500 loss_xy: 3.050810 loss_wh: 0.724620 loss_obj: 3.094785 loss_cls: 0.777676 loss: 7.618496 eta: 5:35:10 batch_cost: 0.3637 data_cost: 0.0210 ips: 21.9967 images/s
- [11/22 12:39:44] ppdet.engine INFO: Epoch: [14] [1200/1215] learning_rate: 0.003500 loss_xy: 2.876873 loss_wh: 0.689410 loss_obj: 2.936512 loss_cls: 0.806421 loss: 7.378902 eta: 5:33:02 batch_cost: 0.3604 data_cost: 0.0164 ips: 22.1980 images/s
- [11/22 12:39:50] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 12:39:51] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_2.xml, x1: 217.0, y1: 763.0, x2: 217.0, y2: 764.0.
- [11/22 12:39:52] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_8.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/22 12:39:52] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_881_3.xml, x1: 35.0, y1: 198.0, x2: 75.0, y2: 197.0.
- [11/22 12:39:52] ppdet.engine INFO: Eval iter: 0
- [11/22 12:39:56] ppdet.engine INFO: Eval iter: 100
- [11/22 12:40:00] ppdet.engine INFO: Eval iter: 200
- [11/22 12:40:04] ppdet.engine INFO: Eval iter: 300
- [11/22 12:40:08] ppdet.engine INFO: Eval iter: 400
- [11/22 12:40:12] ppdet.engine INFO: Eval iter: 500
- [11/22 12:40:16] ppdet.engine INFO: Eval iter: 600
- [11/22 12:40:20] ppdet.engine INFO: Eval iter: 700
- [11/22 12:40:24] ppdet.engine INFO: Eval iter: 800
- [11/22 12:40:28] ppdet.engine INFO: Eval iter: 900
- [11/22 12:40:32] ppdet.engine INFO: Eval iter: 1000
- [11/22 12:40:36] ppdet.engine INFO: Eval iter: 1100
- [11/22 12:40:40] ppdet.engine INFO: Eval iter: 1200
- [11/22 12:40:44] ppdet.engine INFO: Eval iter: 1300
- [11/22 12:40:49] ppdet.engine INFO: Eval iter: 1400
- [11/22 12:40:53] ppdet.engine INFO: Eval iter: 1500
- [11/22 12:40:57] ppdet.engine INFO: Eval iter: 1600
- [11/22 12:41:02] ppdet.engine INFO: Eval iter: 1700
- [11/22 12:41:06] ppdet.engine INFO: Eval iter: 1800
- [11/22 12:41:10] ppdet.engine INFO: Eval iter: 1900
- [11/22 12:41:14] ppdet.engine INFO: Eval iter: 2000
- [11/22 12:41:19] ppdet.engine INFO: Eval iter: 2100
- [11/22 12:41:23] ppdet.engine INFO: Eval iter: 2200
- [11/22 12:41:27] ppdet.engine INFO: Eval iter: 2300
- [11/22 12:41:31] ppdet.engine INFO: Eval iter: 2400
- [11/22 12:41:36] ppdet.engine INFO: Eval iter: 2500
- [11/22 12:41:40] ppdet.engine INFO: Eval iter: 2600
- [11/22 12:41:44] ppdet.engine INFO: Eval iter: 2700
- [11/22 12:41:47] ppdet.engine INFO: Eval iter: 2800
- [11/22 12:41:51] ppdet.engine INFO: Eval iter: 2900
- [11/22 12:41:55] ppdet.engine INFO: Eval iter: 3000
- [11/22 12:41:59] ppdet.engine INFO: Eval iter: 3100
- [11/22 12:42:03] ppdet.engine INFO: Eval iter: 3200
- [11/22 12:42:07] ppdet.engine INFO: Eval iter: 3300
- [11/22 12:42:11] ppdet.engine INFO: Eval iter: 3400
- [11/22 12:42:15] ppdet.engine INFO: Eval iter: 3500
- [11/22 12:42:19] ppdet.engine INFO: Eval iter: 3600
- [11/22 12:42:23] ppdet.engine INFO: Eval iter: 3700
- [11/22 12:42:27] ppdet.engine INFO: Eval iter: 3800
- [11/22 12:42:31] ppdet.engine INFO: Eval iter: 3900
- [11/22 12:42:35] ppdet.engine INFO: Eval iter: 4000
- [11/22 12:42:39] ppdet.engine INFO: Eval iter: 4100
- [11/22 12:42:43] ppdet.engine INFO: Eval iter: 4200
- [11/22 12:42:47] ppdet.engine INFO: Eval iter: 4300
- [11/22 12:42:48] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 12:42:48] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 81.38%
- [11/22 12:42:48] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.505838628208725
- [11/22 12:42:48] ppdet.engine INFO: Best test bbox ap is 0.814.
- [11/22 12:42:52] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 12:42:52] ppdet.engine INFO: Epoch: [15] [ 0/1215] learning_rate: 0.003500 loss_xy: 2.936551 loss_wh: 0.689993 loss_obj: 2.942612 loss_cls: 0.810245 loss: 7.486867 eta: 5:33:05 batch_cost: 0.3612 data_cost: 0.0196 ips: 22.1499 images/s
- [11/22 12:44:05] ppdet.engine INFO: Epoch: [15] [ 200/1215] learning_rate: 0.003500 loss_xy: 2.902883 loss_wh: 0.689825 loss_obj: 3.608374 loss_cls: 0.982685 loss: 8.358641 eta: 5:31:23 batch_cost: 0.3618 data_cost: 0.0129 ips: 22.1128 images/s
- [11/22 12:45:19] ppdet.engine INFO: Epoch: [15] [ 400/1215] learning_rate: 0.003500 loss_xy: 3.001361 loss_wh: 0.691264 loss_obj: 3.653822 loss_cls: 1.013671 loss: 8.427697 eta: 5:30:47 batch_cost: 0.3705 data_cost: 0.0143 ips: 21.5923 images/s
- [11/22 12:46:33] ppdet.engine INFO: Epoch: [15] [ 600/1215] learning_rate: 0.003500 loss_xy: 3.129739 loss_wh: 0.739622 loss_obj: 3.835099 loss_cls: 1.068498 loss: 8.963745 eta: 5:29:45 batch_cost: 0.3676 data_cost: 0.0125 ips: 21.7601 images/s
- [11/22 12:47:47] ppdet.engine INFO: Epoch: [15] [ 800/1215] learning_rate: 0.003500 loss_xy: 3.102491 loss_wh: 0.684545 loss_obj: 3.927583 loss_cls: 1.024463 loss: 8.843790 eta: 5:28:49 batch_cost: 0.3690 data_cost: 0.0117 ips: 21.6818 images/s
- [11/22 12:49:00] ppdet.engine INFO: Epoch: [15] [1000/1215] learning_rate: 0.003500 loss_xy: 3.219447 loss_wh: 0.728584 loss_obj: 4.008489 loss_cls: 1.115929 loss: 8.953494 eta: 5:27:17 batch_cost: 0.3624 data_cost: 0.0343 ips: 22.0730 images/s
- [11/22 12:50:13] ppdet.engine INFO: Epoch: [15] [1200/1215] learning_rate: 0.003500 loss_xy: 3.013716 loss_wh: 0.657826 loss_obj: 3.670567 loss_cls: 1.012216 loss: 8.295240 eta: 5:25:47 batch_cost: 0.3621 data_cost: 0.0100 ips: 22.0918 images/s
- [11/22 12:50:19] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 12:50:19] ppdet.engine INFO: Eval iter: 0
- [11/22 12:50:23] ppdet.engine INFO: Eval iter: 100
- [11/22 12:50:27] ppdet.engine INFO: Eval iter: 200
- [11/22 12:50:31] ppdet.engine INFO: Eval iter: 300
- [11/22 12:50:35] ppdet.engine INFO: Eval iter: 400
- [11/22 12:50:39] ppdet.engine INFO: Eval iter: 500
- [11/22 12:50:43] ppdet.engine INFO: Eval iter: 600
- [11/22 12:50:47] ppdet.engine INFO: Eval iter: 700
- [11/22 12:50:51] ppdet.engine INFO: Eval iter: 800
- [11/22 12:50:55] ppdet.engine INFO: Eval iter: 900
- [11/22 12:50:59] ppdet.engine INFO: Eval iter: 1000
- [11/22 12:51:03] ppdet.engine INFO: Eval iter: 1100
- [11/22 12:51:07] ppdet.engine INFO: Eval iter: 1200
- [11/22 12:51:11] ppdet.engine INFO: Eval iter: 1300
- [11/22 12:51:15] ppdet.engine INFO: Eval iter: 1400
- [11/22 12:51:19] ppdet.engine INFO: Eval iter: 1500
- [11/22 12:51:23] ppdet.engine INFO: Eval iter: 1600
- [11/22 12:51:27] ppdet.engine INFO: Eval iter: 1700
- [11/22 12:51:31] ppdet.engine INFO: Eval iter: 1800
- [11/22 12:51:35] ppdet.engine INFO: Eval iter: 1900
- [11/22 12:51:39] ppdet.engine INFO: Eval iter: 2000
- [11/22 12:51:43] ppdet.engine INFO: Eval iter: 2100
- [11/22 12:51:47] ppdet.engine INFO: Eval iter: 2200
- [11/22 12:51:51] ppdet.engine INFO: Eval iter: 2300
- [11/22 12:51:55] ppdet.engine INFO: Eval iter: 2400
- [11/22 12:51:59] ppdet.engine INFO: Eval iter: 2500
- [11/22 12:52:03] ppdet.engine INFO: Eval iter: 2600
- [11/22 12:52:07] ppdet.engine INFO: Eval iter: 2700
- [11/22 12:52:11] ppdet.engine INFO: Eval iter: 2800
- [11/22 12:52:15] ppdet.engine INFO: Eval iter: 2900
- [11/22 12:52:19] ppdet.engine INFO: Eval iter: 3000
- [11/22 12:52:23] ppdet.engine INFO: Eval iter: 3100
- [11/22 12:52:27] ppdet.engine INFO: Eval iter: 3200
- [11/22 12:52:31] ppdet.engine INFO: Eval iter: 3300
- [11/22 12:52:35] ppdet.engine INFO: Eval iter: 3400
- [11/22 12:52:39] ppdet.engine INFO: Eval iter: 3500
- [11/22 12:52:43] ppdet.engine INFO: Eval iter: 3600
- [11/22 12:52:48] ppdet.engine INFO: Eval iter: 3700
- [11/22 12:52:52] ppdet.engine INFO: Eval iter: 3800
- [11/22 12:52:56] ppdet.engine INFO: Eval iter: 3900
- [11/22 12:53:00] ppdet.engine INFO: Eval iter: 4000
- [11/22 12:53:04] ppdet.engine INFO: Eval iter: 4100
- [11/22 12:53:08] ppdet.engine INFO: Eval iter: 4200
- [11/22 12:53:12] ppdet.engine INFO: Eval iter: 4300
- [11/22 12:53:13] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 12:53:13] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 82.16%
- [11/22 12:53:13] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.818711816469968
- [11/22 12:53:13] ppdet.engine INFO: Best test bbox ap is 0.822.
- [11/22 12:53:17] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 12:53:17] ppdet.engine INFO: Epoch: [16] [ 0/1215] learning_rate: 0.003500 loss_xy: 3.040336 loss_wh: 0.688699 loss_obj: 3.654066 loss_cls: 1.040305 loss: 8.295240 eta: 5:25:43 batch_cost: 0.3629 data_cost: 0.0107 ips: 22.0446 images/s
- [11/22 12:54:29] ppdet.engine INFO: Epoch: [16] [ 200/1215] learning_rate: 0.003500 loss_xy: 3.187690 loss_wh: 0.739485 loss_obj: 3.989048 loss_cls: 1.041618 loss: 9.300484 eta: 5:24:09 batch_cost: 0.3604 data_cost: 0.0276 ips: 22.1959 images/s
- [11/22 12:55:43] ppdet.engine INFO: Epoch: [16] [ 400/1215] learning_rate: 0.003500 loss_xy: 3.129346 loss_wh: 0.780476 loss_obj: 3.804833 loss_cls: 0.919589 loss: 8.683457 eta: 5:23:05 batch_cost: 0.3678 data_cost: 0.0027 ips: 21.7535 images/s
- [11/22 12:56:55] ppdet.engine INFO: Epoch: [16] [ 600/1215] learning_rate: 0.003500 loss_xy: 3.129284 loss_wh: 0.700736 loss_obj: 3.580670 loss_cls: 1.043138 loss: 8.514053 eta: 5:21:19 batch_cost: 0.3558 data_cost: 0.0191 ips: 22.4816 images/s
- [11/22 12:58:07] ppdet.engine INFO: Epoch: [16] [ 800/1215] learning_rate: 0.003500 loss_xy: 2.933323 loss_wh: 0.666489 loss_obj: 3.508916 loss_cls: 0.981952 loss: 8.113819 eta: 5:19:44 batch_cost: 0.3580 data_cost: 0.0067 ips: 22.3463 images/s
- [11/22 12:59:21] ppdet.engine INFO: Epoch: [16] [1000/1215] learning_rate: 0.003500 loss_xy: 3.182017 loss_wh: 0.741631 loss_obj: 3.933079 loss_cls: 1.024101 loss: 9.111794 eta: 5:18:50 batch_cost: 0.3705 data_cost: 0.0168 ips: 21.5936 images/s
- [11/22 13:00:34] ppdet.engine INFO: Epoch: [16] [1200/1215] learning_rate: 0.003500 loss_xy: 3.045088 loss_wh: 0.731767 loss_obj: 3.702673 loss_cls: 1.107288 loss: 8.740210 eta: 5:17:35 batch_cost: 0.3640 data_cost: 0.0331 ips: 21.9782 images/s
- [11/22 13:00:40] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 13:00:40] ppdet.engine INFO: Eval iter: 0
- [11/22 13:00:44] ppdet.engine INFO: Eval iter: 100
- [11/22 13:00:49] ppdet.engine INFO: Eval iter: 200
- [11/22 13:00:53] ppdet.engine INFO: Eval iter: 300
- [11/22 13:00:57] ppdet.engine INFO: Eval iter: 400
- [11/22 13:01:01] ppdet.engine INFO: Eval iter: 500
- [11/22 13:01:05] ppdet.engine INFO: Eval iter: 600
- [11/22 13:01:09] ppdet.engine INFO: Eval iter: 700
- [11/22 13:01:13] ppdet.engine INFO: Eval iter: 800
- [11/22 13:01:17] ppdet.engine INFO: Eval iter: 900
- [11/22 13:01:21] ppdet.engine INFO: Eval iter: 1000
- [11/22 13:01:25] ppdet.engine INFO: Eval iter: 1100
- [11/22 13:01:29] ppdet.engine INFO: Eval iter: 1200
- [11/22 13:01:33] ppdet.engine INFO: Eval iter: 1300
- [11/22 13:01:37] ppdet.engine INFO: Eval iter: 1400
- [11/22 13:01:41] ppdet.engine INFO: Eval iter: 1500
- [11/22 13:01:45] ppdet.engine INFO: Eval iter: 1600
- [11/22 13:01:49] ppdet.engine INFO: Eval iter: 1700
- [11/22 13:01:54] ppdet.engine INFO: Eval iter: 1800
- [11/22 13:01:58] ppdet.engine INFO: Eval iter: 1900
- [11/22 13:02:02] ppdet.engine INFO: Eval iter: 2000
- [11/22 13:02:06] ppdet.engine INFO: Eval iter: 2100
- [11/22 13:02:10] ppdet.engine INFO: Eval iter: 2200
- [11/22 13:02:14] ppdet.engine INFO: Eval iter: 2300
- [11/22 13:02:18] ppdet.engine INFO: Eval iter: 2400
- [11/22 13:02:22] ppdet.engine INFO: Eval iter: 2500
- [11/22 13:02:26] ppdet.engine INFO: Eval iter: 2600
- [11/22 13:02:30] ppdet.engine INFO: Eval iter: 2700
- [11/22 13:02:34] ppdet.engine INFO: Eval iter: 2800
- [11/22 13:02:38] ppdet.engine INFO: Eval iter: 2900
- [11/22 13:02:43] ppdet.engine INFO: Eval iter: 3000
- [11/22 13:02:47] ppdet.engine INFO: Eval iter: 3100
- [11/22 13:02:51] ppdet.engine INFO: Eval iter: 3200
- [11/22 13:02:55] ppdet.engine INFO: Eval iter: 3300
- [11/22 13:02:59] ppdet.engine INFO: Eval iter: 3400
- [11/22 13:03:03] ppdet.engine INFO: Eval iter: 3500
- [11/22 13:03:07] ppdet.engine INFO: Eval iter: 3600
- [11/22 13:03:11] ppdet.engine INFO: Eval iter: 3700
- [11/22 13:03:15] ppdet.engine INFO: Eval iter: 3800
- [11/22 13:03:19] ppdet.engine INFO: Eval iter: 3900
- [11/22 13:03:23] ppdet.engine INFO: Eval iter: 4000
- [11/22 13:03:27] ppdet.engine INFO: Eval iter: 4100
- [11/22 13:03:31] ppdet.engine INFO: Eval iter: 4200
- [11/22 13:03:35] ppdet.engine INFO: Eval iter: 4300
- [11/22 13:03:36] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 13:03:36] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 82.04%
- [11/22 13:03:36] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.535694313436988
- [11/22 13:03:36] ppdet.engine INFO: Best test bbox ap is 0.822.
- [11/22 13:03:37] ppdet.engine INFO: Epoch: [17] [ 0/1215] learning_rate: 0.003500 loss_xy: 3.023299 loss_wh: 0.713594 loss_obj: 3.694011 loss_cls: 1.109199 loss: 8.584542 eta: 5:17:30 batch_cost: 0.3613 data_cost: 0.0253 ips: 22.1448 images/s
- [11/22 13:04:49] ppdet.engine INFO: Epoch: [17] [ 200/1215] learning_rate: 0.003500 loss_xy: 2.924578 loss_wh: 0.675483 loss_obj: 3.388441 loss_cls: 0.958851 loss: 8.160095 eta: 5:16:06 batch_cost: 0.3604 data_cost: 0.0158 ips: 22.1958 images/s
- [11/22 13:06:03] ppdet.engine INFO: Epoch: [17] [ 400/1215] learning_rate: 0.003500 loss_xy: 3.056283 loss_wh: 0.758128 loss_obj: 3.849988 loss_cls: 0.962615 loss: 8.748558 eta: 5:14:57 batch_cost: 0.3661 data_cost: 0.0238 ips: 21.8530 images/s
- [11/22 13:07:15] ppdet.engine INFO: Epoch: [17] [ 600/1215] learning_rate: 0.003500 loss_xy: 2.927267 loss_wh: 0.653851 loss_obj: 3.341533 loss_cls: 1.012116 loss: 8.098165 eta: 5:13:32 batch_cost: 0.3596 data_cost: 0.0169 ips: 22.2460 images/s
- [11/22 13:08:27] ppdet.engine INFO: Epoch: [17] [ 800/1215] learning_rate: 0.003500 loss_xy: 3.026126 loss_wh: 0.683576 loss_obj: 3.607949 loss_cls: 0.943609 loss: 8.441807 eta: 5:12:03 batch_cost: 0.3573 data_cost: 0.0112 ips: 22.3886 images/s
- [11/22 13:09:39] ppdet.engine INFO: Epoch: [17] [1000/1215] learning_rate: 0.003500 loss_xy: 3.124389 loss_wh: 0.715599 loss_obj: 3.603878 loss_cls: 1.031467 loss: 8.448759 eta: 5:10:49 batch_cost: 0.3632 data_cost: 0.0279 ips: 22.0283 images/s
- [11/22 13:10:51] ppdet.engine INFO: Epoch: [17] [1200/1215] learning_rate: 0.003500 loss_xy: 3.146523 loss_wh: 0.753553 loss_obj: 3.641109 loss_cls: 0.986030 loss: 8.627457 eta: 5:09:22 batch_cost: 0.3574 data_cost: 0.0148 ips: 22.3835 images/s
- [11/22 13:10:58] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 13:10:58] ppdet.engine INFO: Eval iter: 0
- [11/22 13:11:02] ppdet.engine INFO: Eval iter: 100
- [11/22 13:11:06] ppdet.engine INFO: Eval iter: 200
- [11/22 13:11:10] ppdet.engine INFO: Eval iter: 300
- [11/22 13:11:14] ppdet.engine INFO: Eval iter: 400
- [11/22 13:11:18] ppdet.engine INFO: Eval iter: 500
- [11/22 13:11:23] ppdet.engine INFO: Eval iter: 600
- [11/22 13:11:27] ppdet.engine INFO: Eval iter: 700
- [11/22 13:11:31] ppdet.engine INFO: Eval iter: 800
- [11/22 13:11:35] ppdet.engine INFO: Eval iter: 900
- [11/22 13:11:40] ppdet.engine INFO: Eval iter: 1000
- [11/22 13:11:44] ppdet.engine INFO: Eval iter: 1100
- [11/22 13:11:48] ppdet.engine INFO: Eval iter: 1200
- [11/22 13:11:52] ppdet.engine INFO: Eval iter: 1300
- [11/22 13:11:56] ppdet.engine INFO: Eval iter: 1400
- [11/22 13:12:00] ppdet.engine INFO: Eval iter: 1500
- [11/22 13:12:04] ppdet.engine INFO: Eval iter: 1600
- [11/22 13:12:08] ppdet.engine INFO: Eval iter: 1700
- [11/22 13:12:12] ppdet.engine INFO: Eval iter: 1800
- [11/22 13:12:17] ppdet.engine INFO: Eval iter: 1900
- [11/22 13:12:21] ppdet.engine INFO: Eval iter: 2000
- [11/22 13:12:25] ppdet.engine INFO: Eval iter: 2100
- [11/22 13:12:29] ppdet.engine INFO: Eval iter: 2200
- [11/22 13:12:33] ppdet.engine INFO: Eval iter: 2300
- [11/22 13:12:37] ppdet.engine INFO: Eval iter: 2400
- [11/22 13:12:41] ppdet.engine INFO: Eval iter: 2500
- [11/22 13:12:45] ppdet.engine INFO: Eval iter: 2600
- [11/22 13:12:49] ppdet.engine INFO: Eval iter: 2700
- [11/22 13:12:54] ppdet.engine INFO: Eval iter: 2800
- [11/22 13:12:58] ppdet.engine INFO: Eval iter: 2900
- [11/22 13:13:02] ppdet.engine INFO: Eval iter: 3000
- [11/22 13:13:06] ppdet.engine INFO: Eval iter: 3100
- [11/22 13:13:10] ppdet.engine INFO: Eval iter: 3200
- [11/22 13:13:14] ppdet.engine INFO: Eval iter: 3300
- [11/22 13:13:18] ppdet.engine INFO: Eval iter: 3400
- [11/22 13:13:22] ppdet.engine INFO: Eval iter: 3500
- [11/22 13:13:26] ppdet.engine INFO: Eval iter: 3600
- [11/22 13:13:30] ppdet.engine INFO: Eval iter: 3700
- [11/22 13:13:35] ppdet.engine INFO: Eval iter: 3800
- [11/22 13:13:39] ppdet.engine INFO: Eval iter: 3900
- [11/22 13:13:43] ppdet.engine INFO: Eval iter: 4000
- [11/22 13:13:47] ppdet.engine INFO: Eval iter: 4100
- [11/22 13:13:51] ppdet.engine INFO: Eval iter: 4200
- [11/22 13:13:55] ppdet.engine INFO: Eval iter: 4300
- [11/22 13:13:56] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 13:13:56] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 83.16%
- [11/22 13:13:56] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.264586277461518
- [11/22 13:13:56] ppdet.engine INFO: Best test bbox ap is 0.832.
- [11/22 13:14:00] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 13:14:00] ppdet.engine INFO: Epoch: [18] [ 0/1215] learning_rate: 0.003500 loss_xy: 3.147645 loss_wh: 0.755351 loss_obj: 3.693794 loss_cls: 0.989430 loss: 8.745901 eta: 5:09:19 batch_cost: 0.3590 data_cost: 0.0121 ips: 22.2846 images/s
- [11/22 13:15:12] ppdet.engine INFO: Epoch: [18] [ 200/1215] learning_rate: 0.003500 loss_xy: 3.109331 loss_wh: 0.709688 loss_obj: 3.474004 loss_cls: 0.891882 loss: 8.235905 eta: 5:07:55 batch_cost: 0.3579 data_cost: 0.0457 ips: 22.3519 images/s
- [11/22 13:16:26] ppdet.engine INFO: Epoch: [18] [ 400/1215] learning_rate: 0.003500 loss_xy: 2.872918 loss_wh: 0.650949 loss_obj: 3.154339 loss_cls: 0.883507 loss: 7.551907 eta: 5:06:49 batch_cost: 0.3673 data_cost: 0.0301 ips: 21.7826 images/s
- [11/22 13:17:37] ppdet.engine INFO: Epoch: [18] [ 600/1215] learning_rate: 0.003500 loss_xy: 3.052910 loss_wh: 0.731001 loss_obj: 3.722553 loss_cls: 0.960788 loss: 8.588685 eta: 5:05:19 batch_cost: 0.3539 data_cost: 0.0257 ips: 22.6049 images/s
- [11/22 13:18:51] ppdet.engine INFO: Epoch: [18] [ 800/1215] learning_rate: 0.003500 loss_xy: 2.943198 loss_wh: 0.687072 loss_obj: 3.587214 loss_cls: 0.908008 loss: 8.168530 eta: 5:04:17 batch_cost: 0.3696 data_cost: 0.0333 ips: 21.6453 images/s
- [11/22 13:20:04] ppdet.engine INFO: Epoch: [18] [1000/1215] learning_rate: 0.003500 loss_xy: 2.848819 loss_wh: 0.655689 loss_obj: 3.299960 loss_cls: 0.912188 loss: 7.728662 eta: 5:03:04 batch_cost: 0.3630 data_cost: 0.0121 ips: 22.0368 images/s
- [11/22 13:21:16] ppdet.engine INFO: Epoch: [18] [1200/1215] learning_rate: 0.003500 loss_xy: 3.177651 loss_wh: 0.756071 loss_obj: 3.616183 loss_cls: 0.970536 loss: 8.832527 eta: 5:01:44 batch_cost: 0.3595 data_cost: 0.0282 ips: 22.2558 images/s
- [11/22 13:21:22] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 13:21:22] ppdet.engine INFO: Eval iter: 0
- [11/22 13:21:26] ppdet.engine INFO: Eval iter: 100
- [11/22 13:21:30] ppdet.engine INFO: Eval iter: 200
- [11/22 13:21:34] ppdet.engine INFO: Eval iter: 300
- [11/22 13:21:38] ppdet.engine INFO: Eval iter: 400
- [11/22 13:21:42] ppdet.engine INFO: Eval iter: 500
- [11/22 13:21:46] ppdet.engine INFO: Eval iter: 600
- [11/22 13:21:50] ppdet.engine INFO: Eval iter: 700
- [11/22 13:21:54] ppdet.engine INFO: Eval iter: 800
- [11/22 13:21:58] ppdet.engine INFO: Eval iter: 900
- [11/22 13:22:02] ppdet.engine INFO: Eval iter: 1000
- [11/22 13:22:06] ppdet.engine INFO: Eval iter: 1100
- [11/22 13:22:10] ppdet.engine INFO: Eval iter: 1200
- [11/22 13:22:15] ppdet.engine INFO: Eval iter: 1300
- [11/22 13:22:19] ppdet.engine INFO: Eval iter: 1400
- [11/22 13:22:23] ppdet.engine INFO: Eval iter: 1500
- [11/22 13:22:27] ppdet.engine INFO: Eval iter: 1600
- [11/22 13:22:31] ppdet.engine INFO: Eval iter: 1700
- [11/22 13:22:35] ppdet.engine INFO: Eval iter: 1800
- [11/22 13:22:40] ppdet.engine INFO: Eval iter: 1900
- [11/22 13:22:45] ppdet.engine INFO: Eval iter: 2000
- [11/22 13:22:49] ppdet.engine INFO: Eval iter: 2100
- [11/22 13:22:53] ppdet.engine INFO: Eval iter: 2200
- [11/22 13:22:57] ppdet.engine INFO: Eval iter: 2300
- [11/22 13:23:01] ppdet.engine INFO: Eval iter: 2400
- [11/22 13:23:05] ppdet.engine INFO: Eval iter: 2500
- [11/22 13:23:09] ppdet.engine INFO: Eval iter: 2600
- [11/22 13:23:13] ppdet.engine INFO: Eval iter: 2700
- [11/22 13:23:17] ppdet.engine INFO: Eval iter: 2800
- [11/22 13:23:21] ppdet.engine INFO: Eval iter: 2900
- [11/22 13:23:25] ppdet.engine INFO: Eval iter: 3000
- [11/22 13:23:29] ppdet.engine INFO: Eval iter: 3100
- [11/22 13:23:33] ppdet.engine INFO: Eval iter: 3200
- [11/22 13:23:38] ppdet.engine INFO: Eval iter: 3300
- [11/22 13:23:42] ppdet.engine INFO: Eval iter: 3400
- [11/22 13:23:46] ppdet.engine INFO: Eval iter: 3500
- [11/22 13:23:50] ppdet.engine INFO: Eval iter: 3600
- [11/22 13:23:54] ppdet.engine INFO: Eval iter: 3700
- [11/22 13:23:58] ppdet.engine INFO: Eval iter: 3800
- [11/22 13:24:02] ppdet.engine INFO: Eval iter: 3900
- [11/22 13:24:06] ppdet.engine INFO: Eval iter: 4000
- [11/22 13:24:10] ppdet.engine INFO: Eval iter: 4100
- [11/22 13:24:14] ppdet.engine INFO: Eval iter: 4200
- [11/22 13:24:18] ppdet.engine INFO: Eval iter: 4300
- [11/22 13:24:19] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 13:24:19] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 82.07%
- [11/22 13:24:19] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.33417371633038
- [11/22 13:24:19] ppdet.engine INFO: Best test bbox ap is 0.832.
- [11/22 13:24:20] ppdet.engine INFO: Epoch: [19] [ 0/1215] learning_rate: 0.003500 loss_xy: 3.184964 loss_wh: 0.751376 loss_obj: 3.651384 loss_cls: 0.955303 loss: 8.810658 eta: 5:01:35 batch_cost: 0.3584 data_cost: 0.0309 ips: 22.3218 images/s
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::BasicEngine::Execute()
- 1 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVariableWrapperMap const&, paddle::imperative::NameVariableWrapperMap const&, paddle::framework::AttributeMap const&, paddle::framework::AttributeMap const&)
- 2 phi::KernelImpl<void (*)(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*), &(void phi::ConvCudnnGradKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*))>::Compute(phi::KernelContext*)
- 3 void phi::ConvCudnnGradKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669094669 (unix time) try "date -d @1669094669" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e8000002e6) received by PID 824 (TID 0x7f8cfcc88700) from PID 742 ***]
-
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:58769', '127.0.0.1:53067', '127.0.0.1:39803']
- I1122 13:25:56.588228 30894 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1122 13:25:57.255868 30894 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1122 13:25:57.258916 30894 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/22 13:25:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- [11/22 13:25:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/22 13:25:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- [11/22 13:25:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- [11/22 13:26:01] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- [11/22 13:26:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- [11/22 13:26:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- [11/22 13:26:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- [11/22 13:26:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- [11/22 13:26:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- [11/22 13:26:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/22 13:26:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- [11/22 13:26:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- [11/22 13:26:05] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/22 13:26:05] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- [11/22 13:26:06] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- [11/22 13:26:08] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/22 13:26:09] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/18.pdparams
- [11/22 13:26:12] ppdet.engine INFO: Epoch: [19] [ 0/1215] learning_rate: 0.003500 loss_xy: 4.867394 loss_wh: 1.533396 loss_obj: 6.615178 loss_cls: 0.956380 loss: 13.972348 eta: 1 day, 9:03:07 batch_cost: 2.3886 data_cost: 0.0005 ips: 3.3493 images/s
- [11/22 13:27:24] ppdet.engine INFO: Epoch: [19] [ 200/1215] learning_rate: 0.002500 loss_xy: 2.886417 loss_wh: 0.629142 loss_obj: 3.006885 loss_cls: 0.790217 loss: 7.596091 eta: 5:03:51 batch_cost: 0.3574 data_cost: 0.0106 ips: 22.3867 images/s
- [11/22 13:28:39] ppdet.engine INFO: Epoch: [19] [ 400/1215] learning_rate: 0.002500 loss_xy: 2.900329 loss_wh: 0.631821 loss_obj: 2.881810 loss_cls: 0.760697 loss: 7.355360 eta: 5:05:27 batch_cost: 0.3743 data_cost: 0.0112 ips: 21.3720 images/s
- [11/22 13:29:52] ppdet.engine INFO: Epoch: [19] [ 600/1215] learning_rate: 0.002500 loss_xy: 2.925161 loss_wh: 0.643954 loss_obj: 2.768742 loss_cls: 0.673384 loss: 7.126351 eta: 5:02:47 batch_cost: 0.3657 data_cost: 0.0170 ips: 21.8779 images/s
- [11/22 13:31:05] ppdet.engine INFO: Epoch: [19] [ 800/1215] learning_rate: 0.002500 loss_xy: 2.892232 loss_wh: 0.604385 loss_obj: 2.599072 loss_cls: 0.652203 loss: 6.764188 eta: 5:00:24 batch_cost: 0.3635 data_cost: 0.0180 ips: 22.0089 images/s
- [11/22 13:32:17] ppdet.engine INFO: Epoch: [19] [1000/1215] learning_rate: 0.002500 loss_xy: 3.084985 loss_wh: 0.648046 loss_obj: 2.855505 loss_cls: 0.671061 loss: 7.254824 eta: 4:57:47 batch_cost: 0.3592 data_cost: 0.0158 ips: 22.2737 images/s
- [11/22 13:33:30] ppdet.engine INFO: Epoch: [19] [1200/1215] learning_rate: 0.002500 loss_xy: 2.885878 loss_wh: 0.621406 loss_obj: 2.623840 loss_cls: 0.717046 loss: 6.846200 eta: 4:56:15 batch_cost: 0.3637 data_cost: 0.0132 ips: 21.9946 images/s
- [11/22 13:33:37] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 13:33:37] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_2.xml, x1: 217.0, y1: 763.0, x2: 217.0, y2: 764.0.
- [11/22 13:33:38] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_8.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/22 13:33:38] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_881_3.xml, x1: 35.0, y1: 198.0, x2: 75.0, y2: 197.0.
- [11/22 13:33:38] ppdet.engine INFO: Eval iter: 0
- [11/22 13:33:42] ppdet.engine INFO: Eval iter: 100
- [11/22 13:33:46] ppdet.engine INFO: Eval iter: 200
- [11/22 13:33:50] ppdet.engine INFO: Eval iter: 300
- [11/22 13:33:54] ppdet.engine INFO: Eval iter: 400
- [11/22 13:33:58] ppdet.engine INFO: Eval iter: 500
- [11/22 13:34:02] ppdet.engine INFO: Eval iter: 600
- [11/22 13:34:06] ppdet.engine INFO: Eval iter: 700
- [11/22 13:34:10] ppdet.engine INFO: Eval iter: 800
- [11/22 13:34:14] ppdet.engine INFO: Eval iter: 900
- [11/22 13:34:18] ppdet.engine INFO: Eval iter: 1000
- [11/22 13:34:22] ppdet.engine INFO: Eval iter: 1100
- [11/22 13:34:26] ppdet.engine INFO: Eval iter: 1200
- [11/22 13:34:29] ppdet.engine INFO: Eval iter: 1300
- [11/22 13:34:33] ppdet.engine INFO: Eval iter: 1400
- [11/22 13:34:37] ppdet.engine INFO: Eval iter: 1500
- [11/22 13:34:41] ppdet.engine INFO: Eval iter: 1600
- [11/22 13:34:45] ppdet.engine INFO: Eval iter: 1700
- [11/22 13:34:49] ppdet.engine INFO: Eval iter: 1800
- [11/22 13:34:53] ppdet.engine INFO: Eval iter: 1900
- [11/22 13:34:57] ppdet.engine INFO: Eval iter: 2000
- [11/22 13:35:01] ppdet.engine INFO: Eval iter: 2100
- [11/22 13:35:05] ppdet.engine INFO: Eval iter: 2200
- [11/22 13:35:10] ppdet.engine INFO: Eval iter: 2300
- [11/22 13:35:14] ppdet.engine INFO: Eval iter: 2400
- [11/22 13:35:18] ppdet.engine INFO: Eval iter: 2500
- [11/22 13:35:23] ppdet.engine INFO: Eval iter: 2600
- [11/22 13:35:27] ppdet.engine INFO: Eval iter: 2700
- [11/22 13:35:31] ppdet.engine INFO: Eval iter: 2800
- [11/22 13:35:35] ppdet.engine INFO: Eval iter: 2900
- [11/22 13:35:40] ppdet.engine INFO: Eval iter: 3000
- [11/22 13:35:44] ppdet.engine INFO: Eval iter: 3100
- [11/22 13:35:48] ppdet.engine INFO: Eval iter: 3200
- [11/22 13:35:52] ppdet.engine INFO: Eval iter: 3300
- [11/22 13:35:57] ppdet.engine INFO: Eval iter: 3400
- [11/22 13:36:01] ppdet.engine INFO: Eval iter: 3500
- [11/22 13:36:05] ppdet.engine INFO: Eval iter: 3600
- [11/22 13:36:10] ppdet.engine INFO: Eval iter: 3700
- [11/22 13:36:14] ppdet.engine INFO: Eval iter: 3800
- [11/22 13:36:18] ppdet.engine INFO: Eval iter: 3900
- [11/22 13:36:22] ppdet.engine INFO: Eval iter: 4000
- [11/22 13:36:26] ppdet.engine INFO: Eval iter: 4100
- [11/22 13:36:30] ppdet.engine INFO: Eval iter: 4200
- [11/22 13:36:34] ppdet.engine INFO: Eval iter: 4300
- [11/22 13:36:35] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 13:36:35] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 83.95%
- [11/22 13:36:35] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.4412829988374
- [11/22 13:36:35] ppdet.engine INFO: Best test bbox ap is 0.839.
- [11/22 13:36:39] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 13:36:39] ppdet.engine INFO: Epoch: [20] [ 0/1215] learning_rate: 0.002500 loss_xy: 2.934909 loss_wh: 0.630300 loss_obj: 2.623840 loss_cls: 0.717046 loss: 6.864637 eta: 4:56:10 batch_cost: 0.3650 data_cost: 0.0161 ips: 21.9161 images/s
- [11/22 13:37:52] ppdet.engine INFO: Epoch: [20] [ 200/1215] learning_rate: 0.002500 loss_xy: 2.862699 loss_wh: 0.628425 loss_obj: 2.685523 loss_cls: 0.704978 loss: 7.044815 eta: 4:54:30 batch_cost: 0.3616 data_cost: 0.0196 ips: 22.1215 images/s
- [11/22 13:39:07] ppdet.engine INFO: Epoch: [20] [ 400/1215] learning_rate: 0.002500 loss_xy: 2.931030 loss_wh: 0.619112 loss_obj: 2.864744 loss_cls: 0.707356 loss: 7.223612 eta: 4:54:20 batch_cost: 0.3757 data_cost: 0.0179 ips: 21.2963 images/s
- [11/22 13:40:21] ppdet.engine INFO: Epoch: [20] [ 600/1215] learning_rate: 0.002500 loss_xy: 3.090825 loss_wh: 0.653836 loss_obj: 2.976973 loss_cls: 0.795417 loss: 7.754862 eta: 4:53:19 batch_cost: 0.3687 data_cost: 0.0159 ips: 21.6987 images/s
- [11/22 13:41:34] ppdet.engine INFO: Epoch: [20] [ 800/1215] learning_rate: 0.002500 loss_xy: 3.069372 loss_wh: 0.639603 loss_obj: 3.017100 loss_cls: 0.735152 loss: 7.514993 eta: 4:51:57 batch_cost: 0.3650 data_cost: 0.0152 ips: 21.9202 images/s
- [11/22 13:42:47] ppdet.engine INFO: Epoch: [20] [1000/1215] learning_rate: 0.002500 loss_xy: 3.143850 loss_wh: 0.663826 loss_obj: 3.053492 loss_cls: 0.806362 loss: 7.890793 eta: 4:50:16 batch_cost: 0.3600 data_cost: 0.0306 ips: 22.2248 images/s
- [11/22 13:44:00] ppdet.engine INFO: Epoch: [20] [1200/1215] learning_rate: 0.002500 loss_xy: 2.966608 loss_wh: 0.638135 loss_obj: 2.906694 loss_cls: 0.764046 loss: 7.183378 eta: 4:48:56 batch_cost: 0.3642 data_cost: 0.0154 ips: 21.9668 images/s
- [11/22 13:44:06] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 13:44:06] ppdet.engine INFO: Eval iter: 0
- [11/22 13:44:10] ppdet.engine INFO: Eval iter: 100
- [11/22 13:44:14] ppdet.engine INFO: Eval iter: 200
- [11/22 13:44:18] ppdet.engine INFO: Eval iter: 300
- [11/22 13:44:22] ppdet.engine INFO: Eval iter: 400
- [11/22 13:44:26] ppdet.engine INFO: Eval iter: 500
- [11/22 13:44:31] ppdet.engine INFO: Eval iter: 600
- [11/22 13:44:35] ppdet.engine INFO: Eval iter: 700
- [11/22 13:44:39] ppdet.engine INFO: Eval iter: 800
- [11/22 13:44:43] ppdet.engine INFO: Eval iter: 900
- [11/22 13:44:47] ppdet.engine INFO: Eval iter: 1000
- [11/22 13:44:51] ppdet.engine INFO: Eval iter: 1100
- [11/22 13:44:55] ppdet.engine INFO: Eval iter: 1200
- [11/22 13:44:59] ppdet.engine INFO: Eval iter: 1300
- [11/22 13:45:03] ppdet.engine INFO: Eval iter: 1400
- [11/22 13:45:07] ppdet.engine INFO: Eval iter: 1500
- [11/22 13:45:11] ppdet.engine INFO: Eval iter: 1600
- [11/22 13:45:15] ppdet.engine INFO: Eval iter: 1700
- [11/22 13:45:19] ppdet.engine INFO: Eval iter: 1800
- [11/22 13:45:23] ppdet.engine INFO: Eval iter: 1900
- [11/22 13:45:27] ppdet.engine INFO: Eval iter: 2000
- [11/22 13:45:31] ppdet.engine INFO: Eval iter: 2100
- [11/22 13:45:35] ppdet.engine INFO: Eval iter: 2200
- [11/22 13:45:39] ppdet.engine INFO: Eval iter: 2300
- [11/22 13:45:43] ppdet.engine INFO: Eval iter: 2400
- [11/22 13:45:47] ppdet.engine INFO: Eval iter: 2500
- [11/22 13:45:51] ppdet.engine INFO: Eval iter: 2600
- [11/22 13:45:55] ppdet.engine INFO: Eval iter: 2700
- [11/22 13:45:59] ppdet.engine INFO: Eval iter: 2800
- [11/22 13:46:03] ppdet.engine INFO: Eval iter: 2900
- [11/22 13:46:07] ppdet.engine INFO: Eval iter: 3000
- [11/22 13:46:11] ppdet.engine INFO: Eval iter: 3100
- [11/22 13:46:15] ppdet.engine INFO: Eval iter: 3200
- [11/22 13:46:19] ppdet.engine INFO: Eval iter: 3300
- [11/22 13:46:23] ppdet.engine INFO: Eval iter: 3400
- [11/22 13:46:27] ppdet.engine INFO: Eval iter: 3500
- [11/22 13:46:31] ppdet.engine INFO: Eval iter: 3600
- [11/22 13:46:35] ppdet.engine INFO: Eval iter: 3700
- [11/22 13:46:39] ppdet.engine INFO: Eval iter: 3800
- [11/22 13:46:43] ppdet.engine INFO: Eval iter: 3900
- [11/22 13:46:47] ppdet.engine INFO: Eval iter: 4000
- [11/22 13:46:51] ppdet.engine INFO: Eval iter: 4100
- [11/22 13:46:55] ppdet.engine INFO: Eval iter: 4200
- [11/22 13:46:59] ppdet.engine INFO: Eval iter: 4300
- [11/22 13:47:00] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 13:47:00] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 83.75%
- [11/22 13:47:00] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.803246119561873
- [11/22 13:47:00] ppdet.engine INFO: Best test bbox ap is 0.839.
- [11/22 13:47:00] ppdet.engine INFO: Epoch: [21] [ 0/1215] learning_rate: 0.002500 loss_xy: 3.036470 loss_wh: 0.652121 loss_obj: 2.790349 loss_cls: 0.749184 loss: 7.183378 eta: 4:48:45 batch_cost: 0.3624 data_cost: 0.0155 ips: 22.0742 images/s
- [11/22 13:48:13] ppdet.engine INFO: Epoch: [21] [ 200/1215] learning_rate: 0.002500 loss_xy: 3.157483 loss_wh: 0.670195 loss_obj: 3.324836 loss_cls: 0.742973 loss: 7.818409 eta: 4:47:11 batch_cost: 0.3598 data_cost: 0.0200 ips: 22.2359 images/s
- [11/22 13:49:26] ppdet.engine INFO: Epoch: [21] [ 400/1215] learning_rate: 0.002500 loss_xy: 3.068971 loss_wh: 0.683188 loss_obj: 3.043363 loss_cls: 0.706486 loss: 7.628610 eta: 4:46:02 batch_cost: 0.3665 data_cost: 0.0049 ips: 21.8256 images/s
- [11/22 13:50:38] ppdet.engine INFO: Epoch: [21] [ 600/1215] learning_rate: 0.002500 loss_xy: 3.070149 loss_wh: 0.663999 loss_obj: 2.750218 loss_cls: 0.765522 loss: 7.253464 eta: 4:44:25 batch_cost: 0.3574 data_cost: 0.0146 ips: 22.3852 images/s
- [11/22 13:51:49] ppdet.engine INFO: Epoch: [21] [ 800/1215] learning_rate: 0.002500 loss_xy: 2.903234 loss_wh: 0.637626 loss_obj: 2.717014 loss_cls: 0.762010 loss: 7.184401 eta: 4:42:37 batch_cost: 0.3529 data_cost: 0.0051 ips: 22.6715 images/s
- [11/22 13:53:04] ppdet.engine INFO: Epoch: [21] [1000/1215] learning_rate: 0.002500 loss_xy: 3.163116 loss_wh: 0.693153 loss_obj: 3.286148 loss_cls: 0.798944 loss: 8.199278 eta: 4:41:57 batch_cost: 0.3760 data_cost: 0.0203 ips: 21.2756 images/s
- [11/22 13:54:16] ppdet.engine INFO: Epoch: [21] [1200/1215] learning_rate: 0.002500 loss_xy: 3.001812 loss_wh: 0.678911 loss_obj: 2.948578 loss_cls: 0.849175 loss: 7.626981 eta: 4:40:30 batch_cost: 0.3594 data_cost: 0.0373 ips: 22.2603 images/s
- [11/22 13:54:23] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 13:54:23] ppdet.engine INFO: Eval iter: 0
- [11/22 13:54:27] ppdet.engine INFO: Eval iter: 100
- [11/22 13:54:31] ppdet.engine INFO: Eval iter: 200
- [11/22 13:54:35] ppdet.engine INFO: Eval iter: 300
- [11/22 13:54:39] ppdet.engine INFO: Eval iter: 400
- [11/22 13:54:43] ppdet.engine INFO: Eval iter: 500
- [11/22 13:54:47] ppdet.engine INFO: Eval iter: 600
- [11/22 13:54:51] ppdet.engine INFO: Eval iter: 700
- [11/22 13:54:56] ppdet.engine INFO: Eval iter: 800
- [11/22 13:55:00] ppdet.engine INFO: Eval iter: 900
- [11/22 13:55:04] ppdet.engine INFO: Eval iter: 1000
- [11/22 13:55:09] ppdet.engine INFO: Eval iter: 1100
- [11/22 13:55:13] ppdet.engine INFO: Eval iter: 1200
- [11/22 13:55:17] ppdet.engine INFO: Eval iter: 1300
- [11/22 13:55:21] ppdet.engine INFO: Eval iter: 1400
- [11/22 13:55:26] ppdet.engine INFO: Eval iter: 1500
- [11/22 13:55:30] ppdet.engine INFO: Eval iter: 1600
- [11/22 13:55:34] ppdet.engine INFO: Eval iter: 1700
- [11/22 13:55:38] ppdet.engine INFO: Eval iter: 1800
- [11/22 13:55:42] ppdet.engine INFO: Eval iter: 1900
- [11/22 13:55:46] ppdet.engine INFO: Eval iter: 2000
- [11/22 13:55:50] ppdet.engine INFO: Eval iter: 2100
- [11/22 13:55:54] ppdet.engine INFO: Eval iter: 2200
- [11/22 13:55:58] ppdet.engine INFO: Eval iter: 2300
- [11/22 13:56:02] ppdet.engine INFO: Eval iter: 2400
- [11/22 13:56:06] ppdet.engine INFO: Eval iter: 2500
- [11/22 13:56:10] ppdet.engine INFO: Eval iter: 2600
- [11/22 13:56:14] ppdet.engine INFO: Eval iter: 2700
- [11/22 13:56:18] ppdet.engine INFO: Eval iter: 2800
- [11/22 13:56:22] ppdet.engine INFO: Eval iter: 2900
- [11/22 13:56:26] ppdet.engine INFO: Eval iter: 3000
- [11/22 13:56:30] ppdet.engine INFO: Eval iter: 3100
- [11/22 13:56:34] ppdet.engine INFO: Eval iter: 3200
- [11/22 13:56:38] ppdet.engine INFO: Eval iter: 3300
- [11/22 13:56:43] ppdet.engine INFO: Eval iter: 3400
- [11/22 13:56:47] ppdet.engine INFO: Eval iter: 3500
- [11/22 13:56:51] ppdet.engine INFO: Eval iter: 3600
- [11/22 13:56:55] ppdet.engine INFO: Eval iter: 3700
- [11/22 13:56:59] ppdet.engine INFO: Eval iter: 3800
- [11/22 13:57:03] ppdet.engine INFO: Eval iter: 3900
- [11/22 13:57:07] ppdet.engine INFO: Eval iter: 4000
- [11/22 13:57:11] ppdet.engine INFO: Eval iter: 4100
- [11/22 13:57:15] ppdet.engine INFO: Eval iter: 4200
- [11/22 13:57:19] ppdet.engine INFO: Eval iter: 4300
- [11/22 13:57:20] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 13:57:20] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 84.20%
- [11/22 13:57:20] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.395501047283147
- [11/22 13:57:20] ppdet.engine INFO: Best test bbox ap is 0.842.
- [11/22 13:57:24] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 13:57:24] ppdet.engine INFO: Epoch: [22] [ 0/1215] learning_rate: 0.002500 loss_xy: 2.985542 loss_wh: 0.649180 loss_obj: 2.933593 loss_cls: 0.850841 loss: 7.504622 eta: 4:40:26 batch_cost: 0.3571 data_cost: 0.0294 ips: 22.4026 images/s
- [11/22 13:58:36] ppdet.engine INFO: Epoch: [22] [ 200/1215] learning_rate: 0.002500 loss_xy: 2.857981 loss_wh: 0.602278 loss_obj: 2.741049 loss_cls: 0.700927 loss: 6.983431 eta: 4:39:00 batch_cost: 0.3590 data_cost: 0.0112 ips: 22.2838 images/s
- [11/22 13:59:48] ppdet.engine INFO: Epoch: [22] [ 400/1215] learning_rate: 0.002500 loss_xy: 2.994753 loss_wh: 0.677690 loss_obj: 2.850745 loss_cls: 0.667404 loss: 7.520867 eta: 4:37:33 batch_cost: 0.3579 data_cost: 0.0210 ips: 22.3502 images/s
- [11/22 14:01:00] ppdet.engine INFO: Epoch: [22] [ 600/1215] learning_rate: 0.002500 loss_xy: 2.902802 loss_wh: 0.593407 loss_obj: 2.707242 loss_cls: 0.724745 loss: 6.992822 eta: 4:36:10 batch_cost: 0.3591 data_cost: 0.0170 ips: 22.2792 images/s
- [11/22 14:02:11] ppdet.engine INFO: Epoch: [22] [ 800/1215] learning_rate: 0.002500 loss_xy: 2.980840 loss_wh: 0.652244 loss_obj: 2.912298 loss_cls: 0.661106 loss: 7.201091 eta: 4:34:40 batch_cost: 0.3553 data_cost: 0.0095 ips: 22.5160 images/s
- [11/22 14:03:25] ppdet.engine INFO: Epoch: [22] [1000/1215] learning_rate: 0.002500 loss_xy: 3.077184 loss_wh: 0.669069 loss_obj: 2.877624 loss_cls: 0.679886 loss: 7.398191 eta: 4:33:33 batch_cost: 0.3662 data_cost: 0.0331 ips: 21.8484 images/s
- [11/22 14:04:37] ppdet.engine INFO: Epoch: [22] [1200/1215] learning_rate: 0.002500 loss_xy: 3.111678 loss_wh: 0.653185 loss_obj: 2.971709 loss_cls: 0.731813 loss: 7.521700 eta: 4:32:13 batch_cost: 0.3590 data_cost: 0.0091 ips: 22.2814 images/s
- [11/22 14:04:43] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 14:04:43] ppdet.engine INFO: Eval iter: 0
- [11/22 14:04:48] ppdet.engine INFO: Eval iter: 100
- [11/22 14:04:52] ppdet.engine INFO: Eval iter: 200
- [11/22 14:04:56] ppdet.engine INFO: Eval iter: 300
- [11/22 14:05:01] ppdet.engine INFO: Eval iter: 400
- [11/22 14:05:05] ppdet.engine INFO: Eval iter: 500
- [11/22 14:05:10] ppdet.engine INFO: Eval iter: 600
- [11/22 14:05:14] ppdet.engine INFO: Eval iter: 700
- [11/22 14:05:18] ppdet.engine INFO: Eval iter: 800
- [11/22 14:05:22] ppdet.engine INFO: Eval iter: 900
- [11/22 14:05:26] ppdet.engine INFO: Eval iter: 1000
- [11/22 14:05:30] ppdet.engine INFO: Eval iter: 1100
- [11/22 14:05:34] ppdet.engine INFO: Eval iter: 1200
- [11/22 14:05:38] ppdet.engine INFO: Eval iter: 1300
- [11/22 14:05:42] ppdet.engine INFO: Eval iter: 1400
- [11/22 14:05:46] ppdet.engine INFO: Eval iter: 1500
- [11/22 14:05:50] ppdet.engine INFO: Eval iter: 1600
- [11/22 14:05:54] ppdet.engine INFO: Eval iter: 1700
- [11/22 14:05:58] ppdet.engine INFO: Eval iter: 1800
- [11/22 14:06:02] ppdet.engine INFO: Eval iter: 1900
- [11/22 14:06:06] ppdet.engine INFO: Eval iter: 2000
- [11/22 14:06:10] ppdet.engine INFO: Eval iter: 2100
- [11/22 14:06:14] ppdet.engine INFO: Eval iter: 2200
- [11/22 14:06:18] ppdet.engine INFO: Eval iter: 2300
- [11/22 14:06:22] ppdet.engine INFO: Eval iter: 2400
- [11/22 14:06:26] ppdet.engine INFO: Eval iter: 2500
- [11/22 14:06:30] ppdet.engine INFO: Eval iter: 2600
- [11/22 14:06:34] ppdet.engine INFO: Eval iter: 2700
- [11/22 14:06:38] ppdet.engine INFO: Eval iter: 2800
- [11/22 14:06:42] ppdet.engine INFO: Eval iter: 2900
- [11/22 14:06:46] ppdet.engine INFO: Eval iter: 3000
- [11/22 14:06:50] ppdet.engine INFO: Eval iter: 3100
- [11/22 14:06:54] ppdet.engine INFO: Eval iter: 3200
- [11/22 14:06:58] ppdet.engine INFO: Eval iter: 3300
- [11/22 14:07:02] ppdet.engine INFO: Eval iter: 3400
- [11/22 14:07:06] ppdet.engine INFO: Eval iter: 3500
- [11/22 14:07:10] ppdet.engine INFO: Eval iter: 3600
- [11/22 14:07:14] ppdet.engine INFO: Eval iter: 3700
- [11/22 14:07:18] ppdet.engine INFO: Eval iter: 3800
- [11/22 14:07:22] ppdet.engine INFO: Eval iter: 3900
- [11/22 14:07:26] ppdet.engine INFO: Eval iter: 4000
- [11/22 14:07:30] ppdet.engine INFO: Eval iter: 4100
- [11/22 14:07:34] ppdet.engine INFO: Eval iter: 4200
- [11/22 14:07:38] ppdet.engine INFO: Eval iter: 4300
- [11/22 14:07:39] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 14:07:39] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 85.33%
- [11/22 14:07:39] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.599969079496933
- [11/22 14:07:39] ppdet.engine INFO: Best test bbox ap is 0.853.
- [11/22 14:07:43] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 14:07:43] ppdet.engine INFO: Epoch: [23] [ 0/1215] learning_rate: 0.002500 loss_xy: 3.139543 loss_wh: 0.676327 loss_obj: 3.015043 loss_cls: 0.734748 loss: 7.631557 eta: 4:32:10 batch_cost: 0.3609 data_cost: 0.0087 ips: 22.1662 images/s
- [11/22 14:08:54] ppdet.engine INFO: Epoch: [23] [ 200/1215] learning_rate: 0.002500 loss_xy: 3.055465 loss_wh: 0.652439 loss_obj: 2.760919 loss_cls: 0.704086 loss: 7.148224 eta: 4:30:39 batch_cost: 0.3530 data_cost: 0.0306 ips: 22.6646 images/s
- [11/22 14:10:07] ppdet.engine INFO: Epoch: [23] [ 400/1215] learning_rate: 0.002500 loss_xy: 2.805641 loss_wh: 0.601975 loss_obj: 2.513329 loss_cls: 0.642393 loss: 6.581067 eta: 4:29:31 batch_cost: 0.3657 data_cost: 0.0213 ips: 21.8769 images/s
- [11/22 14:11:17] ppdet.engine INFO: Epoch: [23] [ 600/1215] learning_rate: 0.002500 loss_xy: 3.015815 loss_wh: 0.669939 loss_obj: 2.850561 loss_cls: 0.692184 loss: 7.316739 eta: 4:27:54 batch_cost: 0.3477 data_cost: 0.0170 ips: 23.0075 images/s
- [11/22 14:12:30] ppdet.engine INFO: Epoch: [23] [ 800/1215] learning_rate: 0.002500 loss_xy: 2.938328 loss_wh: 0.641144 loss_obj: 2.838834 loss_cls: 0.648094 loss: 7.124407 eta: 4:26:45 batch_cost: 0.3646 data_cost: 0.0228 ips: 21.9397 images/s
- [11/22 14:13:43] ppdet.engine INFO: Epoch: [23] [1000/1215] learning_rate: 0.002500 loss_xy: 2.803728 loss_wh: 0.606764 loss_obj: 2.392327 loss_cls: 0.627352 loss: 6.511455 eta: 4:25:33 batch_cost: 0.3630 data_cost: 0.0059 ips: 22.0405 images/s
- [11/22 14:14:55] ppdet.engine INFO: Epoch: [23] [1200/1215] learning_rate: 0.002500 loss_xy: 3.110498 loss_wh: 0.688655 loss_obj: 2.782815 loss_cls: 0.723359 loss: 7.545352 eta: 4:24:10 batch_cost: 0.3551 data_cost: 0.0157 ips: 22.5289 images/s
- [11/22 14:15:00] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 14:15:01] ppdet.engine INFO: Eval iter: 0
- [11/22 14:15:05] ppdet.engine INFO: Eval iter: 100
- [11/22 14:15:09] ppdet.engine INFO: Eval iter: 200
- [11/22 14:15:14] ppdet.engine INFO: Eval iter: 300
- [11/22 14:15:18] ppdet.engine INFO: Eval iter: 400
- [11/22 14:15:22] ppdet.engine INFO: Eval iter: 500
- [11/22 14:15:26] ppdet.engine INFO: Eval iter: 600
- [11/22 14:15:30] ppdet.engine INFO: Eval iter: 700
- [11/22 14:15:34] ppdet.engine INFO: Eval iter: 800
- [11/22 14:15:38] ppdet.engine INFO: Eval iter: 900
- [11/22 14:15:42] ppdet.engine INFO: Eval iter: 1000
- [11/22 14:15:46] ppdet.engine INFO: Eval iter: 1100
- [11/22 14:15:50] ppdet.engine INFO: Eval iter: 1200
- [11/22 14:15:54] ppdet.engine INFO: Eval iter: 1300
- [11/22 14:15:58] ppdet.engine INFO: Eval iter: 1400
- [11/22 14:16:02] ppdet.engine INFO: Eval iter: 1500
- [11/22 14:16:06] ppdet.engine INFO: Eval iter: 1600
- [11/22 14:16:10] ppdet.engine INFO: Eval iter: 1700
- [11/22 14:16:14] ppdet.engine INFO: Eval iter: 1800
- [11/22 14:16:18] ppdet.engine INFO: Eval iter: 1900
- [11/22 14:16:22] ppdet.engine INFO: Eval iter: 2000
- [11/22 14:16:26] ppdet.engine INFO: Eval iter: 2100
- [11/22 14:16:30] ppdet.engine INFO: Eval iter: 2200
- [11/22 14:16:34] ppdet.engine INFO: Eval iter: 2300
- [11/22 14:16:38] ppdet.engine INFO: Eval iter: 2400
- [11/22 14:16:42] ppdet.engine INFO: Eval iter: 2500
- [11/22 14:16:46] ppdet.engine INFO: Eval iter: 2600
- [11/22 14:16:50] ppdet.engine INFO: Eval iter: 2700
- [11/22 14:16:54] ppdet.engine INFO: Eval iter: 2800
- [11/22 14:16:58] ppdet.engine INFO: Eval iter: 2900
- [11/22 14:17:02] ppdet.engine INFO: Eval iter: 3000
- [11/22 14:17:06] ppdet.engine INFO: Eval iter: 3100
- [11/22 14:17:10] ppdet.engine INFO: Eval iter: 3200
- [11/22 14:17:14] ppdet.engine INFO: Eval iter: 3300
- [11/22 14:17:18] ppdet.engine INFO: Eval iter: 3400
- [11/22 14:17:22] ppdet.engine INFO: Eval iter: 3500
- [11/22 14:17:26] ppdet.engine INFO: Eval iter: 3600
- [11/22 14:17:30] ppdet.engine INFO: Eval iter: 3700
- [11/22 14:17:34] ppdet.engine INFO: Eval iter: 3800
- [11/22 14:17:37] ppdet.engine INFO: Eval iter: 3900
- [11/22 14:17:41] ppdet.engine INFO: Eval iter: 4000
- [11/22 14:17:45] ppdet.engine INFO: Eval iter: 4100
- [11/22 14:17:49] ppdet.engine INFO: Eval iter: 4200
- [11/22 14:17:53] ppdet.engine INFO: Eval iter: 4300
- [11/22 14:17:54] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 14:17:54] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 83.72%
- [11/22 14:17:54] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.854502528061555
- [11/22 14:17:54] ppdet.engine INFO: Best test bbox ap is 0.853.
- [11/22 14:17:55] ppdet.engine INFO: Epoch: [24] [ 0/1215] learning_rate: 0.002500 loss_xy: 3.119858 loss_wh: 0.688655 loss_obj: 2.889948 loss_cls: 0.712280 loss: 7.545352 eta: 4:24:01 batch_cost: 0.3544 data_cost: 0.0160 ips: 22.5720 images/s
- [11/22 14:19:07] ppdet.engine INFO: Epoch: [24] [ 200/1215] learning_rate: 0.002500 loss_xy: 3.058218 loss_wh: 0.652843 loss_obj: 3.268324 loss_cls: 0.861591 loss: 8.135222 eta: 4:22:45 batch_cost: 0.3595 data_cost: 0.0365 ips: 22.2533 images/s
- [11/22 14:20:20] ppdet.engine INFO: Epoch: [24] [ 400/1215] learning_rate: 0.002500 loss_xy: 2.970448 loss_wh: 0.616359 loss_obj: 3.268284 loss_cls: 0.768616 loss: 7.747204 eta: 4:21:37 batch_cost: 0.3652 data_cost: 0.0162 ips: 21.9061 images/s
- [11/22 14:21:34] ppdet.engine INFO: Epoch: [24] [ 600/1215] learning_rate: 0.002500 loss_xy: 3.001367 loss_wh: 0.608082 loss_obj: 3.277891 loss_cls: 0.876386 loss: 7.829921 eta: 4:20:32 batch_cost: 0.3678 data_cost: 0.0194 ips: 21.7489 images/s
- [11/22 14:22:46] ppdet.engine INFO: Epoch: [24] [ 800/1215] learning_rate: 0.002500 loss_xy: 2.959203 loss_wh: 0.631995 loss_obj: 3.103574 loss_cls: 0.834502 loss: 7.720676 eta: 4:19:13 batch_cost: 0.3570 data_cost: 0.0120 ips: 22.4079 images/s
- [11/22 14:23:59] ppdet.engine INFO: Epoch: [24] [1000/1215] learning_rate: 0.002500 loss_xy: 3.004073 loss_wh: 0.647770 loss_obj: 3.270823 loss_cls: 0.859503 loss: 8.038187 eta: 4:18:05 batch_cost: 0.3666 data_cost: 0.0040 ips: 21.8217 images/s
- [11/22 14:25:14] ppdet.engine INFO: Epoch: [24] [1200/1215] learning_rate: 0.002500 loss_xy: 3.097903 loss_wh: 0.656318 loss_obj: 3.446897 loss_cls: 0.856884 loss: 8.103826 eta: 4:17:05 batch_cost: 0.3728 data_cost: 0.0362 ips: 21.4576 images/s
- [11/22 14:25:20] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 14:25:20] ppdet.engine INFO: Eval iter: 0
- [11/22 14:25:24] ppdet.engine INFO: Eval iter: 100
- [11/22 14:25:28] ppdet.engine INFO: Eval iter: 200
- [11/22 14:25:32] ppdet.engine INFO: Eval iter: 300
- [11/22 14:25:36] ppdet.engine INFO: Eval iter: 400
- [11/22 14:25:40] ppdet.engine INFO: Eval iter: 500
- [11/22 14:25:44] ppdet.engine INFO: Eval iter: 600
- [11/22 14:25:48] ppdet.engine INFO: Eval iter: 700
- [11/22 14:25:52] ppdet.engine INFO: Eval iter: 800
- [11/22 14:25:56] ppdet.engine INFO: Eval iter: 900
- [11/22 14:26:00] ppdet.engine INFO: Eval iter: 1000
- [11/22 14:26:04] ppdet.engine INFO: Eval iter: 1100
- [11/22 14:26:07] ppdet.engine INFO: Eval iter: 1200
- [11/22 14:26:11] ppdet.engine INFO: Eval iter: 1300
- [11/22 14:26:15] ppdet.engine INFO: Eval iter: 1400
- [11/22 14:26:19] ppdet.engine INFO: Eval iter: 1500
- [11/22 14:26:23] ppdet.engine INFO: Eval iter: 1600
- [11/22 14:26:27] ppdet.engine INFO: Eval iter: 1700
- [11/22 14:26:31] ppdet.engine INFO: Eval iter: 1800
- [11/22 14:26:35] ppdet.engine INFO: Eval iter: 1900
- [11/22 14:26:39] ppdet.engine INFO: Eval iter: 2000
- [11/22 14:26:43] ppdet.engine INFO: Eval iter: 2100
- [11/22 14:26:47] ppdet.engine INFO: Eval iter: 2200
- [11/22 14:26:51] ppdet.engine INFO: Eval iter: 2300
- [11/22 14:26:55] ppdet.engine INFO: Eval iter: 2400
- [11/22 14:26:59] ppdet.engine INFO: Eval iter: 2500
- [11/22 14:27:03] ppdet.engine INFO: Eval iter: 2600
- [11/22 14:27:07] ppdet.engine INFO: Eval iter: 2700
- [11/22 14:27:11] ppdet.engine INFO: Eval iter: 2800
- [11/22 14:27:15] ppdet.engine INFO: Eval iter: 2900
- [11/22 14:27:19] ppdet.engine INFO: Eval iter: 3000
- [11/22 14:27:23] ppdet.engine INFO: Eval iter: 3100
- [11/22 14:27:27] ppdet.engine INFO: Eval iter: 3200
- [11/22 14:27:31] ppdet.engine INFO: Eval iter: 3300
- [11/22 14:27:35] ppdet.engine INFO: Eval iter: 3400
- [11/22 14:27:39] ppdet.engine INFO: Eval iter: 3500
- [11/22 14:27:43] ppdet.engine INFO: Eval iter: 3600
- [11/22 14:27:47] ppdet.engine INFO: Eval iter: 3700
- [11/22 14:27:51] ppdet.engine INFO: Eval iter: 3800
- [11/22 14:27:55] ppdet.engine INFO: Eval iter: 3900
- [11/22 14:27:59] ppdet.engine INFO: Eval iter: 4000
- [11/22 14:28:03] ppdet.engine INFO: Eval iter: 4100
- [11/22 14:28:07] ppdet.engine INFO: Eval iter: 4200
- [11/22 14:28:11] ppdet.engine INFO: Eval iter: 4300
- [11/22 14:28:12] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 14:28:12] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 83.20%
- [11/22 14:28:12] ppdet.engine INFO: Total sample number: 4322, averge FPS: 25.133535768874903
- [11/22 14:28:12] ppdet.engine INFO: Best test bbox ap is 0.853.
- [11/22 14:28:12] ppdet.engine INFO: Epoch: [25] [ 0/1215] learning_rate: 0.002500 loss_xy: 3.093482 loss_wh: 0.656318 loss_obj: 3.466780 loss_cls: 0.851478 loss: 8.097486 eta: 4:16:56 batch_cost: 0.3674 data_cost: 0.0371 ips: 21.7748 images/s
- [11/22 14:29:25] ppdet.engine INFO: Epoch: [25] [ 200/1215] learning_rate: 0.002500 loss_xy: 2.991953 loss_wh: 0.641739 loss_obj: 3.397084 loss_cls: 0.844883 loss: 7.952458 eta: 4:15:45 batch_cost: 0.3638 data_cost: 0.0173 ips: 21.9916 images/s
- [11/22 14:30:42] ppdet.engine INFO: Epoch: [25] [ 400/1215] learning_rate: 0.002500 loss_xy: 3.114479 loss_wh: 0.663075 loss_obj: 3.660706 loss_cls: 0.918797 loss: 8.333830 eta: 4:14:53 batch_cost: 0.3813 data_cost: 0.0153 ips: 20.9806 images/s
- [11/22 14:31:55] ppdet.engine INFO: Epoch: [25] [ 600/1215] learning_rate: 0.002500 loss_xy: 3.175154 loss_wh: 0.701379 loss_obj: 3.723315 loss_cls: 0.878565 loss: 8.556681 eta: 4:13:45 batch_cost: 0.3673 data_cost: 0.0007 ips: 21.7805 images/s
- [11/22 14:33:07] ppdet.engine INFO: Epoch: [25] [ 800/1215] learning_rate: 0.002500 loss_xy: 3.070700 loss_wh: 0.654809 loss_obj: 3.354778 loss_cls: 0.887927 loss: 8.081770 eta: 4:12:27 batch_cost: 0.3579 data_cost: 0.0011 ips: 22.3547 images/s
- [11/22 14:34:21] ppdet.engine INFO: Epoch: [25] [1000/1215] learning_rate: 0.002500 loss_xy: 3.203738 loss_wh: 0.671058 loss_obj: 3.531581 loss_cls: 0.818973 loss: 8.278908 eta: 4:11:18 batch_cost: 0.3668 data_cost: 0.0086 ips: 21.8123 images/s
- [11/22 14:35:33] ppdet.engine INFO: Epoch: [25] [1200/1215] learning_rate: 0.002500 loss_xy: 3.048559 loss_wh: 0.638725 loss_obj: 3.412725 loss_cls: 0.870223 loss: 8.065794 eta: 4:10:02 batch_cost: 0.3599 data_cost: 0.0091 ips: 22.2310 images/s
- [11/22 14:35:39] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 14:35:40] ppdet.engine INFO: Eval iter: 0
- [11/22 14:35:44] ppdet.engine INFO: Eval iter: 100
- [11/22 14:35:48] ppdet.engine INFO: Eval iter: 200
- [11/22 14:35:52] ppdet.engine INFO: Eval iter: 300
- [11/22 14:35:56] ppdet.engine INFO: Eval iter: 400
- [11/22 14:36:01] ppdet.engine INFO: Eval iter: 500
- [11/22 14:36:05] ppdet.engine INFO: Eval iter: 600
- [11/22 14:36:09] ppdet.engine INFO: Eval iter: 700
- [11/22 14:36:13] ppdet.engine INFO: Eval iter: 800
- [11/22 14:36:17] ppdet.engine INFO: Eval iter: 900
- [11/22 14:36:21] ppdet.engine INFO: Eval iter: 1000
- [11/22 14:36:25] ppdet.engine INFO: Eval iter: 1100
- [11/22 14:36:29] ppdet.engine INFO: Eval iter: 1200
- [11/22 14:36:33] ppdet.engine INFO: Eval iter: 1300
- [11/22 14:36:37] ppdet.engine INFO: Eval iter: 1400
- [11/22 14:36:41] ppdet.engine INFO: Eval iter: 1500
- [11/22 14:36:45] ppdet.engine INFO: Eval iter: 1600
- [11/22 14:36:49] ppdet.engine INFO: Eval iter: 1700
- [11/22 14:36:53] ppdet.engine INFO: Eval iter: 1800
- [11/22 14:36:57] ppdet.engine INFO: Eval iter: 1900
- [11/22 14:37:01] ppdet.engine INFO: Eval iter: 2000
- [11/22 14:37:05] ppdet.engine INFO: Eval iter: 2100
- [11/22 14:37:09] ppdet.engine INFO: Eval iter: 2200
- [11/22 14:37:13] ppdet.engine INFO: Eval iter: 2300
- [11/22 14:37:17] ppdet.engine INFO: Eval iter: 2400
- [11/22 14:37:21] ppdet.engine INFO: Eval iter: 2500
- [11/22 14:37:25] ppdet.engine INFO: Eval iter: 2600
- [11/22 14:37:29] ppdet.engine INFO: Eval iter: 2700
- [11/22 14:37:33] ppdet.engine INFO: Eval iter: 2800
- [11/22 14:37:37] ppdet.engine INFO: Eval iter: 2900
- [11/22 14:37:41] ppdet.engine INFO: Eval iter: 3000
- [11/22 14:37:45] ppdet.engine INFO: Eval iter: 3100
- [11/22 14:37:50] ppdet.engine INFO: Eval iter: 3200
- [11/22 14:37:54] ppdet.engine INFO: Eval iter: 3300
- [11/22 14:37:58] ppdet.engine INFO: Eval iter: 3400
- [11/22 14:38:02] ppdet.engine INFO: Eval iter: 3500
- [11/22 14:38:06] ppdet.engine INFO: Eval iter: 3600
- [11/22 14:38:10] ppdet.engine INFO: Eval iter: 3700
- [11/22 14:38:14] ppdet.engine INFO: Eval iter: 3800
- [11/22 14:38:18] ppdet.engine INFO: Eval iter: 3900
- [11/22 14:38:22] ppdet.engine INFO: Eval iter: 4000
- [11/22 14:38:26] ppdet.engine INFO: Eval iter: 4100
- [11/22 14:38:30] ppdet.engine INFO: Eval iter: 4200
- [11/22 14:38:34] ppdet.engine INFO: Eval iter: 4300
- [11/22 14:38:35] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 14:38:35] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 84.74%
- [11/22 14:38:35] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.62118482049293
- [11/22 14:38:35] ppdet.engine INFO: Best test bbox ap is 0.853.
- [11/22 14:38:35] ppdet.engine INFO: Epoch: [26] [ 0/1215] learning_rate: 0.002500 loss_xy: 3.061683 loss_wh: 0.635181 loss_obj: 3.453074 loss_cls: 0.872765 loss: 8.054026 eta: 4:09:57 batch_cost: 0.3587 data_cost: 0.0091 ips: 22.3027 images/s
- I1122 14:39:57.016235 39583 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1122 14:39:57.779166 39583 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1122 14:39:57.785562 39583 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/22 14:39:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- [11/22 14:39:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/22 14:40:00] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- [11/22 14:40:00] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- [11/22 14:40:01] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- [11/22 14:40:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- [11/22 14:40:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- [11/22 14:40:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- [11/22 14:40:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- [11/22 14:40:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- [11/22 14:40:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/22 14:40:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- [11/22 14:40:05] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- [11/22 14:40:05] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/22 14:40:05] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- [11/22 14:40:06] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- [11/22 14:40:08] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/22 14:40:09] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/25.pdparams
- [11/22 14:40:12] ppdet.engine INFO: Epoch: [26] [ 0/1215] learning_rate: 0.002500 loss_xy: 4.826826 loss_wh: 1.352595 loss_obj: 5.803462 loss_cls: 1.593733 loss: 13.576617 eta: 1 day, 3:05:18 batch_cost: 2.3606 data_cost: 0.0003 ips: 3.3889 images/s
- [11/22 14:41:23] ppdet.engine INFO: Epoch: [26] [ 200/1215] learning_rate: 0.001000 loss_xy: 2.879446 loss_wh: 0.550086 loss_obj: 2.506320 loss_cls: 0.624995 loss: 6.681121 eta: 4:09:27 batch_cost: 0.3541 data_cost: 0.0171 ips: 22.5917 images/s
- [11/22 14:42:38] ppdet.engine INFO: Epoch: [26] [ 400/1215] learning_rate: 0.001000 loss_xy: 2.878510 loss_wh: 0.525746 loss_obj: 2.287694 loss_cls: 0.582344 loss: 6.617065 eta: 4:11:27 batch_cost: 0.3735 data_cost: 0.0131 ips: 21.4190 images/s
- [11/22 14:43:53] ppdet.engine INFO: Epoch: [26] [ 600/1215] learning_rate: 0.001000 loss_xy: 2.889726 loss_wh: 0.530710 loss_obj: 2.211832 loss_cls: 0.507097 loss: 6.243162 eta: 4:10:55 batch_cost: 0.3719 data_cost: 0.0083 ips: 21.5111 images/s
- [11/22 14:45:07] ppdet.engine INFO: Epoch: [26] [ 800/1215] learning_rate: 0.001000 loss_xy: 2.865095 loss_wh: 0.516392 loss_obj: 2.052738 loss_cls: 0.495065 loss: 5.888078 eta: 4:09:51 batch_cost: 0.3708 data_cost: 0.0142 ips: 21.5731 images/s
- [11/22 14:46:21] ppdet.engine INFO: Epoch: [26] [1000/1215] learning_rate: 0.001000 loss_xy: 2.946200 loss_wh: 0.547422 loss_obj: 2.049998 loss_cls: 0.513949 loss: 6.383658 eta: 4:07:59 batch_cost: 0.3653 data_cost: 0.0258 ips: 21.8991 images/s
- [11/22 14:47:34] ppdet.engine INFO: Epoch: [26] [1200/1215] learning_rate: 0.001000 loss_xy: 2.820389 loss_wh: 0.531532 loss_obj: 1.978957 loss_cls: 0.547486 loss: 6.039793 eta: 4:06:25 batch_cost: 0.3661 data_cost: 0.0118 ips: 21.8540 images/s
- [11/22 14:47:41] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 14:47:41] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_2.xml, x1: 217.0, y1: 763.0, x2: 217.0, y2: 764.0.
- [11/22 14:47:42] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_8.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/22 14:47:42] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_881_3.xml, x1: 35.0, y1: 198.0, x2: 75.0, y2: 197.0.
- [11/22 14:47:42] ppdet.engine INFO: Eval iter: 0
- [11/22 14:47:46] ppdet.engine INFO: Eval iter: 100
- [11/22 14:47:50] ppdet.engine INFO: Eval iter: 200
- [11/22 14:47:54] ppdet.engine INFO: Eval iter: 300
- [11/22 14:47:58] ppdet.engine INFO: Eval iter: 400
- [11/22 14:48:02] ppdet.engine INFO: Eval iter: 500
- [11/22 14:48:07] ppdet.engine INFO: Eval iter: 600
- [11/22 14:48:11] ppdet.engine INFO: Eval iter: 700
- [11/22 14:48:15] ppdet.engine INFO: Eval iter: 800
- [11/22 14:48:19] ppdet.engine INFO: Eval iter: 900
- [11/22 14:48:23] ppdet.engine INFO: Eval iter: 1000
- [11/22 14:48:27] ppdet.engine INFO: Eval iter: 1100
- [11/22 14:48:31] ppdet.engine INFO: Eval iter: 1200
- [11/22 14:48:35] ppdet.engine INFO: Eval iter: 1300
- [11/22 14:48:39] ppdet.engine INFO: Eval iter: 1400
- [11/22 14:48:43] ppdet.engine INFO: Eval iter: 1500
- [11/22 14:48:47] ppdet.engine INFO: Eval iter: 1600
- [11/22 14:48:51] ppdet.engine INFO: Eval iter: 1700
- [11/22 14:48:55] ppdet.engine INFO: Eval iter: 1800
- [11/22 14:48:59] ppdet.engine INFO: Eval iter: 1900
- [11/22 14:49:03] ppdet.engine INFO: Eval iter: 2000
- [11/22 14:49:07] ppdet.engine INFO: Eval iter: 2100
- [11/22 14:49:11] ppdet.engine INFO: Eval iter: 2200
- [11/22 14:49:15] ppdet.engine INFO: Eval iter: 2300
- [11/22 14:49:20] ppdet.engine INFO: Eval iter: 2400
- [11/22 14:49:24] ppdet.engine INFO: Eval iter: 2500
- [11/22 14:49:28] ppdet.engine INFO: Eval iter: 2600
- [11/22 14:49:32] ppdet.engine INFO: Eval iter: 2700
- [11/22 14:49:35] ppdet.engine INFO: Eval iter: 2800
- [11/22 14:49:39] ppdet.engine INFO: Eval iter: 2900
- [11/22 14:49:43] ppdet.engine INFO: Eval iter: 3000
- [11/22 14:49:47] ppdet.engine INFO: Eval iter: 3100
- [11/22 14:49:51] ppdet.engine INFO: Eval iter: 3200
- [11/22 14:49:55] ppdet.engine INFO: Eval iter: 3300
- [11/22 14:49:59] ppdet.engine INFO: Eval iter: 3400
- [11/22 14:50:03] ppdet.engine INFO: Eval iter: 3500
- [11/22 14:50:07] ppdet.engine INFO: Eval iter: 3600
- [11/22 14:50:11] ppdet.engine INFO: Eval iter: 3700
- [11/22 14:50:15] ppdet.engine INFO: Eval iter: 3800
- [11/22 14:50:19] ppdet.engine INFO: Eval iter: 3900
- [11/22 14:50:23] ppdet.engine INFO: Eval iter: 4000
- [11/22 14:50:27] ppdet.engine INFO: Eval iter: 4100
- [11/22 14:50:31] ppdet.engine INFO: Eval iter: 4200
- [11/22 14:50:35] ppdet.engine INFO: Eval iter: 4300
- [11/22 14:50:36] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 14:50:36] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 87.78%
- [11/22 14:50:36] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.852865833822598
- [11/22 14:50:36] ppdet.engine INFO: Best test bbox ap is 0.878.
- [11/22 14:50:40] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 14:50:40] ppdet.engine INFO: Epoch: [27] [ 0/1215] learning_rate: 0.001000 loss_xy: 2.911708 loss_wh: 0.544475 loss_obj: 1.990732 loss_cls: 0.550982 loss: 6.103916 eta: 4:06:19 batch_cost: 0.3672 data_cost: 0.0151 ips: 21.7872 images/s
- [11/22 14:51:54] ppdet.engine INFO: Epoch: [27] [ 200/1215] learning_rate: 0.001000 loss_xy: 2.803940 loss_wh: 0.518575 loss_obj: 2.068616 loss_cls: 0.531949 loss: 5.979707 eta: 4:04:55 batch_cost: 0.3667 data_cost: 0.0138 ips: 21.8161 images/s
- [11/22 14:53:08] ppdet.engine INFO: Epoch: [27] [ 400/1215] learning_rate: 0.001000 loss_xy: 2.919713 loss_wh: 0.542880 loss_obj: 2.099018 loss_cls: 0.496279 loss: 6.146082 eta: 4:04:00 batch_cost: 0.3723 data_cost: 0.0129 ips: 21.4900 images/s
- [11/22 14:54:22] ppdet.engine INFO: Epoch: [27] [ 600/1215] learning_rate: 0.001000 loss_xy: 3.049737 loss_wh: 0.528585 loss_obj: 2.273072 loss_cls: 0.543687 loss: 6.521732 eta: 4:02:37 batch_cost: 0.3666 data_cost: 0.0153 ips: 21.8201 images/s
- [11/22 14:55:35] ppdet.engine INFO: Epoch: [27] [ 800/1215] learning_rate: 0.001000 loss_xy: 2.987645 loss_wh: 0.516047 loss_obj: 2.166022 loss_cls: 0.478473 loss: 6.329563 eta: 4:01:14 batch_cost: 0.3663 data_cost: 0.0135 ips: 21.8385 images/s
- [11/22 14:56:48] ppdet.engine INFO: Epoch: [27] [1000/1215] learning_rate: 0.001000 loss_xy: 3.070595 loss_wh: 0.571001 loss_obj: 2.169991 loss_cls: 0.521880 loss: 6.729041 eta: 3:59:46 batch_cost: 0.3641 data_cost: 0.0336 ips: 21.9723 images/s
- [11/22 14:58:02] ppdet.engine INFO: Epoch: [27] [1200/1215] learning_rate: 0.001000 loss_xy: 2.938412 loss_wh: 0.506744 loss_obj: 2.165702 loss_cls: 0.495945 loss: 6.168200 eta: 3:58:23 batch_cost: 0.3652 data_cost: 0.0063 ips: 21.9035 images/s
- [11/22 14:58:08] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 14:58:08] ppdet.engine INFO: Eval iter: 0
- [11/22 14:58:12] ppdet.engine INFO: Eval iter: 100
- [11/22 14:58:16] ppdet.engine INFO: Eval iter: 200
- [11/22 14:58:20] ppdet.engine INFO: Eval iter: 300
- [11/22 14:58:24] ppdet.engine INFO: Eval iter: 400
- [11/22 14:58:29] ppdet.engine INFO: Eval iter: 500
- [11/22 14:58:33] ppdet.engine INFO: Eval iter: 600
- [11/22 14:58:37] ppdet.engine INFO: Eval iter: 700
- [11/22 14:58:41] ppdet.engine INFO: Eval iter: 800
- [11/22 14:58:45] ppdet.engine INFO: Eval iter: 900
- [11/22 14:58:49] ppdet.engine INFO: Eval iter: 1000
- [11/22 14:58:53] ppdet.engine INFO: Eval iter: 1100
- [11/22 14:58:57] ppdet.engine INFO: Eval iter: 1200
- [11/22 14:59:01] ppdet.engine INFO: Eval iter: 1300
- [11/22 14:59:05] ppdet.engine INFO: Eval iter: 1400
- [11/22 14:59:09] ppdet.engine INFO: Eval iter: 1500
- [11/22 14:59:13] ppdet.engine INFO: Eval iter: 1600
- [11/22 14:59:17] ppdet.engine INFO: Eval iter: 1700
- [11/22 14:59:21] ppdet.engine INFO: Eval iter: 1800
- [11/22 14:59:25] ppdet.engine INFO: Eval iter: 1900
- [11/22 14:59:29] ppdet.engine INFO: Eval iter: 2000
- [11/22 14:59:33] ppdet.engine INFO: Eval iter: 2100
- [11/22 14:59:37] ppdet.engine INFO: Eval iter: 2200
- [11/22 14:59:41] ppdet.engine INFO: Eval iter: 2300
- [11/22 14:59:45] ppdet.engine INFO: Eval iter: 2400
- [11/22 14:59:49] ppdet.engine INFO: Eval iter: 2500
- [11/22 14:59:53] ppdet.engine INFO: Eval iter: 2600
- [11/22 14:59:57] ppdet.engine INFO: Eval iter: 2700
- [11/22 15:00:01] ppdet.engine INFO: Eval iter: 2800
- [11/22 15:00:06] ppdet.engine INFO: Eval iter: 2900
- [11/22 15:00:10] ppdet.engine INFO: Eval iter: 3000
- [11/22 15:00:14] ppdet.engine INFO: Eval iter: 3100
- [11/22 15:00:18] ppdet.engine INFO: Eval iter: 3200
- [11/22 15:00:22] ppdet.engine INFO: Eval iter: 3300
- [11/22 15:00:26] ppdet.engine INFO: Eval iter: 3400
- [11/22 15:00:30] ppdet.engine INFO: Eval iter: 3500
- [11/22 15:00:34] ppdet.engine INFO: Eval iter: 3600
- [11/22 15:00:38] ppdet.engine INFO: Eval iter: 3700
- [11/22 15:00:42] ppdet.engine INFO: Eval iter: 3800
- [11/22 15:00:46] ppdet.engine INFO: Eval iter: 3900
- [11/22 15:00:50] ppdet.engine INFO: Eval iter: 4000
- [11/22 15:00:54] ppdet.engine INFO: Eval iter: 4100
- [11/22 15:00:58] ppdet.engine INFO: Eval iter: 4200
- [11/22 15:01:03] ppdet.engine INFO: Eval iter: 4300
- [11/22 15:01:04] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 15:01:04] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 87.56%
- [11/22 15:01:04] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.609765227264347
- [11/22 15:01:04] ppdet.engine INFO: Best test bbox ap is 0.878.
- [11/22 15:01:04] ppdet.engine INFO: Epoch: [28] [ 0/1215] learning_rate: 0.001000 loss_xy: 2.965744 loss_wh: 0.527193 loss_obj: 2.140296 loss_cls: 0.493078 loss: 6.211865 eta: 3:58:12 batch_cost: 0.3641 data_cost: 0.0077 ips: 21.9719 images/s
- [11/22 15:02:17] ppdet.engine INFO: Epoch: [28] [ 200/1215] learning_rate: 0.001000 loss_xy: 3.092305 loss_wh: 0.549862 loss_obj: 2.384075 loss_cls: 0.488391 loss: 6.661283 eta: 3:56:48 batch_cost: 0.3638 data_cost: 0.0226 ips: 21.9884 images/s
- [11/22 15:03:31] ppdet.engine INFO: Epoch: [28] [ 400/1215] learning_rate: 0.001000 loss_xy: 2.997136 loss_wh: 0.599116 loss_obj: 2.208735 loss_cls: 0.470342 loss: 6.390667 eta: 3:55:33 batch_cost: 0.3668 data_cost: 0.0200 ips: 21.8077 images/s
- [11/22 15:04:44] ppdet.engine INFO: Epoch: [28] [ 600/1215] learning_rate: 0.001000 loss_xy: 3.021292 loss_wh: 0.546293 loss_obj: 1.957738 loss_cls: 0.533450 loss: 6.069045 eta: 3:54:14 batch_cost: 0.3653 data_cost: 0.0426 ips: 21.8994 images/s
- [11/22 15:05:55] ppdet.engine INFO: Epoch: [28] [ 800/1215] learning_rate: 0.001000 loss_xy: 2.830721 loss_wh: 0.514474 loss_obj: 1.802497 loss_cls: 0.465394 loss: 5.770278 eta: 3:52:33 batch_cost: 0.3554 data_cost: 0.0171 ips: 22.5094 images/s
- [11/22 15:07:11] ppdet.engine INFO: Epoch: [28] [1000/1215] learning_rate: 0.001000 loss_xy: 3.079796 loss_wh: 0.558441 loss_obj: 2.304625 loss_cls: 0.518403 loss: 6.720607 eta: 3:51:40 batch_cost: 0.3754 data_cost: 0.0320 ips: 21.3108 images/s
- [11/22 15:08:23] ppdet.engine INFO: Epoch: [28] [1200/1215] learning_rate: 0.001000 loss_xy: 2.924245 loss_wh: 0.537242 loss_obj: 1.964025 loss_cls: 0.552637 loss: 6.025584 eta: 3:50:19 batch_cost: 0.3635 data_cost: 0.0407 ips: 22.0062 images/s
- [11/22 15:08:30] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 15:08:30] ppdet.engine INFO: Eval iter: 0
- [11/22 15:08:34] ppdet.engine INFO: Eval iter: 100
- [11/22 15:08:38] ppdet.engine INFO: Eval iter: 200
- [11/22 15:08:42] ppdet.engine INFO: Eval iter: 300
- [11/22 15:08:46] ppdet.engine INFO: Eval iter: 400
- [11/22 15:08:50] ppdet.engine INFO: Eval iter: 500
- [11/22 15:08:54] ppdet.engine INFO: Eval iter: 600
- [11/22 15:08:58] ppdet.engine INFO: Eval iter: 700
- [11/22 15:09:02] ppdet.engine INFO: Eval iter: 800
- [11/22 15:09:06] ppdet.engine INFO: Eval iter: 900
- [11/22 15:09:10] ppdet.engine INFO: Eval iter: 1000
- [11/22 15:09:14] ppdet.engine INFO: Eval iter: 1100
- [11/22 15:09:18] ppdet.engine INFO: Eval iter: 1200
- [11/22 15:09:22] ppdet.engine INFO: Eval iter: 1300
- [11/22 15:09:26] ppdet.engine INFO: Eval iter: 1400
- [11/22 15:09:30] ppdet.engine INFO: Eval iter: 1500
- [11/22 15:09:34] ppdet.engine INFO: Eval iter: 1600
- [11/22 15:09:38] ppdet.engine INFO: Eval iter: 1700
- [11/22 15:09:42] ppdet.engine INFO: Eval iter: 1800
- [11/22 15:09:46] ppdet.engine INFO: Eval iter: 1900
- [11/22 15:09:50] ppdet.engine INFO: Eval iter: 2000
- [11/22 15:09:54] ppdet.engine INFO: Eval iter: 2100
- [11/22 15:09:58] ppdet.engine INFO: Eval iter: 2200
- [11/22 15:10:02] ppdet.engine INFO: Eval iter: 2300
- [11/22 15:10:06] ppdet.engine INFO: Eval iter: 2400
- [11/22 15:10:10] ppdet.engine INFO: Eval iter: 2500
- [11/22 15:10:14] ppdet.engine INFO: Eval iter: 2600
- [11/22 15:10:18] ppdet.engine INFO: Eval iter: 2700
- [11/22 15:10:22] ppdet.engine INFO: Eval iter: 2800
- [11/22 15:10:26] ppdet.engine INFO: Eval iter: 2900
- [11/22 15:10:30] ppdet.engine INFO: Eval iter: 3000
- [11/22 15:10:34] ppdet.engine INFO: Eval iter: 3100
- [11/22 15:10:38] ppdet.engine INFO: Eval iter: 3200
- [11/22 15:10:43] ppdet.engine INFO: Eval iter: 3300
- [11/22 15:10:47] ppdet.engine INFO: Eval iter: 3400
- [11/22 15:10:51] ppdet.engine INFO: Eval iter: 3500
- [11/22 15:10:55] ppdet.engine INFO: Eval iter: 3600
- [11/22 15:10:59] ppdet.engine INFO: Eval iter: 3700
- [11/22 15:11:03] ppdet.engine INFO: Eval iter: 3800
- [11/22 15:11:07] ppdet.engine INFO: Eval iter: 3900
- [11/22 15:11:11] ppdet.engine INFO: Eval iter: 4000
- [11/22 15:11:15] ppdet.engine INFO: Eval iter: 4100
- [11/22 15:11:19] ppdet.engine INFO: Eval iter: 4200
- [11/22 15:11:23] ppdet.engine INFO: Eval iter: 4300
- [11/22 15:11:24] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 15:11:24] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 88.12%
- [11/22 15:11:24] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.81481208698593
- [11/22 15:11:24] ppdet.engine INFO: Best test bbox ap is 0.881.
- [11/22 15:11:28] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 15:11:28] ppdet.engine INFO: Epoch: [29] [ 0/1215] learning_rate: 0.001000 loss_xy: 2.890235 loss_wh: 0.526502 loss_obj: 1.979930 loss_cls: 0.565404 loss: 6.008667 eta: 3:50:14 batch_cost: 0.3607 data_cost: 0.0327 ips: 22.1761 images/s
- [11/22 15:12:42] ppdet.engine INFO: Epoch: [29] [ 200/1215] learning_rate: 0.001000 loss_xy: 2.813323 loss_wh: 0.515029 loss_obj: 1.853236 loss_cls: 0.468016 loss: 5.763780 eta: 3:49:05 batch_cost: 0.3691 data_cost: 0.0037 ips: 21.6728 images/s
- [11/22 15:13:56] ppdet.engine INFO: Epoch: [29] [ 400/1215] learning_rate: 0.001000 loss_xy: 2.947053 loss_wh: 0.542291 loss_obj: 1.944205 loss_cls: 0.433426 loss: 5.960671 eta: 3:47:52 batch_cost: 0.3671 data_cost: 0.0134 ips: 21.7945 images/s
- [11/22 15:15:11] ppdet.engine INFO: Epoch: [29] [ 600/1215] learning_rate: 0.001000 loss_xy: 2.866656 loss_wh: 0.496670 loss_obj: 1.719321 loss_cls: 0.439893 loss: 5.723284 eta: 3:46:49 batch_cost: 0.3727 data_cost: 0.0167 ips: 21.4673 images/s
- [11/22 15:16:25] ppdet.engine INFO: Epoch: [29] [ 800/1215] learning_rate: 0.001000 loss_xy: 2.913915 loss_wh: 0.530617 loss_obj: 1.958437 loss_cls: 0.458597 loss: 5.910068 eta: 3:45:39 batch_cost: 0.3696 data_cost: 0.0022 ips: 21.6464 images/s
- [11/22 15:17:39] ppdet.engine INFO: Epoch: [29] [1000/1215] learning_rate: 0.001000 loss_xy: 3.029404 loss_wh: 0.545731 loss_obj: 1.947728 loss_cls: 0.448753 loss: 6.113163 eta: 3:44:32 batch_cost: 0.3711 data_cost: 0.0279 ips: 21.5552 images/s
- [11/22 15:18:53] ppdet.engine INFO: Epoch: [29] [1200/1215] learning_rate: 0.001000 loss_xy: 3.042196 loss_wh: 0.545417 loss_obj: 1.897175 loss_cls: 0.416654 loss: 6.063687 eta: 3:43:14 batch_cost: 0.3647 data_cost: 0.0152 ips: 21.9340 images/s
- [11/22 15:18:59] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 15:18:59] ppdet.engine INFO: Eval iter: 0
- [11/22 15:19:03] ppdet.engine INFO: Eval iter: 100
- [11/22 15:19:07] ppdet.engine INFO: Eval iter: 200
- [11/22 15:19:11] ppdet.engine INFO: Eval iter: 300
- [11/22 15:19:15] ppdet.engine INFO: Eval iter: 400
- [11/22 15:19:19] ppdet.engine INFO: Eval iter: 500
- [11/22 15:19:23] ppdet.engine INFO: Eval iter: 600
- [11/22 15:19:27] ppdet.engine INFO: Eval iter: 700
- [11/22 15:19:31] ppdet.engine INFO: Eval iter: 800
- [11/22 15:19:35] ppdet.engine INFO: Eval iter: 900
- [11/22 15:19:39] ppdet.engine INFO: Eval iter: 1000
- [11/22 15:19:44] ppdet.engine INFO: Eval iter: 1100
- [11/22 15:19:48] ppdet.engine INFO: Eval iter: 1200
- [11/22 15:19:52] ppdet.engine INFO: Eval iter: 1300
- [11/22 15:19:56] ppdet.engine INFO: Eval iter: 1400
- [11/22 15:20:00] ppdet.engine INFO: Eval iter: 1500
- [11/22 15:20:04] ppdet.engine INFO: Eval iter: 1600
- [11/22 15:20:08] ppdet.engine INFO: Eval iter: 1700
- [11/22 15:20:12] ppdet.engine INFO: Eval iter: 1800
- [11/22 15:20:16] ppdet.engine INFO: Eval iter: 1900
- [11/22 15:20:20] ppdet.engine INFO: Eval iter: 2000
- [11/22 15:20:24] ppdet.engine INFO: Eval iter: 2100
- [11/22 15:20:28] ppdet.engine INFO: Eval iter: 2200
- [11/22 15:20:32] ppdet.engine INFO: Eval iter: 2300
- [11/22 15:20:36] ppdet.engine INFO: Eval iter: 2400
- [11/22 15:20:40] ppdet.engine INFO: Eval iter: 2500
- [11/22 15:20:44] ppdet.engine INFO: Eval iter: 2600
- [11/22 15:20:49] ppdet.engine INFO: Eval iter: 2700
- [11/22 15:20:53] ppdet.engine INFO: Eval iter: 2800
- [11/22 15:20:57] ppdet.engine INFO: Eval iter: 2900
- [11/22 15:21:01] ppdet.engine INFO: Eval iter: 3000
- [11/22 15:21:05] ppdet.engine INFO: Eval iter: 3100
- [11/22 15:21:09] ppdet.engine INFO: Eval iter: 3200
- [11/22 15:21:13] ppdet.engine INFO: Eval iter: 3300
- [11/22 15:21:17] ppdet.engine INFO: Eval iter: 3400
- [11/22 15:21:21] ppdet.engine INFO: Eval iter: 3500
- [11/22 15:21:25] ppdet.engine INFO: Eval iter: 3600
- [11/22 15:21:29] ppdet.engine INFO: Eval iter: 3700
- [11/22 15:21:33] ppdet.engine INFO: Eval iter: 3800
- [11/22 15:21:37] ppdet.engine INFO: Eval iter: 3900
- [11/22 15:21:42] ppdet.engine INFO: Eval iter: 4000
- [11/22 15:21:46] ppdet.engine INFO: Eval iter: 4100
- [11/22 15:21:50] ppdet.engine INFO: Eval iter: 4200
- [11/22 15:21:54] ppdet.engine INFO: Eval iter: 4300
- [11/22 15:21:55] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 15:21:55] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 88.49%
- [11/22 15:21:55] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.606551617446254
- [11/22 15:21:55] ppdet.engine INFO: Best test bbox ap is 0.885.
- [11/22 15:21:59] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 15:21:59] ppdet.engine INFO: Epoch: [30] [ 0/1215] learning_rate: 0.001000 loss_xy: 3.062106 loss_wh: 0.557204 loss_obj: 1.981659 loss_cls: 0.420926 loss: 6.192420 eta: 3:43:09 batch_cost: 0.3652 data_cost: 0.0142 ips: 21.9065 images/s
- [11/22 15:23:11] ppdet.engine INFO: Epoch: [30] [ 200/1215] learning_rate: 0.001000 loss_xy: 2.958700 loss_wh: 0.526691 loss_obj: 1.679857 loss_cls: 0.428477 loss: 5.765636 eta: 3:41:45 batch_cost: 0.3602 data_cost: 0.0279 ips: 22.2095 images/s
- [11/22 15:24:26] ppdet.engine INFO: Epoch: [30] [ 400/1215] learning_rate: 0.001000 loss_xy: 2.766924 loss_wh: 0.466328 loss_obj: 1.704718 loss_cls: 0.407718 loss: 5.437303 eta: 3:40:38 batch_cost: 0.3719 data_cost: 0.0235 ips: 21.5131 images/s
- [11/22 15:25:38] ppdet.engine INFO: Epoch: [30] [ 600/1215] learning_rate: 0.001000 loss_xy: 2.928989 loss_wh: 0.535433 loss_obj: 1.843856 loss_cls: 0.492715 loss: 5.949455 eta: 3:39:12 batch_cost: 0.3578 data_cost: 0.0221 ips: 22.3588 images/s
- [11/22 15:26:53] ppdet.engine INFO: Epoch: [30] [ 800/1215] learning_rate: 0.001000 loss_xy: 2.849595 loss_wh: 0.530350 loss_obj: 1.797164 loss_cls: 0.435509 loss: 5.727885 eta: 3:38:07 batch_cost: 0.3734 data_cost: 0.0261 ips: 21.4271 images/s
- [11/22 15:28:06] ppdet.engine INFO: Epoch: [30] [1000/1215] learning_rate: 0.001000 loss_xy: 2.707886 loss_wh: 0.476222 loss_obj: 1.438630 loss_cls: 0.390370 loss: 5.230768 eta: 3:36:53 batch_cost: 0.3662 data_cost: 0.0160 ips: 21.8468 images/s
- [11/22 15:29:19] ppdet.engine INFO: Epoch: [30] [1200/1215] learning_rate: 0.001000 loss_xy: 3.088341 loss_wh: 0.578901 loss_obj: 1.705412 loss_cls: 0.449099 loss: 6.006550 eta: 3:35:32 batch_cost: 0.3606 data_cost: 0.0131 ips: 22.1864 images/s
- [11/22 15:29:25] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 15:29:25] ppdet.engine INFO: Eval iter: 0
- [11/22 15:29:29] ppdet.engine INFO: Eval iter: 100
- [11/22 15:29:33] ppdet.engine INFO: Eval iter: 200
- [11/22 15:29:37] ppdet.engine INFO: Eval iter: 300
- [11/22 15:29:41] ppdet.engine INFO: Eval iter: 400
- [11/22 15:29:45] ppdet.engine INFO: Eval iter: 500
- [11/22 15:29:49] ppdet.engine INFO: Eval iter: 600
- [11/22 15:29:53] ppdet.engine INFO: Eval iter: 700
- [11/22 15:29:57] ppdet.engine INFO: Eval iter: 800
- [11/22 15:30:01] ppdet.engine INFO: Eval iter: 900
- [11/22 15:30:05] ppdet.engine INFO: Eval iter: 1000
- [11/22 15:30:09] ppdet.engine INFO: Eval iter: 1100
- [11/22 15:30:13] ppdet.engine INFO: Eval iter: 1200
- [11/22 15:30:17] ppdet.engine INFO: Eval iter: 1300
- [11/22 15:30:21] ppdet.engine INFO: Eval iter: 1400
- [11/22 15:30:25] ppdet.engine INFO: Eval iter: 1500
- [11/22 15:30:30] ppdet.engine INFO: Eval iter: 1600
- [11/22 15:30:34] ppdet.engine INFO: Eval iter: 1700
- [11/22 15:30:38] ppdet.engine INFO: Eval iter: 1800
- [11/22 15:30:42] ppdet.engine INFO: Eval iter: 1900
- [11/22 15:30:46] ppdet.engine INFO: Eval iter: 2000
- [11/22 15:30:50] ppdet.engine INFO: Eval iter: 2100
- [11/22 15:30:54] ppdet.engine INFO: Eval iter: 2200
- [11/22 15:30:58] ppdet.engine INFO: Eval iter: 2300
- [11/22 15:31:02] ppdet.engine INFO: Eval iter: 2400
- [11/22 15:31:06] ppdet.engine INFO: Eval iter: 2500
- [11/22 15:31:10] ppdet.engine INFO: Eval iter: 2600
- [11/22 15:31:14] ppdet.engine INFO: Eval iter: 2700
- [11/22 15:31:18] ppdet.engine INFO: Eval iter: 2800
- [11/22 15:31:22] ppdet.engine INFO: Eval iter: 2900
- [11/22 15:31:26] ppdet.engine INFO: Eval iter: 3000
- [11/22 15:31:30] ppdet.engine INFO: Eval iter: 3100
- [11/22 15:31:34] ppdet.engine INFO: Eval iter: 3200
- [11/22 15:31:38] ppdet.engine INFO: Eval iter: 3300
- [11/22 15:31:42] ppdet.engine INFO: Eval iter: 3400
- [11/22 15:31:46] ppdet.engine INFO: Eval iter: 3500
- [11/22 15:31:50] ppdet.engine INFO: Eval iter: 3600
- [11/22 15:31:54] ppdet.engine INFO: Eval iter: 3700
- [11/22 15:31:58] ppdet.engine INFO: Eval iter: 3800
- [11/22 15:32:02] ppdet.engine INFO: Eval iter: 3900
- [11/22 15:32:07] ppdet.engine INFO: Eval iter: 4000
- [11/22 15:32:11] ppdet.engine INFO: Eval iter: 4100
- [11/22 15:32:15] ppdet.engine INFO: Eval iter: 4200
- [11/22 15:32:19] ppdet.engine INFO: Eval iter: 4300
- [11/22 15:32:20] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 15:32:20] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 87.31%
- [11/22 15:32:20] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.681600342797353
- [11/22 15:32:20] ppdet.engine INFO: Best test bbox ap is 0.885.
- [11/22 15:32:20] ppdet.engine INFO: Epoch: [31] [ 0/1215] learning_rate: 0.001000 loss_xy: 3.101678 loss_wh: 0.567089 loss_obj: 1.705412 loss_cls: 0.446714 loss: 6.006550 eta: 3:35:25 batch_cost: 0.3594 data_cost: 0.0133 ips: 22.2565 images/s
- [11/22 15:33:35] ppdet.engine INFO: Epoch: [31] [ 200/1215] learning_rate: 0.001000 loss_xy: 2.999088 loss_wh: 0.527786 loss_obj: 2.001014 loss_cls: 0.483575 loss: 6.152457 eta: 3:34:16 batch_cost: 0.3708 data_cost: 0.0694 ips: 21.5751 images/s
- [11/22 15:34:50] ppdet.engine INFO: Epoch: [31] [ 400/1215] learning_rate: 0.001000 loss_xy: 2.929334 loss_wh: 0.516122 loss_obj: 2.032635 loss_cls: 0.463068 loss: 5.898123 eta: 3:33:09 batch_cost: 0.3727 data_cost: 0.0291 ips: 21.4658 images/s
- [11/22 15:36:04] ppdet.engine INFO: Epoch: [31] [ 600/1215] learning_rate: 0.001000 loss_xy: 2.920765 loss_wh: 0.523389 loss_obj: 2.055344 loss_cls: 0.498977 loss: 6.090296 eta: 3:31:57 batch_cost: 0.3687 data_cost: 0.0181 ips: 21.6950 images/s
- [11/22 15:37:16] ppdet.engine INFO: Epoch: [31] [ 800/1215] learning_rate: 0.001000 loss_xy: 2.901612 loss_wh: 0.498874 loss_obj: 2.009619 loss_cls: 0.441640 loss: 5.934584 eta: 3:30:36 batch_cost: 0.3596 data_cost: 0.0206 ips: 22.2468 images/s
- [11/22 15:38:30] ppdet.engine INFO: Epoch: [31] [1000/1215] learning_rate: 0.001000 loss_xy: 2.959753 loss_wh: 0.549688 loss_obj: 2.060503 loss_cls: 0.490507 loss: 6.221950 eta: 3:29:27 batch_cost: 0.3710 data_cost: 0.0074 ips: 21.5626 images/s
- [11/22 15:39:46] ppdet.engine INFO: Epoch: [31] [1200/1215] learning_rate: 0.001000 loss_xy: 3.004926 loss_wh: 0.540530 loss_obj: 2.114811 loss_cls: 0.479719 loss: 6.261750 eta: 3:28:24 batch_cost: 0.3790 data_cost: 0.0435 ips: 21.1080 images/s
- [11/22 15:39:52] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 15:39:52] ppdet.engine INFO: Eval iter: 0
- [11/22 15:39:56] ppdet.engine INFO: Eval iter: 100
- [11/22 15:40:00] ppdet.engine INFO: Eval iter: 200
- [11/22 15:40:04] ppdet.engine INFO: Eval iter: 300
- [11/22 15:40:08] ppdet.engine INFO: Eval iter: 400
- [11/22 15:40:13] ppdet.engine INFO: Eval iter: 500
- [11/22 15:40:17] ppdet.engine INFO: Eval iter: 600
- [11/22 15:40:21] ppdet.engine INFO: Eval iter: 700
- [11/22 15:40:25] ppdet.engine INFO: Eval iter: 800
- [11/22 15:40:29] ppdet.engine INFO: Eval iter: 900
- [11/22 15:40:33] ppdet.engine INFO: Eval iter: 1000
- [11/22 15:40:37] ppdet.engine INFO: Eval iter: 1100
- [11/22 15:40:41] ppdet.engine INFO: Eval iter: 1200
- [11/22 15:40:45] ppdet.engine INFO: Eval iter: 1300
- [11/22 15:40:50] ppdet.engine INFO: Eval iter: 1400
- [11/22 15:40:54] ppdet.engine INFO: Eval iter: 1500
- [11/22 15:40:58] ppdet.engine INFO: Eval iter: 1600
- [11/22 15:41:02] ppdet.engine INFO: Eval iter: 1700
- [11/22 15:41:06] ppdet.engine INFO: Eval iter: 1800
- [11/22 15:41:10] ppdet.engine INFO: Eval iter: 1900
- [11/22 15:41:14] ppdet.engine INFO: Eval iter: 2000
- [11/22 15:41:18] ppdet.engine INFO: Eval iter: 2100
- [11/22 15:41:22] ppdet.engine INFO: Eval iter: 2200
- [11/22 15:41:26] ppdet.engine INFO: Eval iter: 2300
- [11/22 15:41:30] ppdet.engine INFO: Eval iter: 2400
- [11/22 15:41:34] ppdet.engine INFO: Eval iter: 2500
- [11/22 15:41:38] ppdet.engine INFO: Eval iter: 2600
- [11/22 15:41:43] ppdet.engine INFO: Eval iter: 2700
- [11/22 15:41:47] ppdet.engine INFO: Eval iter: 2800
- [11/22 15:41:51] ppdet.engine INFO: Eval iter: 2900
- [11/22 15:41:55] ppdet.engine INFO: Eval iter: 3000
- [11/22 15:41:59] ppdet.engine INFO: Eval iter: 3100
- [11/22 15:42:03] ppdet.engine INFO: Eval iter: 3200
- [11/22 15:42:07] ppdet.engine INFO: Eval iter: 3300
- [11/22 15:42:11] ppdet.engine INFO: Eval iter: 3400
- [11/22 15:42:15] ppdet.engine INFO: Eval iter: 3500
- [11/22 15:42:19] ppdet.engine INFO: Eval iter: 3600
- [11/22 15:42:23] ppdet.engine INFO: Eval iter: 3700
- [11/22 15:42:27] ppdet.engine INFO: Eval iter: 3800
- [11/22 15:42:31] ppdet.engine INFO: Eval iter: 3900
- [11/22 15:42:35] ppdet.engine INFO: Eval iter: 4000
- [11/22 15:42:39] ppdet.engine INFO: Eval iter: 4100
- [11/22 15:42:44] ppdet.engine INFO: Eval iter: 4200
- [11/22 15:42:48] ppdet.engine INFO: Eval iter: 4300
- [11/22 15:42:49] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 15:42:49] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 88.57%
- [11/22 15:42:49] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.468984671115646
- [11/22 15:42:49] ppdet.engine INFO: Best test bbox ap is 0.886.
- [11/22 15:42:52] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 15:42:53] ppdet.engine INFO: Epoch: [32] [ 0/1215] learning_rate: 0.001000 loss_xy: 2.993201 loss_wh: 0.537813 loss_obj: 2.178911 loss_cls: 0.474981 loss: 6.251920 eta: 3:28:16 batch_cost: 0.3738 data_cost: 0.0456 ips: 21.4001 images/s
- [11/22 15:44:07] ppdet.engine INFO: Epoch: [32] [ 200/1215] learning_rate: 0.001000 loss_xy: 2.919894 loss_wh: 0.512570 loss_obj: 2.081382 loss_cls: 0.486753 loss: 6.060265 eta: 3:27:04 batch_cost: 0.3694 data_cost: 0.0276 ips: 21.6571 images/s
- [11/22 15:45:23] ppdet.engine INFO: Epoch: [32] [ 400/1215] learning_rate: 0.001000 loss_xy: 3.038823 loss_wh: 0.555772 loss_obj: 2.175719 loss_cls: 0.452670 loss: 6.271052 eta: 3:26:02 batch_cost: 0.3804 data_cost: 0.0218 ips: 21.0325 images/s
- [11/22 15:46:37] ppdet.engine INFO: Epoch: [32] [ 600/1215] learning_rate: 0.001000 loss_xy: 3.048429 loss_wh: 0.553949 loss_obj: 2.235826 loss_cls: 0.480535 loss: 6.548030 eta: 3:24:48 batch_cost: 0.3664 data_cost: 0.0009 ips: 21.8328 images/s
- [11/22 15:47:49] ppdet.engine INFO: Epoch: [32] [ 800/1215] learning_rate: 0.001000 loss_xy: 2.969094 loss_wh: 0.519589 loss_obj: 2.088734 loss_cls: 0.510143 loss: 6.023759 eta: 3:23:29 batch_cost: 0.3608 data_cost: 0.0129 ips: 22.1723 images/s
- [11/22 15:49:03] ppdet.engine INFO: Epoch: [32] [1000/1215] learning_rate: 0.001000 loss_xy: 3.082748 loss_wh: 0.546118 loss_obj: 2.104764 loss_cls: 0.412480 loss: 6.366436 eta: 3:22:15 batch_cost: 0.3677 data_cost: 0.0127 ips: 21.7568 images/s
- [11/22 15:50:16] ppdet.engine INFO: Epoch: [32] [1200/1215] learning_rate: 0.001000 loss_xy: 2.991449 loss_wh: 0.529073 loss_obj: 2.047304 loss_cls: 0.451543 loss: 6.056098 eta: 3:20:58 batch_cost: 0.3627 data_cost: 0.0137 ips: 22.0582 images/s
- [11/22 15:50:22] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 15:50:22] ppdet.engine INFO: Eval iter: 0
- [11/22 15:50:27] ppdet.engine INFO: Eval iter: 100
- [11/22 15:50:31] ppdet.engine INFO: Eval iter: 200
- [11/22 15:50:35] ppdet.engine INFO: Eval iter: 300
- [11/22 15:50:39] ppdet.engine INFO: Eval iter: 400
- [11/22 15:50:43] ppdet.engine INFO: Eval iter: 500
- [11/22 15:50:47] ppdet.engine INFO: Eval iter: 600
- [11/22 15:50:51] ppdet.engine INFO: Eval iter: 700
- [11/22 15:50:55] ppdet.engine INFO: Eval iter: 800
- [11/22 15:50:59] ppdet.engine INFO: Eval iter: 900
- [11/22 15:51:03] ppdet.engine INFO: Eval iter: 1000
- [11/22 15:51:07] ppdet.engine INFO: Eval iter: 1100
- [11/22 15:51:11] ppdet.engine INFO: Eval iter: 1200
- [11/22 15:51:15] ppdet.engine INFO: Eval iter: 1300
- [11/22 15:51:19] ppdet.engine INFO: Eval iter: 1400
- [11/22 15:51:23] ppdet.engine INFO: Eval iter: 1500
- [11/22 15:51:27] ppdet.engine INFO: Eval iter: 1600
- [11/22 15:51:31] ppdet.engine INFO: Eval iter: 1700
- [11/22 15:51:35] ppdet.engine INFO: Eval iter: 1800
- [11/22 15:51:39] ppdet.engine INFO: Eval iter: 1900
- [11/22 15:51:43] ppdet.engine INFO: Eval iter: 2000
- [11/22 15:51:47] ppdet.engine INFO: Eval iter: 2100
- [11/22 15:51:51] ppdet.engine INFO: Eval iter: 2200
- [11/22 15:51:55] ppdet.engine INFO: Eval iter: 2300
- [11/22 15:52:00] ppdet.engine INFO: Eval iter: 2400
- [11/22 15:52:04] ppdet.engine INFO: Eval iter: 2500
- [11/22 15:52:07] ppdet.engine INFO: Eval iter: 2600
- [11/22 15:52:11] ppdet.engine INFO: Eval iter: 2700
- [11/22 15:52:16] ppdet.engine INFO: Eval iter: 2800
- [11/22 15:52:20] ppdet.engine INFO: Eval iter: 2900
- [11/22 15:52:24] ppdet.engine INFO: Eval iter: 3000
- [11/22 15:52:28] ppdet.engine INFO: Eval iter: 3100
- [11/22 15:52:32] ppdet.engine INFO: Eval iter: 3200
- [11/22 15:52:36] ppdet.engine INFO: Eval iter: 3300
- [11/22 15:52:40] ppdet.engine INFO: Eval iter: 3400
- [11/22 15:52:44] ppdet.engine INFO: Eval iter: 3500
- [11/22 15:52:48] ppdet.engine INFO: Eval iter: 3600
- [11/22 15:52:52] ppdet.engine INFO: Eval iter: 3700
- [11/22 15:52:56] ppdet.engine INFO: Eval iter: 3800
- [11/22 15:53:00] ppdet.engine INFO: Eval iter: 3900
- [11/22 15:53:04] ppdet.engine INFO: Eval iter: 4000
- [11/22 15:53:08] ppdet.engine INFO: Eval iter: 4100
- [11/22 15:53:13] ppdet.engine INFO: Eval iter: 4200
- [11/22 15:53:17] ppdet.engine INFO: Eval iter: 4300
- [11/22 15:53:18] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 15:53:18] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 88.27%
- [11/22 15:53:18] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.65692275415189
- [11/22 15:53:18] ppdet.engine INFO: Best test bbox ap is 0.886.
- [11/22 15:53:18] ppdet.engine INFO: Epoch: [33] [ 0/1215] learning_rate: 0.001000 loss_xy: 2.998051 loss_wh: 0.524304 loss_obj: 2.066272 loss_cls: 0.451543 loss: 6.071773 eta: 3:20:52 batch_cost: 0.3624 data_cost: 0.0137 ips: 22.0735 images/s
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::BasicEngine::Execute()
- 1 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVariableWrapperMap const&, paddle::imperative::NameVariableWrapperMap const&, paddle::framework::AttributeMap const&, paddle::framework::AttributeMap const&)
- 2 phi::KernelImpl<void (*)(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*), &(void phi::ConvCudnnGradKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*))>::Compute(phi::KernelContext*)
- 3 void phi::ConvCudnnGradKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669103620 (unix time) try "date -d @1669103620" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e800009a4e) received by PID 39583 (TID 0x7f6d2119f700) from PID 39502 ***]
-
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:57529', '127.0.0.1:54665', '127.0.0.1:35961']
- I1122 15:54:11.295094 49968 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1122 15:54:12.072072 49968 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1122 15:54:12.075454 49968 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/22 15:54:13] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- [11/22 15:54:13] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/22 15:54:14] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- [11/22 15:54:14] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- [11/22 15:54:16] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- [11/22 15:54:16] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- [11/22 15:54:17] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- [11/22 15:54:17] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- [11/22 15:54:17] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- [11/22 15:54:18] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- [11/22 15:54:18] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/22 15:54:19] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- [11/22 15:54:19] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- [11/22 15:54:19] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/22 15:54:20] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- [11/22 15:54:20] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- [11/22 15:54:22] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/22 15:54:24] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/32.pdparams
- [11/22 15:54:27] ppdet.engine INFO: Epoch: [33] [ 0/1215] learning_rate: 0.001000 loss_xy: 4.756993 loss_wh: 1.136884 loss_obj: 3.546198 loss_cls: 0.713581 loss: 10.153656 eta: 20:52:11 batch_cost: 2.2902 data_cost: 0.0006 ips: 3.4931 images/s
- [11/22 15:55:38] ppdet.engine INFO: Epoch: [33] [ 200/1215] learning_rate: 0.000600 loss_xy: 2.850090 loss_wh: 0.493709 loss_obj: 1.747190 loss_cls: 0.379438 loss: 5.573355 eta: 3:19:17 batch_cost: 0.3571 data_cost: 0.0156 ips: 22.4017 images/s
- [11/22 15:56:53] ppdet.engine INFO: Epoch: [33] [ 400/1215] learning_rate: 0.000600 loss_xy: 2.796171 loss_wh: 0.462800 loss_obj: 1.640987 loss_cls: 0.376587 loss: 5.453889 eta: 3:19:16 batch_cost: 0.3712 data_cost: 0.0119 ips: 21.5517 images/s
- [11/22 15:58:06] ppdet.engine INFO: Epoch: [33] [ 600/1215] learning_rate: 0.000600 loss_xy: 2.857778 loss_wh: 0.471169 loss_obj: 1.526320 loss_cls: 0.359192 loss: 5.182627 eta: 3:17:28 batch_cost: 0.3658 data_cost: 0.0221 ips: 21.8697 images/s
- [11/22 15:59:19] ppdet.engine INFO: Epoch: [33] [ 800/1215] learning_rate: 0.000600 loss_xy: 2.813833 loss_wh: 0.465194 loss_obj: 1.405106 loss_cls: 0.342342 loss: 5.008584 eta: 3:15:44 batch_cost: 0.3641 data_cost: 0.0370 ips: 21.9711 images/s
- [11/22 16:00:34] ppdet.engine INFO: Epoch: [33] [1000/1215] learning_rate: 0.000600 loss_xy: 2.914004 loss_wh: 0.486179 loss_obj: 1.441943 loss_cls: 0.360289 loss: 5.386833 eta: 3:14:43 batch_cost: 0.3690 data_cost: 0.0255 ips: 21.6829 images/s
- [11/22 16:01:46] ppdet.engine INFO: Epoch: [33] [1200/1215] learning_rate: 0.000600 loss_xy: 2.774689 loss_wh: 0.468303 loss_obj: 1.419216 loss_cls: 0.368051 loss: 5.246994 eta: 3:12:58 batch_cost: 0.3613 data_cost: 0.0176 ips: 22.1429 images/s
- [11/22 16:01:53] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 16:01:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_2.xml, x1: 217.0, y1: 763.0, x2: 217.0, y2: 764.0.
- [11/22 16:01:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_8.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/22 16:01:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_881_3.xml, x1: 35.0, y1: 198.0, x2: 75.0, y2: 197.0.
- [11/22 16:01:54] ppdet.engine INFO: Eval iter: 0
- [11/22 16:01:58] ppdet.engine INFO: Eval iter: 100
- [11/22 16:02:03] ppdet.engine INFO: Eval iter: 200
- [11/22 16:02:07] ppdet.engine INFO: Eval iter: 300
- [11/22 16:02:11] ppdet.engine INFO: Eval iter: 400
- [11/22 16:02:15] ppdet.engine INFO: Eval iter: 500
- [11/22 16:02:19] ppdet.engine INFO: Eval iter: 600
- [11/22 16:02:23] ppdet.engine INFO: Eval iter: 700
- [11/22 16:02:27] ppdet.engine INFO: Eval iter: 800
- [11/22 16:02:31] ppdet.engine INFO: Eval iter: 900
- [11/22 16:02:35] ppdet.engine INFO: Eval iter: 1000
- [11/22 16:02:39] ppdet.engine INFO: Eval iter: 1100
- [11/22 16:02:43] ppdet.engine INFO: Eval iter: 1200
- [11/22 16:02:46] ppdet.engine INFO: Eval iter: 1300
- [11/22 16:02:50] ppdet.engine INFO: Eval iter: 1400
- [11/22 16:02:54] ppdet.engine INFO: Eval iter: 1500
- [11/22 16:02:58] ppdet.engine INFO: Eval iter: 1600
- [11/22 16:03:02] ppdet.engine INFO: Eval iter: 1700
- [11/22 16:03:06] ppdet.engine INFO: Eval iter: 1800
- [11/22 16:03:10] ppdet.engine INFO: Eval iter: 1900
- [11/22 16:03:14] ppdet.engine INFO: Eval iter: 2000
- [11/22 16:03:18] ppdet.engine INFO: Eval iter: 2100
- [11/22 16:03:22] ppdet.engine INFO: Eval iter: 2200
- [11/22 16:03:27] ppdet.engine INFO: Eval iter: 2300
- [11/22 16:03:31] ppdet.engine INFO: Eval iter: 2400
- [11/22 16:03:35] ppdet.engine INFO: Eval iter: 2500
- [11/22 16:03:39] ppdet.engine INFO: Eval iter: 2600
- [11/22 16:03:43] ppdet.engine INFO: Eval iter: 2700
- [11/22 16:03:47] ppdet.engine INFO: Eval iter: 2800
- [11/22 16:03:51] ppdet.engine INFO: Eval iter: 2900
- [11/22 16:03:55] ppdet.engine INFO: Eval iter: 3000
- [11/22 16:03:59] ppdet.engine INFO: Eval iter: 3100
- [11/22 16:04:03] ppdet.engine INFO: Eval iter: 3200
- [11/22 16:04:07] ppdet.engine INFO: Eval iter: 3300
- [11/22 16:04:11] ppdet.engine INFO: Eval iter: 3400
- [11/22 16:04:15] ppdet.engine INFO: Eval iter: 3500
- [11/22 16:04:19] ppdet.engine INFO: Eval iter: 3600
- [11/22 16:04:23] ppdet.engine INFO: Eval iter: 3700
- [11/22 16:04:26] ppdet.engine INFO: Eval iter: 3800
- [11/22 16:04:30] ppdet.engine INFO: Eval iter: 3900
- [11/22 16:04:34] ppdet.engine INFO: Eval iter: 4000
- [11/22 16:04:38] ppdet.engine INFO: Eval iter: 4100
- [11/22 16:04:42] ppdet.engine INFO: Eval iter: 4200
- [11/22 16:04:46] ppdet.engine INFO: Eval iter: 4300
- [11/22 16:04:47] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 16:04:47] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 88.90%
- [11/22 16:04:47] ppdet.engine INFO: Total sample number: 4322, averge FPS: 25.015078518147043
- [11/22 16:04:47] ppdet.engine INFO: Best test bbox ap is 0.889.
- [11/22 16:04:51] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 16:04:51] ppdet.engine INFO: Epoch: [34] [ 0/1215] learning_rate: 0.000600 loss_xy: 2.884902 loss_wh: 0.471645 loss_obj: 1.346317 loss_cls: 0.373805 loss: 5.241834 eta: 3:12:58 batch_cost: 0.3637 data_cost: 0.0224 ips: 21.9975 images/s
- [11/22 16:06:07] ppdet.engine INFO: Epoch: [34] [ 200/1215] learning_rate: 0.000600 loss_xy: 2.777908 loss_wh: 0.462670 loss_obj: 1.431625 loss_cls: 0.402149 loss: 5.144156 eta: 3:12:46 batch_cost: 0.3802 data_cost: 0.0155 ips: 21.0397 images/s
- [11/22 16:07:25] ppdet.engine INFO: Epoch: [34] [ 400/1215] learning_rate: 0.000600 loss_xy: 2.849908 loss_wh: 0.467346 loss_obj: 1.448280 loss_cls: 0.344219 loss: 5.249464 eta: 3:12:50 batch_cost: 0.3886 data_cost: 0.0323 ips: 20.5855 images/s
- [11/22 16:08:42] ppdet.engine INFO: Epoch: [34] [ 600/1215] learning_rate: 0.000600 loss_xy: 2.993378 loss_wh: 0.495236 loss_obj: 1.558689 loss_cls: 0.351341 loss: 5.557890 eta: 3:12:13 batch_cost: 0.3819 data_cost: 0.0135 ips: 20.9466 images/s
- [11/22 16:09:57] ppdet.engine INFO: Epoch: [34] [ 800/1215] learning_rate: 0.000600 loss_xy: 2.949233 loss_wh: 0.480606 loss_obj: 1.388230 loss_cls: 0.321013 loss: 5.216791 eta: 3:10:58 batch_cost: 0.3719 data_cost: 0.0284 ips: 21.5105 images/s
- [11/22 16:11:11] ppdet.engine INFO: Epoch: [34] [1000/1215] learning_rate: 0.000600 loss_xy: 3.055724 loss_wh: 0.528439 loss_obj: 1.471105 loss_cls: 0.387565 loss: 5.590464 eta: 3:09:37 batch_cost: 0.3698 data_cost: 0.0452 ips: 21.6362 images/s
- [11/22 16:12:25] ppdet.engine INFO: Epoch: [34] [1200/1215] learning_rate: 0.000600 loss_xy: 2.893901 loss_wh: 0.465954 loss_obj: 1.451095 loss_cls: 0.349516 loss: 5.285330 eta: 3:08:16 batch_cost: 0.3693 data_cost: 0.0112 ips: 21.6643 images/s
- [11/22 16:12:31] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 16:12:31] ppdet.engine INFO: Eval iter: 0
- [11/22 16:12:35] ppdet.engine INFO: Eval iter: 100
- [11/22 16:12:39] ppdet.engine INFO: Eval iter: 200
- [11/22 16:12:44] ppdet.engine INFO: Eval iter: 300
- [11/22 16:12:47] ppdet.engine INFO: Eval iter: 400
- [11/22 16:12:52] ppdet.engine INFO: Eval iter: 500
- [11/22 16:12:56] ppdet.engine INFO: Eval iter: 600
- [11/22 16:13:00] ppdet.engine INFO: Eval iter: 700
- [11/22 16:13:04] ppdet.engine INFO: Eval iter: 800
- [11/22 16:13:07] ppdet.engine INFO: Eval iter: 900
- [11/22 16:13:11] ppdet.engine INFO: Eval iter: 1000
- [11/22 16:13:15] ppdet.engine INFO: Eval iter: 1100
- [11/22 16:13:19] ppdet.engine INFO: Eval iter: 1200
- [11/22 16:13:23] ppdet.engine INFO: Eval iter: 1300
- [11/22 16:13:27] ppdet.engine INFO: Eval iter: 1400
- [11/22 16:13:31] ppdet.engine INFO: Eval iter: 1500
- [11/22 16:13:35] ppdet.engine INFO: Eval iter: 1600
- [11/22 16:13:39] ppdet.engine INFO: Eval iter: 1700
- [11/22 16:13:43] ppdet.engine INFO: Eval iter: 1800
- [11/22 16:13:47] ppdet.engine INFO: Eval iter: 1900
- [11/22 16:13:51] ppdet.engine INFO: Eval iter: 2000
- [11/22 16:13:55] ppdet.engine INFO: Eval iter: 2100
- [11/22 16:14:00] ppdet.engine INFO: Eval iter: 2200
- [11/22 16:14:03] ppdet.engine INFO: Eval iter: 2300
- [11/22 16:14:07] ppdet.engine INFO: Eval iter: 2400
- [11/22 16:14:11] ppdet.engine INFO: Eval iter: 2500
- [11/22 16:14:15] ppdet.engine INFO: Eval iter: 2600
- [11/22 16:14:19] ppdet.engine INFO: Eval iter: 2700
- [11/22 16:14:23] ppdet.engine INFO: Eval iter: 2800
- [11/22 16:14:28] ppdet.engine INFO: Eval iter: 2900
- [11/22 16:14:32] ppdet.engine INFO: Eval iter: 3000
- [11/22 16:14:37] ppdet.engine INFO: Eval iter: 3100
- [11/22 16:14:41] ppdet.engine INFO: Eval iter: 3200
- [11/22 16:14:45] ppdet.engine INFO: Eval iter: 3300
- [11/22 16:14:49] ppdet.engine INFO: Eval iter: 3400
- [11/22 16:14:53] ppdet.engine INFO: Eval iter: 3500
- [11/22 16:14:57] ppdet.engine INFO: Eval iter: 3600
- [11/22 16:15:01] ppdet.engine INFO: Eval iter: 3700
- [11/22 16:15:05] ppdet.engine INFO: Eval iter: 3800
- [11/22 16:15:09] ppdet.engine INFO: Eval iter: 3900
- [11/22 16:15:13] ppdet.engine INFO: Eval iter: 4000
- [11/22 16:15:18] ppdet.engine INFO: Eval iter: 4100
- [11/22 16:15:22] ppdet.engine INFO: Eval iter: 4200
- [11/22 16:15:26] ppdet.engine INFO: Eval iter: 4300
- [11/22 16:15:27] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 16:15:27] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 88.55%
- [11/22 16:15:27] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.622251185046004
- [11/22 16:15:27] ppdet.engine INFO: Best test bbox ap is 0.889.
- [11/22 16:15:27] ppdet.engine INFO: Epoch: [35] [ 0/1215] learning_rate: 0.000600 loss_xy: 2.941689 loss_wh: 0.471325 loss_obj: 1.436371 loss_cls: 0.360015 loss: 5.285330 eta: 3:08:06 batch_cost: 0.3680 data_cost: 0.0123 ips: 21.7392 images/s
- [11/22 16:16:41] ppdet.engine INFO: Epoch: [35] [ 200/1215] learning_rate: 0.000600 loss_xy: 3.016262 loss_wh: 0.507664 loss_obj: 1.648500 loss_cls: 0.343616 loss: 5.797842 eta: 3:06:49 batch_cost: 0.3702 data_cost: 0.0288 ips: 21.6073 images/s
- [11/22 16:17:56] ppdet.engine INFO: Epoch: [35] [ 400/1215] learning_rate: 0.000600 loss_xy: 2.976949 loss_wh: 0.509849 loss_obj: 1.452898 loss_cls: 0.337603 loss: 5.351853 eta: 3:05:36 batch_cost: 0.3723 data_cost: 0.0064 ips: 21.4879 images/s
- [11/22 16:19:09] ppdet.engine INFO: Epoch: [35] [ 600/1215] learning_rate: 0.000600 loss_xy: 2.971060 loss_wh: 0.489161 loss_obj: 1.328299 loss_cls: 0.347722 loss: 5.303437 eta: 3:04:07 batch_cost: 0.3639 data_cost: 0.0340 ips: 21.9865 images/s
- [11/22 16:20:21] ppdet.engine INFO: Epoch: [35] [ 800/1215] learning_rate: 0.000600 loss_xy: 2.774609 loss_wh: 0.471628 loss_obj: 1.214364 loss_cls: 0.331950 loss: 5.013632 eta: 3:02:28 batch_cost: 0.3575 data_cost: 0.0118 ips: 22.3756 images/s
- [11/22 16:21:36] ppdet.engine INFO: Epoch: [35] [1000/1215] learning_rate: 0.000600 loss_xy: 3.057374 loss_wh: 0.507365 loss_obj: 1.512011 loss_cls: 0.369127 loss: 5.664075 eta: 3:01:18 batch_cost: 0.3727 data_cost: 0.0315 ips: 21.4678 images/s
- [11/22 16:22:49] ppdet.engine INFO: Epoch: [35] [1200/1215] learning_rate: 0.000600 loss_xy: 2.899421 loss_wh: 0.483992 loss_obj: 1.272325 loss_cls: 0.363271 loss: 5.169531 eta: 2:59:54 batch_cost: 0.3641 data_cost: 0.0355 ips: 21.9725 images/s
- [11/22 16:22:55] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 16:22:55] ppdet.engine INFO: Eval iter: 0
- [11/22 16:23:01] ppdet.engine INFO: Eval iter: 100
- [11/22 16:23:05] ppdet.engine INFO: Eval iter: 200
- [11/22 16:23:09] ppdet.engine INFO: Eval iter: 300
- [11/22 16:23:13] ppdet.engine INFO: Eval iter: 400
- [11/22 16:23:17] ppdet.engine INFO: Eval iter: 500
- [11/22 16:23:21] ppdet.engine INFO: Eval iter: 600
- [11/22 16:23:25] ppdet.engine INFO: Eval iter: 700
- [11/22 16:23:29] ppdet.engine INFO: Eval iter: 800
- [11/22 16:23:33] ppdet.engine INFO: Eval iter: 900
- [11/22 16:23:37] ppdet.engine INFO: Eval iter: 1000
- [11/22 16:23:41] ppdet.engine INFO: Eval iter: 1100
- [11/22 16:23:45] ppdet.engine INFO: Eval iter: 1200
- [11/22 16:23:50] ppdet.engine INFO: Eval iter: 1300
- [11/22 16:23:54] ppdet.engine INFO: Eval iter: 1400
- [11/22 16:23:58] ppdet.engine INFO: Eval iter: 1500
- [11/22 16:24:02] ppdet.engine INFO: Eval iter: 1600
- [11/22 16:24:06] ppdet.engine INFO: Eval iter: 1700
- [11/22 16:24:10] ppdet.engine INFO: Eval iter: 1800
- [11/22 16:24:14] ppdet.engine INFO: Eval iter: 1900
- [11/22 16:24:18] ppdet.engine INFO: Eval iter: 2000
- [11/22 16:24:22] ppdet.engine INFO: Eval iter: 2100
- [11/22 16:24:26] ppdet.engine INFO: Eval iter: 2200
- [11/22 16:24:31] ppdet.engine INFO: Eval iter: 2300
- [11/22 16:24:35] ppdet.engine INFO: Eval iter: 2400
- [11/22 16:24:39] ppdet.engine INFO: Eval iter: 2500
- [11/22 16:24:43] ppdet.engine INFO: Eval iter: 2600
- [11/22 16:24:48] ppdet.engine INFO: Eval iter: 2700
- [11/22 16:24:52] ppdet.engine INFO: Eval iter: 2800
- [11/22 16:24:56] ppdet.engine INFO: Eval iter: 2900
- [11/22 16:25:00] ppdet.engine INFO: Eval iter: 3000
- [11/22 16:25:04] ppdet.engine INFO: Eval iter: 3100
- [11/22 16:25:08] ppdet.engine INFO: Eval iter: 3200
- [11/22 16:25:12] ppdet.engine INFO: Eval iter: 3300
- [11/22 16:25:16] ppdet.engine INFO: Eval iter: 3400
- [11/22 16:25:21] ppdet.engine INFO: Eval iter: 3500
- [11/22 16:25:25] ppdet.engine INFO: Eval iter: 3600
- [11/22 16:25:29] ppdet.engine INFO: Eval iter: 3700
- [11/22 16:25:33] ppdet.engine INFO: Eval iter: 3800
- [11/22 16:25:37] ppdet.engine INFO: Eval iter: 3900
- [11/22 16:25:41] ppdet.engine INFO: Eval iter: 4000
- [11/22 16:25:45] ppdet.engine INFO: Eval iter: 4100
- [11/22 16:25:49] ppdet.engine INFO: Eval iter: 4200
- [11/22 16:25:53] ppdet.engine INFO: Eval iter: 4300
- [11/22 16:25:54] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 16:25:54] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 88.87%
- [11/22 16:25:54] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.191846146377372
- [11/22 16:25:54] ppdet.engine INFO: Best test bbox ap is 0.889.
- [11/22 16:25:54] ppdet.engine INFO: Epoch: [36] [ 0/1215] learning_rate: 0.000600 loss_xy: 2.884248 loss_wh: 0.473073 loss_obj: 1.280063 loss_cls: 0.350199 loss: 5.139583 eta: 2:59:49 batch_cost: 0.3614 data_cost: 0.0303 ips: 22.1335 images/s
- [11/22 16:27:08] ppdet.engine INFO: Epoch: [36] [ 200/1215] learning_rate: 0.000600 loss_xy: 2.780643 loss_wh: 0.452469 loss_obj: 1.271268 loss_cls: 0.321421 loss: 4.934721 eta: 2:58:29 batch_cost: 0.3659 data_cost: 0.0007 ips: 21.8614 images/s
- [11/22 16:28:20] ppdet.engine INFO: Epoch: [36] [ 400/1215] learning_rate: 0.000600 loss_xy: 2.882918 loss_wh: 0.506200 loss_obj: 1.190213 loss_cls: 0.309639 loss: 4.943405 eta: 2:57:03 batch_cost: 0.3615 data_cost: 0.0175 ips: 22.1281 images/s
- [11/22 16:29:33] ppdet.engine INFO: Epoch: [36] [ 600/1215] learning_rate: 0.000600 loss_xy: 2.819391 loss_wh: 0.439610 loss_obj: 1.156984 loss_cls: 0.312338 loss: 4.813684 eta: 2:55:41 batch_cost: 0.3632 data_cost: 0.0145 ips: 22.0272 images/s
- [11/22 16:30:45] ppdet.engine INFO: Epoch: [36] [ 800/1215] learning_rate: 0.000600 loss_xy: 2.889311 loss_wh: 0.478548 loss_obj: 1.294588 loss_cls: 0.285415 loss: 5.061630 eta: 2:54:13 batch_cost: 0.3582 data_cost: 0.0030 ips: 22.3363 images/s
- [11/22 16:31:59] ppdet.engine INFO: Epoch: [36] [1000/1215] learning_rate: 0.000600 loss_xy: 3.002582 loss_wh: 0.507995 loss_obj: 1.186814 loss_cls: 0.332258 loss: 5.222070 eta: 2:52:57 batch_cost: 0.3664 data_cost: 0.0115 ips: 21.8344 images/s
- [11/22 16:33:11] ppdet.engine INFO: Epoch: [36] [1200/1215] learning_rate: 0.000600 loss_xy: 2.991751 loss_wh: 0.499685 loss_obj: 1.290710 loss_cls: 0.297318 loss: 5.237018 eta: 2:51:33 batch_cost: 0.3602 data_cost: 0.0098 ips: 22.2124 images/s
- [11/22 16:33:18] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 16:33:18] ppdet.engine INFO: Eval iter: 0
- [11/22 16:33:22] ppdet.engine INFO: Eval iter: 100
- [11/22 16:33:26] ppdet.engine INFO: Eval iter: 200
- [11/22 16:33:30] ppdet.engine INFO: Eval iter: 300
- [11/22 16:33:34] ppdet.engine INFO: Eval iter: 400
- [11/22 16:33:38] ppdet.engine INFO: Eval iter: 500
- [11/22 16:33:42] ppdet.engine INFO: Eval iter: 600
- [11/22 16:33:46] ppdet.engine INFO: Eval iter: 700
- [11/22 16:33:50] ppdet.engine INFO: Eval iter: 800
- [11/22 16:33:54] ppdet.engine INFO: Eval iter: 900
- [11/22 16:33:58] ppdet.engine INFO: Eval iter: 1000
- [11/22 16:34:02] ppdet.engine INFO: Eval iter: 1100
- [11/22 16:34:06] ppdet.engine INFO: Eval iter: 1200
- [11/22 16:34:10] ppdet.engine INFO: Eval iter: 1300
- [11/22 16:34:15] ppdet.engine INFO: Eval iter: 1400
- [11/22 16:34:19] ppdet.engine INFO: Eval iter: 1500
- [11/22 16:34:23] ppdet.engine INFO: Eval iter: 1600
- [11/22 16:34:27] ppdet.engine INFO: Eval iter: 1700
- [11/22 16:34:31] ppdet.engine INFO: Eval iter: 1800
- [11/22 16:34:35] ppdet.engine INFO: Eval iter: 1900
- [11/22 16:34:39] ppdet.engine INFO: Eval iter: 2000
- [11/22 16:34:43] ppdet.engine INFO: Eval iter: 2100
- [11/22 16:34:47] ppdet.engine INFO: Eval iter: 2200
- [11/22 16:34:52] ppdet.engine INFO: Eval iter: 2300
- [11/22 16:34:56] ppdet.engine INFO: Eval iter: 2400
- [11/22 16:35:00] ppdet.engine INFO: Eval iter: 2500
- [11/22 16:35:04] ppdet.engine INFO: Eval iter: 2600
- [11/22 16:35:08] ppdet.engine INFO: Eval iter: 2700
- [11/22 16:35:12] ppdet.engine INFO: Eval iter: 2800
- [11/22 16:35:16] ppdet.engine INFO: Eval iter: 2900
- [11/22 16:35:20] ppdet.engine INFO: Eval iter: 3000
- [11/22 16:35:24] ppdet.engine INFO: Eval iter: 3100
- [11/22 16:35:28] ppdet.engine INFO: Eval iter: 3200
- [11/22 16:35:32] ppdet.engine INFO: Eval iter: 3300
- [11/22 16:35:36] ppdet.engine INFO: Eval iter: 3400
- [11/22 16:35:40] ppdet.engine INFO: Eval iter: 3500
- [11/22 16:35:44] ppdet.engine INFO: Eval iter: 3600
- [11/22 16:35:48] ppdet.engine INFO: Eval iter: 3700
- [11/22 16:35:52] ppdet.engine INFO: Eval iter: 3800
- [11/22 16:35:56] ppdet.engine INFO: Eval iter: 3900
- [11/22 16:36:00] ppdet.engine INFO: Eval iter: 4000
- [11/22 16:36:04] ppdet.engine INFO: Eval iter: 4100
- [11/22 16:36:08] ppdet.engine INFO: Eval iter: 4200
- [11/22 16:36:12] ppdet.engine INFO: Eval iter: 4300
- [11/22 16:36:14] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 16:36:14] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 88.74%
- [11/22 16:36:14] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.58571378700362
- [11/22 16:36:14] ppdet.engine INFO: Best test bbox ap is 0.889.
- [11/22 16:36:14] ppdet.engine INFO: Epoch: [37] [ 0/1215] learning_rate: 0.000600 loss_xy: 2.992700 loss_wh: 0.508905 loss_obj: 1.320094 loss_cls: 0.298335 loss: 5.282531 eta: 2:51:29 batch_cost: 0.3604 data_cost: 0.0098 ips: 22.1978 images/s
- [11/22 16:37:25] ppdet.engine INFO: Epoch: [37] [ 200/1215] learning_rate: 0.000600 loss_xy: 2.899779 loss_wh: 0.485529 loss_obj: 1.218431 loss_cls: 0.306706 loss: 4.981825 eta: 2:50:01 batch_cost: 0.3551 data_cost: 0.0364 ips: 22.5304 images/s
- [11/22 16:38:40] ppdet.engine INFO: Epoch: [37] [ 400/1215] learning_rate: 0.000600 loss_xy: 2.735030 loss_wh: 0.434400 loss_obj: 1.032057 loss_cls: 0.277934 loss: 4.649773 eta: 2:48:55 batch_cost: 0.3743 data_cost: 0.0300 ips: 21.3754 images/s
- [11/22 16:39:51] ppdet.engine INFO: Epoch: [37] [ 600/1215] learning_rate: 0.000600 loss_xy: 2.894847 loss_wh: 0.491895 loss_obj: 1.127379 loss_cls: 0.321380 loss: 5.055849 eta: 2:47:27 batch_cost: 0.3536 data_cost: 0.0225 ips: 22.6216 images/s
- [11/22 16:41:05] ppdet.engine INFO: Epoch: [37] [ 800/1215] learning_rate: 0.000600 loss_xy: 2.849626 loss_wh: 0.490556 loss_obj: 1.181945 loss_cls: 0.286397 loss: 4.865444 eta: 2:46:14 batch_cost: 0.3680 data_cost: 0.0270 ips: 21.7394 images/s
- [11/22 16:42:18] ppdet.engine INFO: Epoch: [37] [1000/1215] learning_rate: 0.000600 loss_xy: 2.711101 loss_wh: 0.431923 loss_obj: 0.963883 loss_cls: 0.264923 loss: 4.457994 eta: 2:44:54 batch_cost: 0.3610 data_cost: 0.0150 ips: 22.1604 images/s
- [11/22 16:43:30] ppdet.engine INFO: Epoch: [37] [1200/1215] learning_rate: 0.000600 loss_xy: 3.033143 loss_wh: 0.506645 loss_obj: 1.036403 loss_cls: 0.327973 loss: 5.151760 eta: 2:43:34 batch_cost: 0.3590 data_cost: 0.0150 ips: 22.2828 images/s
- [11/22 16:43:36] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 16:43:36] ppdet.engine INFO: Eval iter: 0
- [11/22 16:43:40] ppdet.engine INFO: Eval iter: 100
- [11/22 16:43:44] ppdet.engine INFO: Eval iter: 200
- [11/22 16:43:48] ppdet.engine INFO: Eval iter: 300
- [11/22 16:43:52] ppdet.engine INFO: Eval iter: 400
- [11/22 16:43:56] ppdet.engine INFO: Eval iter: 500
- [11/22 16:44:01] ppdet.engine INFO: Eval iter: 600
- [11/22 16:44:05] ppdet.engine INFO: Eval iter: 700
- [11/22 16:44:09] ppdet.engine INFO: Eval iter: 800
- [11/22 16:44:13] ppdet.engine INFO: Eval iter: 900
- [11/22 16:44:17] ppdet.engine INFO: Eval iter: 1000
- [11/22 16:44:21] ppdet.engine INFO: Eval iter: 1100
- [11/22 16:44:25] ppdet.engine INFO: Eval iter: 1200
- [11/22 16:44:29] ppdet.engine INFO: Eval iter: 1300
- [11/22 16:44:34] ppdet.engine INFO: Eval iter: 1400
- [11/22 16:44:38] ppdet.engine INFO: Eval iter: 1500
- [11/22 16:44:43] ppdet.engine INFO: Eval iter: 1600
- [11/22 16:44:47] ppdet.engine INFO: Eval iter: 1700
- [11/22 16:44:52] ppdet.engine INFO: Eval iter: 1800
- [11/22 16:44:56] ppdet.engine INFO: Eval iter: 1900
- [11/22 16:45:00] ppdet.engine INFO: Eval iter: 2000
- [11/22 16:45:04] ppdet.engine INFO: Eval iter: 2100
- [11/22 16:45:08] ppdet.engine INFO: Eval iter: 2200
- [11/22 16:45:12] ppdet.engine INFO: Eval iter: 2300
- [11/22 16:45:16] ppdet.engine INFO: Eval iter: 2400
- [11/22 16:45:20] ppdet.engine INFO: Eval iter: 2500
- [11/22 16:45:24] ppdet.engine INFO: Eval iter: 2600
- [11/22 16:45:29] ppdet.engine INFO: Eval iter: 2700
- [11/22 16:45:33] ppdet.engine INFO: Eval iter: 2800
- [11/22 16:45:37] ppdet.engine INFO: Eval iter: 2900
- [11/22 16:45:41] ppdet.engine INFO: Eval iter: 3000
- [11/22 16:45:45] ppdet.engine INFO: Eval iter: 3100
- [11/22 16:45:49] ppdet.engine INFO: Eval iter: 3200
- [11/22 16:45:53] ppdet.engine INFO: Eval iter: 3300
- [11/22 16:45:57] ppdet.engine INFO: Eval iter: 3400
- [11/22 16:46:01] ppdet.engine INFO: Eval iter: 3500
- [11/22 16:46:05] ppdet.engine INFO: Eval iter: 3600
- [11/22 16:46:10] ppdet.engine INFO: Eval iter: 3700
- [11/22 16:46:14] ppdet.engine INFO: Eval iter: 3800
- [11/22 16:46:18] ppdet.engine INFO: Eval iter: 3900
- [11/22 16:46:22] ppdet.engine INFO: Eval iter: 4000
- [11/22 16:46:26] ppdet.engine INFO: Eval iter: 4100
- [11/22 16:46:30] ppdet.engine INFO: Eval iter: 4200
- [11/22 16:46:34] ppdet.engine INFO: Eval iter: 4300
- [11/22 16:46:35] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 16:46:35] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 87.69%
- [11/22 16:46:35] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.10041376340245
- [11/22 16:46:35] ppdet.engine INFO: Best test bbox ap is 0.889.
- [11/22 16:46:35] ppdet.engine INFO: Epoch: [38] [ 0/1215] learning_rate: 0.000600 loss_xy: 3.065905 loss_wh: 0.505445 loss_obj: 1.019490 loss_cls: 0.320955 loss: 5.139336 eta: 2:43:27 batch_cost: 0.3588 data_cost: 0.0152 ips: 22.2993 images/s
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:35341']
- I1122 16:47:56.418871 31347 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1122 16:47:57.193220 31347 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1122 16:47:57.196779 31347 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/22 16:47:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- [11/22 16:47:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/22 16:47:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- [11/22 16:47:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- [11/22 16:48:01] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- [11/22 16:48:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- [11/22 16:48:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- [11/22 16:48:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- [11/22 16:48:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- [11/22 16:48:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- [11/22 16:48:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/22 16:48:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- [11/22 16:48:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- [11/22 16:48:05] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/22 16:48:05] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- [11/22 16:48:06] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- [11/22 16:48:08] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Traceback (most recent call last):
- File "tools/train.py", line 172, in <module>
- main()
- File "tools/train.py", line 168, in main
- run(FLAGS, cfg)
- File "tools/train.py", line 127, in run
- trainer.resume_weights(FLAGS.resume)
- File "/home/aistudio/work/myDemo/ppdet/engine/trainer.py", line 392, in resume_weights
- self.ema if self.use_ema else None)
- File "/home/aistudio/work/myDemo/ppdet/utils/checkpoint.py", line 73, in load_weight
- "exists.".format(pdparam_path))
- ValueError: Model pretrain path output/AUG/yolov3_darknet53_270e_voc_garbage/38.pdparams does not exists.
- I1122 16:48:48.726787 31800 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1122 16:48:49.506100 31800 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1122 16:48:49.509584 31800 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/22 16:48:51] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- [11/22 16:48:51] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/22 16:48:51] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- [11/22 16:48:51] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- [11/22 16:48:53] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- [11/22 16:48:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- [11/22 16:48:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- [11/22 16:48:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- [11/22 16:48:55] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- [11/22 16:48:55] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- [11/22 16:48:55] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/22 16:48:56] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- [11/22 16:48:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- [11/22 16:48:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/22 16:48:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- [11/22 16:48:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- [11/22 16:49:00] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/22 16:49:01] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/37.pdparams
- [11/22 16:49:05] ppdet.engine INFO: Epoch: [38] [ 0/1215] learning_rate: 0.000600 loss_xy: 4.757658 loss_wh: 1.039333 loss_obj: 2.808673 loss_cls: 0.329558 loss: 8.935222 eta: 18:06:52 batch_cost: 2.4397 data_cost: 0.0007 ips: 3.2791 images/s
- [11/22 16:50:20] ppdet.engine INFO: Epoch: [38] [ 200/1215] learning_rate: 0.000300 loss_xy: 2.832492 loss_wh: 0.435504 loss_obj: 1.342734 loss_cls: 0.279747 loss: 5.054839 eta: 2:50:13 batch_cost: 0.3747 data_cost: 0.0225 ips: 21.3506 images/s
- [11/22 16:51:38] ppdet.engine INFO: Epoch: [38] [ 400/1215] learning_rate: 0.000300 loss_xy: 2.765042 loss_wh: 0.433608 loss_obj: 1.191946 loss_cls: 0.296642 loss: 4.890218 eta: 2:50:00 batch_cost: 0.3899 data_cost: 0.0130 ips: 20.5185 images/s
- [11/22 16:52:56] ppdet.engine INFO: Epoch: [38] [ 600/1215] learning_rate: 0.000300 loss_xy: 2.794686 loss_wh: 0.438123 loss_obj: 1.074615 loss_cls: 0.262215 loss: 4.545331 eta: 2:49:13 batch_cost: 0.3909 data_cost: 0.0213 ips: 20.4657 images/s
- [11/22 16:54:14] ppdet.engine INFO: Epoch: [38] [ 800/1215] learning_rate: 0.000300 loss_xy: 2.772954 loss_wh: 0.412199 loss_obj: 0.969136 loss_cls: 0.253439 loss: 4.434338 eta: 2:47:27 batch_cost: 0.3843 data_cost: 0.0299 ips: 20.8187 images/s
- [11/22 16:55:30] ppdet.engine INFO: Epoch: [38] [1000/1215] learning_rate: 0.000300 loss_xy: 2.891928 loss_wh: 0.449051 loss_obj: 1.010336 loss_cls: 0.254598 loss: 4.789045 eta: 2:45:35 batch_cost: 0.3808 data_cost: 0.0172 ips: 21.0111 images/s
- [11/22 16:56:46] ppdet.engine INFO: Epoch: [38] [1200/1215] learning_rate: 0.000300 loss_xy: 2.771225 loss_wh: 0.414747 loss_obj: 0.944147 loss_cls: 0.267409 loss: 4.580356 eta: 2:43:54 batch_cost: 0.3806 data_cost: 0.0209 ips: 21.0200 images/s
- [11/22 16:56:53] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 16:56:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_2.xml, x1: 217.0, y1: 763.0, x2: 217.0, y2: 764.0.
- [11/22 16:56:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_8.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/22 16:56:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_881_3.xml, x1: 35.0, y1: 198.0, x2: 75.0, y2: 197.0.
- [11/22 16:56:55] ppdet.engine INFO: Eval iter: 0
- [11/22 16:56:59] ppdet.engine INFO: Eval iter: 100
- [11/22 16:57:03] ppdet.engine INFO: Eval iter: 200
- [11/22 16:57:07] ppdet.engine INFO: Eval iter: 300
- [11/22 16:57:11] ppdet.engine INFO: Eval iter: 400
- [11/22 16:57:15] ppdet.engine INFO: Eval iter: 500
- [11/22 16:57:19] ppdet.engine INFO: Eval iter: 600
- [11/22 16:57:23] ppdet.engine INFO: Eval iter: 700
- [11/22 16:57:27] ppdet.engine INFO: Eval iter: 800
- [11/22 16:57:31] ppdet.engine INFO: Eval iter: 900
- [11/22 16:57:35] ppdet.engine INFO: Eval iter: 1000
- [11/22 16:57:39] ppdet.engine INFO: Eval iter: 1100
- [11/22 16:57:43] ppdet.engine INFO: Eval iter: 1200
- [11/22 16:57:47] ppdet.engine INFO: Eval iter: 1300
- [11/22 16:57:51] ppdet.engine INFO: Eval iter: 1400
- [11/22 16:57:54] ppdet.engine INFO: Eval iter: 1500
- [11/22 16:57:58] ppdet.engine INFO: Eval iter: 1600
- [11/22 16:58:02] ppdet.engine INFO: Eval iter: 1700
- [11/22 16:58:06] ppdet.engine INFO: Eval iter: 1800
- [11/22 16:58:10] ppdet.engine INFO: Eval iter: 1900
- [11/22 16:58:14] ppdet.engine INFO: Eval iter: 2000
- [11/22 16:58:18] ppdet.engine INFO: Eval iter: 2100
- [11/22 16:58:22] ppdet.engine INFO: Eval iter: 2200
- [11/22 16:58:26] ppdet.engine INFO: Eval iter: 2300
- [11/22 16:58:30] ppdet.engine INFO: Eval iter: 2400
- [11/22 16:58:34] ppdet.engine INFO: Eval iter: 2500
- [11/22 16:58:38] ppdet.engine INFO: Eval iter: 2600
- [11/22 16:58:42] ppdet.engine INFO: Eval iter: 2700
- [11/22 16:58:46] ppdet.engine INFO: Eval iter: 2800
- [11/22 16:58:50] ppdet.engine INFO: Eval iter: 2900
- [11/22 16:58:54] ppdet.engine INFO: Eval iter: 3000
- [11/22 16:58:58] ppdet.engine INFO: Eval iter: 3100
- [11/22 16:59:02] ppdet.engine INFO: Eval iter: 3200
- [11/22 16:59:06] ppdet.engine INFO: Eval iter: 3300
- [11/22 16:59:10] ppdet.engine INFO: Eval iter: 3400
- [11/22 16:59:14] ppdet.engine INFO: Eval iter: 3500
- [11/22 16:59:18] ppdet.engine INFO: Eval iter: 3600
- [11/22 16:59:22] ppdet.engine INFO: Eval iter: 3700
- [11/22 16:59:26] ppdet.engine INFO: Eval iter: 3800
- [11/22 16:59:30] ppdet.engine INFO: Eval iter: 3900
- [11/22 16:59:34] ppdet.engine INFO: Eval iter: 4000
- [11/22 16:59:38] ppdet.engine INFO: Eval iter: 4100
- [11/22 16:59:42] ppdet.engine INFO: Eval iter: 4200
- [11/22 16:59:46] ppdet.engine INFO: Eval iter: 4300
- [11/22 16:59:47] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 16:59:47] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 88.64%
- [11/22 16:59:47] ppdet.engine INFO: Total sample number: 4322, averge FPS: 25.00783309717054
- [11/22 16:59:47] ppdet.engine INFO: Best test bbox ap is 0.886.
- [11/22 16:59:51] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 16:59:51] ppdet.engine INFO: Epoch: [39] [ 0/1215] learning_rate: 0.000300 loss_xy: 2.832992 loss_wh: 0.427000 loss_obj: 0.932274 loss_cls: 0.267409 loss: 4.629354 eta: 2:43:49 batch_cost: 0.3819 data_cost: 0.0216 ips: 20.9488 images/s
- [11/22 17:01:06] ppdet.engine INFO: Epoch: [39] [ 200/1215] learning_rate: 0.000300 loss_xy: 2.733222 loss_wh: 0.419586 loss_obj: 0.935735 loss_cls: 0.297693 loss: 4.521574 eta: 2:41:46 batch_cost: 0.3723 data_cost: 0.0219 ips: 21.4861 images/s
- [11/22 17:02:22] ppdet.engine INFO: Epoch: [39] [ 400/1215] learning_rate: 0.000300 loss_xy: 2.809821 loss_wh: 0.436238 loss_obj: 1.005738 loss_cls: 0.233768 loss: 4.575593 eta: 2:40:08 batch_cost: 0.3764 data_cost: 0.0319 ips: 21.2522 images/s
- [11/22 17:03:37] ppdet.engine INFO: Epoch: [39] [ 600/1215] learning_rate: 0.000300 loss_xy: 2.938396 loss_wh: 0.452353 loss_obj: 1.020262 loss_cls: 0.266713 loss: 4.758173 eta: 2:38:28 batch_cost: 0.3742 data_cost: 0.0311 ips: 21.3777 images/s
- [11/22 17:04:53] ppdet.engine INFO: Epoch: [39] [ 800/1215] learning_rate: 0.000300 loss_xy: 2.904861 loss_wh: 0.422296 loss_obj: 0.843555 loss_cls: 0.218774 loss: 4.513139 eta: 2:37:09 batch_cost: 0.3806 data_cost: 0.0220 ips: 21.0196 images/s
- [11/22 17:06:08] ppdet.engine INFO: Epoch: [39] [1000/1215] learning_rate: 0.000300 loss_xy: 2.973337 loss_wh: 0.463434 loss_obj: 0.970507 loss_cls: 0.267879 loss: 4.807652 eta: 2:35:37 batch_cost: 0.3741 data_cost: 0.0497 ips: 21.3831 images/s
- [11/22 17:07:25] ppdet.engine INFO: Epoch: [39] [1200/1215] learning_rate: 0.000300 loss_xy: 2.868380 loss_wh: 0.421394 loss_obj: 0.932787 loss_cls: 0.238705 loss: 4.597235 eta: 2:34:19 batch_cost: 0.3802 data_cost: 0.0147 ips: 21.0433 images/s
- [11/22 17:07:31] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 17:07:31] ppdet.engine INFO: Eval iter: 0
- [11/22 17:07:35] ppdet.engine INFO: Eval iter: 100
- [11/22 17:07:40] ppdet.engine INFO: Eval iter: 200
- [11/22 17:07:44] ppdet.engine INFO: Eval iter: 300
- [11/22 17:07:48] ppdet.engine INFO: Eval iter: 400
- [11/22 17:07:52] ppdet.engine INFO: Eval iter: 500
- [11/22 17:07:56] ppdet.engine INFO: Eval iter: 600
- [11/22 17:08:00] ppdet.engine INFO: Eval iter: 700
- [11/22 17:08:04] ppdet.engine INFO: Eval iter: 800
- [11/22 17:08:08] ppdet.engine INFO: Eval iter: 900
- [11/22 17:08:12] ppdet.engine INFO: Eval iter: 1000
- [11/22 17:08:16] ppdet.engine INFO: Eval iter: 1100
- [11/22 17:08:20] ppdet.engine INFO: Eval iter: 1200
- [11/22 17:08:24] ppdet.engine INFO: Eval iter: 1300
- [11/22 17:08:28] ppdet.engine INFO: Eval iter: 1400
- [11/22 17:08:32] ppdet.engine INFO: Eval iter: 1500
- [11/22 17:08:36] ppdet.engine INFO: Eval iter: 1600
- [11/22 17:08:40] ppdet.engine INFO: Eval iter: 1700
- [11/22 17:08:44] ppdet.engine INFO: Eval iter: 1800
- [11/22 17:08:48] ppdet.engine INFO: Eval iter: 1900
- [11/22 17:08:52] ppdet.engine INFO: Eval iter: 2000
- [11/22 17:08:56] ppdet.engine INFO: Eval iter: 2100
- [11/22 17:09:00] ppdet.engine INFO: Eval iter: 2200
- [11/22 17:09:04] ppdet.engine INFO: Eval iter: 2300
- [11/22 17:09:08] ppdet.engine INFO: Eval iter: 2400
- [11/22 17:09:12] ppdet.engine INFO: Eval iter: 2500
- [11/22 17:09:16] ppdet.engine INFO: Eval iter: 2600
- [11/22 17:09:20] ppdet.engine INFO: Eval iter: 2700
- [11/22 17:09:24] ppdet.engine INFO: Eval iter: 2800
- [11/22 17:09:28] ppdet.engine INFO: Eval iter: 2900
- [11/22 17:09:32] ppdet.engine INFO: Eval iter: 3000
- [11/22 17:09:36] ppdet.engine INFO: Eval iter: 3100
- [11/22 17:09:40] ppdet.engine INFO: Eval iter: 3200
- [11/22 17:09:45] ppdet.engine INFO: Eval iter: 3300
- [11/22 17:09:49] ppdet.engine INFO: Eval iter: 3400
- [11/22 17:09:53] ppdet.engine INFO: Eval iter: 3500
- [11/22 17:09:57] ppdet.engine INFO: Eval iter: 3600
- [11/22 17:10:01] ppdet.engine INFO: Eval iter: 3700
- [11/22 17:10:05] ppdet.engine INFO: Eval iter: 3800
- [11/22 17:10:09] ppdet.engine INFO: Eval iter: 3900
- [11/22 17:10:13] ppdet.engine INFO: Eval iter: 4000
- [11/22 17:10:17] ppdet.engine INFO: Eval iter: 4100
- [11/22 17:10:21] ppdet.engine INFO: Eval iter: 4200
- [11/22 17:10:25] ppdet.engine INFO: Eval iter: 4300
- [11/22 17:10:26] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 17:10:26] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 89.10%
- [11/22 17:10:26] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.751557604167857
- [11/22 17:10:26] ppdet.engine INFO: Best test bbox ap is 0.891.
- [11/22 17:10:30] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 17:10:30] ppdet.engine INFO: Epoch: [40] [ 0/1215] learning_rate: 0.000300 loss_xy: 2.901842 loss_wh: 0.427175 loss_obj: 0.929656 loss_cls: 0.238705 loss: 4.634724 eta: 2:34:12 batch_cost: 0.3800 data_cost: 0.0162 ips: 21.0516 images/s
- [11/22 17:11:44] ppdet.engine INFO: Epoch: [40] [ 200/1215] learning_rate: 0.000300 loss_xy: 3.002567 loss_wh: 0.462780 loss_obj: 0.976339 loss_cls: 0.247232 loss: 4.831436 eta: 2:32:37 batch_cost: 0.3705 data_cost: 0.0336 ips: 21.5929 images/s
- [11/22 17:12:58] ppdet.engine INFO: Epoch: [40] [ 400/1215] learning_rate: 0.000300 loss_xy: 2.925387 loss_wh: 0.465218 loss_obj: 0.863821 loss_cls: 0.244206 loss: 4.704347 eta: 2:31:00 batch_cost: 0.3675 data_cost: 0.0313 ips: 21.7663 images/s
- [11/22 17:14:13] ppdet.engine INFO: Epoch: [40] [ 600/1215] learning_rate: 0.000300 loss_xy: 2.941616 loss_wh: 0.456673 loss_obj: 0.826305 loss_cls: 0.241292 loss: 4.653247 eta: 2:29:33 batch_cost: 0.3719 data_cost: 0.0542 ips: 21.5105 images/s
- [11/22 17:15:24] ppdet.engine INFO: Epoch: [40] [ 800/1215] learning_rate: 0.000300 loss_xy: 2.706295 loss_wh: 0.426353 loss_obj: 0.737206 loss_cls: 0.226682 loss: 4.381104 eta: 2:27:46 batch_cost: 0.3569 data_cost: 0.0163 ips: 22.4122 images/s
- [11/22 17:16:40] ppdet.engine INFO: Epoch: [40] [1000/1215] learning_rate: 0.000300 loss_xy: 3.047088 loss_wh: 0.472221 loss_obj: 0.892682 loss_cls: 0.247746 loss: 4.885479 eta: 2:26:31 batch_cost: 0.3782 data_cost: 0.0413 ips: 21.1547 images/s
- [11/22 17:17:55] ppdet.engine INFO: Epoch: [40] [1200/1215] learning_rate: 0.000300 loss_xy: 2.870801 loss_wh: 0.450908 loss_obj: 0.814159 loss_cls: 0.247687 loss: 4.559628 eta: 2:25:08 batch_cost: 0.3714 data_cost: 0.0529 ips: 21.5388 images/s
- [11/22 17:18:01] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 17:18:02] ppdet.engine INFO: Eval iter: 0
- [11/22 17:18:06] ppdet.engine INFO: Eval iter: 100
- [11/22 17:18:10] ppdet.engine INFO: Eval iter: 200
- [11/22 17:18:14] ppdet.engine INFO: Eval iter: 300
- [11/22 17:18:18] ppdet.engine INFO: Eval iter: 400
- [11/22 17:18:22] ppdet.engine INFO: Eval iter: 500
- [11/22 17:18:27] ppdet.engine INFO: Eval iter: 600
- [11/22 17:18:31] ppdet.engine INFO: Eval iter: 700
- [11/22 17:18:35] ppdet.engine INFO: Eval iter: 800
- [11/22 17:18:39] ppdet.engine INFO: Eval iter: 900
- [11/22 17:18:43] ppdet.engine INFO: Eval iter: 1000
- [11/22 17:18:47] ppdet.engine INFO: Eval iter: 1100
- [11/22 17:18:51] ppdet.engine INFO: Eval iter: 1200
- [11/22 17:18:55] ppdet.engine INFO: Eval iter: 1300
- [11/22 17:18:59] ppdet.engine INFO: Eval iter: 1400
- [11/22 17:19:03] ppdet.engine INFO: Eval iter: 1500
- [11/22 17:19:07] ppdet.engine INFO: Eval iter: 1600
- [11/22 17:19:11] ppdet.engine INFO: Eval iter: 1700
- [11/22 17:19:15] ppdet.engine INFO: Eval iter: 1800
- [11/22 17:19:19] ppdet.engine INFO: Eval iter: 1900
- [11/22 17:19:23] ppdet.engine INFO: Eval iter: 2000
- [11/22 17:19:27] ppdet.engine INFO: Eval iter: 2100
- [11/22 17:19:31] ppdet.engine INFO: Eval iter: 2200
- [11/22 17:19:35] ppdet.engine INFO: Eval iter: 2300
- [11/22 17:19:39] ppdet.engine INFO: Eval iter: 2400
- [11/22 17:19:43] ppdet.engine INFO: Eval iter: 2500
- [11/22 17:19:47] ppdet.engine INFO: Eval iter: 2600
- [11/22 17:19:51] ppdet.engine INFO: Eval iter: 2700
- [11/22 17:19:55] ppdet.engine INFO: Eval iter: 2800
- [11/22 17:19:59] ppdet.engine INFO: Eval iter: 2900
- [11/22 17:20:03] ppdet.engine INFO: Eval iter: 3000
- [11/22 17:20:07] ppdet.engine INFO: Eval iter: 3100
- [11/22 17:20:11] ppdet.engine INFO: Eval iter: 3200
- [11/22 17:20:15] ppdet.engine INFO: Eval iter: 3300
- [11/22 17:20:19] ppdet.engine INFO: Eval iter: 3400
- [11/22 17:20:23] ppdet.engine INFO: Eval iter: 3500
- [11/22 17:20:27] ppdet.engine INFO: Eval iter: 3600
- [11/22 17:20:31] ppdet.engine INFO: Eval iter: 3700
- [11/22 17:20:35] ppdet.engine INFO: Eval iter: 3800
- [11/22 17:20:39] ppdet.engine INFO: Eval iter: 3900
- [11/22 17:20:43] ppdet.engine INFO: Eval iter: 4000
- [11/22 17:20:47] ppdet.engine INFO: Eval iter: 4100
- [11/22 17:20:51] ppdet.engine INFO: Eval iter: 4200
- [11/22 17:20:55] ppdet.engine INFO: Eval iter: 4300
- [11/22 17:20:56] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 17:20:56] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 88.62%
- [11/22 17:20:56] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.77029691933352
- [11/22 17:20:56] ppdet.engine INFO: Best test bbox ap is 0.891.
- [11/22 17:20:56] ppdet.engine INFO: Epoch: [41] [ 0/1215] learning_rate: 0.000300 loss_xy: 2.846153 loss_wh: 0.444852 loss_obj: 0.851282 loss_cls: 0.241825 loss: 4.487465 eta: 2:25:02 batch_cost: 0.3688 data_cost: 0.0450 ips: 21.6949 images/s
- [11/22 17:22:09] ppdet.engine INFO: Epoch: [41] [ 200/1215] learning_rate: 0.000300 loss_xy: 2.727654 loss_wh: 0.395189 loss_obj: 0.746467 loss_cls: 0.214390 loss: 4.227363 eta: 2:23:27 batch_cost: 0.3600 data_cost: 0.0180 ips: 22.2231 images/s
- [11/22 17:23:23] ppdet.engine INFO: Epoch: [41] [ 400/1215] learning_rate: 0.000300 loss_xy: 2.853829 loss_wh: 0.438539 loss_obj: 0.692992 loss_cls: 0.195392 loss: 4.291681 eta: 2:22:04 batch_cost: 0.3693 data_cost: 0.0367 ips: 21.6641 images/s
- [11/22 17:24:36] ppdet.engine INFO: Epoch: [41] [ 600/1215] learning_rate: 0.000300 loss_xy: 2.787365 loss_wh: 0.401696 loss_obj: 0.657409 loss_cls: 0.200075 loss: 4.275901 eta: 2:20:35 batch_cost: 0.3624 data_cost: 0.0362 ips: 22.0744 images/s
- [11/22 17:25:48] ppdet.engine INFO: Epoch: [41] [ 800/1215] learning_rate: 0.000300 loss_xy: 2.849099 loss_wh: 0.426828 loss_obj: 0.790488 loss_cls: 0.185473 loss: 4.316904 eta: 2:19:06 batch_cost: 0.3616 data_cost: 0.0269 ips: 22.1253 images/s
- [11/22 17:27:02] ppdet.engine INFO: Epoch: [41] [1000/1215] learning_rate: 0.000300 loss_xy: 2.952274 loss_wh: 0.445310 loss_obj: 0.724442 loss_cls: 0.220953 loss: 4.429514 eta: 2:17:46 batch_cost: 0.3688 data_cost: 0.0395 ips: 21.6921 images/s
- [11/22 17:28:15] ppdet.engine INFO: Epoch: [41] [1200/1215] learning_rate: 0.000300 loss_xy: 2.930414 loss_wh: 0.454826 loss_obj: 0.698178 loss_cls: 0.202040 loss: 4.428688 eta: 2:16:21 batch_cost: 0.3639 data_cost: 0.0232 ips: 21.9842 images/s
- [11/22 17:28:22] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 17:28:22] ppdet.engine INFO: Eval iter: 0
- [11/22 17:28:26] ppdet.engine INFO: Eval iter: 100
- [11/22 17:28:30] ppdet.engine INFO: Eval iter: 200
- [11/22 17:28:34] ppdet.engine INFO: Eval iter: 300
- [11/22 17:28:39] ppdet.engine INFO: Eval iter: 400
- [11/22 17:28:43] ppdet.engine INFO: Eval iter: 500
- [11/22 17:28:47] ppdet.engine INFO: Eval iter: 600
- [11/22 17:28:51] ppdet.engine INFO: Eval iter: 700
- [11/22 17:28:55] ppdet.engine INFO: Eval iter: 800
- [11/22 17:28:59] ppdet.engine INFO: Eval iter: 900
- [11/22 17:29:03] ppdet.engine INFO: Eval iter: 1000
- [11/22 17:29:07] ppdet.engine INFO: Eval iter: 1100
- [11/22 17:29:11] ppdet.engine INFO: Eval iter: 1200
- [11/22 17:29:15] ppdet.engine INFO: Eval iter: 1300
- [11/22 17:29:19] ppdet.engine INFO: Eval iter: 1400
- [11/22 17:29:23] ppdet.engine INFO: Eval iter: 1500
- [11/22 17:29:27] ppdet.engine INFO: Eval iter: 1600
- [11/22 17:29:31] ppdet.engine INFO: Eval iter: 1700
- [11/22 17:29:35] ppdet.engine INFO: Eval iter: 1800
- [11/22 17:29:39] ppdet.engine INFO: Eval iter: 1900
- [11/22 17:29:43] ppdet.engine INFO: Eval iter: 2000
- [11/22 17:29:47] ppdet.engine INFO: Eval iter: 2100
- [11/22 17:29:51] ppdet.engine INFO: Eval iter: 2200
- [11/22 17:29:55] ppdet.engine INFO: Eval iter: 2300
- [11/22 17:29:59] ppdet.engine INFO: Eval iter: 2400
- [11/22 17:30:03] ppdet.engine INFO: Eval iter: 2500
- [11/22 17:30:07] ppdet.engine INFO: Eval iter: 2600
- [11/22 17:30:11] ppdet.engine INFO: Eval iter: 2700
- [11/22 17:30:15] ppdet.engine INFO: Eval iter: 2800
- [11/22 17:30:19] ppdet.engine INFO: Eval iter: 2900
- [11/22 17:30:23] ppdet.engine INFO: Eval iter: 3000
- [11/22 17:30:27] ppdet.engine INFO: Eval iter: 3100
- [11/22 17:30:31] ppdet.engine INFO: Eval iter: 3200
- [11/22 17:30:35] ppdet.engine INFO: Eval iter: 3300
- [11/22 17:30:39] ppdet.engine INFO: Eval iter: 3400
- [11/22 17:30:43] ppdet.engine INFO: Eval iter: 3500
- [11/22 17:30:47] ppdet.engine INFO: Eval iter: 3600
- [11/22 17:30:51] ppdet.engine INFO: Eval iter: 3700
- [11/22 17:30:55] ppdet.engine INFO: Eval iter: 3800
- [11/22 17:30:59] ppdet.engine INFO: Eval iter: 3900
- [11/22 17:31:03] ppdet.engine INFO: Eval iter: 4000
- [11/22 17:31:07] ppdet.engine INFO: Eval iter: 4100
- [11/22 17:31:11] ppdet.engine INFO: Eval iter: 4200
- [11/22 17:31:15] ppdet.engine INFO: Eval iter: 4300
- [11/22 17:31:16] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 17:31:16] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 89.05%
- [11/22 17:31:16] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.757098588332518
- [11/22 17:31:16] ppdet.engine INFO: Best test bbox ap is 0.891.
- [11/22 17:31:17] ppdet.engine INFO: Epoch: [42] [ 0/1215] learning_rate: 0.000300 loss_xy: 2.946214 loss_wh: 0.464069 loss_obj: 0.722307 loss_cls: 0.202040 loss: 4.541815 eta: 2:16:16 batch_cost: 0.3653 data_cost: 0.0216 ips: 21.9016 images/s
- [11/22 17:32:30] ppdet.engine INFO: Epoch: [42] [ 200/1215] learning_rate: 0.000300 loss_xy: 2.867256 loss_wh: 0.429211 loss_obj: 0.646895 loss_cls: 0.206356 loss: 4.291636 eta: 2:14:51 batch_cost: 0.3620 data_cost: 0.0416 ips: 22.0978 images/s
- [11/22 17:33:44] ppdet.engine INFO: Epoch: [42] [ 400/1215] learning_rate: 0.000300 loss_xy: 2.710164 loss_wh: 0.392446 loss_obj: 0.633768 loss_cls: 0.177844 loss: 4.018091 eta: 2:13:34 batch_cost: 0.3704 data_cost: 0.0380 ips: 21.6003 images/s
- [11/22 17:34:56] ppdet.engine INFO: Epoch: [42] [ 600/1215] learning_rate: 0.000300 loss_xy: 2.876477 loss_wh: 0.453317 loss_obj: 0.698773 loss_cls: 0.198220 loss: 4.369637 eta: 2:12:09 batch_cost: 0.3592 data_cost: 0.0306 ips: 22.2733 images/s
- [11/22 17:36:11] ppdet.engine INFO: Epoch: [42] [ 800/1215] learning_rate: 0.000300 loss_xy: 2.801945 loss_wh: 0.431412 loss_obj: 0.651441 loss_cls: 0.211426 loss: 4.257954 eta: 2:10:55 batch_cost: 0.3744 data_cost: 0.0455 ips: 21.3663 images/s
- [11/22 17:37:24] ppdet.engine INFO: Epoch: [42] [1000/1215] learning_rate: 0.000300 loss_xy: 2.671589 loss_wh: 0.398729 loss_obj: 0.545010 loss_cls: 0.149841 loss: 3.890659 eta: 2:09:33 batch_cost: 0.3626 data_cost: 0.0287 ips: 22.0608 images/s
- [11/22 17:38:36] ppdet.engine INFO: Epoch: [42] [1200/1215] learning_rate: 0.000300 loss_xy: 2.989680 loss_wh: 0.447498 loss_obj: 0.632102 loss_cls: 0.212391 loss: 4.459252 eta: 2:08:11 batch_cost: 0.3606 data_cost: 0.0257 ips: 22.1851 images/s
- [11/22 17:38:42] ppdet.utils.checkpoint INFO: Save checkpoint: output/AUG/yolov3_darknet53_270e_voc_garbage
- [11/22 17:38:42] ppdet.engine INFO: Eval iter: 0
- [11/22 17:38:47] ppdet.engine INFO: Eval iter: 100
- [11/22 17:38:51] ppdet.engine INFO: Eval iter: 200
- [11/22 17:38:55] ppdet.engine INFO: Eval iter: 300
- [11/22 17:38:59] ppdet.engine INFO: Eval iter: 400
- [11/22 17:39:03] ppdet.engine INFO: Eval iter: 500
- [11/22 17:39:07] ppdet.engine INFO: Eval iter: 600
- [11/22 17:39:11] ppdet.engine INFO: Eval iter: 700
- [11/22 17:39:15] ppdet.engine INFO: Eval iter: 800
- [11/22 17:39:19] ppdet.engine INFO: Eval iter: 900
- [11/22 17:39:23] ppdet.engine INFO: Eval iter: 1000
- [11/22 17:39:27] ppdet.engine INFO: Eval iter: 1100
- [11/22 17:39:31] ppdet.engine INFO: Eval iter: 1200
- [11/22 17:39:35] ppdet.engine INFO: Eval iter: 1300
- [11/22 17:39:39] ppdet.engine INFO: Eval iter: 1400
- [11/22 17:39:43] ppdet.engine INFO: Eval iter: 1500
- [11/22 17:39:47] ppdet.engine INFO: Eval iter: 1600
- [11/22 17:39:51] ppdet.engine INFO: Eval iter: 1700
- [11/22 17:39:55] ppdet.engine INFO: Eval iter: 1800
- [11/22 17:39:59] ppdet.engine INFO: Eval iter: 1900
- [11/22 17:40:03] ppdet.engine INFO: Eval iter: 2000
- [11/22 17:40:07] ppdet.engine INFO: Eval iter: 2100
- [11/22 17:40:11] ppdet.engine INFO: Eval iter: 2200
- [11/22 17:40:15] ppdet.engine INFO: Eval iter: 2300
- [11/22 17:40:18] ppdet.engine INFO: Eval iter: 2400
- [11/22 17:40:22] ppdet.engine INFO: Eval iter: 2500
- [11/22 17:40:26] ppdet.engine INFO: Eval iter: 2600
- [11/22 17:40:30] ppdet.engine INFO: Eval iter: 2700
- [11/22 17:40:34] ppdet.engine INFO: Eval iter: 2800
- [11/22 17:40:38] ppdet.engine INFO: Eval iter: 2900
- [11/22 17:40:43] ppdet.engine INFO: Eval iter: 3000
- [11/22 17:40:47] ppdet.engine INFO: Eval iter: 3100
- [11/22 17:40:51] ppdet.engine INFO: Eval iter: 3200
- [11/22 17:40:55] ppdet.engine INFO: Eval iter: 3300
- [11/22 17:40:59] ppdet.engine INFO: Eval iter: 3400
- [11/22 17:41:03] ppdet.engine INFO: Eval iter: 3500
- [11/22 17:41:07] ppdet.engine INFO: Eval iter: 3600
- [11/22 17:41:11] ppdet.engine INFO: Eval iter: 3700
- [11/22 17:41:15] ppdet.engine INFO: Eval iter: 3800
- [11/22 17:41:20] ppdet.engine INFO: Eval iter: 3900
- [11/22 17:41:24] ppdet.engine INFO: Eval iter: 4000
- [11/22 17:41:28] ppdet.engine INFO: Eval iter: 4100
- [11/22 17:41:32] ppdet.engine INFO: Eval iter: 4200
- [11/22 17:41:37] ppdet.engine INFO: Eval iter: 4300
- [11/22 17:41:38] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/22 17:41:38] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 88.35%
- [11/22 17:41:38] ppdet.engine INFO: Total sample number: 4322, averge FPS: 24.67561676992244
- [11/22 17:41:38] ppdet.engine INFO: Best test bbox ap is 0.891.
- [11/22 17:41:38] ppdet.engine INFO: Epoch: [43] [ 0/1215] learning_rate: 0.000300 loss_xy: 2.993100 loss_wh: 0.447498 loss_obj: 0.628424 loss_cls: 0.209732 loss: 4.422963 eta: 2:08:03 batch_cost: 0.3587 data_cost: 0.0286 ips: 22.3025 images/s
- [11/22 17:42:53] ppdet.engine INFO: Epoch: [43] [ 200/1215] learning_rate: 0.000300 loss_xy: 2.971164 loss_wh: 0.451260 loss_obj: 1.269556 loss_cls: 0.308130 loss: 5.112073 eta: 2:06:49 batch_cost: 0.3728 data_cost: 0.0509 ips: 21.4596 images/s
- [11/22 17:44:08] ppdet.engine INFO: Epoch: [43] [ 400/1215] learning_rate: 0.000300 loss_xy: 2.870448 loss_wh: 0.433644 loss_obj: 1.293476 loss_cls: 0.300367 loss: 4.826726 eta: 2:05:36 batch_cost: 0.3737 data_cost: 0.0340 ips: 21.4054 images/s
- [11/22 17:45:23] ppdet.engine INFO: Epoch: [43] [ 600/1215] learning_rate: 0.000300 loss_xy: 2.884181 loss_wh: 0.434100 loss_obj: 1.312097 loss_cls: 0.332880 loss: 5.043772 eta: 2:04:22 batch_cost: 0.3735 data_cost: 0.0240 ips: 21.4208 images/s
- [11/22 17:46:35] ppdet.engine INFO: Epoch: [43] [ 800/1215] learning_rate: 0.000300 loss_xy: 2.815363 loss_wh: 0.430236 loss_obj: 1.174507 loss_cls: 0.303352 loss: 4.859997 eta: 2:03:02 batch_cost: 0.3618 data_cost: 0.0244 ips: 22.1132 images/s
- [11/22 17:47:50] ppdet.engine INFO: Epoch: [43] [1000/1215] learning_rate: 0.000300 loss_xy: 2.881468 loss_wh: 0.470888 loss_obj: 1.244443 loss_cls: 0.312684 loss: 5.088619 eta: 2:01:47 batch_cost: 0.3706 data_cost: 0.0095 ips: 21.5880 images/s
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::BasicEngine::Execute()
- 1 paddle::imperative::EagerGradientAccumulator::SumGrad(std::shared_ptr<paddle::imperative::VariableWrapper>, unsigned long, bool)
- 2 paddle::imperative::VariableWrapperAdd(std::shared_ptr<paddle::imperative::VariableWrapper>, paddle::imperative::VariableWrapper*, bool)
- 3 void paddle::imperative::TensorAdd<paddle::framework::Variable>(paddle::framework::Variable const&, paddle::framework::Variable*)
- 4 paddle::imperative::TensorAddFunctor<float>::result_type paddle::platform::VisitPlace<paddle::imperative::TensorAddFunctor<float> >(phi::Place const&, paddle::imperative::TensorAddFunctor<float> const&)
- 5 paddle::platform::CUDADeviceContext::CublasCall(std::function<void (cublasContext*)> const&) const
- 6 phi::GPUContext::CublasCall(std::function<void (cublasContext*)> const&) const
- 7 std::_Function_handler<void (cublasContext*), phi::funcs::Blas<paddle::platform::CUDADeviceContext>::AXPY<float>(int, float, float const*, float*) const::{lambda(cublasContext*)#1}>::_M_invoke(std::_Any_data const&, cublasContext*&&)
- 8 cublasSaxpy_v2
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669110483 (unix time) try "date -d @1669110483" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e800007be7) received by PID 31800 (TID 0x7f6f04321700) from PID 31719 ***]
-
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:57285', '127.0.0.1:56687']
- I1124 13:46:59.537812 924 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1124 13:47:00.208931 924 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 13:47:00.212136 924 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/24 13:47:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 13:47:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/24 13:47:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/24 13:47:05] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/24 13:47:06] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/24 13:47:06] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/24 13:47:07] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/24 13:47:09] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/24 13:47:09] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/24 13:47:09] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- [11/24 13:47:10] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/24 13:47:10] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/24 13:47:10] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/24 13:47:10] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/24 13:47:11] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 13:47:12] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- [11/24 13:47:13] ppdet.utils.checkpoint INFO: The shape [54] in pretrained weight yolo_head.yolo_output.0.bias is unmatched with the shape [48] in model yolo_head.yolo_output.0.bias. And the weight yolo_head.yolo_output.0.bias will not be loaded
- [11/24 13:47:13] ppdet.utils.checkpoint INFO: The shape [54, 1024, 1, 1] in pretrained weight yolo_head.yolo_output.0.weight is unmatched with the shape [48, 1024, 1, 1] in model yolo_head.yolo_output.0.weight. And the weight yolo_head.yolo_output.0.weight will not be loaded
- [11/24 13:47:13] ppdet.utils.checkpoint INFO: The shape [54] in pretrained weight yolo_head.yolo_output.1.bias is unmatched with the shape [48] in model yolo_head.yolo_output.1.bias. And the weight yolo_head.yolo_output.1.bias will not be loaded
- [11/24 13:47:13] ppdet.utils.checkpoint INFO: The shape [54, 512, 1, 1] in pretrained weight yolo_head.yolo_output.1.weight is unmatched with the shape [48, 512, 1, 1] in model yolo_head.yolo_output.1.weight. And the weight yolo_head.yolo_output.1.weight will not be loaded
- [11/24 13:47:13] ppdet.utils.checkpoint INFO: The shape [54] in pretrained weight yolo_head.yolo_output.2.bias is unmatched with the shape [48] in model yolo_head.yolo_output.2.bias. And the weight yolo_head.yolo_output.2.bias will not be loaded
- [11/24 13:47:13] ppdet.utils.checkpoint INFO: The shape [54, 256, 1, 1] in pretrained weight yolo_head.yolo_output.2.weight is unmatched with the shape [48, 256, 1, 1] in model yolo_head.yolo_output.2.weight. And the weight yolo_head.yolo_output.2.weight will not be loaded
- [11/24 13:47:13] ppdet.utils.checkpoint INFO: Finish loading model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/41.pdparams
- [11/24 13:47:18] ppdet.engine INFO: Epoch: [0] [ 0/1342] learning_rate: 0.000000 loss_xy: 3.583959 loss_wh: 2.960508 loss_obj: 12988.907227 loss_cls: 11.460690 loss: 13006.913086 eta: 3 days, 16:25:40 batch_cost: 3.9536 data_cost: 0.0022 ips: 2.0235 images/s
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::Tracer::TraceOp(std::string const&, paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap, std::map<std::string, std::string, std::less<std::string >, std::allocator<std::pair<std::string const, std::string > > > const&)
- 1 void paddle::imperative::Tracer::TraceOpImpl<paddle::imperative::VarBase>(std::string const&, paddle::imperative::details::NameVarMapTrait<paddle::imperative::VarBase>::Type const&, paddle::imperative::details::NameVarMapTrait<paddle::imperative::VarBase>::Type const&, paddle::framework::AttributeMap&, phi::Place const&, bool, std::map<std::string, std::string, std::less<std::string >, std::allocator<std::pair<std::string const, std::string > > > const&, paddle::framework::AttributeMap*, bool)
- 2 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap const&, paddle::framework::AttributeMap const&)
- 3 void phi::SumKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, std::vector<long, std::allocator<long> > const&, paddle::experimental::DataType, bool, phi::DenseTensor*)
- 4 void phi::funcs::ReduceKernel<float, float, phi::kps::AddFunctor, phi::kps::IdentityFunctor<float, float> >(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor*, phi::kps::IdentityFunctor<float, float> const&, std::vector<int, std::allocator<int> > const&, bool)
- 5 phi::backends::gpu::GetGpuMaxGridDimSize(int)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669268941 (unix time) try "date -d @1669268941" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e80000034c) received by PID 924 (TID 0x7f229edf8700) from PID 844 ***]
-
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:46149', '127.0.0.1:51481']
- I1124 13:49:11.833938 20567 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1124 13:49:12.545950 20567 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 13:49:12.548810 20567 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/24 13:49:14] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 13:49:15] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- No stack trace in paddle, may be caused by external reasons.
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669268956 (unix time) try "date -d @1669268956" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e800005004) received by PID 20567 (TID 0x7fc83ab19700) from PID 20484 ***]
-
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:45051', '127.0.0.1:50917', '127.0.0.1:52063']
- I1124 13:49:48.889228 20812 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1124 13:49:49.530889 20812 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 13:49:49.533885 20812 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/24 13:49:51] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 13:49:52] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/24 13:49:53] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/24 13:49:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/24 13:49:55] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/24 13:49:55] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/24 13:49:56] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/24 13:49:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/24 13:49:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/24 13:49:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- [11/24 13:49:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/24 13:50:00] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/24 13:50:00] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/24 13:50:00] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/24 13:50:01] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 13:50:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- [11/24 13:50:03] ppdet.utils.checkpoint INFO: The shape [54] in pretrained weight yolo_head.yolo_output.0.bias is unmatched with the shape [48] in model yolo_head.yolo_output.0.bias. And the weight yolo_head.yolo_output.0.bias will not be loaded
- [11/24 13:50:03] ppdet.utils.checkpoint INFO: The shape [54, 1024, 1, 1] in pretrained weight yolo_head.yolo_output.0.weight is unmatched with the shape [48, 1024, 1, 1] in model yolo_head.yolo_output.0.weight. And the weight yolo_head.yolo_output.0.weight will not be loaded
- [11/24 13:50:03] ppdet.utils.checkpoint INFO: The shape [54] in pretrained weight yolo_head.yolo_output.1.bias is unmatched with the shape [48] in model yolo_head.yolo_output.1.bias. And the weight yolo_head.yolo_output.1.bias will not be loaded
- [11/24 13:50:03] ppdet.utils.checkpoint INFO: The shape [54, 512, 1, 1] in pretrained weight yolo_head.yolo_output.1.weight is unmatched with the shape [48, 512, 1, 1] in model yolo_head.yolo_output.1.weight. And the weight yolo_head.yolo_output.1.weight will not be loaded
- [11/24 13:50:03] ppdet.utils.checkpoint INFO: The shape [54] in pretrained weight yolo_head.yolo_output.2.bias is unmatched with the shape [48] in model yolo_head.yolo_output.2.bias. And the weight yolo_head.yolo_output.2.bias will not be loaded
- [11/24 13:50:03] ppdet.utils.checkpoint INFO: The shape [54, 256, 1, 1] in pretrained weight yolo_head.yolo_output.2.weight is unmatched with the shape [48, 256, 1, 1] in model yolo_head.yolo_output.2.weight. And the weight yolo_head.yolo_output.2.weight will not be loaded
- [11/24 13:50:03] ppdet.utils.checkpoint INFO: Finish loading model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/41.pdparams
- [11/24 13:50:07] ppdet.engine INFO: Epoch: [0] [ 0/1193] learning_rate: 0.000000 loss_xy: 3.401058 loss_wh: 2.279600 loss_obj: 6679.437012 loss_cls: 10.240681 loss: 6695.358398 eta: 2 days, 22:58:21 batch_cost: 3.5695 data_cost: 1.6713 ips: 2.5214 images/s
- [11/24 13:52:45] ppdet.engine INFO: Epoch: [0] [ 200/1193] learning_rate: 0.000250 loss_xy: 3.688858 loss_wh: 1.827319 loss_obj: 6.880538 loss_cls: 8.546109 loss: 21.060017 eta: 15:54:19 batch_cost: 0.7883 data_cost: 0.0898 ips: 11.4163 images/s
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:43333', '127.0.0.1:54893', '127.0.0.1:55231']
- I1124 13:54:24.720919 55305 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1124 13:54:25.437672 55305 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 13:54:25.440846 55305 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/24 13:54:27] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 13:54:28] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/24 13:54:29] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/24 13:54:30] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/24 13:54:31] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/24 13:54:31] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/24 13:54:32] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/24 13:54:34] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/24 13:54:34] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/24 13:54:34] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- [11/24 13:54:35] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/24 13:54:35] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/24 13:54:35] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/24 13:54:36] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/24 13:54:36] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 13:54:37] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- [11/24 13:54:39] ppdet.utils.checkpoint INFO: The shape [54] in pretrained weight yolo_head.yolo_output.0.bias is unmatched with the shape [48] in model yolo_head.yolo_output.0.bias. And the weight yolo_head.yolo_output.0.bias will not be loaded
- [11/24 13:54:39] ppdet.utils.checkpoint INFO: The shape [54, 1024, 1, 1] in pretrained weight yolo_head.yolo_output.0.weight is unmatched with the shape [48, 1024, 1, 1] in model yolo_head.yolo_output.0.weight. And the weight yolo_head.yolo_output.0.weight will not be loaded
- [11/24 13:54:39] ppdet.utils.checkpoint INFO: The shape [54] in pretrained weight yolo_head.yolo_output.1.bias is unmatched with the shape [48] in model yolo_head.yolo_output.1.bias. And the weight yolo_head.yolo_output.1.bias will not be loaded
- [11/24 13:54:39] ppdet.utils.checkpoint INFO: The shape [54, 512, 1, 1] in pretrained weight yolo_head.yolo_output.1.weight is unmatched with the shape [48, 512, 1, 1] in model yolo_head.yolo_output.1.weight. And the weight yolo_head.yolo_output.1.weight will not be loaded
- [11/24 13:54:39] ppdet.utils.checkpoint INFO: The shape [54] in pretrained weight yolo_head.yolo_output.2.bias is unmatched with the shape [48] in model yolo_head.yolo_output.2.bias. And the weight yolo_head.yolo_output.2.bias will not be loaded
- [11/24 13:54:39] ppdet.utils.checkpoint INFO: The shape [54, 256, 1, 1] in pretrained weight yolo_head.yolo_output.2.weight is unmatched with the shape [48, 256, 1, 1] in model yolo_head.yolo_output.2.weight. And the weight yolo_head.yolo_output.2.weight will not be loaded
- [11/24 13:54:39] ppdet.utils.checkpoint INFO: Finish loading model weights: output/AUG/yolov3_darknet53_270e_voc_garbage/41.pdparams
- [11/24 13:54:43] ppdet.engine INFO: Epoch: [0] [ 0/1534] learning_rate: 0.000000 loss_xy: 3.361591 loss_wh: 2.633784 loss_obj: 8283.836914 loss_cls: 11.022654 loss: 8300.854492 eta: 4 days, 23:27:29 batch_cost: 4.6724 data_cost: 0.0014 ips: 1.4982 images/s
- [11/24 13:55:18] ppdet.engine INFO: Epoch: [0] [ 50/1534] learning_rate: 0.000063 loss_xy: 3.545822 loss_wh: 3.006289 loss_obj: 15.736348 loss_cls: 10.680412 loss: 35.416321 eta: 19:29:12 batch_cost: 0.6844 data_cost: 0.0646 ips: 10.2277 images/s
- [11/24 13:55:46] ppdet.engine INFO: Epoch: [0] [ 100/1534] learning_rate: 0.000125 loss_xy: 3.783221 loss_wh: 2.025087 loss_obj: 9.759952 loss_cls: 9.736393 loss: 24.656061 eta: 17:00:42 batch_cost: 0.5677 data_cost: 0.0538 ips: 12.3306 images/s
- [11/24 13:56:16] ppdet.engine INFO: Epoch: [0] [ 150/1534] learning_rate: 0.000188 loss_xy: 3.447023 loss_wh: 1.321683 loss_obj: 6.280262 loss_cls: 6.652865 loss: 17.385202 eta: 16:29:19 batch_cost: 0.6053 data_cost: 0.0996 ips: 11.5644 images/s
- [11/24 13:56:50] ppdet.engine INFO: Epoch: [0] [ 200/1534] learning_rate: 0.000250 loss_xy: 3.710813 loss_wh: 1.412882 loss_obj: 5.964203 loss_cls: 6.375683 loss: 17.439186 eta: 16:34:48 batch_cost: 0.6618 data_cost: 0.2417 ips: 10.5773 images/s
- [11/24 13:57:23] ppdet.engine INFO: Epoch: [0] [ 250/1534] learning_rate: 0.000313 loss_xy: 3.273320 loss_wh: 1.168235 loss_obj: 4.883933 loss_cls: 4.393138 loss: 13.443457 eta: 16:40:35 batch_cost: 0.6707 data_cost: 0.0125 ips: 10.4374 images/s
- [11/24 13:58:06] ppdet.engine INFO: Epoch: [0] [ 300/1534] learning_rate: 0.000375 loss_xy: 3.529453 loss_wh: 1.233270 loss_obj: 4.889991 loss_cls: 4.188671 loss: 14.369637 eta: 17:30:52 batch_cost: 0.8542 data_cost: 0.1031 ips: 8.1950 images/s
- [11/24 13:58:38] ppdet.engine INFO: Epoch: [0] [ 350/1534] learning_rate: 0.000437 loss_xy: 3.199776 loss_wh: 1.098254 loss_obj: 4.530910 loss_cls: 3.427329 loss: 12.449068 eta: 17:20:31 batch_cost: 0.6424 data_cost: 0.1852 ips: 10.8967 images/s
- [11/24 13:59:07] ppdet.engine INFO: Epoch: [0] [ 400/1534] learning_rate: 0.000500 loss_xy: 3.473182 loss_wh: 1.167083 loss_obj: 4.939625 loss_cls: 3.281349 loss: 12.990562 eta: 16:58:37 batch_cost: 0.5689 data_cost: 0.1370 ips: 12.3046 images/s
- [11/24 13:59:42] ppdet.engine INFO: Epoch: [0] [ 450/1534] learning_rate: 0.000563 loss_xy: 2.812180 loss_wh: 0.969737 loss_obj: 3.894099 loss_cls: 2.372307 loss: 9.826089 eta: 17:04:22 batch_cost: 0.7042 data_cost: 0.1972 ips: 9.9406 images/s
- [11/24 14:00:14] ppdet.engine INFO: Epoch: [0] [ 500/1534] learning_rate: 0.000625 loss_xy: 3.352195 loss_wh: 1.233680 loss_obj: 5.090685 loss_cls: 2.474102 loss: 12.421247 eta: 17:00:44 batch_cost: 0.6509 data_cost: 0.0802 ips: 10.7541 images/s
- [11/24 14:00:47] ppdet.engine INFO: Epoch: [0] [ 550/1534] learning_rate: 0.000688 loss_xy: 2.988070 loss_wh: 1.174240 loss_obj: 4.249016 loss_cls: 2.293247 loss: 10.689631 eta: 16:58:25 batch_cost: 0.6563 data_cost: 0.1581 ips: 10.6655 images/s
- [11/24 14:01:24] ppdet.engine INFO: Epoch: [0] [ 600/1534] learning_rate: 0.000750 loss_xy: 3.037298 loss_wh: 1.056655 loss_obj: 4.011245 loss_cls: 2.012219 loss: 10.467497 eta: 17:07:18 batch_cost: 0.7423 data_cost: 0.2893 ips: 9.4307 images/s
- [11/24 14:01:57] ppdet.engine INFO: Epoch: [0] [ 650/1534] learning_rate: 0.000813 loss_xy: 3.192178 loss_wh: 1.188125 loss_obj: 4.595175 loss_cls: 2.171480 loss: 11.269007 eta: 17:04:16 batch_cost: 0.6530 data_cost: 0.0304 ips: 10.7194 images/s
- [11/24 14:02:32] ppdet.engine INFO: Epoch: [0] [ 700/1534] learning_rate: 0.000875 loss_xy: 3.222226 loss_wh: 1.225494 loss_obj: 3.966979 loss_cls: 1.884312 loss: 10.640764 eta: 17:07:27 batch_cost: 0.7070 data_cost: 0.0090 ips: 9.9008 images/s
- [11/24 14:03:06] ppdet.engine INFO: Epoch: [0] [ 750/1534] learning_rate: 0.000937 loss_xy: 3.361796 loss_wh: 1.246208 loss_obj: 5.235610 loss_cls: 1.971423 loss: 12.351117 eta: 17:06:52 batch_cost: 0.6747 data_cost: 0.0389 ips: 10.3751 images/s
- [11/24 14:03:43] ppdet.engine INFO: Epoch: [0] [ 800/1534] learning_rate: 0.001000 loss_xy: 2.961443 loss_wh: 1.116301 loss_obj: 4.328202 loss_cls: 1.809745 loss: 10.155407 eta: 17:12:08 batch_cost: 0.7362 data_cost: 0.2413 ips: 9.5082 images/s
- [11/24 14:04:23] ppdet.engine INFO: Epoch: [0] [ 850/1534] learning_rate: 0.001063 loss_xy: 3.018098 loss_wh: 1.123518 loss_obj: 4.168916 loss_cls: 1.600045 loss: 10.072302 eta: 17:21:36 batch_cost: 0.7911 data_cost: 0.3619 ips: 8.8480 images/s
- [11/24 14:04:44] ppdet.engine INFO: Epoch: [0] [ 900/1534] learning_rate: 0.001125 loss_xy: 3.271696 loss_wh: 1.110174 loss_obj: 4.461338 loss_cls: 1.745728 loss: 10.689318 eta: 16:59:12 batch_cost: 0.4263 data_cost: 0.0319 ips: 16.4191 images/s
- [11/24 14:05:21] ppdet.engine INFO: Epoch: [0] [ 950/1534] learning_rate: 0.001187 loss_xy: 3.050296 loss_wh: 1.090838 loss_obj: 3.887352 loss_cls: 1.533561 loss: 9.644025 eta: 17:04:05 batch_cost: 0.7393 data_cost: 0.2111 ips: 9.4686 images/s
- [11/24 14:05:54] ppdet.engine INFO: Epoch: [0] [1000/1534] learning_rate: 0.001250 loss_xy: 3.171063 loss_wh: 1.097297 loss_obj: 4.271413 loss_cls: 1.571276 loss: 10.414698 eta: 17:01:55 batch_cost: 0.6534 data_cost: 0.2131 ips: 10.7136 images/s
- [11/24 14:06:28] ppdet.engine INFO: Epoch: [0] [1050/1534] learning_rate: 0.001313 loss_xy: 2.991643 loss_wh: 1.146133 loss_obj: 4.441251 loss_cls: 1.570835 loss: 10.067602 eta: 17:01:32 batch_cost: 0.6760 data_cost: 0.0536 ips: 10.3547 images/s
- [11/24 14:07:04] ppdet.engine INFO: Epoch: [0] [1100/1534] learning_rate: 0.001375 loss_xy: 3.064510 loss_wh: 1.153867 loss_obj: 4.355071 loss_cls: 1.397327 loss: 9.626194 eta: 17:05:02 batch_cost: 0.7325 data_cost: 0.1191 ips: 9.5569 images/s
- [11/24 14:07:31] ppdet.engine INFO: Epoch: [0] [1150/1534] learning_rate: 0.001437 loss_xy: 2.963858 loss_wh: 0.978681 loss_obj: 3.869754 loss_cls: 1.646774 loss: 9.867052 eta: 16:54:53 batch_cost: 0.5307 data_cost: 0.1055 ips: 13.1896 images/s
- [11/24 14:08:04] ppdet.engine INFO: Epoch: [0] [1200/1534] learning_rate: 0.001500 loss_xy: 2.929232 loss_wh: 1.032489 loss_obj: 4.126871 loss_cls: 1.410820 loss: 9.704563 eta: 16:54:09 batch_cost: 0.6671 data_cost: 0.0149 ips: 10.4939 images/s
- [11/24 14:08:37] ppdet.engine INFO: Epoch: [0] [1250/1534] learning_rate: 0.001563 loss_xy: 2.947800 loss_wh: 1.118047 loss_obj: 4.042204 loss_cls: 1.363262 loss: 9.748967 eta: 16:52:19 batch_cost: 0.6488 data_cost: 0.1809 ips: 10.7894 images/s
- [11/24 14:09:06] ppdet.engine INFO: Epoch: [0] [1300/1534] learning_rate: 0.001625 loss_xy: 2.788626 loss_wh: 0.942713 loss_obj: 3.512803 loss_cls: 1.187420 loss: 8.788980 eta: 16:47:13 batch_cost: 0.5909 data_cost: 0.0117 ips: 11.8466 images/s
- [11/24 14:09:36] ppdet.engine INFO: Epoch: [0] [1350/1534] learning_rate: 0.001688 loss_xy: 2.933552 loss_wh: 0.966883 loss_obj: 4.461253 loss_cls: 1.221154 loss: 9.233853 eta: 16:42:00 batch_cost: 0.5828 data_cost: 0.0084 ips: 12.0104 images/s
- [11/24 14:10:08] ppdet.engine INFO: Epoch: [0] [1400/1534] learning_rate: 0.001750 loss_xy: 3.234232 loss_wh: 0.984987 loss_obj: 4.364760 loss_cls: 1.415101 loss: 9.971373 eta: 16:40:15 batch_cost: 0.6406 data_cost: 0.1383 ips: 10.9269 images/s
- [11/24 14:10:41] ppdet.engine INFO: Epoch: [0] [1450/1534] learning_rate: 0.001812 loss_xy: 3.116692 loss_wh: 1.212262 loss_obj: 4.214636 loss_cls: 1.414902 loss: 9.582967 eta: 16:40:28 batch_cost: 0.6770 data_cost: 0.1269 ips: 10.3393 images/s
- [11/24 14:11:11] ppdet.engine INFO: Epoch: [0] [1500/1534] learning_rate: 0.001875 loss_xy: 2.922828 loss_wh: 1.154429 loss_obj: 3.964667 loss_cls: 1.330209 loss: 9.545134 eta: 16:36:16 batch_cost: 0.5898 data_cost: 0.0229 ips: 11.8684 images/s
- [11/24 14:11:33] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 14:11:35] ppdet.engine INFO: Eval iter: 0
- [11/24 14:11:39] ppdet.engine INFO: Eval iter: 100
- [11/24 14:11:43] ppdet.engine INFO: Eval iter: 200
- [11/24 14:11:47] ppdet.engine INFO: Eval iter: 300
- [11/24 14:11:50] ppdet.engine INFO: Eval iter: 400
- [11/24 14:11:54] ppdet.engine INFO: Eval iter: 500
- [11/24 14:11:58] ppdet.engine INFO: Eval iter: 600
- [11/24 14:12:02] ppdet.engine INFO: Eval iter: 700
- [11/24 14:12:06] ppdet.engine INFO: Eval iter: 800
- [11/24 14:12:10] ppdet.engine INFO: Eval iter: 900
- [11/24 14:12:14] ppdet.engine INFO: Eval iter: 1000
- [11/24 14:12:18] ppdet.engine INFO: Eval iter: 1100
- [11/24 14:12:22] ppdet.engine INFO: Eval iter: 1200
- [11/24 14:12:26] ppdet.engine INFO: Eval iter: 1300
- [11/24 14:12:30] ppdet.engine INFO: Eval iter: 1400
- [11/24 14:12:35] ppdet.engine INFO: Eval iter: 1500
- [11/24 14:12:39] ppdet.engine INFO: Eval iter: 1600
- [11/24 14:12:43] ppdet.engine INFO: Eval iter: 1700
- [11/24 14:12:47] ppdet.engine INFO: Eval iter: 1800
- [11/24 14:12:51] ppdet.engine INFO: Eval iter: 1900
- [11/24 14:12:55] ppdet.engine INFO: Eval iter: 2000
- [11/24 14:12:59] ppdet.engine INFO: Eval iter: 2100
- [11/24 14:13:03] ppdet.engine INFO: Eval iter: 2200
- [11/24 14:13:07] ppdet.engine INFO: Eval iter: 2300
- [11/24 14:13:11] ppdet.engine INFO: Eval iter: 2400
- [11/24 14:13:15] ppdet.engine INFO: Eval iter: 2500
- [11/24 14:13:19] ppdet.engine INFO: Eval iter: 2600
- [11/24 14:13:23] ppdet.engine INFO: Eval iter: 2700
- [11/24 14:13:27] ppdet.engine INFO: Eval iter: 2800
- [11/24 14:13:31] ppdet.engine INFO: Eval iter: 2900
- [11/24 14:13:35] ppdet.engine INFO: Eval iter: 3000
- [11/24 14:13:39] ppdet.engine INFO: Eval iter: 3100
- [11/24 14:13:44] ppdet.engine INFO: Eval iter: 3200
- [11/24 14:13:47] ppdet.engine INFO: Eval iter: 3300
- [11/24 14:13:51] ppdet.engine INFO: Eval iter: 3400
- [11/24 14:13:56] ppdet.engine INFO: Eval iter: 3500
- [11/24 14:14:00] ppdet.engine INFO: Eval iter: 3600
- [11/24 14:14:03] ppdet.engine INFO: Eval iter: 3700
- [11/24 14:14:08] ppdet.engine INFO: Eval iter: 3800
- [11/24 14:14:12] ppdet.engine INFO: Eval iter: 3900
- [11/24 14:14:16] ppdet.engine INFO: Eval iter: 4000
- [11/24 14:14:19] ppdet.engine INFO: Eval iter: 4100
- [11/24 14:14:23] ppdet.engine INFO: Eval iter: 4200
- [11/24 14:14:28] ppdet.engine INFO: Eval iter: 4300
- [11/24 14:14:32] ppdet.engine INFO: Eval iter: 4400
- [11/24 14:14:36] ppdet.engine INFO: Eval iter: 4500
- [11/24 14:14:40] ppdet.engine INFO: Eval iter: 4600
- [11/24 14:14:44] ppdet.engine INFO: Eval iter: 4700
- [11/24 14:14:47] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 14:14:47] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 79.65%
- [11/24 14:14:47] ppdet.engine INFO: Total sample number: 4773, averge FPS: 24.79599889245089
- [11/24 14:14:47] ppdet.engine INFO: Best test bbox ap is 0.797.
- [11/24 14:14:48] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 14:14:48] ppdet.engine INFO: Epoch: [1] [ 0/1534] learning_rate: 0.001918 loss_xy: 3.083075 loss_wh: 1.156156 loss_obj: 4.186234 loss_cls: 1.429034 loss: 10.035299 eta: 16:34:28 batch_cost: 0.5887 data_cost: 0.0239 ips: 11.8915 images/s
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:59721', '127.0.0.1:59515', '127.0.0.1:46869']
- I1124 16:09:52.743609 1019 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1124 16:09:53.587150 1019 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 16:09:53.591281 1019 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/24 16:09:55] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 16:09:56] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/24 16:09:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/24 16:09:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/24 16:09:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/24 16:09:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/24 16:10:00] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/24 16:10:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/24 16:10:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/24 16:10:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- [11/24 16:10:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/24 16:10:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/24 16:10:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/24 16:10:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/24 16:10:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 16:10:05] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- [11/24 16:10:06] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage/best_model.pdparams
- [11/24 16:10:12] ppdet.engine INFO: Epoch: [1] [ 0/1534] learning_rate: 0.001918 loss_xy: 2.875748 loss_wh: 1.096282 loss_obj: 4.640458 loss_cls: 1.285789 loss: 9.898277 eta: 4 days, 15:35:35 batch_cost: 4.4388 data_cost: 0.0012 ips: 1.5770 images/s
- [11/24 16:10:45] ppdet.engine INFO: Epoch: [1] [ 50/1534] learning_rate: 0.001980 loss_xy: 2.927859 loss_wh: 0.944056 loss_obj: 4.275543 loss_cls: 1.304338 loss: 9.152302 eta: 18:52:48 batch_cost: 0.6777 data_cost: 0.0691 ips: 10.3298 images/s
- [11/24 16:11:14] ppdet.engine INFO: Epoch: [1] [ 100/1534] learning_rate: 0.002043 loss_xy: 3.190293 loss_wh: 1.054523 loss_obj: 4.723373 loss_cls: 1.488155 loss: 10.589924 eta: 16:30:33 batch_cost: 0.5615 data_cost: 0.0544 ips: 12.4657 images/s
- [11/24 16:11:44] ppdet.engine INFO: Epoch: [1] [ 150/1534] learning_rate: 0.002105 loss_xy: 2.965056 loss_wh: 0.997591 loss_obj: 4.388167 loss_cls: 1.214353 loss: 9.271738 eta: 16:00:20 batch_cost: 0.5979 data_cost: 0.0978 ips: 11.7073 images/s
- [11/24 16:12:16] ppdet.engine INFO: Epoch: [1] [ 200/1534] learning_rate: 0.002168 loss_xy: 3.339620 loss_wh: 1.101749 loss_obj: 4.930855 loss_cls: 1.541252 loss: 11.143095 eta: 16:06:51 batch_cost: 0.6565 data_cost: 0.2443 ips: 10.6621 images/s
- [11/24 16:12:50] ppdet.engine INFO: Epoch: [1] [ 250/1534] learning_rate: 0.002230 loss_xy: 3.125120 loss_wh: 1.109073 loss_obj: 3.763535 loss_cls: 1.215374 loss: 9.154163 eta: 16:15:01 batch_cost: 0.6714 data_cost: 0.0121 ips: 10.4260 images/s
- [11/24 16:13:33] ppdet.engine INFO: Epoch: [1] [ 300/1534] learning_rate: 0.002293 loss_xy: 3.282604 loss_wh: 1.093524 loss_obj: 4.491249 loss_cls: 1.544618 loss: 10.211472 eta: 17:06:24 batch_cost: 0.8561 data_cost: 0.1072 ips: 8.1764 images/s
- [11/24 16:14:05] ppdet.engine INFO: Epoch: [1] [ 350/1534] learning_rate: 0.002355 loss_xy: 3.038744 loss_wh: 0.920999 loss_obj: 3.945143 loss_cls: 1.291975 loss: 9.542325 eta: 16:56:14 batch_cost: 0.6379 data_cost: 0.1795 ips: 10.9742 images/s
- [11/24 16:14:33] ppdet.engine INFO: Epoch: [1] [ 400/1534] learning_rate: 0.002417 loss_xy: 3.343896 loss_wh: 1.043453 loss_obj: 4.288301 loss_cls: 1.429583 loss: 10.045492 eta: 16:33:29 batch_cost: 0.5578 data_cost: 0.1371 ips: 12.5494 images/s
- [11/24 16:15:07] ppdet.engine INFO: Epoch: [1] [ 450/1534] learning_rate: 0.002480 loss_xy: 2.684653 loss_wh: 0.832316 loss_obj: 3.652759 loss_cls: 1.174548 loss: 8.260841 eta: 16:38:20 batch_cost: 0.6941 data_cost: 0.1907 ips: 10.0856 images/s
- [11/24 16:15:40] ppdet.engine INFO: Epoch: [1] [ 500/1534] learning_rate: 0.002542 loss_xy: 3.218676 loss_wh: 1.087920 loss_obj: 4.611267 loss_cls: 1.331879 loss: 10.446569 eta: 16:36:58 batch_cost: 0.6597 data_cost: 0.0796 ips: 10.6110 images/s
- [11/24 16:16:13] ppdet.engine INFO: Epoch: [1] [ 550/1534] learning_rate: 0.002605 loss_xy: 2.944452 loss_wh: 1.004444 loss_obj: 3.845105 loss_cls: 1.443704 loss: 9.015618 eta: 16:34:45 batch_cost: 0.6523 data_cost: 0.1490 ips: 10.7305 images/s
- [11/24 16:16:50] ppdet.engine INFO: Epoch: [1] [ 600/1534] learning_rate: 0.002667 loss_xy: 2.990955 loss_wh: 0.885992 loss_obj: 3.556718 loss_cls: 1.125355 loss: 9.198679 eta: 16:44:13 batch_cost: 0.7440 data_cost: 0.2974 ips: 9.4091 images/s
- [11/24 16:17:23] ppdet.engine INFO: Epoch: [1] [ 650/1534] learning_rate: 0.002730 loss_xy: 3.198981 loss_wh: 1.076268 loss_obj: 4.018896 loss_cls: 1.213276 loss: 9.337774 eta: 16:41:04 batch_cost: 0.6476 data_cost: 0.0292 ips: 10.8096 images/s
- [11/24 16:17:58] ppdet.engine INFO: Epoch: [1] [ 700/1534] learning_rate: 0.002792 loss_xy: 3.144958 loss_wh: 0.909832 loss_obj: 3.861422 loss_cls: 1.333979 loss: 9.403290 eta: 16:44:19 batch_cost: 0.7041 data_cost: 0.0149 ips: 9.9416 images/s
- [11/24 16:18:32] ppdet.engine INFO: Epoch: [1] [ 750/1534] learning_rate: 0.002855 loss_xy: 3.393974 loss_wh: 1.168794 loss_obj: 4.604689 loss_cls: 1.407987 loss: 10.856413 eta: 16:43:29 batch_cost: 0.6683 data_cost: 0.0447 ips: 10.4745 images/s
- [11/24 16:19:08] ppdet.engine INFO: Epoch: [1] [ 800/1534] learning_rate: 0.002917 loss_xy: 2.876913 loss_wh: 1.061089 loss_obj: 3.775888 loss_cls: 1.160112 loss: 9.422299 eta: 16:48:05 batch_cost: 0.7261 data_cost: 0.2415 ips: 9.6403 images/s
- [11/24 16:19:47] ppdet.engine INFO: Epoch: [1] [ 850/1534] learning_rate: 0.002980 loss_xy: 2.981766 loss_wh: 0.979215 loss_obj: 3.866754 loss_cls: 1.263777 loss: 9.137622 eta: 16:56:13 batch_cost: 0.7732 data_cost: 0.3512 ips: 9.0530 images/s
- [11/24 16:20:08] ppdet.engine INFO: Epoch: [1] [ 900/1534] learning_rate: 0.003043 loss_xy: 3.269639 loss_wh: 1.160955 loss_obj: 4.252564 loss_cls: 1.524719 loss: 9.158363 eta: 16:34:03 batch_cost: 0.4195 data_cost: 0.0351 ips: 16.6846 images/s
- [11/24 16:20:44] ppdet.engine INFO: Epoch: [1] [ 950/1534] learning_rate: 0.003105 loss_xy: 3.108182 loss_wh: 0.972852 loss_obj: 3.744244 loss_cls: 1.105819 loss: 8.602254 eta: 16:38:04 batch_cost: 0.7238 data_cost: 0.2172 ips: 9.6718 images/s
- [11/24 16:21:17] ppdet.engine INFO: Epoch: [1] [1000/1534] learning_rate: 0.003167 loss_xy: 3.233608 loss_wh: 1.050611 loss_obj: 3.817551 loss_cls: 1.204989 loss: 9.927918 eta: 16:36:45 batch_cost: 0.6585 data_cost: 0.2273 ips: 10.6309 images/s
- [11/24 16:21:50] ppdet.engine INFO: Epoch: [1] [1050/1534] learning_rate: 0.003230 loss_xy: 2.939094 loss_wh: 1.012313 loss_obj: 4.038918 loss_cls: 1.334512 loss: 9.328714 eta: 16:36:28 batch_cost: 0.6722 data_cost: 0.0575 ips: 10.4139 images/s
- [11/24 16:22:27] ppdet.engine INFO: Epoch: [1] [1100/1534] learning_rate: 0.003292 loss_xy: 3.138871 loss_wh: 1.003555 loss_obj: 3.852297 loss_cls: 0.981135 loss: 8.871263 eta: 16:39:46 batch_cost: 0.7253 data_cost: 0.1246 ips: 9.6509 images/s
- [11/24 16:22:53] ppdet.engine INFO: Epoch: [1] [1150/1534] learning_rate: 0.003355 loss_xy: 3.028111 loss_wh: 0.889034 loss_obj: 3.572925 loss_cls: 1.236883 loss: 9.058002 eta: 16:30:05 batch_cost: 0.5300 data_cost: 0.1118 ips: 13.2074 images/s
- [11/24 16:23:26] ppdet.engine INFO: Epoch: [1] [1200/1534] learning_rate: 0.003417 loss_xy: 2.946030 loss_wh: 0.893558 loss_obj: 3.698017 loss_cls: 1.158664 loss: 8.903545 eta: 16:29:20 batch_cost: 0.6615 data_cost: 0.0149 ips: 10.5815 images/s
- [11/24 16:23:59] ppdet.engine INFO: Epoch: [1] [1250/1534] learning_rate: 0.003480 loss_xy: 2.939990 loss_wh: 1.023453 loss_obj: 3.666893 loss_cls: 1.139680 loss: 8.977142 eta: 16:27:56 batch_cost: 0.6504 data_cost: 0.1788 ips: 10.7627 images/s
- [11/24 16:24:28] ppdet.engine INFO: Epoch: [1] [1300/1534] learning_rate: 0.003543 loss_xy: 2.819346 loss_wh: 0.794440 loss_obj: 3.106867 loss_cls: 1.012913 loss: 7.728424 eta: 16:22:48 batch_cost: 0.5841 data_cost: 0.0103 ips: 11.9833 images/s
- [11/24 16:24:57] ppdet.engine INFO: Epoch: [1] [1350/1534] learning_rate: 0.003605 loss_xy: 2.917819 loss_wh: 0.801588 loss_obj: 3.827180 loss_cls: 1.020241 loss: 8.912693 eta: 16:17:40 batch_cost: 0.5777 data_cost: 0.0060 ips: 12.1171 images/s
- [11/24 16:25:29] ppdet.engine INFO: Epoch: [1] [1400/1534] learning_rate: 0.003668 loss_xy: 3.164589 loss_wh: 0.860410 loss_obj: 4.066391 loss_cls: 1.161491 loss: 9.562803 eta: 16:16:21 batch_cost: 0.6434 data_cost: 0.1376 ips: 10.8798 images/s
- [11/24 16:26:03] ppdet.engine INFO: Epoch: [1] [1450/1534] learning_rate: 0.003730 loss_xy: 3.062343 loss_wh: 1.044851 loss_obj: 4.071656 loss_cls: 1.116640 loss: 10.032112 eta: 16:16:46 batch_cost: 0.6762 data_cost: 0.1262 ips: 10.3512 images/s
- [11/24 16:26:33] ppdet.engine INFO: Epoch: [1] [1500/1534] learning_rate: 0.003792 loss_xy: 2.904434 loss_wh: 0.920486 loss_obj: 3.738038 loss_cls: 1.217739 loss: 9.158510 eta: 16:13:08 batch_cost: 0.5958 data_cost: 0.0250 ips: 11.7487 images/s
- [11/24 16:26:55] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 16:26:57] ppdet.engine INFO: Eval iter: 0
- [11/24 16:27:01] ppdet.engine INFO: Eval iter: 100
- [11/24 16:27:05] ppdet.engine INFO: Eval iter: 200
- [11/24 16:27:10] ppdet.engine INFO: Eval iter: 300
- [11/24 16:27:14] ppdet.engine INFO: Eval iter: 400
- [11/24 16:27:18] ppdet.engine INFO: Eval iter: 500
- [11/24 16:27:22] ppdet.engine INFO: Eval iter: 600
- [11/24 16:27:26] ppdet.engine INFO: Eval iter: 700
- [11/24 16:27:30] ppdet.engine INFO: Eval iter: 800
- [11/24 16:27:34] ppdet.engine INFO: Eval iter: 900
- [11/24 16:27:38] ppdet.engine INFO: Eval iter: 1000
- [11/24 16:27:42] ppdet.engine INFO: Eval iter: 1100
- [11/24 16:27:46] ppdet.engine INFO: Eval iter: 1200
- [11/24 16:27:50] ppdet.engine INFO: Eval iter: 1300
- [11/24 16:27:54] ppdet.engine INFO: Eval iter: 1400
- [11/24 16:27:58] ppdet.engine INFO: Eval iter: 1500
- [11/24 16:28:02] ppdet.engine INFO: Eval iter: 1600
- [11/24 16:28:06] ppdet.engine INFO: Eval iter: 1700
- [11/24 16:28:10] ppdet.engine INFO: Eval iter: 1800
- [11/24 16:28:14] ppdet.engine INFO: Eval iter: 1900
- [11/24 16:28:18] ppdet.engine INFO: Eval iter: 2000
- [11/24 16:28:22] ppdet.engine INFO: Eval iter: 2100
- [11/24 16:28:26] ppdet.engine INFO: Eval iter: 2200
- [11/24 16:28:30] ppdet.engine INFO: Eval iter: 2300
- [11/24 16:28:34] ppdet.engine INFO: Eval iter: 2400
- [11/24 16:28:38] ppdet.engine INFO: Eval iter: 2500
- [11/24 16:28:42] ppdet.engine INFO: Eval iter: 2600
- [11/24 16:28:46] ppdet.engine INFO: Eval iter: 2700
- [11/24 16:28:50] ppdet.engine INFO: Eval iter: 2800
- [11/24 16:28:54] ppdet.engine INFO: Eval iter: 2900
- [11/24 16:28:58] ppdet.engine INFO: Eval iter: 3000
- [11/24 16:29:02] ppdet.engine INFO: Eval iter: 3100
- [11/24 16:29:06] ppdet.engine INFO: Eval iter: 3200
- [11/24 16:29:10] ppdet.engine INFO: Eval iter: 3300
- [11/24 16:29:14] ppdet.engine INFO: Eval iter: 3400
- [11/24 16:29:18] ppdet.engine INFO: Eval iter: 3500
- [11/24 16:29:22] ppdet.engine INFO: Eval iter: 3600
- [11/24 16:29:26] ppdet.engine INFO: Eval iter: 3700
- [11/24 16:29:30] ppdet.engine INFO: Eval iter: 3800
- [11/24 16:29:35] ppdet.engine INFO: Eval iter: 3900
- [11/24 16:29:39] ppdet.engine INFO: Eval iter: 4000
- [11/24 16:29:42] ppdet.engine INFO: Eval iter: 4100
- [11/24 16:29:46] ppdet.engine INFO: Eval iter: 4200
- [11/24 16:29:50] ppdet.engine INFO: Eval iter: 4300
- [11/24 16:29:54] ppdet.engine INFO: Eval iter: 4400
- [11/24 16:29:58] ppdet.engine INFO: Eval iter: 4500
- [11/24 16:30:02] ppdet.engine INFO: Eval iter: 4600
- [11/24 16:30:06] ppdet.engine INFO: Eval iter: 4700
- [11/24 16:30:09] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 16:30:09] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 69.95%
- [11/24 16:30:09] ppdet.engine INFO: Total sample number: 4773, averge FPS: 24.82147657593169
- [11/24 16:30:09] ppdet.engine INFO: Best test bbox ap is 0.699.
- [11/24 16:30:13] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 16:30:13] ppdet.engine INFO: Epoch: [2] [ 0/1534] learning_rate: 0.003835 loss_xy: 3.064435 loss_wh: 0.911585 loss_obj: 4.356935 loss_cls: 1.171910 loss: 9.370679 eta: 16:11:27 batch_cost: 0.5893 data_cost: 0.0267 ips: 11.8785 images/s
- [11/24 16:30:45] ppdet.engine INFO: Epoch: [2] [ 50/1534] learning_rate: 0.003897 loss_xy: 2.942944 loss_wh: 0.826261 loss_obj: 4.197628 loss_cls: 1.521263 loss: 9.161444 eta: 16:10:21 batch_cost: 0.6434 data_cost: 0.1126 ips: 10.8792 images/s
- [11/24 16:31:27] ppdet.engine INFO: Epoch: [2] [ 100/1534] learning_rate: 0.003960 loss_xy: 2.940829 loss_wh: 0.968683 loss_obj: 4.360307 loss_cls: 1.552199 loss: 9.970424 eta: 16:18:06 batch_cost: 0.8377 data_cost: 0.3147 ips: 8.3560 images/s
- [11/24 16:32:04] ppdet.engine INFO: Epoch: [2] [ 150/1534] learning_rate: 0.004023 loss_xy: 3.325117 loss_wh: 1.041147 loss_obj: 4.715698 loss_cls: 1.441701 loss: 10.430090 eta: 16:20:40 batch_cost: 0.7313 data_cost: 0.2971 ips: 9.5721 images/s
- [11/24 16:32:37] ppdet.engine INFO: Epoch: [2] [ 200/1534] learning_rate: 0.004085 loss_xy: 3.040336 loss_wh: 0.903486 loss_obj: 4.712743 loss_cls: 1.337516 loss: 9.837658 eta: 16:20:23 batch_cost: 0.6688 data_cost: 0.0461 ips: 10.4661 images/s
- [11/24 16:33:02] ppdet.engine INFO: Epoch: [2] [ 250/1534] learning_rate: 0.004148 loss_xy: 3.234161 loss_wh: 0.892682 loss_obj: 4.828940 loss_cls: 1.439259 loss: 10.424141 eta: 16:12:53 batch_cost: 0.4950 data_cost: 0.0460 ips: 14.1409 images/s
- [11/24 16:33:34] ppdet.engine INFO: Epoch: [2] [ 300/1534] learning_rate: 0.004210 loss_xy: 3.015962 loss_wh: 0.783733 loss_obj: 4.255517 loss_cls: 1.358296 loss: 9.445065 eta: 16:11:41 batch_cost: 0.6417 data_cost: 0.1398 ips: 10.9078 images/s
- [11/24 16:34:06] ppdet.engine INFO: Epoch: [2] [ 350/1534] learning_rate: 0.004273 loss_xy: 3.290705 loss_wh: 0.972952 loss_obj: 4.494318 loss_cls: 1.200344 loss: 9.956854 eta: 16:09:59 batch_cost: 0.6278 data_cost: 0.0006 ips: 11.1509 images/s
- [11/24 16:34:39] ppdet.engine INFO: Epoch: [2] [ 400/1534] learning_rate: 0.004335 loss_xy: 3.415229 loss_wh: 1.101921 loss_obj: 5.703148 loss_cls: 1.756555 loss: 12.422053 eta: 16:10:02 batch_cost: 0.6726 data_cost: 0.1887 ips: 10.4069 images/s
- [11/24 16:35:16] ppdet.engine INFO: Epoch: [2] [ 450/1534] learning_rate: 0.004397 loss_xy: 3.017037 loss_wh: 0.927758 loss_obj: 4.498892 loss_cls: 1.460860 loss: 9.957586 eta: 16:12:12 batch_cost: 0.7300 data_cost: 0.0433 ips: 9.5890 images/s
- [11/24 16:35:48] ppdet.engine INFO: Epoch: [2] [ 500/1534] learning_rate: 0.004460 loss_xy: 3.133540 loss_wh: 0.929954 loss_obj: 4.516765 loss_cls: 1.223262 loss: 10.012350 eta: 16:10:37 batch_cost: 0.6307 data_cost: 0.0628 ips: 11.0992 images/s
- [11/24 16:36:15] ppdet.engine INFO: Epoch: [2] [ 550/1534] learning_rate: 0.004522 loss_xy: 3.134184 loss_wh: 0.904897 loss_obj: 4.959890 loss_cls: 1.407398 loss: 10.607609 eta: 16:05:58 batch_cost: 0.5420 data_cost: 0.1087 ips: 12.9156 images/s
- [11/24 16:36:54] ppdet.engine INFO: Epoch: [2] [ 600/1534] learning_rate: 0.004585 loss_xy: 3.011437 loss_wh: 0.897159 loss_obj: 4.537273 loss_cls: 1.213116 loss: 9.394266 eta: 16:09:37 batch_cost: 0.7770 data_cost: 0.2461 ips: 9.0093 images/s
- [11/24 16:37:23] ppdet.engine INFO: Epoch: [2] [ 650/1534] learning_rate: 0.004647 loss_xy: 3.152078 loss_wh: 0.882895 loss_obj: 5.228784 loss_cls: 1.488928 loss: 11.046423 eta: 16:06:48 batch_cost: 0.5911 data_cost: 0.0955 ips: 11.8414 images/s
- [11/24 16:37:57] ppdet.engine INFO: Epoch: [2] [ 700/1534] learning_rate: 0.004710 loss_xy: 3.099212 loss_wh: 0.830489 loss_obj: 4.552097 loss_cls: 1.521140 loss: 9.785996 eta: 16:06:39 batch_cost: 0.6688 data_cost: 0.0764 ips: 10.4672 images/s
- [11/24 16:38:26] ppdet.engine INFO: Epoch: [2] [ 750/1534] learning_rate: 0.004772 loss_xy: 3.045029 loss_wh: 0.931917 loss_obj: 4.863248 loss_cls: 1.508103 loss: 10.232292 eta: 16:04:04 batch_cost: 0.5938 data_cost: 0.1945 ips: 11.7881 images/s
- [11/24 16:39:01] ppdet.engine INFO: Epoch: [2] [ 800/1534] learning_rate: 0.004835 loss_xy: 2.983632 loss_wh: 0.789689 loss_obj: 4.667772 loss_cls: 1.114963 loss: 9.235991 eta: 16:04:54 batch_cost: 0.6995 data_cost: 0.0625 ips: 10.0071 images/s
- [11/24 16:39:30] ppdet.engine INFO: Epoch: [2] [ 850/1534] learning_rate: 0.004897 loss_xy: 3.496978 loss_wh: 1.090867 loss_obj: 5.102177 loss_cls: 1.568424 loss: 11.487505 eta: 16:01:31 batch_cost: 0.5649 data_cost: 0.0971 ips: 12.3923 images/s
- [11/24 16:40:00] ppdet.engine INFO: Epoch: [2] [ 900/1534] learning_rate: 0.004960 loss_xy: 2.930837 loss_wh: 0.859149 loss_obj: 4.468829 loss_cls: 1.385168 loss: 9.631342 eta: 15:59:32 batch_cost: 0.6067 data_cost: 0.0514 ips: 11.5371 images/s
- [11/24 16:40:31] ppdet.engine INFO: Epoch: [2] [ 950/1534] learning_rate: 0.005000 loss_xy: 3.198170 loss_wh: 0.991565 loss_obj: 5.233963 loss_cls: 1.485520 loss: 10.931957 eta: 15:58:04 batch_cost: 0.6222 data_cost: 0.0338 ips: 11.2499 images/s
- [11/24 16:40:55] ppdet.engine INFO: Epoch: [2] [1000/1534] learning_rate: 0.005000 loss_xy: 3.003151 loss_wh: 0.867462 loss_obj: 4.728297 loss_cls: 1.513713 loss: 10.186551 eta: 15:52:24 batch_cost: 0.4760 data_cost: 0.0213 ips: 14.7064 images/s
- [11/24 16:41:26] ppdet.engine INFO: Epoch: [2] [1050/1534] learning_rate: 0.005000 loss_xy: 3.203890 loss_wh: 0.902714 loss_obj: 5.129824 loss_cls: 1.505837 loss: 10.501303 eta: 15:51:11 batch_cost: 0.6261 data_cost: 0.0005 ips: 11.1805 images/s
- [11/24 16:41:58] ppdet.engine INFO: Epoch: [2] [1100/1534] learning_rate: 0.005000 loss_xy: 3.396695 loss_wh: 1.024233 loss_obj: 5.058123 loss_cls: 1.479914 loss: 11.482101 eta: 15:49:55 batch_cost: 0.6231 data_cost: 0.1452 ips: 11.2342 images/s
- [11/24 16:42:31] ppdet.engine INFO: Epoch: [2] [1150/1534] learning_rate: 0.005000 loss_xy: 3.028393 loss_wh: 1.002453 loss_obj: 5.005142 loss_cls: 1.467812 loss: 10.471587 eta: 15:49:54 batch_cost: 0.6673 data_cost: 0.0641 ips: 10.4896 images/s
- [11/24 16:43:01] ppdet.engine INFO: Epoch: [2] [1200/1534] learning_rate: 0.005000 loss_xy: 3.085010 loss_wh: 0.893541 loss_obj: 4.914080 loss_cls: 1.458287 loss: 10.853613 eta: 15:48:02 batch_cost: 0.5997 data_cost: 0.1514 ips: 11.6732 images/s
- [11/24 16:43:28] ppdet.engine INFO: Epoch: [2] [1250/1534] learning_rate: 0.005000 loss_xy: 3.299938 loss_wh: 0.959754 loss_obj: 4.779429 loss_cls: 1.748444 loss: 11.023569 eta: 15:44:54 batch_cost: 0.5490 data_cost: 0.0715 ips: 12.7501 images/s
- [11/24 16:43:56] ppdet.engine INFO: Epoch: [2] [1300/1534] learning_rate: 0.005000 loss_xy: 3.002660 loss_wh: 1.016485 loss_obj: 5.051055 loss_cls: 1.503773 loss: 10.940964 eta: 15:41:48 batch_cost: 0.5473 data_cost: 0.0572 ips: 12.7890 images/s
- [11/24 16:44:30] ppdet.engine INFO: Epoch: [2] [1350/1534] learning_rate: 0.005000 loss_xy: 3.157627 loss_wh: 0.972432 loss_obj: 4.386884 loss_cls: 1.278664 loss: 10.237175 eta: 15:42:00 batch_cost: 0.6733 data_cost: 0.1212 ips: 10.3967 images/s
- [11/24 16:44:57] ppdet.engine INFO: Epoch: [2] [1400/1534] learning_rate: 0.005000 loss_xy: 3.175933 loss_wh: 0.879464 loss_obj: 5.036693 loss_cls: 1.442580 loss: 10.454319 eta: 15:38:47 batch_cost: 0.5374 data_cost: 0.0836 ips: 13.0248 images/s
- [11/24 16:45:25] ppdet.engine INFO: Epoch: [2] [1450/1534] learning_rate: 0.005000 loss_xy: 3.135676 loss_wh: 0.975937 loss_obj: 5.095954 loss_cls: 1.603059 loss: 11.201767 eta: 15:36:22 batch_cost: 0.5660 data_cost: 0.0493 ips: 12.3677 images/s
- [11/24 16:46:00] ppdet.engine INFO: Epoch: [2] [1500/1534] learning_rate: 0.005000 loss_xy: 2.765433 loss_wh: 0.841492 loss_obj: 4.255256 loss_cls: 1.613544 loss: 9.244644 eta: 15:37:29 batch_cost: 0.7109 data_cost: 0.0752 ips: 9.8471 images/s
- [11/24 16:46:21] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 16:46:21] ppdet.engine INFO: Eval iter: 0
- [11/24 16:46:25] ppdet.engine INFO: Eval iter: 100
- [11/24 16:46:29] ppdet.engine INFO: Eval iter: 200
- [11/24 16:46:33] ppdet.engine INFO: Eval iter: 300
- [11/24 16:46:37] ppdet.engine INFO: Eval iter: 400
- [11/24 16:46:41] ppdet.engine INFO: Eval iter: 500
- [11/24 16:46:45] ppdet.engine INFO: Eval iter: 600
- [11/24 16:46:49] ppdet.engine INFO: Eval iter: 700
- [11/24 16:46:53] ppdet.engine INFO: Eval iter: 800
- [11/24 16:46:57] ppdet.engine INFO: Eval iter: 900
- [11/24 16:47:02] ppdet.engine INFO: Eval iter: 1000
- [11/24 16:47:06] ppdet.engine INFO: Eval iter: 1100
- [11/24 16:47:10] ppdet.engine INFO: Eval iter: 1200
- [11/24 16:47:14] ppdet.engine INFO: Eval iter: 1300
- [11/24 16:47:18] ppdet.engine INFO: Eval iter: 1400
- [11/24 16:47:22] ppdet.engine INFO: Eval iter: 1500
- [11/24 16:47:26] ppdet.engine INFO: Eval iter: 1600
- [11/24 16:47:30] ppdet.engine INFO: Eval iter: 1700
- [11/24 16:47:34] ppdet.engine INFO: Eval iter: 1800
- [11/24 16:47:38] ppdet.engine INFO: Eval iter: 1900
- [11/24 16:47:42] ppdet.engine INFO: Eval iter: 2000
- [11/24 16:47:46] ppdet.engine INFO: Eval iter: 2100
- [11/24 16:47:50] ppdet.engine INFO: Eval iter: 2200
- [11/24 16:47:54] ppdet.engine INFO: Eval iter: 2300
- [11/24 16:47:58] ppdet.engine INFO: Eval iter: 2400
- [11/24 16:48:02] ppdet.engine INFO: Eval iter: 2500
- [11/24 16:48:06] ppdet.engine INFO: Eval iter: 2600
- [11/24 16:48:10] ppdet.engine INFO: Eval iter: 2700
- [11/24 16:48:14] ppdet.engine INFO: Eval iter: 2800
- [11/24 16:48:18] ppdet.engine INFO: Eval iter: 2900
- [11/24 16:48:22] ppdet.engine INFO: Eval iter: 3000
- [11/24 16:48:26] ppdet.engine INFO: Eval iter: 3100
- [11/24 16:48:31] ppdet.engine INFO: Eval iter: 3200
- [11/24 16:48:35] ppdet.engine INFO: Eval iter: 3300
- [11/24 16:48:39] ppdet.engine INFO: Eval iter: 3400
- [11/24 16:48:43] ppdet.engine INFO: Eval iter: 3500
- [11/24 16:48:47] ppdet.engine INFO: Eval iter: 3600
- [11/24 16:48:51] ppdet.engine INFO: Eval iter: 3700
- [11/24 16:48:55] ppdet.engine INFO: Eval iter: 3800
- [11/24 16:48:59] ppdet.engine INFO: Eval iter: 3900
- [11/24 16:49:03] ppdet.engine INFO: Eval iter: 4000
- [11/24 16:49:07] ppdet.engine INFO: Eval iter: 4100
- [11/24 16:49:11] ppdet.engine INFO: Eval iter: 4200
- [11/24 16:49:15] ppdet.engine INFO: Eval iter: 4300
- [11/24 16:49:19] ppdet.engine INFO: Eval iter: 4400
- [11/24 16:49:23] ppdet.engine INFO: Eval iter: 4500
- [11/24 16:49:27] ppdet.engine INFO: Eval iter: 4600
- [11/24 16:49:31] ppdet.engine INFO: Eval iter: 4700
- [11/24 16:49:34] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 16:49:34] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 78.57%
- [11/24 16:49:34] ppdet.engine INFO: Total sample number: 4773, averge FPS: 24.74546814437685
- [11/24 16:49:34] ppdet.engine INFO: Best test bbox ap is 0.786.
- [11/24 16:49:38] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 16:49:38] ppdet.engine INFO: Epoch: [3] [ 0/1534] learning_rate: 0.005000 loss_xy: 3.082482 loss_wh: 0.834353 loss_obj: 4.779385 loss_cls: 1.598262 loss: 10.651309 eta: 15:36:09 batch_cost: 0.5595 data_cost: 0.0293 ips: 12.5114 images/s
- [11/24 16:50:09] ppdet.engine INFO: Epoch: [3] [ 50/1534] learning_rate: 0.005000 loss_xy: 3.229274 loss_wh: 0.888755 loss_obj: 4.530074 loss_cls: 1.264549 loss: 10.330365 eta: 15:34:51 batch_cost: 0.6100 data_cost: 0.0438 ips: 11.4754 images/s
- [11/24 16:50:41] ppdet.engine INFO: Epoch: [3] [ 100/1534] learning_rate: 0.005000 loss_xy: 3.010656 loss_wh: 0.931510 loss_obj: 4.347396 loss_cls: 1.239604 loss: 9.336501 eta: 15:34:30 batch_cost: 0.6496 data_cost: 0.2014 ips: 10.7752 images/s
- [11/24 16:51:09] ppdet.engine INFO: Epoch: [3] [ 150/1534] learning_rate: 0.005000 loss_xy: 3.361638 loss_wh: 0.923779 loss_obj: 4.781902 loss_cls: 1.324539 loss: 10.937794 eta: 15:32:12 batch_cost: 0.5641 data_cost: 0.0005 ips: 12.4099 images/s
- [11/24 16:51:41] ppdet.engine INFO: Epoch: [3] [ 200/1534] learning_rate: 0.005000 loss_xy: 3.096210 loss_wh: 0.810914 loss_obj: 4.287298 loss_cls: 1.217946 loss: 9.949451 eta: 15:31:40 batch_cost: 0.6404 data_cost: 0.0535 ips: 10.9302 images/s
- [11/24 16:52:13] ppdet.engine INFO: Epoch: [3] [ 250/1534] learning_rate: 0.005000 loss_xy: 3.591442 loss_wh: 0.940699 loss_obj: 5.181079 loss_cls: 1.340531 loss: 10.988579 eta: 15:30:46 batch_cost: 0.6245 data_cost: 0.1096 ips: 11.2095 images/s
- [11/24 16:52:41] ppdet.engine INFO: Epoch: [3] [ 300/1534] learning_rate: 0.005000 loss_xy: 3.158279 loss_wh: 1.008348 loss_obj: 4.911231 loss_cls: 1.275126 loss: 10.316002 eta: 15:28:24 batch_cost: 0.5554 data_cost: 0.1194 ips: 12.6032 images/s
- [11/24 16:53:15] ppdet.engine INFO: Epoch: [3] [ 350/1534] learning_rate: 0.005000 loss_xy: 3.171793 loss_wh: 1.029523 loss_obj: 4.373105 loss_cls: 1.388592 loss: 10.045737 eta: 15:28:42 batch_cost: 0.6787 data_cost: 0.2255 ips: 10.3143 images/s
- [11/24 16:53:43] ppdet.engine INFO: Epoch: [3] [ 400/1534] learning_rate: 0.005000 loss_xy: 3.041169 loss_wh: 0.793952 loss_obj: 4.360600 loss_cls: 1.615792 loss: 10.091534 eta: 15:26:40 batch_cost: 0.5680 data_cost: 0.0450 ips: 12.3230 images/s
- [11/24 16:54:11] ppdet.engine INFO: Epoch: [3] [ 450/1534] learning_rate: 0.005000 loss_xy: 2.940388 loss_wh: 0.839669 loss_obj: 4.459917 loss_cls: 1.410544 loss: 9.507497 eta: 15:24:28 batch_cost: 0.5580 data_cost: 0.1034 ips: 12.5455 images/s
- [11/24 16:54:45] ppdet.engine INFO: Epoch: [3] [ 500/1534] learning_rate: 0.005000 loss_xy: 3.402417 loss_wh: 0.876593 loss_obj: 4.935153 loss_cls: 1.567164 loss: 11.184736 eta: 15:24:57 batch_cost: 0.6870 data_cost: 0.0006 ips: 10.1887 images/s
- [11/24 16:55:16] ppdet.engine INFO: Epoch: [3] [ 550/1534] learning_rate: 0.005000 loss_xy: 3.012018 loss_wh: 0.783212 loss_obj: 4.097132 loss_cls: 1.519142 loss: 9.252481 eta: 15:23:58 batch_cost: 0.6162 data_cost: 0.0155 ips: 11.3594 images/s
- [11/24 16:55:46] ppdet.engine INFO: Epoch: [3] [ 600/1534] learning_rate: 0.005000 loss_xy: 2.926747 loss_wh: 0.784934 loss_obj: 4.402675 loss_cls: 1.240540 loss: 9.234055 eta: 15:22:31 batch_cost: 0.5912 data_cost: 0.0571 ips: 11.8396 images/s
- [11/24 16:56:12] ppdet.engine INFO: Epoch: [3] [ 650/1534] learning_rate: 0.005000 loss_xy: 3.223555 loss_wh: 0.835516 loss_obj: 5.066288 loss_cls: 1.261568 loss: 10.232929 eta: 15:19:47 batch_cost: 0.5242 data_cost: 0.0450 ips: 13.3534 images/s
- [11/24 16:56:39] ppdet.engine INFO: Epoch: [3] [ 700/1534] learning_rate: 0.005000 loss_xy: 3.228332 loss_wh: 0.806017 loss_obj: 4.292135 loss_cls: 1.338419 loss: 10.007969 eta: 15:17:29 batch_cost: 0.5433 data_cost: 0.1758 ips: 12.8837 images/s
- [11/24 16:57:16] ppdet.engine INFO: Epoch: [3] [ 750/1534] learning_rate: 0.005000 loss_xy: 3.503234 loss_wh: 0.876408 loss_obj: 5.467278 loss_cls: 1.436247 loss: 10.881039 eta: 15:18:55 batch_cost: 0.7387 data_cost: 0.2040 ips: 9.4759 images/s
- [11/24 16:57:51] ppdet.engine INFO: Epoch: [3] [ 800/1534] learning_rate: 0.005000 loss_xy: 3.075739 loss_wh: 0.870065 loss_obj: 4.032300 loss_cls: 1.430310 loss: 9.449238 eta: 15:19:24 batch_cost: 0.6906 data_cost: 0.0726 ips: 10.1355 images/s
- [11/24 16:58:21] ppdet.engine INFO: Epoch: [3] [ 850/1534] learning_rate: 0.005000 loss_xy: 3.054488 loss_wh: 0.867442 loss_obj: 4.836720 loss_cls: 1.455474 loss: 10.176119 eta: 15:18:16 batch_cost: 0.6037 data_cost: 0.0703 ips: 11.5958 images/s
- [11/24 16:58:58] ppdet.engine INFO: Epoch: [3] [ 900/1534] learning_rate: 0.005000 loss_xy: 3.262928 loss_wh: 0.949972 loss_obj: 4.705281 loss_cls: 1.404888 loss: 10.630437 eta: 15:19:33 batch_cost: 0.7361 data_cost: 0.2460 ips: 9.5102 images/s
- [11/24 16:59:28] ppdet.engine INFO: Epoch: [3] [ 950/1534] learning_rate: 0.005000 loss_xy: 3.289985 loss_wh: 0.867964 loss_obj: 4.439239 loss_cls: 1.556350 loss: 10.351473 eta: 15:18:33 batch_cost: 0.6121 data_cost: 0.0107 ips: 11.4365 images/s
- [11/24 16:59:59] ppdet.engine INFO: Epoch: [3] [1000/1534] learning_rate: 0.005000 loss_xy: 3.151752 loss_wh: 0.909987 loss_obj: 5.100385 loss_cls: 1.399312 loss: 11.062254 eta: 15:17:25 batch_cost: 0.6030 data_cost: 0.0691 ips: 11.6082 images/s
- [11/24 17:00:28] ppdet.engine INFO: Epoch: [3] [1050/1534] learning_rate: 0.005000 loss_xy: 3.172149 loss_wh: 0.805724 loss_obj: 4.461917 loss_cls: 1.391340 loss: 9.972581 eta: 15:15:58 batch_cost: 0.5842 data_cost: 0.0609 ips: 11.9819 images/s
- [11/24 17:01:11] ppdet.engine INFO: Epoch: [3] [1100/1534] learning_rate: 0.005000 loss_xy: 2.905349 loss_wh: 0.756377 loss_obj: 3.792602 loss_cls: 1.009334 loss: 8.415628 eta: 15:19:27 batch_cost: 0.8682 data_cost: 0.3611 ips: 8.0629 images/s
- [11/24 17:01:46] ppdet.engine INFO: Epoch: [3] [1150/1534] learning_rate: 0.005000 loss_xy: 2.983631 loss_wh: 0.780883 loss_obj: 4.032805 loss_cls: 1.257199 loss: 9.245743 eta: 15:19:48 batch_cost: 0.6915 data_cost: 0.0410 ips: 10.1226 images/s
- [11/24 17:02:10] ppdet.engine INFO: Epoch: [3] [1200/1534] learning_rate: 0.005000 loss_xy: 3.132674 loss_wh: 0.926139 loss_obj: 4.337335 loss_cls: 1.323868 loss: 9.838586 eta: 15:16:30 batch_cost: 0.4745 data_cost: 0.1207 ips: 14.7523 images/s
- [11/24 17:02:41] ppdet.engine INFO: Epoch: [3] [1250/1534] learning_rate: 0.005000 loss_xy: 3.075538 loss_wh: 0.929567 loss_obj: 4.412124 loss_cls: 1.235860 loss: 10.088814 eta: 15:15:47 batch_cost: 0.6268 data_cost: 0.1629 ips: 11.1675 images/s
- [11/24 17:03:11] ppdet.engine INFO: Epoch: [3] [1300/1534] learning_rate: 0.005000 loss_xy: 2.770839 loss_wh: 0.715072 loss_obj: 4.198661 loss_cls: 1.113083 loss: 8.774418 eta: 15:14:29 batch_cost: 0.5904 data_cost: 0.0526 ips: 11.8559 images/s
- [11/24 17:03:42] ppdet.engine INFO: Epoch: [3] [1350/1534] learning_rate: 0.005000 loss_xy: 3.665555 loss_wh: 0.942349 loss_obj: 4.626921 loss_cls: 1.384114 loss: 11.047030 eta: 15:13:36 batch_cost: 0.6159 data_cost: 0.1244 ips: 11.3651 images/s
- [11/24 17:04:14] ppdet.engine INFO: Epoch: [3] [1400/1534] learning_rate: 0.005000 loss_xy: 3.263133 loss_wh: 0.905142 loss_obj: 4.887136 loss_cls: 1.091042 loss: 9.946460 eta: 15:13:19 batch_cost: 0.6517 data_cost: 0.1154 ips: 10.7418 images/s
- [11/24 17:04:40] ppdet.engine INFO: Epoch: [3] [1450/1534] learning_rate: 0.005000 loss_xy: 3.000381 loss_wh: 0.810095 loss_obj: 3.978102 loss_cls: 1.414348 loss: 9.113180 eta: 15:10:53 batch_cost: 0.5171 data_cost: 0.0976 ips: 13.5362 images/s
- [11/24 17:05:15] ppdet.engine INFO: Epoch: [3] [1500/1534] learning_rate: 0.005000 loss_xy: 2.890056 loss_wh: 0.829057 loss_obj: 4.311395 loss_cls: 1.336590 loss: 9.436143 eta: 15:11:19 batch_cost: 0.6965 data_cost: 0.0888 ips: 10.0509 images/s
- [11/24 17:05:32] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 17:05:32] ppdet.engine INFO: Eval iter: 0
- [11/24 17:05:36] ppdet.engine INFO: Eval iter: 100
- [11/24 17:05:40] ppdet.engine INFO: Eval iter: 200
- [11/24 17:05:44] ppdet.engine INFO: Eval iter: 300
- [11/24 17:05:48] ppdet.engine INFO: Eval iter: 400
- [11/24 17:05:52] ppdet.engine INFO: Eval iter: 500
- [11/24 17:05:56] ppdet.engine INFO: Eval iter: 600
- [11/24 17:06:00] ppdet.engine INFO: Eval iter: 700
- [11/24 17:06:04] ppdet.engine INFO: Eval iter: 800
- [11/24 17:06:08] ppdet.engine INFO: Eval iter: 900
- [11/24 17:06:12] ppdet.engine INFO: Eval iter: 1000
- [11/24 17:06:16] ppdet.engine INFO: Eval iter: 1100
- [11/24 17:06:20] ppdet.engine INFO: Eval iter: 1200
- [11/24 17:06:24] ppdet.engine INFO: Eval iter: 1300
- [11/24 17:06:28] ppdet.engine INFO: Eval iter: 1400
- [11/24 17:06:32] ppdet.engine INFO: Eval iter: 1500
- [11/24 17:06:36] ppdet.engine INFO: Eval iter: 1600
- [11/24 17:06:40] ppdet.engine INFO: Eval iter: 1700
- [11/24 17:06:44] ppdet.engine INFO: Eval iter: 1800
- [11/24 17:06:48] ppdet.engine INFO: Eval iter: 1900
- [11/24 17:06:52] ppdet.engine INFO: Eval iter: 2000
- [11/24 17:06:56] ppdet.engine INFO: Eval iter: 2100
- [11/24 17:07:00] ppdet.engine INFO: Eval iter: 2200
- [11/24 17:07:04] ppdet.engine INFO: Eval iter: 2300
- [11/24 17:07:08] ppdet.engine INFO: Eval iter: 2400
- [11/24 17:07:12] ppdet.engine INFO: Eval iter: 2500
- [11/24 17:07:16] ppdet.engine INFO: Eval iter: 2600
- [11/24 17:07:20] ppdet.engine INFO: Eval iter: 2700
- [11/24 17:07:24] ppdet.engine INFO: Eval iter: 2800
- [11/24 17:07:28] ppdet.engine INFO: Eval iter: 2900
- [11/24 17:07:32] ppdet.engine INFO: Eval iter: 3000
- [11/24 17:07:36] ppdet.engine INFO: Eval iter: 3100
- [11/24 17:07:40] ppdet.engine INFO: Eval iter: 3200
- [11/24 17:07:44] ppdet.engine INFO: Eval iter: 3300
- [11/24 17:07:48] ppdet.engine INFO: Eval iter: 3400
- [11/24 17:07:52] ppdet.engine INFO: Eval iter: 3500
- [11/24 17:07:56] ppdet.engine INFO: Eval iter: 3600
- [11/24 17:08:00] ppdet.engine INFO: Eval iter: 3700
- [11/24 17:08:04] ppdet.engine INFO: Eval iter: 3800
- [11/24 17:08:08] ppdet.engine INFO: Eval iter: 3900
- [11/24 17:08:12] ppdet.engine INFO: Eval iter: 4000
- [11/24 17:08:16] ppdet.engine INFO: Eval iter: 4100
- [11/24 17:08:20] ppdet.engine INFO: Eval iter: 4200
- [11/24 17:08:24] ppdet.engine INFO: Eval iter: 4300
- [11/24 17:08:28] ppdet.engine INFO: Eval iter: 4400
- [11/24 17:08:32] ppdet.engine INFO: Eval iter: 4500
- [11/24 17:08:36] ppdet.engine INFO: Eval iter: 4600
- [11/24 17:08:40] ppdet.engine INFO: Eval iter: 4700
- [11/24 17:08:43] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 17:08:43] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 79.57%
- [11/24 17:08:43] ppdet.engine INFO: Total sample number: 4773, averge FPS: 24.99741957381966
- [11/24 17:08:43] ppdet.engine INFO: Best test bbox ap is 0.796.
- [11/24 17:08:46] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 17:08:47] ppdet.engine INFO: Epoch: [4] [ 0/1534] learning_rate: 0.005000 loss_xy: 2.967264 loss_wh: 0.783154 loss_obj: 3.914229 loss_cls: 1.173182 loss: 8.997509 eta: 15:09:10 batch_cost: 0.4732 data_cost: 0.1247 ips: 14.7924 images/s
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:44009', '127.0.0.1:59739', '127.0.0.1:43219']
- I1124 17:09:56.237408 48689 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1124 17:09:57.131036 48689 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 17:09:57.135172 48689 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/24 17:09:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 17:09:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/24 17:10:01] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/24 17:10:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/24 17:10:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/24 17:10:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/24 17:10:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/24 17:10:06] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/24 17:10:06] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/24 17:10:06] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- [11/24 17:10:07] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/24 17:10:07] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/24 17:10:07] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/24 17:10:08] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/24 17:10:08] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 17:10:09] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- [11/24 17:10:10] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage/best_model.pdparams
- [11/24 17:10:15] ppdet.engine INFO: Epoch: [4] [ 0/1193] learning_rate: 0.005000 loss_xy: 2.782032 loss_wh: 0.987891 loss_obj: 4.834598 loss_cls: 1.199734 loss: 9.804255 eta: 3 days, 2:09:39 batch_cost: 3.9962 data_cost: 1.7180 ips: 2.2521 images/s
- [11/24 17:10:48] ppdet.engine INFO: Epoch: [4] [ 50/1193] learning_rate: 0.002500 loss_xy: 3.175237 loss_wh: 0.761496 loss_obj: 4.657146 loss_cls: 1.227887 loss: 9.696098 eta: 13:31:00 batch_cost: 0.6636 data_cost: 0.0826 ips: 13.5630 images/s
- [11/24 17:11:27] ppdet.engine INFO: Epoch: [4] [ 100/1193] learning_rate: 0.002500 loss_xy: 3.078290 loss_wh: 0.630624 loss_obj: 3.884529 loss_cls: 1.027899 loss: 8.993387 eta: 14:00:24 batch_cost: 0.7834 data_cost: 0.0969 ips: 11.4882 images/s
- [11/24 17:12:12] ppdet.engine INFO: Epoch: [4] [ 150/1193] learning_rate: 0.002500 loss_xy: 3.120849 loss_wh: 0.663230 loss_obj: 3.790101 loss_cls: 1.120339 loss: 8.649271 eta: 14:50:00 batch_cost: 0.8924 data_cost: 0.1163 ips: 10.0847 images/s
- [11/24 17:12:54] ppdet.engine INFO: Epoch: [4] [ 200/1193] learning_rate: 0.002500 loss_xy: 3.054896 loss_wh: 0.641706 loss_obj: 4.157578 loss_cls: 0.960152 loss: 9.052227 eta: 14:59:22 batch_cost: 0.8375 data_cost: 0.0594 ips: 10.7466 images/s
- [11/24 17:13:34] ppdet.engine INFO: Epoch: [4] [ 250/1193] learning_rate: 0.002500 loss_xy: 3.221839 loss_wh: 0.653757 loss_obj: 3.910824 loss_cls: 0.965342 loss: 8.831301 eta: 14:58:06 batch_cost: 0.8075 data_cost: 0.0862 ips: 11.1457 images/s
- [11/24 17:14:08] ppdet.engine INFO: Epoch: [4] [ 300/1193] learning_rate: 0.002500 loss_xy: 3.212601 loss_wh: 0.725473 loss_obj: 4.345290 loss_cls: 1.088271 loss: 9.628627 eta: 14:31:50 batch_cost: 0.6706 data_cost: 0.0245 ips: 13.4199 images/s
- [11/24 17:14:44] ppdet.engine INFO: Epoch: [4] [ 350/1193] learning_rate: 0.002500 loss_xy: 3.397719 loss_wh: 0.755611 loss_obj: 4.301578 loss_cls: 0.970741 loss: 9.341368 eta: 14:21:28 batch_cost: 0.7249 data_cost: 0.0869 ips: 12.4147 images/s
- [11/24 17:15:27] ppdet.engine INFO: Epoch: [4] [ 400/1193] learning_rate: 0.002500 loss_xy: 3.353548 loss_wh: 0.791999 loss_obj: 4.785318 loss_cls: 1.007849 loss: 10.420635 eta: 14:32:42 batch_cost: 0.8638 data_cost: 0.0597 ips: 10.4189 images/s
- [11/24 17:16:06] ppdet.engine INFO: Epoch: [4] [ 450/1193] learning_rate: 0.002500 loss_xy: 3.214338 loss_wh: 0.656493 loss_obj: 3.572964 loss_cls: 0.885469 loss: 8.411852 eta: 14:31:23 batch_cost: 0.7831 data_cost: 0.0626 ips: 11.4925 images/s
- [11/24 17:16:45] ppdet.engine INFO: Epoch: [4] [ 500/1193] learning_rate: 0.002500 loss_xy: 2.958081 loss_wh: 0.644578 loss_obj: 4.095665 loss_cls: 0.912793 loss: 8.630177 eta: 14:29:14 batch_cost: 0.7744 data_cost: 0.1426 ips: 11.6225 images/s
- [11/24 17:17:22] ppdet.engine INFO: Epoch: [4] [ 550/1193] learning_rate: 0.002500 loss_xy: 3.520280 loss_wh: 0.749334 loss_obj: 4.398980 loss_cls: 0.959582 loss: 9.850580 eta: 14:22:53 batch_cost: 0.7298 data_cost: 0.3070 ips: 12.3325 images/s
- [11/24 17:18:00] ppdet.engine INFO: Epoch: [4] [ 600/1193] learning_rate: 0.002500 loss_xy: 3.259456 loss_wh: 0.690855 loss_obj: 3.909049 loss_cls: 0.853864 loss: 8.805504 eta: 14:21:16 batch_cost: 0.7708 data_cost: 0.0430 ips: 11.6757 images/s
- [11/24 17:18:41] ppdet.engine INFO: Epoch: [4] [ 650/1193] learning_rate: 0.002500 loss_xy: 3.283944 loss_wh: 0.754460 loss_obj: 3.735699 loss_cls: 0.973885 loss: 9.040024 eta: 14:24:10 batch_cost: 0.8225 data_cost: 0.1902 ips: 10.9426 images/s
- [11/24 17:19:20] ppdet.engine INFO: Epoch: [4] [ 700/1193] learning_rate: 0.002500 loss_xy: 3.209212 loss_wh: 0.666984 loss_obj: 3.885943 loss_cls: 0.887271 loss: 8.861429 eta: 14:22:55 batch_cost: 0.7761 data_cost: 0.1911 ips: 11.5970 images/s
- [11/24 17:20:00] ppdet.engine INFO: Epoch: [4] [ 750/1193] learning_rate: 0.002500 loss_xy: 3.241578 loss_wh: 0.653923 loss_obj: 3.742257 loss_cls: 0.863435 loss: 8.854259 eta: 14:22:51 batch_cost: 0.7912 data_cost: 0.0428 ips: 11.3755 images/s
- [11/24 17:20:37] ppdet.engine INFO: Epoch: [4] [ 800/1193] learning_rate: 0.002500 loss_xy: 3.264558 loss_wh: 0.653000 loss_obj: 3.655027 loss_cls: 0.905860 loss: 8.537922 eta: 14:19:28 batch_cost: 0.7440 data_cost: 0.1931 ips: 12.0967 images/s
- [11/24 17:21:14] ppdet.engine INFO: Epoch: [4] [ 850/1193] learning_rate: 0.002500 loss_xy: 3.088553 loss_wh: 0.623869 loss_obj: 3.542712 loss_cls: 1.037661 loss: 8.759368 eta: 14:16:25 batch_cost: 0.7441 data_cost: 0.1712 ips: 12.0954 images/s
- [11/24 17:21:48] ppdet.engine INFO: Epoch: [4] [ 900/1193] learning_rate: 0.002500 loss_xy: 3.086190 loss_wh: 0.656229 loss_obj: 3.426476 loss_cls: 0.885446 loss: 8.233873 eta: 14:09:44 batch_cost: 0.6802 data_cost: 0.0401 ips: 13.2322 images/s
- [11/24 17:22:27] ppdet.engine INFO: Epoch: [4] [ 950/1193] learning_rate: 0.002500 loss_xy: 3.250997 loss_wh: 0.624511 loss_obj: 3.365702 loss_cls: 0.849946 loss: 8.144585 eta: 14:09:00 batch_cost: 0.7721 data_cost: 0.0603 ips: 11.6563 images/s
- [11/24 17:23:05] ppdet.engine INFO: Epoch: [4] [1000/1193] learning_rate: 0.002500 loss_xy: 2.944134 loss_wh: 0.582789 loss_obj: 3.324008 loss_cls: 0.813201 loss: 7.852965 eta: 14:07:37 batch_cost: 0.7599 data_cost: 0.2455 ips: 11.8444 images/s
- [11/24 17:23:45] ppdet.engine INFO: Epoch: [4] [1050/1193] learning_rate: 0.002500 loss_xy: 3.223045 loss_wh: 0.704365 loss_obj: 3.912784 loss_cls: 1.112625 loss: 8.808756 eta: 14:08:31 batch_cost: 0.8025 data_cost: 0.1736 ips: 11.2145 images/s
- [11/24 17:24:27] ppdet.engine INFO: Epoch: [4] [1100/1193] learning_rate: 0.002500 loss_xy: 3.312888 loss_wh: 0.718590 loss_obj: 3.946808 loss_cls: 0.870021 loss: 9.029760 eta: 14:10:33 batch_cost: 0.8280 data_cost: 0.2888 ips: 10.8702 images/s
- [11/24 17:25:05] ppdet.engine INFO: Epoch: [4] [1150/1193] learning_rate: 0.002500 loss_xy: 3.429996 loss_wh: 0.687977 loss_obj: 3.748777 loss_cls: 0.986614 loss: 9.043113 eta: 14:09:10 batch_cost: 0.7614 data_cost: 0.1685 ips: 11.8200 images/s
- [11/24 17:25:31] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 17:25:32] ppdet.engine INFO: Eval iter: 0
- [11/24 17:25:37] ppdet.engine INFO: Eval iter: 100
- [11/24 17:25:41] ppdet.engine INFO: Eval iter: 200
- [11/24 17:25:45] ppdet.engine INFO: Eval iter: 300
- [11/24 17:25:49] ppdet.engine INFO: Eval iter: 400
- [11/24 17:25:53] ppdet.engine INFO: Eval iter: 500
- [11/24 17:25:57] ppdet.engine INFO: Eval iter: 600
- [11/24 17:26:01] ppdet.engine INFO: Eval iter: 700
- [11/24 17:26:05] ppdet.engine INFO: Eval iter: 800
- [11/24 17:26:09] ppdet.engine INFO: Eval iter: 900
- [11/24 17:26:13] ppdet.engine INFO: Eval iter: 1000
- [11/24 17:26:17] ppdet.engine INFO: Eval iter: 1100
- [11/24 17:26:21] ppdet.engine INFO: Eval iter: 1200
- [11/24 17:26:25] ppdet.engine INFO: Eval iter: 1300
- [11/24 17:26:29] ppdet.engine INFO: Eval iter: 1400
- [11/24 17:26:33] ppdet.engine INFO: Eval iter: 1500
- [11/24 17:26:37] ppdet.engine INFO: Eval iter: 1600
- [11/24 17:26:40] ppdet.engine INFO: Eval iter: 1700
- [11/24 17:26:44] ppdet.engine INFO: Eval iter: 1800
- [11/24 17:26:48] ppdet.engine INFO: Eval iter: 1900
- [11/24 17:26:53] ppdet.engine INFO: Eval iter: 2000
- [11/24 17:26:57] ppdet.engine INFO: Eval iter: 2100
- [11/24 17:27:01] ppdet.engine INFO: Eval iter: 2200
- [11/24 17:27:05] ppdet.engine INFO: Eval iter: 2300
- [11/24 17:27:09] ppdet.engine INFO: Eval iter: 2400
- [11/24 17:27:13] ppdet.engine INFO: Eval iter: 2500
- [11/24 17:27:17] ppdet.engine INFO: Eval iter: 2600
- [11/24 17:27:21] ppdet.engine INFO: Eval iter: 2700
- [11/24 17:27:25] ppdet.engine INFO: Eval iter: 2800
- [11/24 17:27:29] ppdet.engine INFO: Eval iter: 2900
- [11/24 17:27:33] ppdet.engine INFO: Eval iter: 3000
- [11/24 17:27:37] ppdet.engine INFO: Eval iter: 3100
- [11/24 17:27:41] ppdet.engine INFO: Eval iter: 3200
- [11/24 17:27:46] ppdet.engine INFO: Eval iter: 3300
- [11/24 17:27:50] ppdet.engine INFO: Eval iter: 3400
- [11/24 17:27:54] ppdet.engine INFO: Eval iter: 3500
- [11/24 17:27:58] ppdet.engine INFO: Eval iter: 3600
- [11/24 17:28:02] ppdet.engine INFO: Eval iter: 3700
- [11/24 17:28:06] ppdet.engine INFO: Eval iter: 3800
- [11/24 17:28:10] ppdet.engine INFO: Eval iter: 3900
- [11/24 17:28:14] ppdet.engine INFO: Eval iter: 4000
- [11/24 17:28:18] ppdet.engine INFO: Eval iter: 4100
- [11/24 17:28:22] ppdet.engine INFO: Eval iter: 4200
- [11/24 17:28:26] ppdet.engine INFO: Eval iter: 4300
- [11/24 17:28:30] ppdet.engine INFO: Eval iter: 4400
- [11/24 17:28:34] ppdet.engine INFO: Eval iter: 4500
- [11/24 17:28:38] ppdet.engine INFO: Eval iter: 4600
- [11/24 17:28:42] ppdet.engine INFO: Eval iter: 4700
- [11/24 17:28:45] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 17:28:45] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 88.44%
- [11/24 17:28:45] ppdet.engine INFO: Total sample number: 4773, averge FPS: 24.76746398473173
- [11/24 17:28:45] ppdet.engine INFO: Best test bbox ap is 0.884.
- [11/24 17:28:49] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 17:28:49] ppdet.engine INFO: Epoch: [5] [ 0/1193] learning_rate: 0.002500 loss_xy: 3.166675 loss_wh: 0.665120 loss_obj: 3.774890 loss_cls: 0.894117 loss: 8.930134 eta: 14:00:56 batch_cost: 0.6123 data_cost: 0.0393 ips: 14.6985 images/s
- [11/24 17:29:32] ppdet.engine INFO: Epoch: [5] [ 50/1193] learning_rate: 0.002500 loss_xy: 3.419957 loss_wh: 0.692450 loss_obj: 4.082058 loss_cls: 0.936661 loss: 9.208252 eta: 14:04:00 batch_cost: 0.8535 data_cost: 0.1527 ips: 10.5454 images/s
- [11/24 17:30:13] ppdet.engine INFO: Epoch: [5] [ 100/1193] learning_rate: 0.002500 loss_xy: 3.217815 loss_wh: 0.654078 loss_obj: 3.621505 loss_cls: 0.900769 loss: 8.642252 eta: 14:05:19 batch_cost: 0.8187 data_cost: 0.3005 ips: 10.9926 images/s
- [11/24 17:30:49] ppdet.engine INFO: Epoch: [5] [ 150/1193] learning_rate: 0.002500 loss_xy: 3.085418 loss_wh: 0.621422 loss_obj: 3.626148 loss_cls: 0.914510 loss: 8.817179 eta: 14:02:15 batch_cost: 0.7143 data_cost: 0.1049 ips: 12.6004 images/s
- [11/24 17:31:24] ppdet.engine INFO: Epoch: [5] [ 200/1193] learning_rate: 0.002500 loss_xy: 3.313715 loss_wh: 0.762771 loss_obj: 3.921145 loss_cls: 0.952807 loss: 9.298550 eta: 13:58:45 batch_cost: 0.6991 data_cost: 0.0955 ips: 12.8741 images/s
- [11/24 17:32:06] ppdet.engine INFO: Epoch: [5] [ 250/1193] learning_rate: 0.002500 loss_xy: 2.975321 loss_wh: 0.622377 loss_obj: 3.637796 loss_cls: 0.822504 loss: 8.262959 eta: 14:01:11 batch_cost: 0.8507 data_cost: 0.0859 ips: 10.5799 images/s
- [11/24 17:32:44] ppdet.engine INFO: Epoch: [5] [ 300/1193] learning_rate: 0.002500 loss_xy: 3.012673 loss_wh: 0.631500 loss_obj: 3.636731 loss_cls: 0.794631 loss: 7.878693 eta: 14:00:00 batch_cost: 0.7573 data_cost: 0.2340 ips: 11.8845 images/s
- [11/24 17:33:36] ppdet.engine INFO: Epoch: [5] [ 350/1193] learning_rate: 0.002500 loss_xy: 3.050128 loss_wh: 0.685606 loss_obj: 3.967741 loss_cls: 0.781684 loss: 8.647797 eta: 14:08:41 batch_cost: 1.0367 data_cost: 0.1172 ips: 8.6816 images/s
- [11/24 17:34:18] ppdet.engine INFO: Epoch: [5] [ 400/1193] learning_rate: 0.002500 loss_xy: 3.039378 loss_wh: 0.643576 loss_obj: 3.187078 loss_cls: 0.778541 loss: 7.478715 eta: 14:10:07 batch_cost: 0.8411 data_cost: 0.0777 ips: 10.7009 images/s
- [11/24 17:35:05] ppdet.engine INFO: Epoch: [5] [ 450/1193] learning_rate: 0.002500 loss_xy: 3.083435 loss_wh: 0.612685 loss_obj: 3.458898 loss_cls: 0.780580 loss: 7.788103 eta: 14:14:33 batch_cost: 0.9363 data_cost: 0.1483 ips: 9.6125 images/s
- [11/24 17:35:42] ppdet.engine INFO: Epoch: [5] [ 500/1193] learning_rate: 0.002500 loss_xy: 3.091077 loss_wh: 0.688975 loss_obj: 3.762993 loss_cls: 0.815407 loss: 8.443010 eta: 14:12:28 batch_cost: 0.7421 data_cost: 0.0494 ips: 12.1280 images/s
- [11/24 17:36:18] ppdet.engine INFO: Epoch: [5] [ 550/1193] learning_rate: 0.002500 loss_xy: 2.691394 loss_wh: 0.598528 loss_obj: 3.416036 loss_cls: 0.773536 loss: 7.940307 eta: 14:09:33 batch_cost: 0.7128 data_cost: 0.0918 ips: 12.6261 images/s
- [11/24 17:36:55] ppdet.engine INFO: Epoch: [5] [ 600/1193] learning_rate: 0.002500 loss_xy: 3.002202 loss_wh: 0.586479 loss_obj: 3.295039 loss_cls: 0.872789 loss: 7.883259 eta: 14:07:52 batch_cost: 0.7495 data_cost: 0.2372 ips: 12.0081 images/s
- [11/24 17:37:43] ppdet.engine INFO: Epoch: [5] [ 650/1193] learning_rate: 0.002500 loss_xy: 3.170389 loss_wh: 0.685436 loss_obj: 3.992542 loss_cls: 0.876974 loss: 8.985737 eta: 14:12:20 batch_cost: 0.9567 data_cost: 0.0676 ips: 9.4070 images/s
- [11/24 17:38:25] ppdet.engine INFO: Epoch: [5] [ 700/1193] learning_rate: 0.002500 loss_xy: 2.806384 loss_wh: 0.617828 loss_obj: 3.161753 loss_cls: 0.864342 loss: 7.495540 eta: 14:12:48 batch_cost: 0.8266 data_cost: 0.0873 ips: 10.8878 images/s
- [11/24 17:39:09] ppdet.engine INFO: Epoch: [5] [ 750/1193] learning_rate: 0.002500 loss_xy: 3.027729 loss_wh: 0.723910 loss_obj: 3.728714 loss_cls: 0.925330 loss: 8.351479 eta: 14:14:45 batch_cost: 0.8820 data_cost: 0.1751 ips: 10.2042 images/s
- [11/24 17:39:41] ppdet.engine INFO: Epoch: [5] [ 800/1193] learning_rate: 0.002500 loss_xy: 3.130845 loss_wh: 0.667499 loss_obj: 3.974711 loss_cls: 0.921871 loss: 8.790509 eta: 14:10:23 batch_cost: 0.6535 data_cost: 0.1048 ips: 13.7722 images/s
- [11/24 17:40:17] ppdet.engine INFO: Epoch: [5] [ 850/1193] learning_rate: 0.002500 loss_xy: 3.007344 loss_wh: 0.650794 loss_obj: 3.476666 loss_cls: 0.867903 loss: 8.291872 eta: 14:07:34 batch_cost: 0.7057 data_cost: 0.0800 ips: 12.7541 images/s
- [11/24 17:40:54] ppdet.engine INFO: Epoch: [5] [ 900/1193] learning_rate: 0.002500 loss_xy: 2.805111 loss_wh: 0.546873 loss_obj: 3.335837 loss_cls: 0.812117 loss: 7.482882 eta: 14:05:58 batch_cost: 0.7482 data_cost: 0.0950 ips: 12.0288 images/s
- [11/24 17:41:32] ppdet.engine INFO: Epoch: [5] [ 950/1193] learning_rate: 0.002500 loss_xy: 3.023979 loss_wh: 0.624782 loss_obj: 3.538837 loss_cls: 0.868785 loss: 7.992905 eta: 14:04:27 batch_cost: 0.7504 data_cost: 0.1812 ips: 11.9929 images/s
- [11/24 17:42:08] ppdet.engine INFO: Epoch: [5] [1000/1193] learning_rate: 0.002500 loss_xy: 3.241925 loss_wh: 0.641299 loss_obj: 3.819619 loss_cls: 0.980778 loss: 8.588409 eta: 14:02:04 batch_cost: 0.7131 data_cost: 0.1623 ips: 12.6201 images/s
- [11/24 17:42:48] ppdet.engine INFO: Epoch: [5] [1050/1193] learning_rate: 0.002500 loss_xy: 2.781797 loss_wh: 0.585033 loss_obj: 3.377254 loss_cls: 0.808229 loss: 7.618470 eta: 14:02:17 batch_cost: 0.8182 data_cost: 0.2508 ips: 11.0002 images/s
- [11/24 17:43:27] ppdet.engine INFO: Epoch: [5] [1100/1193] learning_rate: 0.002500 loss_xy: 3.023147 loss_wh: 0.587171 loss_obj: 3.258878 loss_cls: 0.761093 loss: 8.119644 eta: 14:01:22 batch_cost: 0.7708 data_cost: 0.0979 ips: 11.6756 images/s
- [11/24 17:44:06] ppdet.engine INFO: Epoch: [5] [1150/1193] learning_rate: 0.002500 loss_xy: 2.932893 loss_wh: 0.555989 loss_obj: 3.463820 loss_cls: 0.829841 loss: 7.875789 eta: 14:00:48 batch_cost: 0.7862 data_cost: 0.0107 ips: 11.4469 images/s
- [11/24 17:44:43] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 17:44:43] ppdet.engine INFO: Eval iter: 0
- [11/24 17:44:47] ppdet.engine INFO: Eval iter: 100
- [11/24 17:44:51] ppdet.engine INFO: Eval iter: 200
- [11/24 17:44:55] ppdet.engine INFO: Eval iter: 300
- [11/24 17:44:59] ppdet.engine INFO: Eval iter: 400
- [11/24 17:45:03] ppdet.engine INFO: Eval iter: 500
- [11/24 17:45:07] ppdet.engine INFO: Eval iter: 600
- [11/24 17:45:12] ppdet.engine INFO: Eval iter: 700
- [11/24 17:45:16] ppdet.engine INFO: Eval iter: 800
- [11/24 17:45:20] ppdet.engine INFO: Eval iter: 900
- [11/24 17:45:24] ppdet.engine INFO: Eval iter: 1000
- [11/24 17:45:28] ppdet.engine INFO: Eval iter: 1100
- [11/24 17:45:32] ppdet.engine INFO: Eval iter: 1200
- [11/24 17:45:36] ppdet.engine INFO: Eval iter: 1300
- [11/24 17:45:40] ppdet.engine INFO: Eval iter: 1400
- [11/24 17:45:44] ppdet.engine INFO: Eval iter: 1500
- [11/24 17:45:48] ppdet.engine INFO: Eval iter: 1600
- [11/24 17:45:52] ppdet.engine INFO: Eval iter: 1700
- [11/24 17:45:57] ppdet.engine INFO: Eval iter: 1800
- [11/24 17:46:01] ppdet.engine INFO: Eval iter: 1900
- [11/24 17:46:05] ppdet.engine INFO: Eval iter: 2000
- [11/24 17:46:09] ppdet.engine INFO: Eval iter: 2100
- [11/24 17:46:13] ppdet.engine INFO: Eval iter: 2200
- [11/24 17:46:17] ppdet.engine INFO: Eval iter: 2300
- [11/24 17:46:21] ppdet.engine INFO: Eval iter: 2400
- [11/24 17:46:25] ppdet.engine INFO: Eval iter: 2500
- [11/24 17:46:29] ppdet.engine INFO: Eval iter: 2600
- [11/24 17:46:33] ppdet.engine INFO: Eval iter: 2700
- [11/24 17:46:37] ppdet.engine INFO: Eval iter: 2800
- [11/24 17:46:41] ppdet.engine INFO: Eval iter: 2900
- [11/24 17:46:45] ppdet.engine INFO: Eval iter: 3000
- [11/24 17:46:49] ppdet.engine INFO: Eval iter: 3100
- [11/24 17:46:53] ppdet.engine INFO: Eval iter: 3200
- [11/24 17:46:57] ppdet.engine INFO: Eval iter: 3300
- [11/24 17:47:01] ppdet.engine INFO: Eval iter: 3400
- [11/24 17:47:05] ppdet.engine INFO: Eval iter: 3500
- [11/24 17:47:09] ppdet.engine INFO: Eval iter: 3600
- [11/24 17:47:13] ppdet.engine INFO: Eval iter: 3700
- [11/24 17:47:17] ppdet.engine INFO: Eval iter: 3800
- [11/24 17:47:21] ppdet.engine INFO: Eval iter: 3900
- [11/24 17:47:25] ppdet.engine INFO: Eval iter: 4000
- [11/24 17:47:29] ppdet.engine INFO: Eval iter: 4100
- [11/24 17:47:33] ppdet.engine INFO: Eval iter: 4200
- [11/24 17:47:37] ppdet.engine INFO: Eval iter: 4300
- [11/24 17:47:41] ppdet.engine INFO: Eval iter: 4400
- [11/24 17:47:45] ppdet.engine INFO: Eval iter: 4500
- [11/24 17:47:49] ppdet.engine INFO: Eval iter: 4600
- [11/24 17:47:53] ppdet.engine INFO: Eval iter: 4700
- [11/24 17:47:56] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 17:47:56] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 87.66%
- [11/24 17:47:56] ppdet.engine INFO: Total sample number: 4773, averge FPS: 24.638837971745968
- [11/24 17:47:56] ppdet.engine INFO: Best test bbox ap is 0.884.
- [11/24 17:47:57] ppdet.engine INFO: Epoch: [6] [ 0/1193] learning_rate: 0.002500 loss_xy: 3.003227 loss_wh: 0.578355 loss_obj: 2.959613 loss_cls: 0.879506 loss: 7.068338 eta: 14:00:59 batch_cost: 0.7799 data_cost: 0.0756 ips: 11.5395 images/s
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:49429', '127.0.0.1:46561']
- I1124 17:48:55.717113 20888 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1124 17:48:56.658488 20888 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 17:48:56.662817 20888 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/24 17:48:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 17:48:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/24 17:49:00] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/24 17:49:01] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/24 17:49:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/24 17:49:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/24 17:49:03] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/24 17:49:05] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/24 17:49:05] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/24 17:49:05] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- [11/24 17:49:06] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/24 17:49:06] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/24 17:49:07] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/24 17:49:07] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/24 17:49:07] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 17:49:08] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- [11/24 17:49:09] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage/5.pdparams
- [11/24 17:49:14] ppdet.engine INFO: Epoch: [6] [ 0/1193] learning_rate: 0.002500 loss_xy: 2.694128 loss_wh: 0.678380 loss_obj: 3.512159 loss_cls: 0.925522 loss: 7.810188 eta: 2 days, 19:21:20 batch_cost: 3.7639 data_cost: 1.5254 ips: 2.3911 images/s
- [11/24 17:49:47] ppdet.engine INFO: Epoch: [6] [ 50/1193] learning_rate: 0.002500 loss_xy: 3.059959 loss_wh: 0.667645 loss_obj: 3.362279 loss_cls: 0.942315 loss: 7.945555 eta: 12:54:52 batch_cost: 0.6614 data_cost: 0.0769 ips: 13.6072 images/s
- [11/24 17:50:25] ppdet.engine INFO: Epoch: [6] [ 100/1193] learning_rate: 0.002500 loss_xy: 3.062032 loss_wh: 0.596642 loss_obj: 2.881240 loss_cls: 0.725412 loss: 7.145204 eta: 13:15:10 batch_cost: 0.7616 data_cost: 0.1003 ips: 11.8168 images/s
- [11/24 17:51:09] ppdet.engine INFO: Epoch: [6] [ 150/1193] learning_rate: 0.002500 loss_xy: 3.085433 loss_wh: 0.652369 loss_obj: 2.955109 loss_cls: 0.651376 loss: 7.332201 eta: 14:03:11 batch_cost: 0.8789 data_cost: 0.1240 ips: 10.2403 images/s
- [11/24 17:51:51] ppdet.engine INFO: Epoch: [6] [ 200/1193] learning_rate: 0.002500 loss_xy: 3.067231 loss_wh: 0.622180 loss_obj: 2.942375 loss_cls: 0.703672 loss: 7.307162 eta: 14:17:24 batch_cost: 0.8430 data_cost: 0.0663 ips: 10.6762 images/s
- [11/24 17:52:31] ppdet.engine INFO: Epoch: [6] [ 250/1193] learning_rate: 0.002500 loss_xy: 3.099130 loss_wh: 0.638370 loss_obj: 2.857677 loss_cls: 0.595978 loss: 7.252436 eta: 14:15:21 batch_cost: 0.7945 data_cost: 0.0815 ips: 11.3277 images/s
- [11/24 17:53:04] ppdet.engine INFO: Epoch: [6] [ 300/1193] learning_rate: 0.002500 loss_xy: 3.166929 loss_wh: 0.662091 loss_obj: 3.349864 loss_cls: 0.714772 loss: 8.056497 eta: 13:50:31 batch_cost: 0.6637 data_cost: 0.0262 ips: 13.5613 images/s
- [11/24 17:53:41] ppdet.engine INFO: Epoch: [6] [ 350/1193] learning_rate: 0.002500 loss_xy: 3.319693 loss_wh: 0.752482 loss_obj: 3.135375 loss_cls: 0.649560 loss: 8.069536 eta: 13:42:46 batch_cost: 0.7304 data_cost: 0.0903 ips: 12.3212 images/s
- [11/24 17:54:24] ppdet.engine INFO: Epoch: [6] [ 400/1193] learning_rate: 0.002500 loss_xy: 3.291668 loss_wh: 0.696229 loss_obj: 3.515773 loss_cls: 0.636547 loss: 8.362307 eta: 13:54:34 batch_cost: 0.8640 data_cost: 0.0664 ips: 10.4167 images/s
- [11/24 17:55:04] ppdet.engine INFO: Epoch: [6] [ 450/1193] learning_rate: 0.002500 loss_xy: 3.124334 loss_wh: 0.612725 loss_obj: 2.885318 loss_cls: 0.650783 loss: 7.443854 eta: 13:55:12 batch_cost: 0.7930 data_cost: 0.0613 ips: 11.3493 images/s
- [11/24 17:55:43] ppdet.engine INFO: Epoch: [6] [ 500/1193] learning_rate: 0.002500 loss_xy: 2.933896 loss_wh: 0.651538 loss_obj: 2.932065 loss_cls: 0.617790 loss: 7.278274 eta: 13:54:06 batch_cost: 0.7791 data_cost: 0.1370 ips: 11.5517 images/s
- [11/24 17:56:20] ppdet.engine INFO: Epoch: [6] [ 550/1193] learning_rate: 0.002500 loss_xy: 3.448697 loss_wh: 0.756120 loss_obj: 3.247299 loss_cls: 0.619901 loss: 8.471077 eta: 13:48:54 batch_cost: 0.7358 data_cost: 0.3165 ips: 12.2313 images/s
- [11/24 17:56:58] ppdet.engine INFO: Epoch: [6] [ 600/1193] learning_rate: 0.002500 loss_xy: 3.182579 loss_wh: 0.646946 loss_obj: 2.764172 loss_cls: 0.509334 loss: 7.573401 eta: 13:46:58 batch_cost: 0.7643 data_cost: 0.0449 ips: 11.7755 images/s
- [11/24 17:57:38] ppdet.engine INFO: Epoch: [6] [ 650/1193] learning_rate: 0.002500 loss_xy: 3.256794 loss_wh: 0.698019 loss_obj: 3.020157 loss_cls: 0.611534 loss: 7.696426 eta: 13:48:40 batch_cost: 0.8061 data_cost: 0.1789 ips: 11.1643 images/s
- [11/24 17:58:17] ppdet.engine INFO: Epoch: [6] [ 700/1193] learning_rate: 0.002500 loss_xy: 3.167058 loss_wh: 0.680303 loss_obj: 2.813475 loss_cls: 0.572599 loss: 7.360578 eta: 13:47:47 batch_cost: 0.7766 data_cost: 0.1853 ips: 11.5892 images/s
- [11/24 17:58:56] ppdet.engine INFO: Epoch: [6] [ 750/1193] learning_rate: 0.002500 loss_xy: 3.205678 loss_wh: 0.630601 loss_obj: 3.081669 loss_cls: 0.575048 loss: 7.852473 eta: 13:47:08 batch_cost: 0.7794 data_cost: 0.0495 ips: 11.5477 images/s
- [11/24 17:59:33] ppdet.engine INFO: Epoch: [6] [ 800/1193] learning_rate: 0.002500 loss_xy: 3.211811 loss_wh: 0.697717 loss_obj: 2.887270 loss_cls: 0.512476 loss: 7.401710 eta: 13:43:16 batch_cost: 0.7309 data_cost: 0.1879 ips: 12.3143 images/s
- [11/24 18:00:10] ppdet.engine INFO: Epoch: [6] [ 850/1193] learning_rate: 0.002500 loss_xy: 3.042780 loss_wh: 0.634478 loss_obj: 2.805276 loss_cls: 0.573808 loss: 7.278664 eta: 13:40:17 batch_cost: 0.7389 data_cost: 0.1595 ips: 12.1800 images/s
- [11/24 18:00:43] ppdet.engine INFO: Epoch: [6] [ 900/1193] learning_rate: 0.002500 loss_xy: 3.032159 loss_wh: 0.604935 loss_obj: 2.361621 loss_cls: 0.583872 loss: 6.720060 eta: 13:33:51 batch_cost: 0.6756 data_cost: 0.0377 ips: 13.3223 images/s
- [11/24 18:01:22] ppdet.engine INFO: Epoch: [6] [ 950/1193] learning_rate: 0.002500 loss_xy: 3.240172 loss_wh: 0.664820 loss_obj: 2.725234 loss_cls: 0.582989 loss: 6.989074 eta: 13:33:12 batch_cost: 0.7687 data_cost: 0.0622 ips: 11.7076 images/s
- [11/24 18:02:01] ppdet.engine INFO: Epoch: [6] [1000/1193] learning_rate: 0.002500 loss_xy: 2.876031 loss_wh: 0.578255 loss_obj: 2.626191 loss_cls: 0.518396 loss: 6.937197 eta: 13:32:49 batch_cost: 0.7736 data_cost: 0.2477 ips: 11.6337 images/s
- [11/24 18:02:41] ppdet.engine INFO: Epoch: [6] [1050/1193] learning_rate: 0.002500 loss_xy: 3.118509 loss_wh: 0.653020 loss_obj: 3.001262 loss_cls: 0.754450 loss: 7.773034 eta: 13:33:36 batch_cost: 0.7971 data_cost: 0.1778 ips: 11.2910 images/s
- [11/24 18:03:22] ppdet.engine INFO: Epoch: [6] [1100/1193] learning_rate: 0.002500 loss_xy: 3.292206 loss_wh: 0.692713 loss_obj: 3.112415 loss_cls: 0.621992 loss: 8.108994 eta: 13:35:44 batch_cost: 0.8283 data_cost: 0.2995 ips: 10.8652 images/s
- [11/24 18:04:01] ppdet.engine INFO: Epoch: [6] [1150/1193] learning_rate: 0.002500 loss_xy: 3.365832 loss_wh: 0.694184 loss_obj: 3.047990 loss_cls: 0.582553 loss: 7.782279 eta: 13:34:56 batch_cost: 0.7696 data_cost: 0.1930 ips: 11.6938 images/s
- [11/24 18:04:26] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 18:04:27] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/24 18:04:28] ppdet.engine INFO: Eval iter: 0
- [11/24 18:04:32] ppdet.engine INFO: Eval iter: 100
- [11/24 18:04:36] ppdet.engine INFO: Eval iter: 200
- [11/24 18:04:40] ppdet.engine INFO: Eval iter: 300
- [11/24 18:04:43] ppdet.engine INFO: Eval iter: 400
- [11/24 18:04:47] ppdet.engine INFO: Eval iter: 500
- [11/24 18:04:51] ppdet.engine INFO: Eval iter: 600
- [11/24 18:04:55] ppdet.engine INFO: Eval iter: 700
- [11/24 18:04:59] ppdet.engine INFO: Eval iter: 800
- [11/24 18:05:03] ppdet.engine INFO: Eval iter: 900
- [11/24 18:05:07] ppdet.engine INFO: Eval iter: 1000
- [11/24 18:05:11] ppdet.engine INFO: Eval iter: 1100
- [11/24 18:05:14] ppdet.engine INFO: Eval iter: 1200
- [11/24 18:05:18] ppdet.engine INFO: Eval iter: 1300
- [11/24 18:05:22] ppdet.engine INFO: Eval iter: 1400
- [11/24 18:05:26] ppdet.engine INFO: Eval iter: 1500
- [11/24 18:05:30] ppdet.engine INFO: Eval iter: 1600
- [11/24 18:05:34] ppdet.engine INFO: Eval iter: 1700
- [11/24 18:05:38] ppdet.engine INFO: Eval iter: 1800
- [11/24 18:05:42] ppdet.engine INFO: Eval iter: 1900
- [11/24 18:05:46] ppdet.engine INFO: Eval iter: 2000
- [11/24 18:05:50] ppdet.engine INFO: Eval iter: 2100
- [11/24 18:05:55] ppdet.engine INFO: Eval iter: 2200
- [11/24 18:05:59] ppdet.engine INFO: Eval iter: 2300
- [11/24 18:06:03] ppdet.engine INFO: Eval iter: 2400
- [11/24 18:06:07] ppdet.engine INFO: Eval iter: 2500
- [11/24 18:06:11] ppdet.engine INFO: Eval iter: 2600
- [11/24 18:06:16] ppdet.engine INFO: Eval iter: 2700
- [11/24 18:06:20] ppdet.engine INFO: Eval iter: 2800
- [11/24 18:06:23] ppdet.engine INFO: Eval iter: 2900
- [11/24 18:06:28] ppdet.engine INFO: Eval iter: 3000
- [11/24 18:06:31] ppdet.engine INFO: Eval iter: 3100
- [11/24 18:06:35] ppdet.engine INFO: Eval iter: 3200
- [11/24 18:06:40] ppdet.engine INFO: Eval iter: 3300
- [11/24 18:06:43] ppdet.engine INFO: Eval iter: 3400
- [11/24 18:06:47] ppdet.engine INFO: Eval iter: 3500
- [11/24 18:06:51] ppdet.engine INFO: Eval iter: 3600
- [11/24 18:06:56] ppdet.engine INFO: Eval iter: 3700
- [11/24 18:07:00] ppdet.engine INFO: Eval iter: 3800
- [11/24 18:07:05] ppdet.engine INFO: Eval iter: 3900
- [11/24 18:07:09] ppdet.engine INFO: Eval iter: 4000
- [11/24 18:07:13] ppdet.engine INFO: Eval iter: 4100
- [11/24 18:07:17] ppdet.engine INFO: Eval iter: 4200
- [11/24 18:07:21] ppdet.engine INFO: Eval iter: 4300
- [11/24 18:07:23] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 18:07:23] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 87.39%
- [11/24 18:07:23] ppdet.engine INFO: Total sample number: 4339, averge FPS: 24.771590176807376
- [11/24 18:07:23] ppdet.engine INFO: Best test bbox ap is 0.874.
- [11/24 18:07:27] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 18:07:27] ppdet.engine INFO: Epoch: [7] [ 0/1193] learning_rate: 0.002500 loss_xy: 3.099675 loss_wh: 0.640436 loss_obj: 2.864761 loss_cls: 0.676608 loss: 7.687221 eta: 13:26:50 batch_cost: 0.6055 data_cost: 0.0421 ips: 14.8643 images/s
- [11/24 18:08:09] ppdet.engine INFO: Epoch: [7] [ 50/1193] learning_rate: 0.002500 loss_xy: 3.387426 loss_wh: 0.719090 loss_obj: 3.067526 loss_cls: 0.609379 loss: 7.448108 eta: 13:29:17 batch_cost: 0.8388 data_cost: 0.1517 ips: 10.7295 images/s
- [11/24 18:08:50] ppdet.engine INFO: Epoch: [7] [ 100/1193] learning_rate: 0.002500 loss_xy: 3.183581 loss_wh: 0.610323 loss_obj: 3.023068 loss_cls: 0.601894 loss: 7.362257 eta: 13:31:00 batch_cost: 0.8265 data_cost: 0.3070 ips: 10.8897 images/s
- [11/24 18:09:26] ppdet.engine INFO: Epoch: [7] [ 150/1193] learning_rate: 0.002500 loss_xy: 3.048188 loss_wh: 0.574545 loss_obj: 2.703791 loss_cls: 0.720643 loss: 7.588591 eta: 13:28:07 batch_cost: 0.7135 data_cost: 0.0979 ips: 12.6135 images/s
- [11/24 18:10:02] ppdet.engine INFO: Epoch: [7] [ 200/1193] learning_rate: 0.002500 loss_xy: 3.282278 loss_wh: 0.723815 loss_obj: 2.795896 loss_cls: 0.729070 loss: 7.882648 eta: 13:25:15 batch_cost: 0.7095 data_cost: 0.0982 ips: 12.6852 images/s
- [11/24 18:10:44] ppdet.engine INFO: Epoch: [7] [ 250/1193] learning_rate: 0.002500 loss_xy: 2.917607 loss_wh: 0.571877 loss_obj: 2.852618 loss_cls: 0.661818 loss: 7.363335 eta: 13:27:40 batch_cost: 0.8505 data_cost: 0.0948 ips: 10.5817 images/s
- [11/24 18:11:23] ppdet.engine INFO: Epoch: [7] [ 300/1193] learning_rate: 0.002500 loss_xy: 2.949164 loss_wh: 0.568135 loss_obj: 2.609501 loss_cls: 0.606382 loss: 6.526048 eta: 13:27:11 batch_cost: 0.7739 data_cost: 0.2531 ips: 11.6289 images/s
- [11/24 18:12:14] ppdet.engine INFO: Epoch: [7] [ 350/1193] learning_rate: 0.002500 loss_xy: 3.030170 loss_wh: 0.665473 loss_obj: 3.049253 loss_cls: 0.541527 loss: 7.371579 eta: 13:35:26 batch_cost: 1.0317 data_cost: 0.1181 ips: 8.7233 images/s
- [11/24 18:12:56] ppdet.engine INFO: Epoch: [7] [ 400/1193] learning_rate: 0.002500 loss_xy: 2.992084 loss_wh: 0.623734 loss_obj: 2.452892 loss_cls: 0.555943 loss: 6.722635 eta: 13:36:40 batch_cost: 0.8356 data_cost: 0.0817 ips: 10.7713 images/s
- [11/24 18:13:43] ppdet.engine INFO: Epoch: [7] [ 450/1193] learning_rate: 0.002500 loss_xy: 3.039697 loss_wh: 0.609087 loss_obj: 2.634409 loss_cls: 0.528205 loss: 7.092788 eta: 13:41:02 batch_cost: 0.9374 data_cost: 0.1521 ips: 9.6013 images/s
- [11/24 18:14:20] ppdet.engine INFO: Epoch: [7] [ 500/1193] learning_rate: 0.002500 loss_xy: 3.054078 loss_wh: 0.675802 loss_obj: 2.923771 loss_cls: 0.581150 loss: 7.117271 eta: 13:38:57 batch_cost: 0.7384 data_cost: 0.0535 ips: 12.1888 images/s
- [11/24 18:14:56] ppdet.engine INFO: Epoch: [7] [ 550/1193] learning_rate: 0.002500 loss_xy: 2.661842 loss_wh: 0.600500 loss_obj: 2.694170 loss_cls: 0.572016 loss: 6.803842 eta: 13:36:22 batch_cost: 0.7190 data_cost: 0.1027 ips: 12.5172 images/s
- [11/24 18:15:34] ppdet.engine INFO: Epoch: [7] [ 600/1193] learning_rate: 0.002500 loss_xy: 2.946920 loss_wh: 0.545319 loss_obj: 2.933699 loss_cls: 0.573865 loss: 7.083235 eta: 13:34:47 batch_cost: 0.7491 data_cost: 0.2412 ips: 12.0148 images/s
- [11/24 18:16:21] ppdet.engine INFO: Epoch: [7] [ 650/1193] learning_rate: 0.002500 loss_xy: 3.115234 loss_wh: 0.702421 loss_obj: 3.148805 loss_cls: 0.596371 loss: 7.798341 eta: 13:39:02 batch_cost: 0.9544 data_cost: 0.0773 ips: 9.4303 images/s
- [11/24 18:17:03] ppdet.engine INFO: Epoch: [7] [ 700/1193] learning_rate: 0.002500 loss_xy: 2.769791 loss_wh: 0.609502 loss_obj: 2.403167 loss_cls: 0.674008 loss: 6.571776 eta: 13:39:28 batch_cost: 0.8246 data_cost: 0.0937 ips: 10.9142 images/s
- [11/24 18:17:48] ppdet.engine INFO: Epoch: [7] [ 750/1193] learning_rate: 0.002500 loss_xy: 2.931687 loss_wh: 0.652383 loss_obj: 2.835293 loss_cls: 0.611508 loss: 7.068840 eta: 13:41:48 batch_cost: 0.8977 data_cost: 0.1903 ips: 10.0253 images/s
- [11/24 18:18:21] ppdet.engine INFO: Epoch: [7] [ 800/1193] learning_rate: 0.002500 loss_xy: 3.083824 loss_wh: 0.671487 loss_obj: 3.189439 loss_cls: 0.614979 loss: 7.689487 eta: 13:37:54 batch_cost: 0.6650 data_cost: 0.1163 ips: 13.5345 images/s
- [11/24 18:18:56] ppdet.engine INFO: Epoch: [7] [ 850/1193] learning_rate: 0.002500 loss_xy: 3.037673 loss_wh: 0.663396 loss_obj: 2.922084 loss_cls: 0.620173 loss: 7.041429 eta: 13:35:18 batch_cost: 0.7098 data_cost: 0.1071 ips: 12.6803 images/s
- [11/24 18:19:34] ppdet.engine INFO: Epoch: [7] [ 900/1193] learning_rate: 0.002500 loss_xy: 2.750330 loss_wh: 0.549786 loss_obj: 2.492316 loss_cls: 0.543848 loss: 6.245554 eta: 13:33:54 batch_cost: 0.7539 data_cost: 0.1143 ips: 11.9381 images/s
- [11/24 18:20:12] ppdet.engine INFO: Epoch: [7] [ 950/1193] learning_rate: 0.002500 loss_xy: 2.952173 loss_wh: 0.610946 loss_obj: 2.896821 loss_cls: 0.662380 loss: 7.204007 eta: 13:32:39 batch_cost: 0.7591 data_cost: 0.1838 ips: 11.8565 images/s
- [11/24 18:20:48] ppdet.engine INFO: Epoch: [7] [1000/1193] learning_rate: 0.002500 loss_xy: 3.198756 loss_wh: 0.640813 loss_obj: 2.855005 loss_cls: 0.743121 loss: 7.671157 eta: 13:30:14 batch_cost: 0.7080 data_cost: 0.1578 ips: 12.7118 images/s
- [11/24 18:21:29] ppdet.engine INFO: Epoch: [7] [1050/1193] learning_rate: 0.002500 loss_xy: 2.764298 loss_wh: 0.556088 loss_obj: 2.771038 loss_cls: 0.604925 loss: 6.859374 eta: 13:30:35 batch_cost: 0.8248 data_cost: 0.2732 ips: 10.9112 images/s
- [11/24 18:22:07] ppdet.engine INFO: Epoch: [7] [1100/1193] learning_rate: 0.002500 loss_xy: 2.959965 loss_wh: 0.568026 loss_obj: 2.553903 loss_cls: 0.601028 loss: 6.806147 eta: 13:29:29 batch_cost: 0.7620 data_cost: 0.1006 ips: 11.8115 images/s
- [11/24 18:22:47] ppdet.engine INFO: Epoch: [7] [1150/1193] learning_rate: 0.002500 loss_xy: 2.908010 loss_wh: 0.529201 loss_obj: 2.559623 loss_cls: 0.590713 loss: 6.913291 eta: 13:29:02 batch_cost: 0.7914 data_cost: 0.0074 ips: 11.3715 images/s
- [11/24 18:23:22] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 18:23:22] ppdet.engine INFO: Eval iter: 0
- [11/24 18:23:27] ppdet.engine INFO: Eval iter: 100
- [11/24 18:23:31] ppdet.engine INFO: Eval iter: 200
- [11/24 18:23:35] ppdet.engine INFO: Eval iter: 300
- [11/24 18:23:39] ppdet.engine INFO: Eval iter: 400
- [11/24 18:23:43] ppdet.engine INFO: Eval iter: 500
- [11/24 18:23:47] ppdet.engine INFO: Eval iter: 600
- [11/24 18:23:51] ppdet.engine INFO: Eval iter: 700
- [11/24 18:23:55] ppdet.engine INFO: Eval iter: 800
- [11/24 18:23:59] ppdet.engine INFO: Eval iter: 900
- [11/24 18:24:03] ppdet.engine INFO: Eval iter: 1000
- [11/24 18:24:07] ppdet.engine INFO: Eval iter: 1100
- [11/24 18:24:11] ppdet.engine INFO: Eval iter: 1200
- [11/24 18:24:15] ppdet.engine INFO: Eval iter: 1300
- [11/24 18:24:19] ppdet.engine INFO: Eval iter: 1400
- [11/24 18:24:23] ppdet.engine INFO: Eval iter: 1500
- [11/24 18:24:27] ppdet.engine INFO: Eval iter: 1600
- [11/24 18:24:32] ppdet.engine INFO: Eval iter: 1700
- [11/24 18:24:36] ppdet.engine INFO: Eval iter: 1800
- [11/24 18:24:40] ppdet.engine INFO: Eval iter: 1900
- [11/24 18:24:45] ppdet.engine INFO: Eval iter: 2000
- [11/24 18:24:49] ppdet.engine INFO: Eval iter: 2100
- [11/24 18:24:53] ppdet.engine INFO: Eval iter: 2200
- [11/24 18:24:58] ppdet.engine INFO: Eval iter: 2300
- [11/24 18:25:02] ppdet.engine INFO: Eval iter: 2400
- [11/24 18:25:06] ppdet.engine INFO: Eval iter: 2500
- [11/24 18:25:10] ppdet.engine INFO: Eval iter: 2600
- [11/24 18:25:14] ppdet.engine INFO: Eval iter: 2700
- [11/24 18:25:18] ppdet.engine INFO: Eval iter: 2800
- [11/24 18:25:22] ppdet.engine INFO: Eval iter: 2900
- [11/24 18:25:26] ppdet.engine INFO: Eval iter: 3000
- [11/24 18:25:30] ppdet.engine INFO: Eval iter: 3100
- [11/24 18:25:34] ppdet.engine INFO: Eval iter: 3200
- [11/24 18:25:38] ppdet.engine INFO: Eval iter: 3300
- [11/24 18:25:42] ppdet.engine INFO: Eval iter: 3400
- [11/24 18:25:46] ppdet.engine INFO: Eval iter: 3500
- [11/24 18:25:50] ppdet.engine INFO: Eval iter: 3600
- [11/24 18:25:55] ppdet.engine INFO: Eval iter: 3700
- [11/24 18:25:59] ppdet.engine INFO: Eval iter: 3800
- [11/24 18:26:04] ppdet.engine INFO: Eval iter: 3900
- [11/24 18:26:08] ppdet.engine INFO: Eval iter: 4000
- [11/24 18:26:12] ppdet.engine INFO: Eval iter: 4100
- [11/24 18:26:17] ppdet.engine INFO: Eval iter: 4200
- [11/24 18:26:21] ppdet.engine INFO: Eval iter: 4300
- [11/24 18:26:23] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 18:26:23] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 87.87%
- [11/24 18:26:23] ppdet.engine INFO: Total sample number: 4339, averge FPS: 24.03857991897561
- [11/24 18:26:23] ppdet.engine INFO: Best test bbox ap is 0.879.
- [11/24 18:26:27] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 18:26:27] ppdet.engine INFO: Epoch: [8] [ 0/1193] learning_rate: 0.002500 loss_xy: 2.981578 loss_wh: 0.586009 loss_obj: 2.406670 loss_cls: 0.683970 loss: 6.804967 eta: 13:28:59 batch_cost: 0.7713 data_cost: 0.0663 ips: 11.6693 images/s
- [11/24 18:27:04] ppdet.engine INFO: Epoch: [8] [ 50/1193] learning_rate: 0.002500 loss_xy: 3.091776 loss_wh: 0.627818 loss_obj: 3.600734 loss_cls: 0.758967 loss: 8.275541 eta: 13:27:19 batch_cost: 0.7349 data_cost: 0.0527 ips: 12.2473 images/s
- [11/24 18:27:51] ppdet.engine INFO: Epoch: [8] [ 100/1193] learning_rate: 0.002500 loss_xy: 3.219908 loss_wh: 0.690348 loss_obj: 3.584692 loss_cls: 0.867003 loss: 8.012008 eta: 13:29:58 batch_cost: 0.9400 data_cost: 0.0897 ips: 9.5742 images/s
- [11/24 18:28:21] ppdet.engine INFO: Epoch: [8] [ 150/1193] learning_rate: 0.002500 loss_xy: 3.181452 loss_wh: 0.750214 loss_obj: 3.675479 loss_cls: 0.846274 loss: 8.745619 eta: 13:25:41 batch_cost: 0.6068 data_cost: 0.0551 ips: 14.8328 images/s
- [11/24 18:28:58] ppdet.engine INFO: Epoch: [8] [ 200/1193] learning_rate: 0.002500 loss_xy: 3.125288 loss_wh: 0.687802 loss_obj: 3.440178 loss_cls: 0.810678 loss: 8.180218 eta: 13:24:05 batch_cost: 0.7334 data_cost: 0.1593 ips: 12.2714 images/s
- [11/24 18:29:33] ppdet.engine INFO: Epoch: [8] [ 250/1193] learning_rate: 0.002500 loss_xy: 3.510186 loss_wh: 0.684087 loss_obj: 3.930123 loss_cls: 0.972532 loss: 9.313890 eta: 13:22:01 batch_cost: 0.7074 data_cost: 0.0448 ips: 12.7233 images/s
- [11/24 18:30:11] ppdet.engine INFO: Epoch: [8] [ 300/1193] learning_rate: 0.002500 loss_xy: 3.250080 loss_wh: 0.697119 loss_obj: 3.759696 loss_cls: 0.698403 loss: 8.787844 eta: 13:20:49 batch_cost: 0.7504 data_cost: 0.0613 ips: 11.9934 images/s
- [11/24 18:30:54] ppdet.engine INFO: Epoch: [8] [ 350/1193] learning_rate: 0.002500 loss_xy: 3.130602 loss_wh: 0.584293 loss_obj: 3.903580 loss_cls: 0.732524 loss: 8.535461 eta: 13:21:38 batch_cost: 0.8561 data_cost: 0.0951 ips: 10.5127 images/s
- [11/24 18:31:31] ppdet.engine INFO: Epoch: [8] [ 400/1193] learning_rate: 0.002500 loss_xy: 3.136198 loss_wh: 0.677543 loss_obj: 4.113268 loss_cls: 0.842503 loss: 8.506281 eta: 13:20:26 batch_cost: 0.7499 data_cost: 0.1404 ips: 12.0024 images/s
- [11/24 18:32:03] ppdet.engine INFO: Epoch: [8] [ 450/1193] learning_rate: 0.002500 loss_xy: 3.374500 loss_wh: 0.596781 loss_obj: 3.497336 loss_cls: 0.917605 loss: 8.257936 eta: 13:17:03 batch_cost: 0.6277 data_cost: 0.0167 ips: 14.3387 images/s
- [11/24 18:32:38] ppdet.engine INFO: Epoch: [8] [ 500/1193] learning_rate: 0.002500 loss_xy: 2.784250 loss_wh: 0.609281 loss_obj: 3.403935 loss_cls: 0.712559 loss: 8.116672 eta: 13:15:02 batch_cost: 0.7000 data_cost: 0.2018 ips: 12.8570 images/s
- [11/24 18:33:16] ppdet.engine INFO: Epoch: [8] [ 550/1193] learning_rate: 0.002500 loss_xy: 2.789708 loss_wh: 0.607625 loss_obj: 3.259711 loss_cls: 0.808318 loss: 7.545938 eta: 13:14:08 batch_cost: 0.7600 data_cost: 0.1022 ips: 11.8425 images/s
- [11/24 18:33:54] ppdet.engine INFO: Epoch: [8] [ 600/1193] learning_rate: 0.002500 loss_xy: 3.190320 loss_wh: 0.633922 loss_obj: 3.327226 loss_cls: 0.823421 loss: 8.069119 eta: 13:13:09 batch_cost: 0.7559 data_cost: 0.1511 ips: 11.9058 images/s
- [11/24 18:34:27] ppdet.engine INFO: Epoch: [8] [ 650/1193] learning_rate: 0.002500 loss_xy: 3.089112 loss_wh: 0.657874 loss_obj: 3.439831 loss_cls: 0.990747 loss: 7.833358 eta: 13:10:41 batch_cost: 0.6663 data_cost: 0.1676 ips: 13.5072 images/s
- [11/24 18:35:03] ppdet.engine INFO: Epoch: [8] [ 700/1193] learning_rate: 0.002500 loss_xy: 3.147774 loss_wh: 0.644878 loss_obj: 3.780623 loss_cls: 0.910699 loss: 8.170158 eta: 13:08:56 batch_cost: 0.7061 data_cost: 0.1455 ips: 12.7452 images/s
- [11/24 18:35:41] ppdet.engine INFO: Epoch: [8] [ 750/1193] learning_rate: 0.002500 loss_xy: 2.907655 loss_wh: 0.659291 loss_obj: 3.119017 loss_cls: 0.853881 loss: 7.737969 eta: 13:08:22 batch_cost: 0.7760 data_cost: 0.0464 ips: 11.5982 images/s
- [11/24 18:36:25] ppdet.engine INFO: Epoch: [8] [ 800/1193] learning_rate: 0.002500 loss_xy: 3.235289 loss_wh: 0.637764 loss_obj: 3.802845 loss_cls: 0.829717 loss: 8.389987 eta: 13:09:15 batch_cost: 0.8670 data_cost: 0.0523 ips: 10.3811 images/s
- [11/24 18:37:05] ppdet.engine INFO: Epoch: [8] [ 850/1193] learning_rate: 0.002500 loss_xy: 2.764842 loss_wh: 0.626398 loss_obj: 3.390954 loss_cls: 0.748131 loss: 8.190071 eta: 13:09:04 batch_cost: 0.8030 data_cost: 0.0836 ips: 11.2075 images/s
- [11/24 18:37:46] ppdet.engine INFO: Epoch: [8] [ 900/1193] learning_rate: 0.002500 loss_xy: 2.913828 loss_wh: 0.601892 loss_obj: 3.178063 loss_cls: 0.937097 loss: 7.630213 eta: 13:09:12 batch_cost: 0.8235 data_cost: 0.0014 ips: 10.9286 images/s
- [11/24 18:38:28] ppdet.engine INFO: Epoch: [8] [ 950/1193] learning_rate: 0.002500 loss_xy: 3.388792 loss_wh: 0.759510 loss_obj: 4.144102 loss_cls: 0.799746 loss: 9.113113 eta: 13:09:32 batch_cost: 0.8391 data_cost: 0.1913 ips: 10.7258 images/s
- [11/24 18:39:08] ppdet.engine INFO: Epoch: [8] [1000/1193] learning_rate: 0.002500 loss_xy: 3.516383 loss_wh: 0.778221 loss_obj: 4.058579 loss_cls: 1.059421 loss: 9.565088 eta: 13:09:08 batch_cost: 0.7915 data_cost: 0.0961 ips: 11.3710 images/s
- [11/24 18:39:47] ppdet.engine INFO: Epoch: [8] [1050/1193] learning_rate: 0.002500 loss_xy: 3.062243 loss_wh: 0.582370 loss_obj: 3.229296 loss_cls: 0.946282 loss: 7.818372 eta: 13:08:35 batch_cost: 0.7829 data_cost: 0.1276 ips: 11.4964 images/s
- [11/24 18:40:23] ppdet.engine INFO: Epoch: [8] [1100/1193] learning_rate: 0.002500 loss_xy: 3.198160 loss_wh: 0.717839 loss_obj: 3.778241 loss_cls: 1.013733 loss: 8.710672 eta: 13:07:04 batch_cost: 0.7165 data_cost: 0.0494 ips: 12.5613 images/s
- [11/24 18:41:04] ppdet.engine INFO: Epoch: [8] [1150/1193] learning_rate: 0.002500 loss_xy: 3.350298 loss_wh: 0.706465 loss_obj: 3.593665 loss_cls: 0.938602 loss: 8.875691 eta: 13:07:06 batch_cost: 0.8217 data_cost: 0.1655 ips: 10.9526 images/s
- [11/24 18:41:41] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 18:41:41] ppdet.engine INFO: Eval iter: 0
- [11/24 18:41:45] ppdet.engine INFO: Eval iter: 100
- [11/24 18:41:49] ppdet.engine INFO: Eval iter: 200
- [11/24 18:41:53] ppdet.engine INFO: Eval iter: 300
- [11/24 18:41:58] ppdet.engine INFO: Eval iter: 400
- [11/24 18:42:02] ppdet.engine INFO: Eval iter: 500
- [11/24 18:42:06] ppdet.engine INFO: Eval iter: 600
- [11/24 18:42:10] ppdet.engine INFO: Eval iter: 700
- [11/24 18:42:14] ppdet.engine INFO: Eval iter: 800
- [11/24 18:42:18] ppdet.engine INFO: Eval iter: 900
- [11/24 18:42:22] ppdet.engine INFO: Eval iter: 1000
- [11/24 18:42:26] ppdet.engine INFO: Eval iter: 1100
- [11/24 18:42:30] ppdet.engine INFO: Eval iter: 1200
- [11/24 18:42:34] ppdet.engine INFO: Eval iter: 1300
- [11/24 18:42:38] ppdet.engine INFO: Eval iter: 1400
- [11/24 18:42:42] ppdet.engine INFO: Eval iter: 1500
- [11/24 18:42:46] ppdet.engine INFO: Eval iter: 1600
- [11/24 18:42:50] ppdet.engine INFO: Eval iter: 1700
- [11/24 18:42:54] ppdet.engine INFO: Eval iter: 1800
- [11/24 18:42:58] ppdet.engine INFO: Eval iter: 1900
- [11/24 18:43:02] ppdet.engine INFO: Eval iter: 2000
- [11/24 18:43:06] ppdet.engine INFO: Eval iter: 2100
- [11/24 18:43:10] ppdet.engine INFO: Eval iter: 2200
- [11/24 18:43:14] ppdet.engine INFO: Eval iter: 2300
- [11/24 18:43:18] ppdet.engine INFO: Eval iter: 2400
- [11/24 18:43:22] ppdet.engine INFO: Eval iter: 2500
- [11/24 18:43:26] ppdet.engine INFO: Eval iter: 2600
- [11/24 18:43:30] ppdet.engine INFO: Eval iter: 2700
- [11/24 18:43:34] ppdet.engine INFO: Eval iter: 2800
- [11/24 18:43:38] ppdet.engine INFO: Eval iter: 2900
- [11/24 18:43:42] ppdet.engine INFO: Eval iter: 3000
- [11/24 18:43:46] ppdet.engine INFO: Eval iter: 3100
- [11/24 18:43:50] ppdet.engine INFO: Eval iter: 3200
- [11/24 18:43:54] ppdet.engine INFO: Eval iter: 3300
- [11/24 18:43:58] ppdet.engine INFO: Eval iter: 3400
- [11/24 18:44:02] ppdet.engine INFO: Eval iter: 3500
- [11/24 18:44:06] ppdet.engine INFO: Eval iter: 3600
- [11/24 18:44:11] ppdet.engine INFO: Eval iter: 3700
- [11/24 18:44:15] ppdet.engine INFO: Eval iter: 3800
- [11/24 18:44:19] ppdet.engine INFO: Eval iter: 3900
- [11/24 18:44:23] ppdet.engine INFO: Eval iter: 4000
- [11/24 18:44:27] ppdet.engine INFO: Eval iter: 4100
- [11/24 18:44:31] ppdet.engine INFO: Eval iter: 4200
- [11/24 18:44:35] ppdet.engine INFO: Eval iter: 4300
- [11/24 18:44:37] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 18:44:37] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 87.20%
- [11/24 18:44:37] ppdet.engine INFO: Total sample number: 4339, averge FPS: 24.664651458817325
- [11/24 18:44:37] ppdet.engine INFO: Best test bbox ap is 0.879.
- [11/24 18:44:38] ppdet.engine INFO: Epoch: [9] [ 0/1193] learning_rate: 0.002500 loss_xy: 3.147556 loss_wh: 0.600962 loss_obj: 3.325010 loss_cls: 0.784851 loss: 7.925399 eta: 13:07:20 batch_cost: 0.8469 data_cost: 0.0794 ips: 10.6272 images/s
- [11/24 18:45:10] ppdet.engine INFO: Epoch: [9] [ 50/1193] learning_rate: 0.002500 loss_xy: 3.668642 loss_wh: 0.812164 loss_obj: 4.035004 loss_cls: 0.948881 loss: 9.441910 eta: 13:04:58 batch_cost: 0.6533 data_cost: 0.0097 ips: 13.7771 images/s
- [11/24 18:45:51] ppdet.engine INFO: Epoch: [9] [ 100/1193] learning_rate: 0.002500 loss_xy: 3.193462 loss_wh: 0.645566 loss_obj: 3.464133 loss_cls: 0.830881 loss: 8.487825 eta: 13:04:45 batch_cost: 0.8061 data_cost: 0.1789 ips: 11.1646 images/s
- [11/24 18:46:29] ppdet.engine INFO: Epoch: [9] [ 150/1193] learning_rate: 0.002500 loss_xy: 3.351757 loss_wh: 0.742658 loss_obj: 3.390878 loss_cls: 0.758468 loss: 8.430350 eta: 13:03:56 batch_cost: 0.7628 data_cost: 0.1682 ips: 11.7988 images/s
- [11/24 18:47:05] ppdet.engine INFO: Epoch: [9] [ 200/1193] learning_rate: 0.002500 loss_xy: 3.112890 loss_wh: 0.681163 loss_obj: 3.588068 loss_cls: 0.885239 loss: 8.375417 eta: 13:02:40 batch_cost: 0.7279 data_cost: 0.0216 ips: 12.3649 images/s
- [11/24 18:47:44] ppdet.engine INFO: Epoch: [9] [ 250/1193] learning_rate: 0.002500 loss_xy: 2.773605 loss_wh: 0.539353 loss_obj: 3.257517 loss_cls: 0.814175 loss: 7.182824 eta: 13:01:59 batch_cost: 0.7722 data_cost: 0.0496 ips: 11.6546 images/s
- [11/24 18:48:16] ppdet.engine INFO: Epoch: [9] [ 300/1193] learning_rate: 0.002500 loss_xy: 2.817196 loss_wh: 0.598432 loss_obj: 3.273714 loss_cls: 0.713158 loss: 7.382885 eta: 12:59:44 batch_cost: 0.6504 data_cost: 0.0169 ips: 13.8385 images/s
- [11/24 18:49:00] ppdet.engine INFO: Epoch: [9] [ 350/1193] learning_rate: 0.002500 loss_xy: 3.303812 loss_wh: 0.698300 loss_obj: 3.736388 loss_cls: 0.811553 loss: 8.596446 eta: 13:00:14 batch_cost: 0.8617 data_cost: 0.0896 ips: 10.4443 images/s
- [11/24 18:49:52] ppdet.engine INFO: Epoch: [9] [ 400/1193] learning_rate: 0.002500 loss_xy: 3.223893 loss_wh: 0.759322 loss_obj: 3.782935 loss_cls: 0.753964 loss: 8.774516 eta: 13:03:01 batch_cost: 1.0451 data_cost: 0.6699 ips: 8.6119 images/s
- [11/24 18:50:32] ppdet.engine INFO: Epoch: [9] [ 450/1193] learning_rate: 0.002500 loss_xy: 3.067329 loss_wh: 0.651756 loss_obj: 3.415229 loss_cls: 0.897262 loss: 8.611059 eta: 13:02:48 batch_cost: 0.8111 data_cost: 0.2538 ips: 11.0954 images/s
- [11/24 18:51:14] ppdet.engine INFO: Epoch: [9] [ 500/1193] learning_rate: 0.002500 loss_xy: 3.324458 loss_wh: 0.631995 loss_obj: 3.845495 loss_cls: 0.917228 loss: 8.918983 eta: 13:02:49 batch_cost: 0.8321 data_cost: 0.2135 ips: 10.8157 images/s
- [11/24 18:51:44] ppdet.engine INFO: Epoch: [9] [ 550/1193] learning_rate: 0.002500 loss_xy: 3.341464 loss_wh: 0.657904 loss_obj: 3.768493 loss_cls: 0.709142 loss: 8.634344 eta: 12:59:51 batch_cost: 0.5881 data_cost: 0.1047 ips: 15.3045 images/s
- [11/24 18:52:22] ppdet.engine INFO: Epoch: [9] [ 600/1193] learning_rate: 0.002500 loss_xy: 3.026342 loss_wh: 0.615081 loss_obj: 3.429566 loss_cls: 0.714749 loss: 7.853089 eta: 12:59:01 batch_cost: 0.7603 data_cost: 0.0082 ips: 11.8380 images/s
- [11/24 18:53:01] ppdet.engine INFO: Epoch: [9] [ 650/1193] learning_rate: 0.002500 loss_xy: 3.151995 loss_wh: 0.648491 loss_obj: 3.627375 loss_cls: 0.872864 loss: 8.386497 eta: 12:58:26 batch_cost: 0.7810 data_cost: 0.1389 ips: 11.5238 images/s
- [11/24 18:53:37] ppdet.engine INFO: Epoch: [9] [ 700/1193] learning_rate: 0.002500 loss_xy: 3.116327 loss_wh: 0.607402 loss_obj: 3.267810 loss_cls: 0.719090 loss: 7.885116 eta: 12:57:11 batch_cost: 0.7245 data_cost: 0.0781 ips: 12.4228 images/s
- [11/24 18:54:14] ppdet.engine INFO: Epoch: [9] [ 750/1193] learning_rate: 0.002500 loss_xy: 3.176022 loss_wh: 0.679226 loss_obj: 3.762330 loss_cls: 0.770995 loss: 8.565205 eta: 12:56:09 batch_cost: 0.7416 data_cost: 0.3395 ips: 12.1357 images/s
- [11/24 18:54:48] ppdet.engine INFO: Epoch: [9] [ 800/1193] learning_rate: 0.002500 loss_xy: 3.071365 loss_wh: 0.674233 loss_obj: 3.572978 loss_cls: 1.024542 loss: 8.601196 eta: 12:54:28 batch_cost: 0.6854 data_cost: 0.1347 ips: 13.1314 images/s
- [11/24 18:55:25] ppdet.engine INFO: Epoch: [9] [ 850/1193] learning_rate: 0.002500 loss_xy: 3.425933 loss_wh: 0.702688 loss_obj: 3.736235 loss_cls: 0.858050 loss: 8.260995 eta: 12:53:26 batch_cost: 0.7390 data_cost: 0.0590 ips: 12.1779 images/s
- [11/24 18:56:05] ppdet.engine INFO: Epoch: [9] [ 900/1193] learning_rate: 0.002500 loss_xy: 3.181363 loss_wh: 0.676707 loss_obj: 3.673333 loss_cls: 0.800638 loss: 8.382175 eta: 12:53:04 batch_cost: 0.7988 data_cost: 0.0733 ips: 11.2666 images/s
- [11/24 18:56:48] ppdet.engine INFO: Epoch: [9] [ 950/1193] learning_rate: 0.002500 loss_xy: 2.924166 loss_wh: 0.629299 loss_obj: 3.297728 loss_cls: 0.799468 loss: 7.573365 eta: 12:53:12 batch_cost: 0.8439 data_cost: 0.1446 ips: 10.6648 images/s
- [11/24 18:57:30] ppdet.engine INFO: Epoch: [9] [1000/1193] learning_rate: 0.002500 loss_xy: 3.206161 loss_wh: 0.718164 loss_obj: 3.808277 loss_cls: 0.785948 loss: 9.196308 eta: 12:53:18 batch_cost: 0.8426 data_cost: 0.2017 ips: 10.6816 images/s
- [11/24 18:58:17] ppdet.engine INFO: Epoch: [9] [1050/1193] learning_rate: 0.002500 loss_xy: 3.388050 loss_wh: 0.671284 loss_obj: 3.824037 loss_cls: 0.814036 loss: 8.873586 eta: 12:54:29 batch_cost: 0.9460 data_cost: 0.1204 ips: 9.5137 images/s
- [11/24 18:58:57] ppdet.engine INFO: Epoch: [9] [1100/1193] learning_rate: 0.002500 loss_xy: 3.264235 loss_wh: 0.683192 loss_obj: 3.531415 loss_cls: 0.788942 loss: 8.033610 eta: 12:54:06 batch_cost: 0.8017 data_cost: 0.2223 ips: 11.2260 images/s
- [11/24 18:59:30] ppdet.engine INFO: Epoch: [9] [1150/1193] learning_rate: 0.002500 loss_xy: 3.289911 loss_wh: 0.749884 loss_obj: 3.703246 loss_cls: 0.751098 loss: 8.690506 eta: 12:52:14 batch_cost: 0.6623 data_cost: 0.1059 ips: 13.5881 images/s
- [11/24 19:00:03] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 19:00:03] ppdet.engine INFO: Eval iter: 0
- [11/24 19:00:07] ppdet.engine INFO: Eval iter: 100
- [11/24 19:00:11] ppdet.engine INFO: Eval iter: 200
- [11/24 19:00:15] ppdet.engine INFO: Eval iter: 300
- [11/24 19:00:19] ppdet.engine INFO: Eval iter: 400
- [11/24 19:00:23] ppdet.engine INFO: Eval iter: 500
- [11/24 19:00:27] ppdet.engine INFO: Eval iter: 600
- [11/24 19:00:31] ppdet.engine INFO: Eval iter: 700
- [11/24 19:00:35] ppdet.engine INFO: Eval iter: 800
- [11/24 19:00:39] ppdet.engine INFO: Eval iter: 900
- [11/24 19:00:44] ppdet.engine INFO: Eval iter: 1000
- [11/24 19:00:48] ppdet.engine INFO: Eval iter: 1100
- [11/24 19:00:53] ppdet.engine INFO: Eval iter: 1200
- [11/24 19:00:58] ppdet.engine INFO: Eval iter: 1300
- [11/24 19:01:02] ppdet.engine INFO: Eval iter: 1400
- [11/24 19:01:06] ppdet.engine INFO: Eval iter: 1500
- [11/24 19:01:10] ppdet.engine INFO: Eval iter: 1600
- [11/24 19:01:14] ppdet.engine INFO: Eval iter: 1700
- [11/24 19:01:18] ppdet.engine INFO: Eval iter: 1800
- [11/24 19:01:22] ppdet.engine INFO: Eval iter: 1900
- [11/24 19:01:26] ppdet.engine INFO: Eval iter: 2000
- [11/24 19:01:30] ppdet.engine INFO: Eval iter: 2100
- [11/24 19:01:34] ppdet.engine INFO: Eval iter: 2200
- [11/24 19:01:38] ppdet.engine INFO: Eval iter: 2300
- [11/24 19:01:43] ppdet.engine INFO: Eval iter: 2400
- [11/24 19:01:47] ppdet.engine INFO: Eval iter: 2500
- [11/24 19:01:51] ppdet.engine INFO: Eval iter: 2600
- [11/24 19:01:55] ppdet.engine INFO: Eval iter: 2700
- [11/24 19:01:59] ppdet.engine INFO: Eval iter: 2800
- [11/24 19:02:03] ppdet.engine INFO: Eval iter: 2900
- [11/24 19:02:07] ppdet.engine INFO: Eval iter: 3000
- [11/24 19:02:11] ppdet.engine INFO: Eval iter: 3100
- [11/24 19:02:15] ppdet.engine INFO: Eval iter: 3200
- [11/24 19:02:19] ppdet.engine INFO: Eval iter: 3300
- [11/24 19:02:23] ppdet.engine INFO: Eval iter: 3400
- [11/24 19:02:27] ppdet.engine INFO: Eval iter: 3500
- [11/24 19:02:31] ppdet.engine INFO: Eval iter: 3600
- [11/24 19:02:36] ppdet.engine INFO: Eval iter: 3700
- [11/24 19:02:40] ppdet.engine INFO: Eval iter: 3800
- [11/24 19:02:43] ppdet.engine INFO: Eval iter: 3900
- [11/24 19:02:47] ppdet.engine INFO: Eval iter: 4000
- [11/24 19:02:52] ppdet.engine INFO: Eval iter: 4100
- [11/24 19:02:55] ppdet.engine INFO: Eval iter: 4200
- [11/24 19:02:59] ppdet.engine INFO: Eval iter: 4300
- [11/24 19:03:01] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 19:03:01] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 88.13%
- [11/24 19:03:01] ppdet.engine INFO: Total sample number: 4339, averge FPS: 24.35630491664306
- [11/24 19:03:01] ppdet.engine INFO: Best test bbox ap is 0.881.
- [11/24 19:03:05] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 19:03:05] ppdet.engine INFO: Epoch: [10] [ 0/1193] learning_rate: 0.002500 loss_xy: 2.863069 loss_wh: 0.531043 loss_obj: 3.104683 loss_cls: 0.689964 loss: 7.184703 eta: 12:51:19 batch_cost: 0.7348 data_cost: 0.1184 ips: 12.2485 images/s
- [11/24 19:03:42] ppdet.engine INFO: Epoch: [10] [ 50/1193] learning_rate: 0.002500 loss_xy: 2.931909 loss_wh: 0.635336 loss_obj: 3.318968 loss_cls: 0.828208 loss: 7.283414 eta: 12:50:12 batch_cost: 0.7294 data_cost: 0.0917 ips: 12.3387 images/s
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::Tracer::TraceOp(std::string const&, paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap, std::map<std::string, std::string, std::less<std::string >, std::allocator<std::pair<std::string const, std::string > > > const&)
- 1 void paddle::imperative::Tracer::TraceOpImpl<paddle::imperative::VarBase>(std::string const&, paddle::imperative::details::NameVarMapTrait<paddle::imperative::VarBase>::Type const&, paddle::imperative::details::NameVarMapTrait<paddle::imperative::VarBase>::Type const&, paddle::framework::AttributeMap&, phi::Place const&, bool, std::map<std::string, std::string, std::less<std::string >, std::allocator<std::pair<std::string const, std::string > > > const&, paddle::framework::AttributeMap*, bool)
- 2 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap const&, paddle::framework::AttributeMap const&)
- 3 phi::KernelImpl<void (*)(phi::GPUContext const&, std::vector<phi::DenseTensor const*, std::allocator<phi::DenseTensor const*> > const&, std::vector<phi::DenseTensor*, std::allocator<phi::DenseTensor*> >), &(void phi::MeshgridKernel<long, phi::GPUContext>(phi::GPUContext const&, std::vector<phi::DenseTensor const*, std::allocator<phi::DenseTensor const*> > const&, std::vector<phi::DenseTensor*, std::allocator<phi::DenseTensor*> >))>::Compute(phi::KernelContext*)
- 4 void phi::MeshgridKernel<long, phi::GPUContext>(phi::GPUContext const&, std::vector<phi::DenseTensor const*, std::allocator<phi::DenseTensor const*> > const&, std::vector<phi::DenseTensor*, std::allocator<phi::DenseTensor*> >)
- 5 void phi::MeshgridForward<long, phi::GPUContext, 2>(phi::GPUContext const&, std::vector<phi::DenseTensor const*, std::allocator<phi::DenseTensor const*> > const&, std::vector<phi::DenseTensor*, std::allocator<phi::DenseTensor*> >)
- 6 void paddle::framework::TensorCopyImpl<phi::DenseTensor>(phi::DenseTensor const&, phi::Place const&, phi::DeviceContext const&, phi::DenseTensor*)
- 7 void paddle::memory::Copy<phi::Place, phi::Place>(phi::Place, void*, phi::Place, void const*, unsigned long, void*)
- 8 void paddle::memory::Copy<phi::GPUPlace, phi::GPUPlace>(phi::GPUPlace, void*, phi::GPUPlace, void const*, unsigned long, void*)
- 9 phi::backends::gpu::GpuMemcpyAsync(void*, void const*, unsigned long, cudaMemcpyKind, CUstream_st*)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669287849 (unix time) try "date -d @1669287849" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e800005148) received by PID 20888 (TID 0x7f5489386700) from PID 20808 ***]
-
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:33389', '127.0.0.1:53373', '127.0.0.1:47055']
- I1124 19:14:46.530498 41910 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1124 19:14:47.342681 41910 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 19:14:47.346956 41910 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/24 19:14:49] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 19:14:49] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/24 19:14:51] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/24 19:14:52] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/24 19:14:53] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/24 19:14:53] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/24 19:14:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/24 19:14:56] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/24 19:14:56] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/24 19:14:56] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- [11/24 19:14:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/24 19:14:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/24 19:14:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/24 19:14:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/24 19:14:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 19:14:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- [11/24 19:15:00] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage/9.pdparams
- [11/24 19:15:05] ppdet.engine INFO: Epoch: [10] [ 0/1193] learning_rate: 0.002500 loss_xy: 2.633298 loss_wh: 0.735424 loss_obj: 3.165926 loss_cls: 0.822666 loss: 7.357314 eta: 2 days, 15:18:21 batch_cost: 3.8206 data_cost: 1.6678 ips: 2.3556 images/s
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:40411', '127.0.0.1:46327', '127.0.0.1:41287']
- I1124 19:15:47.317837 48517 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1124 19:15:48.224442 48517 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 19:15:48.229154 48517 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/24 19:15:50] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 19:15:50] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/24 19:15:52] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/24 19:15:53] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/24 19:15:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/24 19:15:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/24 19:15:55] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/24 19:15:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/24 19:15:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/24 19:15:57] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- [11/24 19:15:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/24 19:15:58] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/24 19:15:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/24 19:15:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/24 19:15:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/24 19:16:00] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- [11/24 19:16:01] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage/9.pdparams
- [11/24 19:16:06] ppdet.engine INFO: Epoch: [10] [ 0/1193] learning_rate: 0.002500 loss_xy: 2.633298 loss_wh: 0.735424 loss_obj: 3.165926 loss_cls: 0.822666 loss: 7.357314 eta: 2 days, 15:36:35 batch_cost: 3.8390 data_cost: 1.6273 ips: 2.3444 images/s
- [11/24 19:16:39] ppdet.engine INFO: Epoch: [10] [ 50/1193] learning_rate: 0.001000 loss_xy: 3.019864 loss_wh: 0.577202 loss_obj: 3.052632 loss_cls: 0.749310 loss: 7.478045 eta: 11:59:43 batch_cost: 0.6623 data_cost: 0.0843 ips: 13.5898 images/s
- [11/24 19:17:17] ppdet.engine INFO: Epoch: [10] [ 100/1193] learning_rate: 0.001000 loss_xy: 3.023602 loss_wh: 0.471206 loss_obj: 2.270442 loss_cls: 0.568563 loss: 6.284634 eta: 12:18:57 batch_cost: 0.7649 data_cost: 0.1000 ips: 11.7657 images/s
- [11/24 19:18:01] ppdet.engine INFO: Epoch: [10] [ 150/1193] learning_rate: 0.001000 loss_xy: 3.046696 loss_wh: 0.502875 loss_obj: 2.366838 loss_cls: 0.525318 loss: 6.323416 eta: 13:05:16 batch_cost: 0.8875 data_cost: 0.1221 ips: 10.1408 images/s
- [11/24 19:18:44] ppdet.engine INFO: Epoch: [10] [ 200/1193] learning_rate: 0.001000 loss_xy: 3.009668 loss_wh: 0.503713 loss_obj: 2.449362 loss_cls: 0.509303 loss: 6.526912 eta: 13:17:33 batch_cost: 0.8444 data_cost: 0.0596 ips: 10.6589 images/s
- [11/24 19:19:24] ppdet.engine INFO: Epoch: [10] [ 250/1193] learning_rate: 0.001000 loss_xy: 3.035593 loss_wh: 0.505458 loss_obj: 2.287102 loss_cls: 0.489740 loss: 6.153751 eta: 13:17:24 batch_cost: 0.8076 data_cost: 0.0818 ips: 11.1445 images/s
- [11/24 19:19:57] ppdet.engine INFO: Epoch: [10] [ 300/1193] learning_rate: 0.001000 loss_xy: 3.069946 loss_wh: 0.570567 loss_obj: 2.610641 loss_cls: 0.489374 loss: 6.957041 eta: 12:53:48 batch_cost: 0.6660 data_cost: 0.0232 ips: 13.5144 images/s
- [11/24 19:20:34] ppdet.engine INFO: Epoch: [10] [ 350/1193] learning_rate: 0.001000 loss_xy: 3.232972 loss_wh: 0.583163 loss_obj: 2.598116 loss_cls: 0.566802 loss: 6.954273 eta: 12:44:35 batch_cost: 0.7214 data_cost: 0.0885 ips: 12.4752 images/s
- [11/24 19:21:17] ppdet.engine INFO: Epoch: [10] [ 400/1193] learning_rate: 0.001000 loss_xy: 3.283921 loss_wh: 0.619227 loss_obj: 2.836220 loss_cls: 0.524691 loss: 7.344953 eta: 12:55:24 batch_cost: 0.8667 data_cost: 0.0598 ips: 10.3842 images/s
- [11/24 19:21:56] ppdet.engine INFO: Epoch: [10] [ 450/1193] learning_rate: 0.001000 loss_xy: 3.078543 loss_wh: 0.512563 loss_obj: 2.135256 loss_cls: 0.465552 loss: 6.319977 eta: 12:55:04 batch_cost: 0.7882 data_cost: 0.0563 ips: 11.4182 images/s
- [11/24 19:22:35] ppdet.engine INFO: Epoch: [10] [ 500/1193] learning_rate: 0.001000 loss_xy: 2.886384 loss_wh: 0.549664 loss_obj: 2.374732 loss_cls: 0.454816 loss: 6.337524 eta: 12:53:03 batch_cost: 0.7716 data_cost: 0.1371 ips: 11.6639 images/s
- [11/24 19:23:11] ppdet.engine INFO: Epoch: [10] [ 550/1193] learning_rate: 0.001000 loss_xy: 3.452388 loss_wh: 0.659024 loss_obj: 2.581985 loss_cls: 0.456576 loss: 7.340144 eta: 12:46:40 batch_cost: 0.7200 data_cost: 0.3006 ips: 12.4992 images/s
- [11/24 19:23:49] ppdet.engine INFO: Epoch: [10] [ 600/1193] learning_rate: 0.001000 loss_xy: 3.128875 loss_wh: 0.545760 loss_obj: 2.302933 loss_cls: 0.414509 loss: 6.730107 eta: 12:45:08 batch_cost: 0.7676 data_cost: 0.0438 ips: 11.7254 images/s
- [11/24 19:24:30] ppdet.engine INFO: Epoch: [10] [ 650/1193] learning_rate: 0.001000 loss_xy: 3.199617 loss_wh: 0.620074 loss_obj: 2.307325 loss_cls: 0.425628 loss: 6.714039 eta: 12:46:55 batch_cost: 0.8097 data_cost: 0.1805 ips: 11.1159 images/s
- [11/24 19:25:09] ppdet.engine INFO: Epoch: [10] [ 700/1193] learning_rate: 0.001000 loss_xy: 3.144461 loss_wh: 0.578956 loss_obj: 2.297560 loss_cls: 0.422145 loss: 6.220817 eta: 12:46:20 batch_cost: 0.7808 data_cost: 0.1971 ips: 11.5261 images/s
- [11/24 19:25:49] ppdet.engine INFO: Epoch: [10] [ 750/1193] learning_rate: 0.001000 loss_xy: 3.168547 loss_wh: 0.528016 loss_obj: 2.244240 loss_cls: 0.473239 loss: 6.465210 eta: 12:46:10 batch_cost: 0.7874 data_cost: 0.0452 ips: 11.4300 images/s
- [11/24 19:26:26] ppdet.engine INFO: Epoch: [10] [ 800/1193] learning_rate: 0.001000 loss_xy: 3.124054 loss_wh: 0.563979 loss_obj: 2.147023 loss_cls: 0.394354 loss: 6.234438 eta: 12:43:01 batch_cost: 0.7398 data_cost: 0.1992 ips: 12.1656 images/s
- [11/24 19:27:03] ppdet.engine INFO: Epoch: [10] [ 850/1193] learning_rate: 0.001000 loss_xy: 3.003409 loss_wh: 0.516604 loss_obj: 2.016832 loss_cls: 0.444888 loss: 6.160168 eta: 12:40:10 batch_cost: 0.7396 data_cost: 0.1574 ips: 12.1682 images/s
- [11/24 19:27:37] ppdet.engine INFO: Epoch: [10] [ 900/1193] learning_rate: 0.001000 loss_xy: 2.983660 loss_wh: 0.514929 loss_obj: 1.889101 loss_cls: 0.448983 loss: 5.969903 eta: 12:34:22 batch_cost: 0.6808 data_cost: 0.0357 ips: 13.2192 images/s
- [11/24 19:28:15] ppdet.engine INFO: Epoch: [10] [ 950/1193] learning_rate: 0.001000 loss_xy: 3.177066 loss_wh: 0.571153 loss_obj: 1.901495 loss_cls: 0.438500 loss: 6.017012 eta: 12:33:44 batch_cost: 0.7706 data_cost: 0.0683 ips: 11.6799 images/s
- [11/24 19:28:54] ppdet.engine INFO: Epoch: [10] [1000/1193] learning_rate: 0.001000 loss_xy: 2.828249 loss_wh: 0.494700 loss_obj: 1.810877 loss_cls: 0.367063 loss: 5.859869 eta: 12:33:02 batch_cost: 0.7693 data_cost: 0.2531 ips: 11.6997 images/s
- [11/24 19:29:34] ppdet.engine INFO: Epoch: [10] [1050/1193] learning_rate: 0.001000 loss_xy: 3.084188 loss_wh: 0.573530 loss_obj: 2.339965 loss_cls: 0.489436 loss: 6.468423 eta: 12:33:45 batch_cost: 0.7994 data_cost: 0.1780 ips: 11.2578 images/s
- [11/24 19:30:16] ppdet.engine INFO: Epoch: [10] [1100/1193] learning_rate: 0.001000 loss_xy: 3.208300 loss_wh: 0.558032 loss_obj: 2.193960 loss_cls: 0.416249 loss: 6.482358 eta: 12:36:05 batch_cost: 0.8390 data_cost: 0.3184 ips: 10.7270 images/s
- [11/24 19:30:54] ppdet.engine INFO: Epoch: [10] [1150/1193] learning_rate: 0.001000 loss_xy: 3.257915 loss_wh: 0.573444 loss_obj: 2.210128 loss_cls: 0.400949 loss: 6.397307 eta: 12:34:44 batch_cost: 0.7585 data_cost: 0.1847 ips: 11.8657 images/s
- [11/24 19:31:19] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 19:31:21] ppdet.engine INFO: Eval iter: 0
- [11/24 19:31:25] ppdet.engine INFO: Eval iter: 100
- [11/24 19:31:29] ppdet.engine INFO: Eval iter: 200
- [11/24 19:31:33] ppdet.engine INFO: Eval iter: 300
- [11/24 19:31:37] ppdet.engine INFO: Eval iter: 400
- [11/24 19:31:41] ppdet.engine INFO: Eval iter: 500
- [11/24 19:31:45] ppdet.engine INFO: Eval iter: 600
- [11/24 19:31:49] ppdet.engine INFO: Eval iter: 700
- [11/24 19:31:53] ppdet.engine INFO: Eval iter: 800
- [11/24 19:31:57] ppdet.engine INFO: Eval iter: 900
- [11/24 19:32:01] ppdet.engine INFO: Eval iter: 1000
- [11/24 19:32:05] ppdet.engine INFO: Eval iter: 1100
- [11/24 19:32:09] ppdet.engine INFO: Eval iter: 1200
- [11/24 19:32:13] ppdet.engine INFO: Eval iter: 1300
- [11/24 19:32:17] ppdet.engine INFO: Eval iter: 1400
- [11/24 19:32:21] ppdet.engine INFO: Eval iter: 1500
- [11/24 19:32:25] ppdet.engine INFO: Eval iter: 1600
- [11/24 19:32:29] ppdet.engine INFO: Eval iter: 1700
- [11/24 19:32:33] ppdet.engine INFO: Eval iter: 1800
- [11/24 19:32:37] ppdet.engine INFO: Eval iter: 1900
- [11/24 19:32:41] ppdet.engine INFO: Eval iter: 2000
- [11/24 19:32:45] ppdet.engine INFO: Eval iter: 2100
- [11/24 19:32:49] ppdet.engine INFO: Eval iter: 2200
- [11/24 19:32:53] ppdet.engine INFO: Eval iter: 2300
- [11/24 19:32:57] ppdet.engine INFO: Eval iter: 2400
- [11/24 19:33:01] ppdet.engine INFO: Eval iter: 2500
- [11/24 19:33:05] ppdet.engine INFO: Eval iter: 2600
- [11/24 19:33:09] ppdet.engine INFO: Eval iter: 2700
- [11/24 19:33:13] ppdet.engine INFO: Eval iter: 2800
- [11/24 19:33:17] ppdet.engine INFO: Eval iter: 2900
- [11/24 19:33:20] ppdet.engine INFO: Eval iter: 3000
- [11/24 19:33:24] ppdet.engine INFO: Eval iter: 3100
- [11/24 19:33:28] ppdet.engine INFO: Eval iter: 3200
- [11/24 19:33:32] ppdet.engine INFO: Eval iter: 3300
- [11/24 19:33:36] ppdet.engine INFO: Eval iter: 3400
- [11/24 19:33:40] ppdet.engine INFO: Eval iter: 3500
- [11/24 19:33:44] ppdet.engine INFO: Eval iter: 3600
- [11/24 19:33:47] ppdet.engine INFO: Eval iter: 3700
- [11/24 19:33:51] ppdet.engine INFO: Eval iter: 3800
- [11/24 19:33:55] ppdet.engine INFO: Eval iter: 3900
- [11/24 19:33:59] ppdet.engine INFO: Eval iter: 4000
- [11/24 19:34:03] ppdet.engine INFO: Eval iter: 4100
- [11/24 19:34:07] ppdet.engine INFO: Eval iter: 4200
- [11/24 19:34:11] ppdet.engine INFO: Eval iter: 4300
- [11/24 19:34:16] ppdet.engine INFO: Eval iter: 4400
- [11/24 19:34:20] ppdet.engine INFO: Eval iter: 4500
- [11/24 19:34:24] ppdet.engine INFO: Eval iter: 4600
- [11/24 19:34:28] ppdet.engine INFO: Eval iter: 4700
- [11/24 19:34:31] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 19:34:31] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 90.84%
- [11/24 19:34:31] ppdet.engine INFO: Total sample number: 4773, averge FPS: 25.132688374906603
- [11/24 19:34:31] ppdet.engine INFO: Best test bbox ap is 0.908.
- [11/24 19:34:35] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 19:34:35] ppdet.engine INFO: Epoch: [11] [ 0/1193] learning_rate: 0.001000 loss_xy: 2.975803 loss_wh: 0.531264 loss_obj: 2.056997 loss_cls: 0.433056 loss: 6.518139 eta: 12:27:13 batch_cost: 0.6051 data_cost: 0.0375 ips: 14.8729 images/s
- [11/24 19:35:18] ppdet.engine INFO: Epoch: [11] [ 50/1193] learning_rate: 0.001000 loss_xy: 3.334692 loss_wh: 0.612705 loss_obj: 2.108571 loss_cls: 0.408164 loss: 6.222170 eta: 12:30:01 batch_cost: 0.8549 data_cost: 0.1640 ips: 10.5273 images/s
- [11/24 19:36:00] ppdet.engine INFO: Epoch: [11] [ 100/1193] learning_rate: 0.001000 loss_xy: 3.152706 loss_wh: 0.533832 loss_obj: 2.173587 loss_cls: 0.419289 loss: 5.956573 eta: 12:31:43 batch_cost: 0.8328 data_cost: 0.3162 ips: 10.8073 images/s
- [11/24 19:36:36] ppdet.engine INFO: Epoch: [11] [ 150/1193] learning_rate: 0.001000 loss_xy: 3.007984 loss_wh: 0.487776 loss_obj: 1.735890 loss_cls: 0.406996 loss: 6.053475 eta: 12:29:10 batch_cost: 0.7205 data_cost: 0.1113 ips: 12.4909 images/s
- [11/24 19:37:11] ppdet.engine INFO: Epoch: [11] [ 200/1193] learning_rate: 0.001000 loss_xy: 3.217873 loss_wh: 0.590280 loss_obj: 2.022943 loss_cls: 0.432537 loss: 6.422501 eta: 12:26:16 batch_cost: 0.7061 data_cost: 0.0983 ips: 12.7462 images/s
- [11/24 19:37:54] ppdet.engine INFO: Epoch: [11] [ 250/1193] learning_rate: 0.001000 loss_xy: 2.916278 loss_wh: 0.472518 loss_obj: 1.920506 loss_cls: 0.433259 loss: 5.810373 eta: 12:28:25 batch_cost: 0.8517 data_cost: 0.0962 ips: 10.5670 images/s
- [11/24 19:38:32] ppdet.engine INFO: Epoch: [11] [ 300/1193] learning_rate: 0.001000 loss_xy: 2.926742 loss_wh: 0.458957 loss_obj: 1.697838 loss_cls: 0.370459 loss: 5.257497 eta: 12:27:33 batch_cost: 0.7645 data_cost: 0.2433 ips: 11.7721 images/s
- [11/24 19:39:23] ppdet.engine INFO: Epoch: [11] [ 350/1193] learning_rate: 0.001000 loss_xy: 3.001630 loss_wh: 0.523982 loss_obj: 2.035818 loss_cls: 0.277442 loss: 6.080102 eta: 12:34:35 batch_cost: 1.0156 data_cost: 0.1193 ips: 8.8618 images/s
- [11/24 19:40:04] ppdet.engine INFO: Epoch: [11] [ 400/1193] learning_rate: 0.001000 loss_xy: 2.942963 loss_wh: 0.544755 loss_obj: 1.708154 loss_cls: 0.360713 loss: 5.726896 eta: 12:35:27 batch_cost: 0.8296 data_cost: 0.0837 ips: 10.8487 images/s
- [11/24 19:40:51] ppdet.engine INFO: Epoch: [11] [ 450/1193] learning_rate: 0.001000 loss_xy: 2.957951 loss_wh: 0.522987 loss_obj: 1.946262 loss_cls: 0.439821 loss: 5.984333 eta: 12:39:31 batch_cost: 0.9407 data_cost: 0.1543 ips: 9.5672 images/s
- [11/24 19:41:28] ppdet.engine INFO: Epoch: [11] [ 500/1193] learning_rate: 0.001000 loss_xy: 3.046415 loss_wh: 0.555777 loss_obj: 2.012070 loss_cls: 0.379411 loss: 6.125405 eta: 12:37:17 batch_cost: 0.7307 data_cost: 0.0575 ips: 12.3164 images/s
- [11/24 19:42:04] ppdet.engine INFO: Epoch: [11] [ 550/1193] learning_rate: 0.001000 loss_xy: 2.610965 loss_wh: 0.476904 loss_obj: 1.827362 loss_cls: 0.349296 loss: 5.464478 eta: 12:34:39 batch_cost: 0.7124 data_cost: 0.1024 ips: 12.6335 images/s
- [11/24 19:42:41] ppdet.engine INFO: Epoch: [11] [ 600/1193] learning_rate: 0.001000 loss_xy: 2.928674 loss_wh: 0.481849 loss_obj: 1.961908 loss_cls: 0.416910 loss: 5.923745 eta: 12:33:16 batch_cost: 0.7544 data_cost: 0.2451 ips: 11.9301 images/s
- [11/24 19:43:30] ppdet.engine INFO: Epoch: [11] [ 650/1193] learning_rate: 0.001000 loss_xy: 3.040953 loss_wh: 0.599175 loss_obj: 2.258060 loss_cls: 0.385344 loss: 6.288223 eta: 12:37:21 batch_cost: 0.9626 data_cost: 0.0764 ips: 9.3492 images/s
- [11/24 19:44:11] ppdet.engine INFO: Epoch: [11] [ 700/1193] learning_rate: 0.001000 loss_xy: 2.736906 loss_wh: 0.505708 loss_obj: 1.744038 loss_cls: 0.438717 loss: 5.634615 eta: 12:37:33 batch_cost: 0.8198 data_cost: 0.0930 ips: 10.9785 images/s
- [11/24 19:44:55] ppdet.engine INFO: Epoch: [11] [ 750/1193] learning_rate: 0.001000 loss_xy: 2.929405 loss_wh: 0.554722 loss_obj: 1.815686 loss_cls: 0.491523 loss: 5.670375 eta: 12:39:33 batch_cost: 0.8944 data_cost: 0.1793 ips: 10.0625 images/s
- [11/24 19:45:29] ppdet.engine INFO: Epoch: [11] [ 800/1193] learning_rate: 0.001000 loss_xy: 2.973066 loss_wh: 0.524755 loss_obj: 2.004826 loss_cls: 0.483970 loss: 6.091640 eta: 12:35:49 batch_cost: 0.6616 data_cost: 0.1148 ips: 13.6030 images/s
- [11/24 19:46:04] ppdet.engine INFO: Epoch: [11] [ 850/1193] learning_rate: 0.001000 loss_xy: 3.003738 loss_wh: 0.517400 loss_obj: 1.880893 loss_cls: 0.383601 loss: 5.721092 eta: 12:33:29 batch_cost: 0.7155 data_cost: 0.0857 ips: 12.5782 images/s
- [11/24 19:46:42] ppdet.engine INFO: Epoch: [11] [ 900/1193] learning_rate: 0.001000 loss_xy: 2.684209 loss_wh: 0.438146 loss_obj: 1.495702 loss_cls: 0.394118 loss: 5.210511 eta: 12:31:56 batch_cost: 0.7452 data_cost: 0.1099 ips: 12.0767 images/s
- [11/24 19:47:19] ppdet.engine INFO: Epoch: [11] [ 950/1193] learning_rate: 0.001000 loss_xy: 2.904229 loss_wh: 0.481289 loss_obj: 2.119429 loss_cls: 0.414555 loss: 5.986637 eta: 12:30:31 batch_cost: 0.7500 data_cost: 0.1887 ips: 12.0005 images/s
- [11/24 19:47:55] ppdet.engine INFO: Epoch: [11] [1000/1193] learning_rate: 0.001000 loss_xy: 3.159341 loss_wh: 0.548778 loss_obj: 1.971804 loss_cls: 0.496628 loss: 6.318345 eta: 12:28:29 batch_cost: 0.7201 data_cost: 0.1680 ips: 12.4980 images/s
- [11/24 19:48:37] ppdet.engine INFO: Epoch: [11] [1050/1193] learning_rate: 0.001000 loss_xy: 2.697896 loss_wh: 0.467173 loss_obj: 1.691426 loss_cls: 0.405456 loss: 5.405315 eta: 12:29:00 batch_cost: 0.8359 data_cost: 0.2775 ips: 10.7669 images/s
- [11/24 19:49:15] ppdet.engine INFO: Epoch: [11] [1100/1193] learning_rate: 0.001000 loss_xy: 2.924149 loss_wh: 0.503933 loss_obj: 1.803269 loss_cls: 0.428773 loss: 5.710451 eta: 12:27:56 batch_cost: 0.7629 data_cost: 0.0966 ips: 11.7966 images/s
- [11/24 19:49:55] ppdet.engine INFO: Epoch: [11] [1150/1193] learning_rate: 0.001000 loss_xy: 2.789368 loss_wh: 0.442644 loss_obj: 1.684477 loss_cls: 0.374376 loss: 5.573234 eta: 12:27:35 batch_cost: 0.7972 data_cost: 0.0266 ips: 11.2901 images/s
- [11/24 19:50:31] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 19:50:31] ppdet.engine INFO: Eval iter: 0
- [11/24 19:50:35] ppdet.engine INFO: Eval iter: 100
- [11/24 19:50:39] ppdet.engine INFO: Eval iter: 200
- [11/24 19:50:43] ppdet.engine INFO: Eval iter: 300
- [11/24 19:50:47] ppdet.engine INFO: Eval iter: 400
- [11/24 19:50:52] ppdet.engine INFO: Eval iter: 500
- [11/24 19:50:56] ppdet.engine INFO: Eval iter: 600
- [11/24 19:51:00] ppdet.engine INFO: Eval iter: 700
- [11/24 19:51:04] ppdet.engine INFO: Eval iter: 800
- [11/24 19:51:09] ppdet.engine INFO: Eval iter: 900
- [11/24 19:51:13] ppdet.engine INFO: Eval iter: 1000
- [11/24 19:51:17] ppdet.engine INFO: Eval iter: 1100
- [11/24 19:51:22] ppdet.engine INFO: Eval iter: 1200
- [11/24 19:51:26] ppdet.engine INFO: Eval iter: 1300
- [11/24 19:51:30] ppdet.engine INFO: Eval iter: 1400
- [11/24 19:51:34] ppdet.engine INFO: Eval iter: 1500
- [11/24 19:51:38] ppdet.engine INFO: Eval iter: 1600
- [11/24 19:51:42] ppdet.engine INFO: Eval iter: 1700
- [11/24 19:51:46] ppdet.engine INFO: Eval iter: 1800
- [11/24 19:51:50] ppdet.engine INFO: Eval iter: 1900
- [11/24 19:51:54] ppdet.engine INFO: Eval iter: 2000
- [11/24 19:51:58] ppdet.engine INFO: Eval iter: 2100
- [11/24 19:52:02] ppdet.engine INFO: Eval iter: 2200
- [11/24 19:52:06] ppdet.engine INFO: Eval iter: 2300
- [11/24 19:52:10] ppdet.engine INFO: Eval iter: 2400
- [11/24 19:52:14] ppdet.engine INFO: Eval iter: 2500
- [11/24 19:52:18] ppdet.engine INFO: Eval iter: 2600
- [11/24 19:52:22] ppdet.engine INFO: Eval iter: 2700
- [11/24 19:52:26] ppdet.engine INFO: Eval iter: 2800
- [11/24 19:52:30] ppdet.engine INFO: Eval iter: 2900
- [11/24 19:52:34] ppdet.engine INFO: Eval iter: 3000
- [11/24 19:52:38] ppdet.engine INFO: Eval iter: 3100
- [11/24 19:52:42] ppdet.engine INFO: Eval iter: 3200
- [11/24 19:52:46] ppdet.engine INFO: Eval iter: 3300
- [11/24 19:52:50] ppdet.engine INFO: Eval iter: 3400
- [11/24 19:52:54] ppdet.engine INFO: Eval iter: 3500
- [11/24 19:52:58] ppdet.engine INFO: Eval iter: 3600
- [11/24 19:53:02] ppdet.engine INFO: Eval iter: 3700
- [11/24 19:53:06] ppdet.engine INFO: Eval iter: 3800
- [11/24 19:53:10] ppdet.engine INFO: Eval iter: 3900
- [11/24 19:53:14] ppdet.engine INFO: Eval iter: 4000
- [11/24 19:53:18] ppdet.engine INFO: Eval iter: 4100
- [11/24 19:53:22] ppdet.engine INFO: Eval iter: 4200
- [11/24 19:53:26] ppdet.engine INFO: Eval iter: 4300
- [11/24 19:53:30] ppdet.engine INFO: Eval iter: 4400
- [11/24 19:53:34] ppdet.engine INFO: Eval iter: 4500
- [11/24 19:53:38] ppdet.engine INFO: Eval iter: 4600
- [11/24 19:53:42] ppdet.engine INFO: Eval iter: 4700
- [11/24 19:53:45] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 19:53:45] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 90.91%
- [11/24 19:53:45] ppdet.engine INFO: Total sample number: 4773, averge FPS: 24.601953866555274
- [11/24 19:53:45] ppdet.engine INFO: Best test bbox ap is 0.909.
- [11/24 19:53:48] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 19:53:49] ppdet.engine INFO: Epoch: [12] [ 0/1193] learning_rate: 0.001000 loss_xy: 2.888885 loss_wh: 0.433086 loss_obj: 1.546715 loss_cls: 0.341084 loss: 5.118978 eta: 12:27:26 batch_cost: 0.7696 data_cost: 0.0658 ips: 11.6947 images/s
- [11/24 19:54:26] ppdet.engine INFO: Epoch: [12] [ 50/1193] learning_rate: 0.001000 loss_xy: 3.013127 loss_wh: 0.509983 loss_obj: 2.031313 loss_cls: 0.430104 loss: 6.402806 eta: 12:26:02 batch_cost: 0.7452 data_cost: 0.0475 ips: 12.0768 images/s
- [11/24 19:55:13] ppdet.engine INFO: Epoch: [12] [ 100/1193] learning_rate: 0.001000 loss_xy: 3.188355 loss_wh: 0.551366 loss_obj: 2.258231 loss_cls: 0.458871 loss: 6.132111 eta: 12:28:25 batch_cost: 0.9410 data_cost: 0.0867 ips: 9.5646 images/s
- [11/24 19:55:44] ppdet.engine INFO: Epoch: [12] [ 150/1193] learning_rate: 0.001000 loss_xy: 3.124197 loss_wh: 0.545797 loss_obj: 2.248470 loss_cls: 0.443731 loss: 6.548149 eta: 12:24:33 batch_cost: 0.6138 data_cost: 0.0510 ips: 14.6623 images/s
- [11/24 19:56:21] ppdet.engine INFO: Epoch: [12] [ 200/1193] learning_rate: 0.001000 loss_xy: 3.016167 loss_wh: 0.567618 loss_obj: 2.312507 loss_cls: 0.441242 loss: 6.516082 eta: 12:23:03 batch_cost: 0.7365 data_cost: 0.1545 ips: 12.2207 images/s
- [11/24 19:56:56] ppdet.engine INFO: Epoch: [12] [ 250/1193] learning_rate: 0.001000 loss_xy: 3.342626 loss_wh: 0.592045 loss_obj: 2.552302 loss_cls: 0.495489 loss: 7.152474 eta: 12:21:03 batch_cost: 0.7060 data_cost: 0.0480 ips: 12.7480 images/s
- [11/24 19:57:34] ppdet.engine INFO: Epoch: [12] [ 300/1193] learning_rate: 0.001000 loss_xy: 3.159328 loss_wh: 0.635617 loss_obj: 2.525003 loss_cls: 0.439540 loss: 6.914573 eta: 12:19:51 batch_cost: 0.7493 data_cost: 0.0576 ips: 12.0110 images/s
- [11/24 19:58:17] ppdet.engine INFO: Epoch: [12] [ 350/1193] learning_rate: 0.001000 loss_xy: 3.060862 loss_wh: 0.496314 loss_obj: 2.375208 loss_cls: 0.384929 loss: 6.482912 eta: 12:20:35 batch_cost: 0.8587 data_cost: 0.0985 ips: 10.4814 images/s
- [11/24 19:58:55] ppdet.engine INFO: Epoch: [12] [ 400/1193] learning_rate: 0.001000 loss_xy: 3.080884 loss_wh: 0.549626 loss_obj: 2.218699 loss_cls: 0.410015 loss: 6.211788 eta: 12:19:37 batch_cost: 0.7626 data_cost: 0.1265 ips: 11.8014 images/s
- [11/24 19:59:27] ppdet.engine INFO: Epoch: [12] [ 450/1193] learning_rate: 0.001000 loss_xy: 3.338418 loss_wh: 0.571806 loss_obj: 2.122081 loss_cls: 0.534058 loss: 6.028563 eta: 12:16:39 batch_cost: 0.6417 data_cost: 0.0153 ips: 14.0246 images/s
- [11/24 20:00:02] ppdet.engine INFO: Epoch: [12] [ 500/1193] learning_rate: 0.001000 loss_xy: 2.774027 loss_wh: 0.483647 loss_obj: 2.096784 loss_cls: 0.474504 loss: 6.089504 eta: 12:14:46 batch_cost: 0.7022 data_cost: 0.1998 ips: 12.8177 images/s
- [11/24 20:00:40] ppdet.engine INFO: Epoch: [12] [ 550/1193] learning_rate: 0.001000 loss_xy: 2.711678 loss_wh: 0.484255 loss_obj: 2.212201 loss_cls: 0.548408 loss: 5.846521 eta: 12:13:55 batch_cost: 0.7637 data_cost: 0.1033 ips: 11.7842 images/s
- [11/24 20:01:18] ppdet.engine INFO: Epoch: [12] [ 600/1193] learning_rate: 0.001000 loss_xy: 3.163697 loss_wh: 0.520940 loss_obj: 1.986634 loss_cls: 0.484286 loss: 6.287736 eta: 12:12:53 batch_cost: 0.7525 data_cost: 0.1553 ips: 11.9594 images/s
- [11/24 20:01:52] ppdet.engine INFO: Epoch: [12] [ 650/1193] learning_rate: 0.001000 loss_xy: 3.039299 loss_wh: 0.559992 loss_obj: 2.127084 loss_cls: 0.503429 loss: 6.193305 eta: 12:10:39 batch_cost: 0.6737 data_cost: 0.1814 ips: 13.3593 images/s
- [11/24 20:02:28] ppdet.engine INFO: Epoch: [12] [ 700/1193] learning_rate: 0.001000 loss_xy: 3.020166 loss_wh: 0.534707 loss_obj: 2.255481 loss_cls: 0.493763 loss: 6.673656 eta: 12:09:06 batch_cost: 0.7157 data_cost: 0.1534 ips: 12.5743 images/s
- [11/24 20:03:08] ppdet.engine INFO: Epoch: [12] [ 750/1193] learning_rate: 0.001000 loss_xy: 2.857195 loss_wh: 0.492348 loss_obj: 2.003945 loss_cls: 0.483659 loss: 5.988478 eta: 12:08:53 batch_cost: 0.8016 data_cost: 0.0419 ips: 11.2273 images/s
- [11/24 20:03:52] ppdet.engine INFO: Epoch: [12] [ 800/1193] learning_rate: 0.001000 loss_xy: 3.121712 loss_wh: 0.523413 loss_obj: 2.276805 loss_cls: 0.450694 loss: 6.222584 eta: 12:09:51 batch_cost: 0.8826 data_cost: 0.0640 ips: 10.1974 images/s
- [11/24 20:04:33] ppdet.engine INFO: Epoch: [12] [ 850/1193] learning_rate: 0.001000 loss_xy: 2.655248 loss_wh: 0.496822 loss_obj: 2.098945 loss_cls: 0.416382 loss: 6.117629 eta: 12:09:44 batch_cost: 0.8126 data_cost: 0.0851 ips: 11.0752 images/s
- [11/24 20:05:13] ppdet.engine INFO: Epoch: [12] [ 900/1193] learning_rate: 0.001000 loss_xy: 2.842899 loss_wh: 0.511952 loss_obj: 1.936010 loss_cls: 0.540036 loss: 6.383679 eta: 12:09:36 batch_cost: 0.8111 data_cost: 0.0009 ips: 11.0960 images/s
- [11/24 20:05:55] ppdet.engine INFO: Epoch: [12] [ 950/1193] learning_rate: 0.001000 loss_xy: 3.191643 loss_wh: 0.596091 loss_obj: 2.701982 loss_cls: 0.498486 loss: 7.071540 eta: 12:09:50 batch_cost: 0.8401 data_cost: 0.1872 ips: 10.7134 images/s
- [11/24 20:06:35] ppdet.engine INFO: Epoch: [12] [1000/1193] learning_rate: 0.001000 loss_xy: 3.380233 loss_wh: 0.608738 loss_obj: 2.475329 loss_cls: 0.513832 loss: 7.421263 eta: 12:09:28 batch_cost: 0.7976 data_cost: 0.1005 ips: 11.2841 images/s
- [11/24 20:07:14] ppdet.engine INFO: Epoch: [12] [1050/1193] learning_rate: 0.001000 loss_xy: 2.945461 loss_wh: 0.494672 loss_obj: 1.869576 loss_cls: 0.576910 loss: 6.081127 eta: 12:08:56 batch_cost: 0.7861 data_cost: 0.1225 ips: 11.4485 images/s
- [11/24 20:07:51] ppdet.engine INFO: Epoch: [12] [1100/1193] learning_rate: 0.001000 loss_xy: 3.138844 loss_wh: 0.568511 loss_obj: 2.547186 loss_cls: 0.489937 loss: 6.755592 eta: 12:07:35 batch_cost: 0.7264 data_cost: 0.0517 ips: 12.3894 images/s
- [11/24 20:08:32] ppdet.engine INFO: Epoch: [12] [1150/1193] learning_rate: 0.001000 loss_xy: 3.248686 loss_wh: 0.626189 loss_obj: 2.253979 loss_cls: 0.574661 loss: 6.567486 eta: 12:07:34 batch_cost: 0.8242 data_cost: 0.1625 ips: 10.9196 images/s
- [11/24 20:09:09] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 20:09:09] ppdet.engine INFO: Eval iter: 0
- [11/24 20:09:13] ppdet.engine INFO: Eval iter: 100
- [11/24 20:09:17] ppdet.engine INFO: Eval iter: 200
- [11/24 20:09:22] ppdet.engine INFO: Eval iter: 300
- [11/24 20:09:26] ppdet.engine INFO: Eval iter: 400
- [11/24 20:09:30] ppdet.engine INFO: Eval iter: 500
- [11/24 20:09:34] ppdet.engine INFO: Eval iter: 600
- [11/24 20:09:38] ppdet.engine INFO: Eval iter: 700
- [11/24 20:09:42] ppdet.engine INFO: Eval iter: 800
- [11/24 20:09:46] ppdet.engine INFO: Eval iter: 900
- [11/24 20:09:50] ppdet.engine INFO: Eval iter: 1000
- [11/24 20:09:54] ppdet.engine INFO: Eval iter: 1100
- [11/24 20:09:58] ppdet.engine INFO: Eval iter: 1200
- [11/24 20:10:02] ppdet.engine INFO: Eval iter: 1300
- [11/24 20:10:06] ppdet.engine INFO: Eval iter: 1400
- [11/24 20:10:10] ppdet.engine INFO: Eval iter: 1500
- [11/24 20:10:14] ppdet.engine INFO: Eval iter: 1600
- [11/24 20:10:18] ppdet.engine INFO: Eval iter: 1700
- [11/24 20:10:22] ppdet.engine INFO: Eval iter: 1800
- [11/24 20:10:26] ppdet.engine INFO: Eval iter: 1900
- [11/24 20:10:30] ppdet.engine INFO: Eval iter: 2000
- [11/24 20:10:34] ppdet.engine INFO: Eval iter: 2100
- [11/24 20:10:38] ppdet.engine INFO: Eval iter: 2200
- [11/24 20:10:42] ppdet.engine INFO: Eval iter: 2300
- [11/24 20:10:46] ppdet.engine INFO: Eval iter: 2400
- [11/24 20:10:50] ppdet.engine INFO: Eval iter: 2500
- [11/24 20:10:54] ppdet.engine INFO: Eval iter: 2600
- [11/24 20:10:58] ppdet.engine INFO: Eval iter: 2700
- [11/24 20:11:02] ppdet.engine INFO: Eval iter: 2800
- [11/24 20:11:06] ppdet.engine INFO: Eval iter: 2900
- [11/24 20:11:10] ppdet.engine INFO: Eval iter: 3000
- [11/24 20:11:14] ppdet.engine INFO: Eval iter: 3100
- [11/24 20:11:18] ppdet.engine INFO: Eval iter: 3200
- [11/24 20:11:22] ppdet.engine INFO: Eval iter: 3300
- [11/24 20:11:26] ppdet.engine INFO: Eval iter: 3400
- [11/24 20:11:30] ppdet.engine INFO: Eval iter: 3500
- [11/24 20:11:34] ppdet.engine INFO: Eval iter: 3600
- [11/24 20:11:38] ppdet.engine INFO: Eval iter: 3700
- [11/24 20:11:42] ppdet.engine INFO: Eval iter: 3800
- [11/24 20:11:46] ppdet.engine INFO: Eval iter: 3900
- [11/24 20:11:50] ppdet.engine INFO: Eval iter: 4000
- [11/24 20:11:54] ppdet.engine INFO: Eval iter: 4100
- [11/24 20:11:58] ppdet.engine INFO: Eval iter: 4200
- [11/24 20:12:01] ppdet.engine INFO: Eval iter: 4300
- [11/24 20:12:06] ppdet.engine INFO: Eval iter: 4400
- [11/24 20:12:10] ppdet.engine INFO: Eval iter: 4500
- [11/24 20:12:14] ppdet.engine INFO: Eval iter: 4600
- [11/24 20:12:18] ppdet.engine INFO: Eval iter: 4700
- [11/24 20:12:21] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 20:12:21] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 90.88%
- [11/24 20:12:21] ppdet.engine INFO: Total sample number: 4773, averge FPS: 24.897997477428078
- [11/24 20:12:21] ppdet.engine INFO: Best test bbox ap is 0.909.
- [11/24 20:12:21] ppdet.engine INFO: Epoch: [13] [ 0/1193] learning_rate: 0.001000 loss_xy: 2.989296 loss_wh: 0.483017 loss_obj: 2.020629 loss_cls: 0.484875 loss: 6.119867 eta: 12:07:42 batch_cost: 0.8462 data_cost: 0.0787 ips: 10.6362 images/s
- [11/24 20:12:54] ppdet.engine INFO: Epoch: [13] [ 50/1193] learning_rate: 0.001000 loss_xy: 3.547661 loss_wh: 0.668630 loss_obj: 2.432878 loss_cls: 0.542647 loss: 7.553561 eta: 12:05:21 batch_cost: 0.6459 data_cost: 0.0075 ips: 13.9351 images/s
- [11/24 20:13:33] ppdet.engine INFO: Epoch: [13] [ 100/1193] learning_rate: 0.001000 loss_xy: 3.130623 loss_wh: 0.544082 loss_obj: 2.112761 loss_cls: 0.517044 loss: 6.310693 eta: 12:04:52 batch_cost: 0.7901 data_cost: 0.1502 ips: 11.3906 images/s
- [11/24 20:14:11] ppdet.engine INFO: Epoch: [13] [ 150/1193] learning_rate: 0.001000 loss_xy: 3.267561 loss_wh: 0.599319 loss_obj: 2.245845 loss_cls: 0.406527 loss: 6.534087 eta: 12:03:51 batch_cost: 0.7477 data_cost: 0.1381 ips: 12.0365 images/s
- [11/24 20:14:48] ppdet.engine INFO: Epoch: [13] [ 200/1193] learning_rate: 0.001000 loss_xy: 3.081948 loss_wh: 0.537203 loss_obj: 2.049520 loss_cls: 0.438231 loss: 6.546575 eta: 12:02:46 batch_cost: 0.7411 data_cost: 0.0102 ips: 12.1448 images/s
- [11/24 20:15:26] ppdet.engine INFO: Epoch: [13] [ 250/1193] learning_rate: 0.001000 loss_xy: 2.771421 loss_wh: 0.441066 loss_obj: 1.736216 loss_cls: 0.402814 loss: 5.400088 eta: 12:02:00 batch_cost: 0.7656 data_cost: 0.0508 ips: 11.7549 images/s
- [11/24 20:15:58] ppdet.engine INFO: Epoch: [13] [ 300/1193] learning_rate: 0.001000 loss_xy: 2.756887 loss_wh: 0.518809 loss_obj: 2.152125 loss_cls: 0.360970 loss: 6.024402 eta: 11:59:49 batch_cost: 0.6480 data_cost: 0.0183 ips: 13.8883 images/s
- [11/24 20:16:41] ppdet.engine INFO: Epoch: [13] [ 350/1193] learning_rate: 0.001000 loss_xy: 3.179984 loss_wh: 0.582688 loss_obj: 2.112536 loss_cls: 0.373369 loss: 6.508628 eta: 12:00:01 batch_cost: 0.8469 data_cost: 0.0812 ips: 10.6266 images/s
- [11/24 20:17:31] ppdet.engine INFO: Epoch: [13] [ 400/1193] learning_rate: 0.001000 loss_xy: 3.179877 loss_wh: 0.548593 loss_obj: 2.047122 loss_cls: 0.474818 loss: 6.566978 eta: 12:02:00 batch_cost: 1.0001 data_cost: 0.6287 ips: 8.9987 images/s
- [11/24 20:18:11] ppdet.engine INFO: Epoch: [13] [ 450/1193] learning_rate: 0.001000 loss_xy: 2.984452 loss_wh: 0.510714 loss_obj: 2.077857 loss_cls: 0.510255 loss: 6.460220 eta: 12:01:34 batch_cost: 0.7965 data_cost: 0.2306 ips: 11.2992 images/s
- [11/24 20:18:51] ppdet.engine INFO: Epoch: [13] [ 500/1193] learning_rate: 0.001000 loss_xy: 3.146020 loss_wh: 0.556107 loss_obj: 2.242355 loss_cls: 0.411215 loss: 6.432540 eta: 12:01:17 batch_cost: 0.8115 data_cost: 0.1983 ips: 11.0900 images/s
- [11/24 20:19:21] ppdet.engine INFO: Epoch: [13] [ 550/1193] learning_rate: 0.001000 loss_xy: 3.246315 loss_wh: 0.508002 loss_obj: 2.032323 loss_cls: 0.347650 loss: 6.192170 eta: 11:58:29 batch_cost: 0.5867 data_cost: 0.0769 ips: 15.3396 images/s
- [11/24 20:19:59] ppdet.engine INFO: Epoch: [13] [ 600/1193] learning_rate: 0.001000 loss_xy: 2.923283 loss_wh: 0.474050 loss_obj: 1.954283 loss_cls: 0.382424 loss: 5.622492 eta: 11:57:36 batch_cost: 0.7553 data_cost: 0.0056 ips: 11.9153 images/s
- [11/24 20:20:37] ppdet.engine INFO: Epoch: [13] [ 650/1193] learning_rate: 0.001000 loss_xy: 3.066255 loss_wh: 0.527494 loss_obj: 2.097351 loss_cls: 0.485891 loss: 6.254639 eta: 11:56:50 batch_cost: 0.7648 data_cost: 0.1410 ips: 11.7675 images/s
- [11/24 20:21:13] ppdet.engine INFO: Epoch: [13] [ 700/1193] learning_rate: 0.001000 loss_xy: 3.004744 loss_wh: 0.519862 loss_obj: 2.076207 loss_cls: 0.396058 loss: 5.957050 eta: 11:55:36 batch_cost: 0.7218 data_cost: 0.0531 ips: 12.4681 images/s
- [11/24 20:21:49] ppdet.engine INFO: Epoch: [13] [ 750/1193] learning_rate: 0.001000 loss_xy: 3.047910 loss_wh: 0.561538 loss_obj: 2.468589 loss_cls: 0.387749 loss: 6.565693 eta: 11:54:18 batch_cost: 0.7133 data_cost: 0.3197 ips: 12.6182 images/s
- [11/24 20:22:23] ppdet.engine INFO: Epoch: [13] [ 800/1193] learning_rate: 0.001000 loss_xy: 3.017611 loss_wh: 0.534474 loss_obj: 2.479455 loss_cls: 0.572211 loss: 6.656081 eta: 11:52:40 batch_cost: 0.6808 data_cost: 0.1060 ips: 13.2189 images/s
- [11/24 20:22:59] ppdet.engine INFO: Epoch: [13] [ 850/1193] learning_rate: 0.001000 loss_xy: 3.348267 loss_wh: 0.590253 loss_obj: 2.159506 loss_cls: 0.452588 loss: 6.417754 eta: 11:51:34 batch_cost: 0.7297 data_cost: 0.0523 ips: 12.3342 images/s
- [11/24 20:23:39] ppdet.engine INFO: Epoch: [13] [ 900/1193] learning_rate: 0.001000 loss_xy: 3.095862 loss_wh: 0.535831 loss_obj: 2.228060 loss_cls: 0.384546 loss: 6.373940 eta: 11:51:05 batch_cost: 0.7900 data_cost: 0.0589 ips: 11.3928 images/s
- [11/24 20:24:20] ppdet.engine INFO: Epoch: [13] [ 950/1193] learning_rate: 0.001000 loss_xy: 2.846602 loss_wh: 0.506220 loss_obj: 1.871405 loss_cls: 0.485976 loss: 5.739088 eta: 11:51:02 batch_cost: 0.8315 data_cost: 0.1214 ips: 10.8237 images/s
- [11/24 20:25:03] ppdet.engine INFO: Epoch: [13] [1000/1193] learning_rate: 0.001000 loss_xy: 3.129750 loss_wh: 0.598006 loss_obj: 2.396637 loss_cls: 0.442246 loss: 6.648032 eta: 11:51:14 batch_cost: 0.8586 data_cost: 0.1629 ips: 10.4817 images/s
- [11/24 20:25:50] ppdet.engine INFO: Epoch: [13] [1050/1193] learning_rate: 0.001000 loss_xy: 3.354498 loss_wh: 0.560768 loss_obj: 2.282377 loss_cls: 0.494530 loss: 6.698553 eta: 11:52:10 batch_cost: 0.9345 data_cost: 0.1125 ips: 9.6308 images/s
- [11/24 20:26:30] ppdet.engine INFO: Epoch: [13] [1100/1193] learning_rate: 0.001000 loss_xy: 3.137451 loss_wh: 0.577360 loss_obj: 2.122973 loss_cls: 0.461604 loss: 6.481135 eta: 11:51:38 batch_cost: 0.7870 data_cost: 0.2043 ips: 11.4358 images/s
- [11/24 20:27:03] ppdet.engine INFO: Epoch: [13] [1150/1193] learning_rate: 0.001000 loss_xy: 3.220345 loss_wh: 0.587352 loss_obj: 2.069304 loss_cls: 0.377855 loss: 6.397051 eta: 11:49:51 batch_cost: 0.6606 data_cost: 0.0623 ips: 13.6241 images/s
- [11/24 20:27:35] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 20:27:35] ppdet.engine INFO: Eval iter: 0
- [11/24 20:27:40] ppdet.engine INFO: Eval iter: 100
- [11/24 20:27:44] ppdet.engine INFO: Eval iter: 200
- [11/24 20:27:48] ppdet.engine INFO: Eval iter: 300
- [11/24 20:27:52] ppdet.engine INFO: Eval iter: 400
- [11/24 20:27:56] ppdet.engine INFO: Eval iter: 500
- [11/24 20:28:00] ppdet.engine INFO: Eval iter: 600
- [11/24 20:28:04] ppdet.engine INFO: Eval iter: 700
- [11/24 20:28:08] ppdet.engine INFO: Eval iter: 800
- [11/24 20:28:12] ppdet.engine INFO: Eval iter: 900
- [11/24 20:28:16] ppdet.engine INFO: Eval iter: 1000
- [11/24 20:28:20] ppdet.engine INFO: Eval iter: 1100
- [11/24 20:28:24] ppdet.engine INFO: Eval iter: 1200
- [11/24 20:28:28] ppdet.engine INFO: Eval iter: 1300
- [11/24 20:28:32] ppdet.engine INFO: Eval iter: 1400
- [11/24 20:28:36] ppdet.engine INFO: Eval iter: 1500
- [11/24 20:28:40] ppdet.engine INFO: Eval iter: 1600
- [11/24 20:28:44] ppdet.engine INFO: Eval iter: 1700
- [11/24 20:28:48] ppdet.engine INFO: Eval iter: 1800
- [11/24 20:28:53] ppdet.engine INFO: Eval iter: 1900
- [11/24 20:28:57] ppdet.engine INFO: Eval iter: 2000
- [11/24 20:29:01] ppdet.engine INFO: Eval iter: 2100
- [11/24 20:29:05] ppdet.engine INFO: Eval iter: 2200
- [11/24 20:29:09] ppdet.engine INFO: Eval iter: 2300
- [11/24 20:29:13] ppdet.engine INFO: Eval iter: 2400
- [11/24 20:29:17] ppdet.engine INFO: Eval iter: 2500
- [11/24 20:29:21] ppdet.engine INFO: Eval iter: 2600
- [11/24 20:29:25] ppdet.engine INFO: Eval iter: 2700
- [11/24 20:29:29] ppdet.engine INFO: Eval iter: 2800
- [11/24 20:29:33] ppdet.engine INFO: Eval iter: 2900
- [11/24 20:29:37] ppdet.engine INFO: Eval iter: 3000
- [11/24 20:29:41] ppdet.engine INFO: Eval iter: 3100
- [11/24 20:29:45] ppdet.engine INFO: Eval iter: 3200
- [11/24 20:29:49] ppdet.engine INFO: Eval iter: 3300
- [11/24 20:29:53] ppdet.engine INFO: Eval iter: 3400
- [11/24 20:29:57] ppdet.engine INFO: Eval iter: 3500
- [11/24 20:30:01] ppdet.engine INFO: Eval iter: 3600
- [11/24 20:30:05] ppdet.engine INFO: Eval iter: 3700
- [11/24 20:30:09] ppdet.engine INFO: Eval iter: 3800
- [11/24 20:30:13] ppdet.engine INFO: Eval iter: 3900
- [11/24 20:30:17] ppdet.engine INFO: Eval iter: 4000
- [11/24 20:30:21] ppdet.engine INFO: Eval iter: 4100
- [11/24 20:30:25] ppdet.engine INFO: Eval iter: 4200
- [11/24 20:30:29] ppdet.engine INFO: Eval iter: 4300
- [11/24 20:30:33] ppdet.engine INFO: Eval iter: 4400
- [11/24 20:30:37] ppdet.engine INFO: Eval iter: 4500
- [11/24 20:30:41] ppdet.engine INFO: Eval iter: 4600
- [11/24 20:30:45] ppdet.engine INFO: Eval iter: 4700
- [11/24 20:30:48] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 20:30:48] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 90.21%
- [11/24 20:30:48] ppdet.engine INFO: Total sample number: 4773, averge FPS: 24.781908315007914
- [11/24 20:30:48] ppdet.engine INFO: Best test bbox ap is 0.909.
- [11/24 20:30:48] ppdet.engine INFO: Epoch: [14] [ 0/1193] learning_rate: 0.001000 loss_xy: 2.797576 loss_wh: 0.438590 loss_obj: 2.019226 loss_cls: 0.372903 loss: 5.428711 eta: 11:49:03 batch_cost: 0.7372 data_cost: 0.1162 ips: 12.2078 images/s
- [11/24 20:31:25] ppdet.engine INFO: Epoch: [14] [ 50/1193] learning_rate: 0.001000 loss_xy: 2.860710 loss_wh: 0.529813 loss_obj: 3.013076 loss_cls: 0.571862 loss: 6.762403 eta: 11:48:00 batch_cost: 0.7339 data_cost: 0.0785 ips: 12.2631 images/s
- [11/24 20:32:00] ppdet.engine INFO: Epoch: [14] [ 100/1193] learning_rate: 0.001000 loss_xy: 3.047539 loss_wh: 0.538383 loss_obj: 2.879256 loss_cls: 0.596779 loss: 7.006386 eta: 11:46:43 batch_cost: 0.7053 data_cost: 0.1282 ips: 12.7608 images/s
- [11/24 20:32:44] ppdet.engine INFO: Epoch: [14] [ 150/1193] learning_rate: 0.001000 loss_xy: 3.177170 loss_wh: 0.594613 loss_obj: 2.994338 loss_cls: 0.541327 loss: 7.411916 eta: 11:46:58 batch_cost: 0.8718 data_cost: 0.0540 ips: 10.3235 images/s
- [11/24 20:33:23] ppdet.engine INFO: Epoch: [14] [ 200/1193] learning_rate: 0.001000 loss_xy: 2.968200 loss_wh: 0.516383 loss_obj: 2.578860 loss_cls: 0.620225 loss: 6.736318 eta: 11:46:16 batch_cost: 0.7688 data_cost: 0.3356 ips: 11.7063 images/s
- [11/24 20:34:04] ppdet.engine INFO: Epoch: [14] [ 250/1193] learning_rate: 0.001000 loss_xy: 3.156862 loss_wh: 0.565276 loss_obj: 3.026085 loss_cls: 0.634279 loss: 7.386445 eta: 11:46:06 batch_cost: 0.8276 data_cost: 0.0525 ips: 10.8749 images/s
- [11/24 20:34:36] ppdet.engine INFO: Epoch: [14] [ 300/1193] learning_rate: 0.001000 loss_xy: 2.857153 loss_wh: 0.513004 loss_obj: 3.297118 loss_cls: 0.630528 loss: 7.275171 eta: 11:44:11 batch_cost: 0.6334 data_cost: 0.0588 ips: 14.2091 images/s
- [11/24 20:35:19] ppdet.engine INFO: Epoch: [14] [ 350/1193] learning_rate: 0.001000 loss_xy: 3.297027 loss_wh: 0.554924 loss_obj: 3.116811 loss_cls: 0.751980 loss: 7.694474 eta: 11:44:23 batch_cost: 0.8699 data_cost: 0.1016 ips: 10.3463 images/s
- [11/24 20:36:00] ppdet.engine INFO: Epoch: [14] [ 400/1193] learning_rate: 0.001000 loss_xy: 3.004448 loss_wh: 0.511233 loss_obj: 3.174666 loss_cls: 0.733730 loss: 7.549541 eta: 11:44:06 batch_cost: 0.8158 data_cost: 0.3050 ips: 11.0319 images/s
- [11/24 20:36:42] ppdet.engine INFO: Epoch: [14] [ 450/1193] learning_rate: 0.001000 loss_xy: 2.978511 loss_wh: 0.518488 loss_obj: 2.894968 loss_cls: 0.556553 loss: 7.093966 eta: 11:43:55 batch_cost: 0.8300 data_cost: 0.1176 ips: 10.8436 images/s
- [11/24 20:37:17] ppdet.engine INFO: Epoch: [14] [ 500/1193] learning_rate: 0.001000 loss_xy: 3.421138 loss_wh: 0.616855 loss_obj: 3.688421 loss_cls: 0.632838 loss: 8.380663 eta: 11:42:39 batch_cost: 0.7035 data_cost: 0.1906 ips: 12.7937 images/s
- [11/24 20:37:58] ppdet.engine INFO: Epoch: [14] [ 550/1193] learning_rate: 0.001000 loss_xy: 3.033300 loss_wh: 0.495328 loss_obj: 2.810535 loss_cls: 0.615514 loss: 7.149057 eta: 11:42:22 batch_cost: 0.8186 data_cost: 0.0471 ips: 10.9942 images/s
- [11/24 20:38:32] ppdet.engine INFO: Epoch: [14] [ 600/1193] learning_rate: 0.001000 loss_xy: 3.204757 loss_wh: 0.559358 loss_obj: 2.651500 loss_cls: 0.826635 loss: 7.480838 eta: 11:41:00 batch_cost: 0.6888 data_cost: 0.0516 ips: 13.0659 images/s
- [11/24 20:39:15] ppdet.engine INFO: Epoch: [14] [ 650/1193] learning_rate: 0.001000 loss_xy: 3.088540 loss_wh: 0.566782 loss_obj: 3.070604 loss_cls: 0.696476 loss: 7.125803 eta: 11:40:58 batch_cost: 0.8482 data_cost: 0.1606 ips: 10.6104 images/s
- [11/24 20:39:47] ppdet.engine INFO: Epoch: [14] [ 700/1193] learning_rate: 0.001000 loss_xy: 3.134438 loss_wh: 0.550938 loss_obj: 3.090761 loss_cls: 0.580179 loss: 7.600378 eta: 11:39:18 batch_cost: 0.6537 data_cost: 0.1388 ips: 13.7680 images/s
- [11/24 20:40:26] ppdet.engine INFO: Epoch: [14] [ 750/1193] learning_rate: 0.001000 loss_xy: 3.325029 loss_wh: 0.542869 loss_obj: 3.336601 loss_cls: 0.640189 loss: 7.802355 eta: 11:38:35 batch_cost: 0.7651 data_cost: 0.0074 ips: 11.7625 images/s
- [11/24 20:41:04] ppdet.engine INFO: Epoch: [14] [ 800/1193] learning_rate: 0.001000 loss_xy: 3.187299 loss_wh: 0.523898 loss_obj: 3.016399 loss_cls: 0.700549 loss: 7.530972 eta: 11:37:46 batch_cost: 0.7537 data_cost: 0.0787 ips: 11.9404 images/s
- [11/24 20:41:43] ppdet.engine INFO: Epoch: [14] [ 850/1193] learning_rate: 0.001000 loss_xy: 2.993257 loss_wh: 0.543640 loss_obj: 2.967481 loss_cls: 0.685827 loss: 7.036926 eta: 11:37:19 batch_cost: 0.7980 data_cost: 0.1414 ips: 11.2777 images/s
- [11/24 20:42:26] ppdet.engine INFO: Epoch: [14] [ 900/1193] learning_rate: 0.001000 loss_xy: 2.771360 loss_wh: 0.469081 loss_obj: 2.642466 loss_cls: 0.609358 loss: 6.337762 eta: 11:37:19 batch_cost: 0.8547 data_cost: 0.2011 ips: 10.5300 images/s
- [11/24 20:43:08] ppdet.engine INFO: Epoch: [14] [ 950/1193] learning_rate: 0.001000 loss_xy: 2.945602 loss_wh: 0.461175 loss_obj: 3.194650 loss_cls: 0.693077 loss: 7.577425 eta: 11:37:04 batch_cost: 0.8266 data_cost: 0.1640 ips: 10.8879 images/s
- [11/24 20:43:36] ppdet.engine INFO: Epoch: [14] [1000/1193] learning_rate: 0.001000 loss_xy: 3.235318 loss_wh: 0.576658 loss_obj: 3.002337 loss_cls: 0.643576 loss: 7.455699 eta: 11:34:51 batch_cost: 0.5744 data_cost: 0.0197 ips: 15.6675 images/s
- [11/24 20:44:09] ppdet.engine INFO: Epoch: [14] [1050/1193] learning_rate: 0.001000 loss_xy: 2.746946 loss_wh: 0.458146 loss_obj: 2.538843 loss_cls: 0.615329 loss: 6.624098 eta: 11:33:16 batch_cost: 0.6500 data_cost: 0.0196 ips: 13.8459 images/s
- [11/24 20:44:52] ppdet.engine INFO: Epoch: [14] [1100/1193] learning_rate: 0.001000 loss_xy: 3.139718 loss_wh: 0.510973 loss_obj: 3.099700 loss_cls: 0.574478 loss: 7.563831 eta: 11:33:14 batch_cost: 0.8529 data_cost: 0.0729 ips: 10.5522 images/s
- [11/24 20:45:32] ppdet.engine INFO: Epoch: [14] [1150/1193] learning_rate: 0.001000 loss_xy: 3.069796 loss_wh: 0.515391 loss_obj: 2.851407 loss_cls: 0.583296 loss: 7.404709 eta: 11:32:54 batch_cost: 0.8159 data_cost: 0.0502 ips: 11.0309 images/s
- [11/24 20:46:04] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 20:46:05] ppdet.engine INFO: Eval iter: 0
- [11/24 20:46:09] ppdet.engine INFO: Eval iter: 100
- [11/24 20:46:13] ppdet.engine INFO: Eval iter: 200
- [11/24 20:46:17] ppdet.engine INFO: Eval iter: 300
- [11/24 20:46:21] ppdet.engine INFO: Eval iter: 400
- [11/24 20:46:25] ppdet.engine INFO: Eval iter: 500
- [11/24 20:46:29] ppdet.engine INFO: Eval iter: 600
- [11/24 20:46:33] ppdet.engine INFO: Eval iter: 700
- [11/24 20:46:37] ppdet.engine INFO: Eval iter: 800
- [11/24 20:46:41] ppdet.engine INFO: Eval iter: 900
- [11/24 20:46:45] ppdet.engine INFO: Eval iter: 1000
- [11/24 20:46:49] ppdet.engine INFO: Eval iter: 1100
- [11/24 20:46:53] ppdet.engine INFO: Eval iter: 1200
- [11/24 20:46:57] ppdet.engine INFO: Eval iter: 1300
- [11/24 20:47:01] ppdet.engine INFO: Eval iter: 1400
- [11/24 20:47:05] ppdet.engine INFO: Eval iter: 1500
- [11/24 20:47:09] ppdet.engine INFO: Eval iter: 1600
- [11/24 20:47:13] ppdet.engine INFO: Eval iter: 1700
- [11/24 20:47:17] ppdet.engine INFO: Eval iter: 1800
- [11/24 20:47:22] ppdet.engine INFO: Eval iter: 1900
- [11/24 20:47:26] ppdet.engine INFO: Eval iter: 2000
- [11/24 20:47:30] ppdet.engine INFO: Eval iter: 2100
- [11/24 20:47:34] ppdet.engine INFO: Eval iter: 2200
- [11/24 20:47:38] ppdet.engine INFO: Eval iter: 2300
- [11/24 20:47:42] ppdet.engine INFO: Eval iter: 2400
- [11/24 20:47:46] ppdet.engine INFO: Eval iter: 2500
- [11/24 20:47:50] ppdet.engine INFO: Eval iter: 2600
- [11/24 20:47:54] ppdet.engine INFO: Eval iter: 2700
- [11/24 20:47:58] ppdet.engine INFO: Eval iter: 2800
- [11/24 20:48:02] ppdet.engine INFO: Eval iter: 2900
- [11/24 20:48:06] ppdet.engine INFO: Eval iter: 3000
- [11/24 20:48:10] ppdet.engine INFO: Eval iter: 3100
- [11/24 20:48:14] ppdet.engine INFO: Eval iter: 3200
- [11/24 20:48:18] ppdet.engine INFO: Eval iter: 3300
- [11/24 20:48:22] ppdet.engine INFO: Eval iter: 3400
- [11/24 20:48:27] ppdet.engine INFO: Eval iter: 3500
- [11/24 20:48:32] ppdet.engine INFO: Eval iter: 3600
- [11/24 20:48:36] ppdet.engine INFO: Eval iter: 3700
- [11/24 20:48:41] ppdet.engine INFO: Eval iter: 3800
- [11/24 20:48:45] ppdet.engine INFO: Eval iter: 3900
- [11/24 20:48:49] ppdet.engine INFO: Eval iter: 4000
- [11/24 20:48:54] ppdet.engine INFO: Eval iter: 4100
- [11/24 20:48:58] ppdet.engine INFO: Eval iter: 4200
- [11/24 20:49:03] ppdet.engine INFO: Eval iter: 4300
- [11/24 20:49:07] ppdet.engine INFO: Eval iter: 4400
- [11/24 20:49:11] ppdet.engine INFO: Eval iter: 4500
- [11/24 20:49:15] ppdet.engine INFO: Eval iter: 4600
- [11/24 20:49:19] ppdet.engine INFO: Eval iter: 4700
- [11/24 20:49:22] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/24 20:49:22] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 91.11%
- [11/24 20:49:22] ppdet.engine INFO: Total sample number: 4773, averge FPS: 24.183620583836753
- [11/24 20:49:22] ppdet.engine INFO: Best test bbox ap is 0.911.
- [11/24 20:49:26] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/24 20:49:26] ppdet.engine INFO: Epoch: [15] [ 0/1193] learning_rate: 0.001000 loss_xy: 3.320924 loss_wh: 0.593919 loss_obj: 3.276583 loss_cls: 0.575895 loss: 7.806449 eta: 11:32:02 batch_cost: 0.6999 data_cost: 0.0920 ips: 12.8596 images/s
- [11/24 20:50:02] ppdet.engine INFO: Epoch: [15] [ 50/1193] learning_rate: 0.001000 loss_xy: 2.887805 loss_wh: 0.518349 loss_obj: 2.577952 loss_cls: 0.674958 loss: 7.082314 eta: 11:31:03 batch_cost: 0.7267 data_cost: 0.0575 ips: 12.3839 images/s
- [11/24 20:50:38] ppdet.engine INFO: Epoch: [15] [ 100/1193] learning_rate: 0.001000 loss_xy: 3.149673 loss_wh: 0.552850 loss_obj: 3.042676 loss_cls: 0.610760 loss: 7.475282 eta: 11:30:01 batch_cost: 0.7212 data_cost: 0.1371 ips: 12.4789 images/s
- [11/24 20:51:12] ppdet.engine INFO: Epoch: [15] [ 150/1193] learning_rate: 0.001000 loss_xy: 3.291084 loss_wh: 0.606543 loss_obj: 3.161914 loss_cls: 0.655982 loss: 7.552655 eta: 11:28:38 batch_cost: 0.6722 data_cost: 0.1039 ips: 13.3879 images/s
- [11/24 20:51:48] ppdet.engine INFO: Epoch: [15] [ 200/1193] learning_rate: 0.001000 loss_xy: 3.194491 loss_wh: 0.511116 loss_obj: 3.102588 loss_cls: 0.638999 loss: 7.304476 eta: 11:27:40 batch_cost: 0.7251 data_cost: 0.1167 ips: 12.4117 images/s
- [11/24 20:52:26] ppdet.engine INFO: Epoch: [15] [ 250/1193] learning_rate: 0.001000 loss_xy: 3.075841 loss_wh: 0.544373 loss_obj: 3.221847 loss_cls: 0.658527 loss: 8.062576 eta: 11:26:53 batch_cost: 0.7529 data_cost: 0.1773 ips: 11.9540 images/s
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:43035', '127.0.0.1:36509', '127.0.0.1:39663']
- I1125 10:33:18.385959 922 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1125 10:33:19.047327 922 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1125 10:33:19.051342 922 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/25 10:33:21] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/25 10:33:21] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/25 10:33:23] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/25 10:33:24] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/25 10:33:25] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/25 10:33:25] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/25 10:33:26] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/25 10:33:28] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/25 10:33:28] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/25 10:33:28] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- [11/25 10:33:29] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/25 10:33:29] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/25 10:33:29] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/25 10:33:29] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/25 10:33:30] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/25 10:33:31] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- Traceback (most recent call last):
- File "tools/train.py", line 172, in <module>
- main()
- File "tools/train.py", line 168, in main
- run(FLAGS, cfg)
- File "tools/train.py", line 127, in run
- trainer.resume_weights(FLAGS.resume)
- File "/home/aistudio/work/myDemo/ppdet/engine/trainer.py", line 392, in resume_weights
- self.ema if self.use_ema else None)
- File "/home/aistudio/work/myDemo/ppdet/utils/checkpoint.py", line 73, in load_weight
- "exists.".format(pdparam_path))
- ValueError: Model pretrain path output/11_AUG_10/yolov3_darknet53_270e_voc_garbage/.pdparams.pdparams does not exists.
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:37305', '127.0.0.1:60571', '127.0.0.1:38879']
- I1125 10:39:04.614998 2124 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1125 10:39:05.281859 2124 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1125 10:39:05.284545 2124 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/25 10:39:07] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/25 10:39:07] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/25 10:39:09] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/25 10:39:10] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/25 10:39:11] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/25 10:39:11] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/25 10:39:12] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/25 10:39:14] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/25 10:39:14] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/25 10:39:14] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- [11/25 10:39:15] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/25 10:39:15] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/25 10:39:15] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/25 10:39:16] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/25 10:39:16] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/25 10:39:17] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- [11/25 10:39:18] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage/14.pdparams
- [11/25 10:39:22] ppdet.engine INFO: Epoch: [15] [ 0/1197] learning_rate: 0.001000 loss_xy: 3.942423 loss_wh: 0.515443 loss_obj: 3.632747 loss_cls: 0.900556 loss: 8.991169 eta: 1 day, 12:24:55 batch_cost: 2.4338 data_cost: 0.0019 ips: 3.6979 images/s
- [11/25 10:40:05] ppdet.engine INFO: Epoch: [15] [ 50/1197] learning_rate: 0.001000 loss_xy: 2.969141 loss_wh: 0.496506 loss_obj: 2.571009 loss_cls: 0.560556 loss: 6.646410 eta: 13:31:11 batch_cost: 0.8738 data_cost: 0.1431 ips: 10.2994 images/s
- [11/25 10:40:29] ppdet.data.transform.operators WARNING: The actual image height: 3472 is not equal to the height: 4624.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:40:29] ppdet.data.transform.operators WARNING: The actual image width: 4624 is not equal to the width: 3472.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:40:45] ppdet.engine INFO: Epoch: [15] [ 100/1197] learning_rate: 0.001000 loss_xy: 2.923278 loss_wh: 0.503531 loss_obj: 2.904177 loss_cls: 0.644196 loss: 7.015575 eta: 12:43:14 batch_cost: 0.7980 data_cost: 0.0471 ips: 11.2776 images/s
- [11/25 10:41:25] ppdet.engine INFO: Epoch: [15] [ 150/1197] learning_rate: 0.001000 loss_xy: 3.092587 loss_wh: 0.536748 loss_obj: 2.770131 loss_cls: 0.505289 loss: 7.037395 eta: 12:26:31 batch_cost: 0.7977 data_cost: 0.1519 ips: 11.2818 images/s
- [11/25 10:41:26] ppdet.data.transform.operators WARNING: The actual image height: 3472 is not equal to the height: 4624.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:41:26] ppdet.data.transform.operators WARNING: The actual image width: 4624 is not equal to the width: 3472.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:42:07] ppdet.engine INFO: Epoch: [15] [ 200/1197] learning_rate: 0.001000 loss_xy: 3.194177 loss_wh: 0.551475 loss_obj: 2.858590 loss_cls: 0.641560 loss: 7.022590 eta: 12:25:02 batch_cost: 0.8303 data_cost: 0.1737 ips: 10.8394 images/s
- [11/25 10:42:32] ppdet.data.transform.operators WARNING: The actual image height: 3120 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:42:32] ppdet.data.transform.operators WARNING: The actual image width: 4208 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:42:35] ppdet.data.transform.operators WARNING: The actual image height: 4624 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:42:35] ppdet.data.transform.operators WARNING: The actual image width: 3472 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:42:56] ppdet.engine INFO: Epoch: [15] [ 250/1197] learning_rate: 0.001000 loss_xy: 3.003038 loss_wh: 0.551189 loss_obj: 3.058918 loss_cls: 0.724206 loss: 7.178566 eta: 12:52:05 batch_cost: 0.9889 data_cost: 0.3098 ips: 9.1009 images/s
- [11/25 10:43:05] ppdet.data.transform.operators WARNING: The actual image height: 3472 is not equal to the height: 3120.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:43:05] ppdet.data.transform.operators WARNING: The actual image width: 4624 is not equal to the width: 4208.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:43:14] ppdet.data.transform.operators WARNING: The actual image height: 4624 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:43:14] ppdet.data.transform.operators WARNING: The actual image width: 3472 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:43:35] ppdet.data.transform.operators WARNING: The actual image height: 4624 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:43:35] ppdet.data.transform.operators WARNING: The actual image width: 3472 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:43:42] ppdet.engine INFO: Epoch: [15] [ 300/1197] learning_rate: 0.001000 loss_xy: 2.923375 loss_wh: 0.509345 loss_obj: 2.879940 loss_cls: 0.612574 loss: 7.135861 eta: 12:59:38 batch_cost: 0.9198 data_cost: 0.0541 ips: 9.7851 images/s
- [11/25 10:44:05] ppdet.data.transform.operators WARNING: The actual image height: 4624 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:44:05] ppdet.data.transform.operators WARNING: The actual image width: 3472 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:44:22] ppdet.data.transform.operators WARNING: The actual image height: 3472 is not equal to the height: 3120.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:44:22] ppdet.data.transform.operators WARNING: The actual image width: 4624 is not equal to the width: 4208.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:44:32] ppdet.engine INFO: Epoch: [15] [ 350/1197] learning_rate: 0.001000 loss_xy: 2.756802 loss_wh: 0.464569 loss_obj: 2.647543 loss_cls: 0.681792 loss: 6.550971 eta: 13:13:55 batch_cost: 0.9914 data_cost: 0.3041 ips: 9.0776 images/s
- [11/25 10:44:32] ppdet.data.transform.operators WARNING: The actual image height: 3472 is not equal to the height: 4624.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:44:32] ppdet.data.transform.operators WARNING: The actual image width: 4624 is not equal to the width: 3472.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:44:43] ppdet.data.transform.operators WARNING: The actual image height: 3472 is not equal to the height: 4624.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:44:43] ppdet.data.transform.operators WARNING: The actual image width: 4624 is not equal to the width: 3472.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:45:14] ppdet.engine INFO: Epoch: [15] [ 400/1197] learning_rate: 0.001000 loss_xy: 3.223794 loss_wh: 0.574065 loss_obj: 2.990512 loss_cls: 0.567551 loss: 7.357512 eta: 13:08:18 batch_cost: 0.8462 data_cost: 0.2982 ips: 10.6352 images/s
- [11/25 10:45:26] ppdet.data.transform.operators WARNING: The actual image height: 4624 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:45:26] ppdet.data.transform.operators WARNING: The actual image width: 3472 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:45:28] ppdet.data.transform.operators WARNING: The actual image height: 3120 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:45:28] ppdet.data.transform.operators WARNING: The actual image width: 4208 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:45:32] ppdet.data.transform.operators WARNING: The actual image height: 3472 is not equal to the height: 3120.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:45:32] ppdet.data.transform.operators WARNING: The actual image width: 4624 is not equal to the width: 4208.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:45:57] ppdet.data.transform.operators WARNING: The actual image height: 4624 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:45:57] ppdet.data.transform.operators WARNING: The actual image width: 3472 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:45:59] ppdet.data.transform.operators WARNING: The actual image height: 3120 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:45:59] ppdet.data.transform.operators WARNING: The actual image width: 4208 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:46:01] ppdet.engine INFO: Epoch: [15] [ 450/1197] learning_rate: 0.001000 loss_xy: 3.109291 loss_wh: 0.558986 loss_obj: 3.140603 loss_cls: 0.613683 loss: 7.416597 eta: 13:13:05 batch_cost: 0.9407 data_cost: 0.2548 ips: 9.5678 images/s
- [11/25 10:46:12] ppdet.data.transform.operators WARNING: The actual image height: 3120 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:46:12] ppdet.data.transform.operators WARNING: The actual image width: 4208 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:46:16] ppdet.data.transform.operators WARNING: The actual image height: 3472 is not equal to the height: 3120.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:46:16] ppdet.data.transform.operators WARNING: The actual image width: 4624 is not equal to the width: 4208.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:46:46] ppdet.engine INFO: Epoch: [15] [ 500/1197] learning_rate: 0.001000 loss_xy: 2.977286 loss_wh: 0.523140 loss_obj: 2.809267 loss_cls: 0.696206 loss: 7.135792 eta: 13:12:14 batch_cost: 0.8895 data_cost: 0.2751 ips: 10.1178 images/s
- [11/25 10:46:53] ppdet.data.transform.operators WARNING: The actual image height: 3120 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:46:53] ppdet.data.transform.operators WARNING: The actual image width: 4208 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:47:26] ppdet.engine INFO: Epoch: [15] [ 550/1197] learning_rate: 0.001000 loss_xy: 2.809802 loss_wh: 0.472043 loss_obj: 2.678264 loss_cls: 0.597038 loss: 6.797999 eta: 13:04:04 batch_cost: 0.7987 data_cost: 0.1447 ips: 11.2686 images/s
- [11/25 10:47:52] ppdet.data.transform.operators WARNING: The actual image height: 4624 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- [11/25 10:47:52] ppdet.data.transform.operators WARNING: The actual image width: 3472 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- [11/25 10:48:06] ppdet.engine INFO: Epoch: [15] [ 600/1197] learning_rate: 0.001000 loss_xy: 3.064888 loss_wh: 0.573923 loss_obj: 3.207803 loss_cls: 0.685900 loss: 7.767478 eta: 12:58:17 batch_cost: 0.8141 data_cost: 0.0322 ips: 11.0549 images/s
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:60027', '127.0.0.1:45589', '127.0.0.1:58983']
- I1125 11:05:27.883289 1026 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1125 11:05:28.741616 1026 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1125 11:05:28.744522 1026 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/25 11:05:29] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/25 11:05:30] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/25 11:05:31] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/25 11:05:31] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/25 11:05:31] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/25 11:05:32] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/25 11:05:32] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/25 11:05:35] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/25 11:05:36] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/25 11:05:37] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/25 11:05:39] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/25 11:05:40] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/25 11:05:41] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/25 11:05:42] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/25 11:05:43] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- [11/25 11:05:45] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage/14.pdparams
- [11/25 11:05:48] ppdet.engine INFO: Epoch: [15] [ 0/1197] learning_rate: 0.001000 loss_xy: 4.567488 loss_wh: 0.864917 loss_obj: 4.711902 loss_cls: 0.979190 loss: 11.123496 eta: 1 day, 18:08:09 batch_cost: 2.8161 data_cost: 0.0004 ips: 3.1959 images/s
- [11/25 11:06:36] ppdet.engine INFO: Epoch: [15] [ 50/1197] learning_rate: 0.001000 loss_xy: 2.885648 loss_wh: 0.471991 loss_obj: 2.484510 loss_cls: 0.468500 loss: 6.425130 eta: 14:47:10 batch_cost: 0.9526 data_cost: 0.1223 ips: 9.4479 images/s
- [11/25 11:07:28] ppdet.engine INFO: Epoch: [15] [ 100/1197] learning_rate: 0.001000 loss_xy: 3.305843 loss_wh: 0.588943 loss_obj: 3.280375 loss_cls: 0.608836 loss: 7.957432 eta: 15:09:45 batch_cost: 1.0419 data_cost: 0.2333 ips: 8.6379 images/s
- [11/25 11:08:18] ppdet.engine INFO: Epoch: [15] [ 150/1197] learning_rate: 0.001000 loss_xy: 3.143818 loss_wh: 0.567166 loss_obj: 3.129631 loss_cls: 0.682574 loss: 7.567833 eta: 15:05:03 batch_cost: 1.0023 data_cost: 0.2999 ips: 8.9796 images/s
- [11/25 11:09:08] ppdet.engine INFO: Epoch: [15] [ 200/1197] learning_rate: 0.001000 loss_xy: 3.067309 loss_wh: 0.545649 loss_obj: 2.983388 loss_cls: 0.747192 loss: 7.161745 eta: 14:57:20 batch_cost: 0.9801 data_cost: 0.3346 ips: 9.1832 images/s
- [11/25 11:10:04] ppdet.engine INFO: Epoch: [15] [ 250/1197] learning_rate: 0.001000 loss_xy: 2.932456 loss_wh: 0.486424 loss_obj: 2.771799 loss_cls: 0.643238 loss: 6.867935 eta: 15:20:27 batch_cost: 1.1378 data_cost: 0.2806 ips: 7.9101 images/s
- [11/25 11:11:00] ppdet.engine INFO: Epoch: [15] [ 300/1197] learning_rate: 0.001000 loss_xy: 2.979955 loss_wh: 0.527365 loss_obj: 2.703617 loss_cls: 0.542279 loss: 7.075900 eta: 15:30:13 batch_cost: 1.1017 data_cost: 0.1879 ips: 8.1690 images/s
- [11/25 11:11:53] ppdet.engine INFO: Epoch: [15] [ 350/1197] learning_rate: 0.001000 loss_xy: 2.969383 loss_wh: 0.504076 loss_obj: 2.773821 loss_cls: 0.616530 loss: 7.310462 eta: 15:32:27 batch_cost: 1.0665 data_cost: 0.1542 ips: 8.4391 images/s
- [11/25 11:12:42] ppdet.engine INFO: Epoch: [15] [ 400/1197] learning_rate: 0.001000 loss_xy: 3.023427 loss_wh: 0.535460 loss_obj: 3.167420 loss_cls: 0.584365 loss: 7.685633 eta: 15:23:28 batch_cost: 0.9724 data_cost: 0.3298 ips: 9.2556 images/s
- [11/25 11:13:38] ppdet.engine INFO: Epoch: [15] [ 450/1197] learning_rate: 0.001000 loss_xy: 3.141979 loss_wh: 0.521567 loss_obj: 2.896013 loss_cls: 0.624175 loss: 6.925860 eta: 15:31:58 batch_cost: 1.1312 data_cost: 0.2676 ips: 7.9559 images/s
- [11/25 11:14:25] ppdet.engine INFO: Epoch: [15] [ 500/1197] learning_rate: 0.001000 loss_xy: 2.854873 loss_wh: 0.505792 loss_obj: 2.672698 loss_cls: 0.759202 loss: 6.873814 eta: 15:20:03 batch_cost: 0.9224 data_cost: 0.1671 ips: 9.7567 images/s
- [11/25 11:15:30] ppdet.engine INFO: Epoch: [15] [ 550/1197] learning_rate: 0.001000 loss_xy: 2.902801 loss_wh: 0.478847 loss_obj: 2.298399 loss_cls: 0.572850 loss: 6.594386 eta: 15:40:23 batch_cost: 1.2974 data_cost: 0.1772 ips: 6.9371 images/s
- [11/25 11:16:20] ppdet.engine INFO: Epoch: [15] [ 600/1197] learning_rate: 0.001000 loss_xy: 2.839031 loss_wh: 0.475137 loss_obj: 2.541215 loss_cls: 0.645648 loss: 6.593143 eta: 15:36:22 batch_cost: 1.0158 data_cost: 0.3449 ips: 8.8602 images/s
- [11/25 11:17:24] ppdet.engine INFO: Epoch: [15] [ 650/1197] learning_rate: 0.001000 loss_xy: 3.306326 loss_wh: 0.588606 loss_obj: 2.924891 loss_cls: 0.645406 loss: 7.514509 eta: 15:50:35 batch_cost: 1.2764 data_cost: 0.2375 ips: 7.0513 images/s
- [11/25 11:18:34] ppdet.engine INFO: Epoch: [15] [ 700/1197] learning_rate: 0.001000 loss_xy: 2.910894 loss_wh: 0.531334 loss_obj: 2.950880 loss_cls: 0.650457 loss: 7.405503 eta: 16:09:48 batch_cost: 1.3900 data_cost: 0.5148 ips: 6.4748 images/s
- [11/25 11:19:23] ppdet.engine INFO: Epoch: [15] [ 750/1197] learning_rate: 0.001000 loss_xy: 3.082434 loss_wh: 0.527367 loss_obj: 3.183603 loss_cls: 0.651022 loss: 7.398579 eta: 16:01:58 batch_cost: 0.9772 data_cost: 0.1060 ips: 9.2103 images/s
- [11/25 11:20:20] ppdet.engine INFO: Epoch: [15] [ 800/1197] learning_rate: 0.001000 loss_xy: 2.885957 loss_wh: 0.529283 loss_obj: 2.984233 loss_cls: 0.569435 loss: 6.956658 eta: 16:04:21 batch_cost: 1.1463 data_cost: 0.3276 ips: 7.8516 images/s
- [11/25 11:21:16] ppdet.engine INFO: Epoch: [15] [ 850/1197] learning_rate: 0.001000 loss_xy: 3.031759 loss_wh: 0.630781 loss_obj: 3.277806 loss_cls: 0.697622 loss: 7.778559 eta: 16:04:16 batch_cost: 1.1060 data_cost: 0.2256 ips: 8.1372 images/s
- [11/25 11:22:09] ppdet.engine INFO: Epoch: [15] [ 900/1197] learning_rate: 0.001000 loss_xy: 3.000478 loss_wh: 0.535396 loss_obj: 2.635451 loss_cls: 0.483114 loss: 7.020946 eta: 16:02:03 batch_cost: 1.0649 data_cost: 0.0980 ips: 8.4519 images/s
- [11/25 11:22:56] ppdet.engine INFO: Epoch: [15] [ 950/1197] learning_rate: 0.001000 loss_xy: 2.894898 loss_wh: 0.485757 loss_obj: 3.091192 loss_cls: 0.653582 loss: 7.615267 eta: 15:53:58 batch_cost: 0.9351 data_cost: 0.0579 ips: 9.6245 images/s
- [11/25 11:23:46] ppdet.engine INFO: Epoch: [15] [1000/1197] learning_rate: 0.001000 loss_xy: 3.041373 loss_wh: 0.564095 loss_obj: 3.123128 loss_cls: 0.671619 loss: 7.477119 eta: 15:49:11 batch_cost: 0.9936 data_cost: 0.1127 ips: 9.0583 images/s
- [11/25 11:24:36] ppdet.engine INFO: Epoch: [15] [1050/1197] learning_rate: 0.001000 loss_xy: 3.117968 loss_wh: 0.554573 loss_obj: 2.878940 loss_cls: 0.685345 loss: 7.149934 eta: 15:44:56 batch_cost: 0.9969 data_cost: 0.1219 ips: 9.0276 images/s
- [11/25 11:25:28] ppdet.engine INFO: Epoch: [15] [1100/1197] learning_rate: 0.001000 loss_xy: 2.944421 loss_wh: 0.492190 loss_obj: 2.533735 loss_cls: 0.517607 loss: 6.927310 eta: 15:43:07 batch_cost: 1.0507 data_cost: 0.0343 ips: 8.5661 images/s
- [11/25 11:26:21] ppdet.engine INFO: Epoch: [15] [1150/1197] learning_rate: 0.001000 loss_xy: 3.002683 loss_wh: 0.507599 loss_obj: 3.001374 loss_cls: 0.641725 loss: 7.269647 eta: 15:41:30 batch_cost: 1.0533 data_cost: 0.5696 ips: 8.5449 images/s
- [11/25 11:27:03] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/25 11:27:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- [11/25 11:27:05] ppdet.engine INFO: Eval iter: 0
- [11/25 11:27:10] ppdet.engine INFO: Eval iter: 100
- [11/25 11:27:15] ppdet.engine INFO: Eval iter: 200
- [11/25 11:27:20] ppdet.engine INFO: Eval iter: 300
- [11/25 11:27:25] ppdet.engine INFO: Eval iter: 400
- [11/25 11:27:30] ppdet.engine INFO: Eval iter: 500
- [11/25 11:27:35] ppdet.engine INFO: Eval iter: 600
- [11/25 11:27:40] ppdet.engine INFO: Eval iter: 700
- [11/25 11:27:45] ppdet.engine INFO: Eval iter: 800
- [11/25 11:27:50] ppdet.engine INFO: Eval iter: 900
- [11/25 11:27:54] ppdet.engine INFO: Eval iter: 1000
- [11/25 11:27:59] ppdet.engine INFO: Eval iter: 1100
- [11/25 11:28:03] ppdet.engine INFO: Eval iter: 1200
- [11/25 11:28:08] ppdet.engine INFO: Eval iter: 1300
- [11/25 11:28:12] ppdet.engine INFO: Eval iter: 1400
- [11/25 11:28:17] ppdet.engine INFO: Eval iter: 1500
- [11/25 11:28:22] ppdet.engine INFO: Eval iter: 1600
- [11/25 11:28:26] ppdet.engine INFO: Eval iter: 1700
- [11/25 11:28:31] ppdet.engine INFO: Eval iter: 1800
- [11/25 11:28:36] ppdet.engine INFO: Eval iter: 1900
- [11/25 11:28:40] ppdet.engine INFO: Eval iter: 2000
- [11/25 11:28:45] ppdet.engine INFO: Eval iter: 2100
- [11/25 11:28:50] ppdet.engine INFO: Eval iter: 2200
- [11/25 11:28:54] ppdet.engine INFO: Eval iter: 2300
- [11/25 11:28:59] ppdet.engine INFO: Eval iter: 2400
- [11/25 11:29:03] ppdet.engine INFO: Eval iter: 2500
- [11/25 11:29:08] ppdet.engine INFO: Eval iter: 2600
- [11/25 11:29:13] ppdet.engine INFO: Eval iter: 2700
- [11/25 11:29:17] ppdet.engine INFO: Eval iter: 2800
- [11/25 11:29:22] ppdet.engine INFO: Eval iter: 2900
- [11/25 11:29:27] ppdet.engine INFO: Eval iter: 3000
- [11/25 11:29:31] ppdet.engine INFO: Eval iter: 3100
- [11/25 11:29:36] ppdet.engine INFO: Eval iter: 3200
- [11/25 11:29:41] ppdet.engine INFO: Eval iter: 3300
- [11/25 11:29:45] ppdet.engine INFO: Eval iter: 3400
- [11/25 11:29:50] ppdet.engine INFO: Eval iter: 3500
- [11/25 11:29:55] ppdet.engine INFO: Eval iter: 3600
- [11/25 11:29:59] ppdet.engine INFO: Eval iter: 3700
- [11/25 11:30:04] ppdet.engine INFO: Eval iter: 3800
- [11/25 11:30:08] ppdet.engine INFO: Eval iter: 3900
- [11/25 11:30:13] ppdet.engine INFO: Eval iter: 4000
- [11/25 11:30:17] ppdet.engine INFO: Eval iter: 4100
- [11/25 11:30:22] ppdet.engine INFO: Eval iter: 4200
- [11/25 11:30:26] ppdet.engine INFO: Eval iter: 4300
- [11/25 11:30:31] ppdet.engine INFO: Eval iter: 4400
- [11/25 11:30:35] ppdet.engine INFO: Eval iter: 4500
- [11/25 11:30:40] ppdet.engine INFO: Eval iter: 4600
- [11/25 11:30:44] ppdet.engine INFO: Eval iter: 4700
- [11/25 11:30:49] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/25 11:30:49] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 90.87%
- [11/25 11:30:49] ppdet.engine INFO: Total sample number: 4792, averge FPS: 21.425251017908913
- [11/25 11:30:49] ppdet.engine INFO: Best test bbox ap is 0.909.
- [11/25 11:30:50] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/25 11:30:50] ppdet.engine INFO: Epoch: [16] [ 0/1197] learning_rate: 0.001000 loss_xy: 2.982295 loss_wh: 0.523584 loss_obj: 3.025615 loss_cls: 0.525852 loss: 7.237786 eta: 15:33:27 batch_cost: 0.8705 data_cost: 0.1947 ips: 10.3391 images/s
- [11/25 11:31:40] ppdet.engine INFO: Epoch: [16] [ 50/1197] learning_rate: 0.001000 loss_xy: 2.852129 loss_wh: 0.476351 loss_obj: 2.842344 loss_cls: 0.676126 loss: 6.835782 eta: 15:29:44 batch_cost: 0.9831 data_cost: 0.1924 ips: 9.1545 images/s
- [11/25 11:32:39] ppdet.engine INFO: Epoch: [16] [ 100/1197] learning_rate: 0.001000 loss_xy: 2.995388 loss_wh: 0.541531 loss_obj: 3.111031 loss_cls: 0.712932 loss: 7.546049 eta: 15:33:20 batch_cost: 1.1930 data_cost: 0.4868 ips: 7.5437 images/s
- [11/25 11:33:31] ppdet.engine INFO: Epoch: [16] [ 150/1197] learning_rate: 0.001000 loss_xy: 3.021135 loss_wh: 0.566816 loss_obj: 2.941602 loss_cls: 0.589393 loss: 7.633080 eta: 15:31:07 batch_cost: 1.0245 data_cost: 0.0071 ips: 8.7849 images/s
- [11/25 11:34:32] ppdet.engine INFO: Epoch: [16] [ 200/1197] learning_rate: 0.001000 loss_xy: 3.183381 loss_wh: 0.604979 loss_obj: 2.833427 loss_cls: 0.713185 loss: 7.475535 eta: 15:35:09 batch_cost: 1.2211 data_cost: 0.3824 ips: 7.3706 images/s
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:44619', '127.0.0.1:59741', '127.0.0.1:51959']
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- No stack trace in paddle, may be caused by external reasons.
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669347330 (unix time) try "date -d @1669347330" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e80000bd1d) received by PID 48494 (TID 0x7f3f207d7700) from PID 48413 ***]
-
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:49555', '127.0.0.1:50571', '127.0.0.1:33613']
- I1125 11:35:41.189498 48643 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1125 11:35:42.111145 48643 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1125 11:35:42.114482 48643 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/25 11:35:43] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/25 11:35:43] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/25 11:35:44] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/25 11:35:45] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/25 11:35:45] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/25 11:35:45] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/25 11:35:46] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/25 11:35:49] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/25 11:35:49] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/25 11:35:50] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/25 11:35:52] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/25 11:35:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/25 11:35:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/25 11:35:55] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/25 11:35:56] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- [11/25 11:35:58] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage/15.pdparams
- [11/25 11:36:01] ppdet.engine INFO: Epoch: [16] [ 0/2156] learning_rate: 0.001000 loss_xy: 6.032628 loss_wh: 0.920102 loss_obj: 5.933374 loss_cls: 1.065818 loss: 13.951922 eta: 2 days, 16:16:58 batch_cost: 2.4395 data_cost: 0.0003 ips: 2.0496 images/s
- [11/25 11:38:22] ppdet.engine INFO: Epoch: [16] [ 200/2156] learning_rate: 0.001000 loss_xy: 3.236153 loss_wh: 0.634406 loss_obj: 3.299679 loss_cls: 0.605783 loss: 7.925270 eta: 18:44:10 batch_cost: 0.7039 data_cost: 0.1481 ips: 7.1034 images/s
- [11/25 11:40:47] ppdet.engine INFO: Epoch: [16] [ 400/2156] learning_rate: 0.001000 loss_xy: 2.960336 loss_wh: 0.498103 loss_obj: 2.826392 loss_cls: 0.619986 loss: 6.971746 eta: 18:48:51 batch_cost: 0.7215 data_cost: 0.2108 ips: 6.9300 images/s
- [11/25 11:42:57] ppdet.engine INFO: Epoch: [16] [ 600/2156] learning_rate: 0.001000 loss_xy: 2.874752 loss_wh: 0.522488 loss_obj: 2.847386 loss_cls: 0.655342 loss: 7.322333 eta: 18:09:08 batch_cost: 0.6456 data_cost: 0.1185 ips: 7.7445 images/s
- [11/25 11:45:23] ppdet.engine INFO: Epoch: [16] [ 800/2156] learning_rate: 0.001000 loss_xy: 2.903294 loss_wh: 0.526906 loss_obj: 2.969540 loss_cls: 0.721945 loss: 7.155731 eta: 18:20:04 batch_cost: 0.7271 data_cost: 0.0710 ips: 6.8769 images/s
- [11/25 11:47:56] ppdet.engine INFO: Epoch: [16] [1000/2156] learning_rate: 0.001000 loss_xy: 2.766132 loss_wh: 0.483121 loss_obj: 2.771874 loss_cls: 0.605242 loss: 6.733770 eta: 18:36:36 batch_cost: 0.7621 data_cost: 0.1162 ips: 6.5607 images/s
- [11/25 11:50:22] ppdet.engine INFO: Epoch: [16] [1200/2156] learning_rate: 0.001000 loss_xy: 2.859393 loss_wh: 0.536218 loss_obj: 2.869074 loss_cls: 0.655440 loss: 7.302467 eta: 18:37:27 batch_cost: 0.7261 data_cost: 0.0708 ips: 6.8857 images/s
- [11/25 11:52:49] ppdet.engine INFO: Epoch: [16] [1400/2156] learning_rate: 0.001000 loss_xy: 2.881591 loss_wh: 0.537615 loss_obj: 2.791451 loss_cls: 0.567375 loss: 6.989482 eta: 18:39:37 batch_cost: 0.7363 data_cost: 0.0547 ips: 6.7903 images/s
- [11/25 11:55:28] ppdet.engine INFO: Epoch: [16] [1600/2156] learning_rate: 0.001000 loss_xy: 2.951868 loss_wh: 0.559177 loss_obj: 3.149070 loss_cls: 0.620591 loss: 7.719795 eta: 18:51:24 batch_cost: 0.7918 data_cost: 0.2121 ips: 6.3151 images/s
- [11/25 11:57:48] ppdet.engine INFO: Epoch: [16] [1800/2156] learning_rate: 0.001000 loss_xy: 2.899755 loss_wh: 0.485683 loss_obj: 3.128998 loss_cls: 0.667850 loss: 7.214990 eta: 18:43:26 batch_cost: 0.6957 data_cost: 0.1265 ips: 7.1870 images/s
- [11/25 12:00:01] ppdet.engine INFO: Epoch: [16] [2000/2156] learning_rate: 0.001000 loss_xy: 3.133654 loss_wh: 0.595798 loss_obj: 3.087416 loss_cls: 0.744989 loss: 7.775926 eta: 18:31:12 batch_cost: 0.6609 data_cost: 0.0351 ips: 7.5657 images/s
- [11/25 12:01:53] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/25 12:01:54] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- [11/25 12:01:55] ppdet.engine INFO: Eval iter: 0
- [11/25 12:02:00] ppdet.engine INFO: Eval iter: 100
- [11/25 12:02:04] ppdet.engine INFO: Eval iter: 200
- [11/25 12:02:09] ppdet.engine INFO: Eval iter: 300
- [11/25 12:02:13] ppdet.engine INFO: Eval iter: 400
- [11/25 12:02:18] ppdet.engine INFO: Eval iter: 500
- [11/25 12:02:22] ppdet.engine INFO: Eval iter: 600
- [11/25 12:02:27] ppdet.engine INFO: Eval iter: 700
- [11/25 12:02:32] ppdet.engine INFO: Eval iter: 800
- [11/25 12:02:36] ppdet.engine INFO: Eval iter: 900
- [11/25 12:02:41] ppdet.engine INFO: Eval iter: 1000
- [11/25 12:02:45] ppdet.engine INFO: Eval iter: 1100
- [11/25 12:02:50] ppdet.engine INFO: Eval iter: 1200
- [11/25 12:02:55] ppdet.engine INFO: Eval iter: 1300
- [11/25 12:02:59] ppdet.engine INFO: Eval iter: 1400
- [11/25 12:03:04] ppdet.engine INFO: Eval iter: 1500
- [11/25 12:03:08] ppdet.engine INFO: Eval iter: 1600
- [11/25 12:03:13] ppdet.engine INFO: Eval iter: 1700
- [11/25 12:03:17] ppdet.engine INFO: Eval iter: 1800
- [11/25 12:03:22] ppdet.engine INFO: Eval iter: 1900
- [11/25 12:03:27] ppdet.engine INFO: Eval iter: 2000
- [11/25 12:03:32] ppdet.engine INFO: Eval iter: 2100
- [11/25 12:03:37] ppdet.engine INFO: Eval iter: 2200
- [11/25 12:03:41] ppdet.engine INFO: Eval iter: 2300
- [11/25 12:03:46] ppdet.engine INFO: Eval iter: 2400
- [11/25 12:03:51] ppdet.engine INFO: Eval iter: 2500
- [11/25 12:03:56] ppdet.engine INFO: Eval iter: 2600
- [11/25 12:04:00] ppdet.engine INFO: Eval iter: 2700
- [11/25 12:04:05] ppdet.engine INFO: Eval iter: 2800
- [11/25 12:04:10] ppdet.engine INFO: Eval iter: 2900
- [11/25 12:04:15] ppdet.engine INFO: Eval iter: 3000
- [11/25 12:04:19] ppdet.engine INFO: Eval iter: 3100
- [11/25 12:04:24] ppdet.engine INFO: Eval iter: 3200
- [11/25 12:04:28] ppdet.engine INFO: Eval iter: 3300
- [11/25 12:04:33] ppdet.engine INFO: Eval iter: 3400
- [11/25 12:04:38] ppdet.engine INFO: Eval iter: 3500
- [11/25 12:04:42] ppdet.engine INFO: Eval iter: 3600
- [11/25 12:04:47] ppdet.engine INFO: Eval iter: 3700
- [11/25 12:04:52] ppdet.engine INFO: Eval iter: 3800
- [11/25 12:04:57] ppdet.engine INFO: Eval iter: 3900
- [11/25 12:05:01] ppdet.engine INFO: Eval iter: 4000
- [11/25 12:05:06] ppdet.engine INFO: Eval iter: 4100
- [11/25 12:05:11] ppdet.engine INFO: Eval iter: 4200
- [11/25 12:05:15] ppdet.engine INFO: Eval iter: 4300
- [11/25 12:05:20] ppdet.engine INFO: Eval iter: 4400
- [11/25 12:05:24] ppdet.engine INFO: Eval iter: 4500
- [11/25 12:05:29] ppdet.engine INFO: Eval iter: 4600
- [11/25 12:05:34] ppdet.engine INFO: Eval iter: 4700
- [11/25 12:05:38] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/25 12:05:38] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 89.88%
- [11/25 12:05:38] ppdet.engine INFO: Total sample number: 4792, averge FPS: 21.460978045186877
- [11/25 12:05:38] ppdet.engine INFO: Best test bbox ap is 0.899.
- [11/25 12:05:40] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/25 12:05:40] ppdet.engine INFO: Epoch: [17] [ 0/2156] learning_rate: 0.001000 loss_xy: 3.040099 loss_wh: 0.583827 loss_obj: 3.253468 loss_cls: 0.720641 loss: 8.059162 eta: 18:28:18 batch_cost: 0.6742 data_cost: 0.0705 ips: 7.4162 images/s
- [11/25 12:07:58] ppdet.engine INFO: Epoch: [17] [ 200/2156] learning_rate: 0.001000 loss_xy: 2.847793 loss_wh: 0.523357 loss_obj: 2.956788 loss_cls: 0.548273 loss: 6.818118 eta: 18:21:52 batch_cost: 0.6865 data_cost: 0.1710 ips: 7.2835 images/s
- [11/25 12:10:13] ppdet.engine INFO: Epoch: [17] [ 400/2156] learning_rate: 0.001000 loss_xy: 3.024113 loss_wh: 0.562863 loss_obj: 3.097179 loss_cls: 0.631975 loss: 7.381402 eta: 18:14:23 batch_cost: 0.6722 data_cost: 0.0999 ips: 7.4387 images/s
- [11/25 12:12:27] ppdet.engine INFO: Epoch: [17] [ 600/2156] learning_rate: 0.001000 loss_xy: 2.904126 loss_wh: 0.508051 loss_obj: 2.854115 loss_cls: 0.630604 loss: 7.072016 eta: 18:07:02 batch_cost: 0.6667 data_cost: 0.0912 ips: 7.4991 images/s
- [11/25 12:14:53] ppdet.engine INFO: Epoch: [17] [ 800/2156] learning_rate: 0.001000 loss_xy: 2.999079 loss_wh: 0.568251 loss_obj: 3.041273 loss_cls: 0.693016 loss: 7.392010 eta: 18:07:00 batch_cost: 0.7306 data_cost: 0.1906 ips: 6.8437 images/s
- [11/25 12:17:14] ppdet.engine INFO: Epoch: [17] [1000/2156] learning_rate: 0.001000 loss_xy: 3.049699 loss_wh: 0.554175 loss_obj: 3.213643 loss_cls: 0.704495 loss: 7.598825 eta: 18:03:38 batch_cost: 0.6992 data_cost: 0.0886 ips: 7.1513 images/s
- [11/25 12:19:26] ppdet.engine INFO: Epoch: [17] [1200/2156] learning_rate: 0.001000 loss_xy: 3.025983 loss_wh: 0.564478 loss_obj: 2.974780 loss_cls: 0.686711 loss: 7.575054 eta: 17:56:41 batch_cost: 0.6585 data_cost: 0.1737 ips: 7.5926 images/s
- [11/25 12:21:48] ppdet.engine INFO: Epoch: [17] [1400/2156] learning_rate: 0.001000 loss_xy: 2.952142 loss_wh: 0.553495 loss_obj: 2.799399 loss_cls: 0.579385 loss: 6.923399 eta: 17:54:43 batch_cost: 0.7105 data_cost: 0.0533 ips: 7.0374 images/s
- [11/25 12:24:11] ppdet.engine INFO: Epoch: [17] [1600/2156] learning_rate: 0.001000 loss_xy: 2.890130 loss_wh: 0.553557 loss_obj: 3.076999 loss_cls: 0.625393 loss: 7.347592 eta: 17:52:34 batch_cost: 0.7087 data_cost: 0.1049 ips: 7.0549 images/s
- [11/25 12:26:23] ppdet.engine INFO: Epoch: [17] [1800/2156] learning_rate: 0.001000 loss_xy: 2.993328 loss_wh: 0.550362 loss_obj: 3.121129 loss_cls: 0.617200 loss: 7.429427 eta: 17:46:44 batch_cost: 0.6609 data_cost: 0.0661 ips: 7.5660 images/s
- [11/25 12:29:16] ppdet.engine INFO: Epoch: [17] [2000/2156] learning_rate: 0.001000 loss_xy: 2.996317 loss_wh: 0.555182 loss_obj: 3.157678 loss_cls: 0.673319 loss: 7.673905 eta: 17:55:48 batch_cost: 0.8611 data_cost: 0.2771 ips: 5.8068 images/s
- [11/25 12:31:06] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/25 12:31:06] ppdet.engine INFO: Eval iter: 0
- [11/25 12:31:11] ppdet.engine INFO: Eval iter: 100
- [11/25 12:31:16] ppdet.engine INFO: Eval iter: 200
- [11/25 12:31:21] ppdet.engine INFO: Eval iter: 300
- [11/25 12:31:26] ppdet.engine INFO: Eval iter: 400
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::Tracer::TraceOp(std::string const&, paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap, std::map<std::string, std::string, std::less<std::string >, std::allocator<std::pair<std::string const, std::string > > > const&)
- 1 void paddle::imperative::Tracer::TraceOpImpl<paddle::imperative::VarBase>(std::string const&, paddle::imperative::details::NameVarMapTrait<paddle::imperative::VarBase>::Type const&, paddle::imperative::details::NameVarMapTrait<paddle::imperative::VarBase>::Type const&, paddle::framework::AttributeMap&, phi::Place const&, bool, std::map<std::string, std::string, std::less<std::string >, std::allocator<std::pair<std::string const, std::string > > > const&, paddle::framework::AttributeMap*, bool)
- 2 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap const&, paddle::framework::AttributeMap const&)
- 3 phi::KernelImpl<void (*)(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*), &(void phi::ConvCudnnKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*))>::Compute(phi::KernelContext*)
- 4 void phi::ConvCudnnKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669350686 (unix time) try "date -d @1669350686" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e80000bdb3) received by PID 48643 (TID 0x7f74b3817700) from PID 48563 ***]
-
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:51931', '127.0.0.1:60277', '127.0.0.1:58863']
- I1125 15:34:58.108011 969 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1125 15:34:58.832266 969 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1125 15:34:58.835554 969 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/25 15:34:59] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/25 15:35:00] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/25 15:35:01] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/25 15:35:01] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/25 15:35:01] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/25 15:35:01] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/25 15:35:02] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/25 15:35:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/25 15:35:04] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/25 15:35:05] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/25 15:35:07] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/25 15:35:08] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/25 15:35:09] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/25 15:35:09] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/25 15:35:11] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- [11/25 15:35:13] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage/21.pdparams
- [11/25 15:35:16] ppdet.engine INFO: Epoch: [22] [ 0/1347] learning_rate: 0.000250 loss_xy: 3.377586 loss_wh: 0.355952 loss_obj: 2.903685 loss_cls: 0.280470 loss: 6.917693 eta: 1 day, 12:33:15 batch_cost: 2.5709 data_cost: 0.0002 ips: 3.1117 images/s
- [11/25 15:36:32] ppdet.engine INFO: Epoch: [22] [ 200/1347] learning_rate: 0.000100 loss_xy: 3.134743 loss_wh: 0.389489 loss_obj: 1.264900 loss_cls: 0.055542 loss: 5.120058 eta: 5:32:16 batch_cost: 0.3801 data_cost: 0.0167 ips: 21.0465 images/s
- [11/25 15:37:52] ppdet.engine INFO: Epoch: [22] [ 400/1347] learning_rate: 0.000100 loss_xy: 2.921003 loss_wh: 0.303631 loss_obj: 1.069089 loss_cls: 0.042591 loss: 4.505855 eta: 5:33:53 batch_cost: 0.3980 data_cost: 0.0201 ips: 20.1023 images/s
- [11/25 15:39:12] ppdet.engine INFO: Epoch: [22] [ 600/1347] learning_rate: 0.000100 loss_xy: 2.887389 loss_wh: 0.330676 loss_obj: 0.933060 loss_cls: 0.028018 loss: 4.299532 eta: 5:33:17 batch_cost: 0.3970 data_cost: 0.0287 ips: 20.1515 images/s
- [11/25 15:40:40] ppdet.engine INFO: Epoch: [22] [ 800/1347] learning_rate: 0.000100 loss_xy: 2.972327 loss_wh: 0.343947 loss_obj: 1.120939 loss_cls: 0.033312 loss: 4.497190 eta: 5:40:29 batch_cost: 0.4360 data_cost: 0.0646 ips: 18.3497 images/s
- [11/25 15:42:03] ppdet.engine INFO: Epoch: [22] [1000/1347] learning_rate: 0.000100 loss_xy: 2.963937 loss_wh: 0.350864 loss_obj: 1.020893 loss_cls: 0.026304 loss: 4.511302 eta: 5:40:45 batch_cost: 0.4151 data_cost: 0.0575 ips: 19.2714 images/s
- [11/25 15:43:22] ppdet.engine INFO: Epoch: [22] [1200/1347] learning_rate: 0.000100 loss_xy: 2.862556 loss_wh: 0.307925 loss_obj: 1.054613 loss_cls: 0.022568 loss: 4.332312 eta: 5:37:25 batch_cost: 0.3931 data_cost: 0.0274 ips: 20.3497 images/s
- [11/25 15:44:19] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/25 15:44:19] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- [11/25 15:44:20] ppdet.engine INFO: Eval iter: 0
- [11/25 15:44:24] ppdet.engine INFO: Eval iter: 100
- [11/25 15:44:29] ppdet.engine INFO: Eval iter: 200
- [11/25 15:44:33] ppdet.engine INFO: Eval iter: 300
- [11/25 15:44:37] ppdet.engine INFO: Eval iter: 400
- [11/25 15:44:41] ppdet.engine INFO: Eval iter: 500
- [11/25 15:44:45] ppdet.engine INFO: Eval iter: 600
- [11/25 15:44:49] ppdet.engine INFO: Eval iter: 700
- [11/25 15:44:53] ppdet.engine INFO: Eval iter: 800
- [11/25 15:44:57] ppdet.engine INFO: Eval iter: 900
- [11/25 15:45:01] ppdet.engine INFO: Eval iter: 1000
- [11/25 15:45:05] ppdet.engine INFO: Eval iter: 1100
- [11/25 15:45:09] ppdet.engine INFO: Eval iter: 1200
- [11/25 15:45:13] ppdet.engine INFO: Eval iter: 1300
- [11/25 15:45:17] ppdet.engine INFO: Eval iter: 1400
- [11/25 15:45:21] ppdet.engine INFO: Eval iter: 1500
- [11/25 15:45:25] ppdet.engine INFO: Eval iter: 1600
- [11/25 15:45:29] ppdet.engine INFO: Eval iter: 1700
- [11/25 15:45:33] ppdet.engine INFO: Eval iter: 1800
- [11/25 15:45:37] ppdet.engine INFO: Eval iter: 1900
- [11/25 15:45:41] ppdet.engine INFO: Eval iter: 2000
- [11/25 15:45:45] ppdet.engine INFO: Eval iter: 2100
- [11/25 15:45:48] ppdet.engine INFO: Eval iter: 2200
- [11/25 15:45:52] ppdet.engine INFO: Eval iter: 2300
- [11/25 15:45:56] ppdet.engine INFO: Eval iter: 2400
- [11/25 15:46:00] ppdet.engine INFO: Eval iter: 2500
- [11/25 15:46:04] ppdet.engine INFO: Eval iter: 2600
- [11/25 15:46:08] ppdet.engine INFO: Eval iter: 2700
- [11/25 15:46:12] ppdet.engine INFO: Eval iter: 2800
- [11/25 15:46:16] ppdet.engine INFO: Eval iter: 2900
- [11/25 15:46:20] ppdet.engine INFO: Eval iter: 3000
- [11/25 15:46:24] ppdet.engine INFO: Eval iter: 3100
- [11/25 15:46:28] ppdet.engine INFO: Eval iter: 3200
- [11/25 15:46:32] ppdet.engine INFO: Eval iter: 3300
- [11/25 15:46:36] ppdet.engine INFO: Eval iter: 3400
- [11/25 15:46:40] ppdet.engine INFO: Eval iter: 3500
- [11/25 15:46:44] ppdet.engine INFO: Eval iter: 3600
- [11/25 15:46:48] ppdet.engine INFO: Eval iter: 3700
- [11/25 15:46:52] ppdet.engine INFO: Eval iter: 3800
- [11/25 15:46:56] ppdet.engine INFO: Eval iter: 3900
- [11/25 15:47:00] ppdet.engine INFO: Eval iter: 4000
- [11/25 15:47:04] ppdet.engine INFO: Eval iter: 4100
- [11/25 15:47:08] ppdet.engine INFO: Eval iter: 4200
- [11/25 15:47:12] ppdet.engine INFO: Eval iter: 4300
- [11/25 15:47:16] ppdet.engine INFO: Eval iter: 4400
- [11/25 15:47:20] ppdet.engine INFO: Eval iter: 4500
- [11/25 15:47:24] ppdet.engine INFO: Eval iter: 4600
- [11/25 15:47:27] ppdet.engine INFO: Eval iter: 4700
- [11/25 15:47:31] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/25 15:47:31] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 91.53%
- [11/25 15:47:31] ppdet.engine INFO: Total sample number: 4792, averge FPS: 25.100505706177906
- [11/25 15:47:31] ppdet.engine INFO: Best test bbox ap is 0.915.
- [11/25 15:47:35] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/25 15:47:35] ppdet.engine INFO: Epoch: [23] [ 0/1347] learning_rate: 0.000100 loss_xy: 2.841593 loss_wh: 0.325987 loss_obj: 1.003956 loss_cls: 0.025675 loss: 4.418137 eta: 5:33:24 batch_cost: 0.3740 data_cost: 0.0382 ips: 21.3920 images/s
- [11/25 15:48:55] ppdet.engine INFO: Epoch: [23] [ 200/1347] learning_rate: 0.000100 loss_xy: 2.861093 loss_wh: 0.300804 loss_obj: 0.887434 loss_cls: 0.020376 loss: 4.169024 eta: 5:31:32 batch_cost: 0.3964 data_cost: 0.0656 ips: 20.1834 images/s
- [11/25 15:50:13] ppdet.engine INFO: Epoch: [23] [ 400/1347] learning_rate: 0.000100 loss_xy: 2.908233 loss_wh: 0.317627 loss_obj: 1.143896 loss_cls: 0.025540 loss: 4.528624 eta: 5:28:49 batch_cost: 0.3862 data_cost: 0.0305 ips: 20.7153 images/s
- [11/25 15:51:38] ppdet.engine INFO: Epoch: [23] [ 600/1347] learning_rate: 0.000100 loss_xy: 3.046040 loss_wh: 0.339165 loss_obj: 0.965652 loss_cls: 0.020380 loss: 4.373465 eta: 5:29:38 batch_cost: 0.4245 data_cost: 0.0377 ips: 18.8458 images/s
- [11/25 15:53:03] ppdet.engine INFO: Epoch: [23] [ 800/1347] learning_rate: 0.000100 loss_xy: 2.824645 loss_wh: 0.296805 loss_obj: 0.772613 loss_cls: 0.019707 loss: 4.002120 eta: 5:29:54 batch_cost: 0.4229 data_cost: 0.0109 ips: 18.9180 images/s
- [11/25 15:54:24] ppdet.engine INFO: Epoch: [23] [1000/1347] learning_rate: 0.000100 loss_xy: 2.668844 loss_wh: 0.305723 loss_obj: 0.893975 loss_cls: 0.022107 loss: 4.219284 eta: 5:28:35 batch_cost: 0.4039 data_cost: 0.0129 ips: 19.8056 images/s
- [11/25 15:55:46] ppdet.engine INFO: Epoch: [23] [1200/1347] learning_rate: 0.000100 loss_xy: 2.856444 loss_wh: 0.301653 loss_obj: 0.860498 loss_cls: 0.017862 loss: 3.986332 eta: 5:27:24 batch_cost: 0.4064 data_cost: 0.0427 ips: 19.6872 images/s
- [11/25 15:56:37] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/25 15:56:37] ppdet.engine INFO: Eval iter: 0
- [11/25 15:56:42] ppdet.engine INFO: Eval iter: 100
- [11/25 15:56:46] ppdet.engine INFO: Eval iter: 200
- [11/25 15:56:50] ppdet.engine INFO: Eval iter: 300
- [11/25 15:56:54] ppdet.engine INFO: Eval iter: 400
- [11/25 15:56:58] ppdet.engine INFO: Eval iter: 500
- [11/25 15:57:02] ppdet.engine INFO: Eval iter: 600
- [11/25 15:57:06] ppdet.engine INFO: Eval iter: 700
- [11/25 15:57:10] ppdet.engine INFO: Eval iter: 800
- [11/25 15:57:14] ppdet.engine INFO: Eval iter: 900
- [11/25 15:57:18] ppdet.engine INFO: Eval iter: 1000
- [11/25 15:57:22] ppdet.engine INFO: Eval iter: 1100
- [11/25 15:57:26] ppdet.engine INFO: Eval iter: 1200
- [11/25 15:57:30] ppdet.engine INFO: Eval iter: 1300
- [11/25 15:57:34] ppdet.engine INFO: Eval iter: 1400
- [11/25 15:57:38] ppdet.engine INFO: Eval iter: 1500
- [11/25 15:57:42] ppdet.engine INFO: Eval iter: 1600
- [11/25 15:57:46] ppdet.engine INFO: Eval iter: 1700
- [11/25 15:57:50] ppdet.engine INFO: Eval iter: 1800
- [11/25 15:57:54] ppdet.engine INFO: Eval iter: 1900
- [11/25 15:57:58] ppdet.engine INFO: Eval iter: 2000
- [11/25 15:58:02] ppdet.engine INFO: Eval iter: 2100
- [11/25 15:58:07] ppdet.engine INFO: Eval iter: 2200
- [11/25 15:58:11] ppdet.engine INFO: Eval iter: 2300
- [11/25 15:58:15] ppdet.engine INFO: Eval iter: 2400
- [11/25 15:58:19] ppdet.engine INFO: Eval iter: 2500
- [11/25 15:58:23] ppdet.engine INFO: Eval iter: 2600
- [11/25 15:58:27] ppdet.engine INFO: Eval iter: 2700
- [11/25 15:58:31] ppdet.engine INFO: Eval iter: 2800
- [11/25 15:58:35] ppdet.engine INFO: Eval iter: 2900
- [11/25 15:58:39] ppdet.engine INFO: Eval iter: 3000
- [11/25 15:58:43] ppdet.engine INFO: Eval iter: 3100
- [11/25 15:58:47] ppdet.engine INFO: Eval iter: 3200
- [11/25 15:58:51] ppdet.engine INFO: Eval iter: 3300
- [11/25 15:58:55] ppdet.engine INFO: Eval iter: 3400
- [11/25 15:58:59] ppdet.engine INFO: Eval iter: 3500
- [11/25 15:59:03] ppdet.engine INFO: Eval iter: 3600
- [11/25 15:59:07] ppdet.engine INFO: Eval iter: 3700
- [11/25 15:59:11] ppdet.engine INFO: Eval iter: 3800
- [11/25 15:59:15] ppdet.engine INFO: Eval iter: 3900
- [11/25 15:59:19] ppdet.engine INFO: Eval iter: 4000
- [11/25 15:59:23] ppdet.engine INFO: Eval iter: 4100
- [11/25 15:59:27] ppdet.engine INFO: Eval iter: 4200
- [11/25 15:59:31] ppdet.engine INFO: Eval iter: 4300
- [11/25 15:59:35] ppdet.engine INFO: Eval iter: 4400
- [11/25 15:59:39] ppdet.engine INFO: Eval iter: 4500
- [11/25 15:59:43] ppdet.engine INFO: Eval iter: 4600
- [11/25 15:59:47] ppdet.engine INFO: Eval iter: 4700
- [11/25 15:59:51] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
- [11/25 15:59:51] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 91.58%
- [11/25 15:59:51] ppdet.engine INFO: Total sample number: 4792, averge FPS: 24.745395861714293
- [11/25 15:59:51] ppdet.engine INFO: Best test bbox ap is 0.916.
- [11/25 15:59:55] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/25 15:59:55] ppdet.engine INFO: Epoch: [24] [ 0/1347] learning_rate: 0.000100 loss_xy: 2.852638 loss_wh: 0.310755 loss_obj: 0.915459 loss_cls: 0.020150 loss: 4.273201 eta: 5:23:24 batch_cost: 0.3408 data_cost: 0.0205 ips: 23.4772 images/s
- [11/25 16:01:21] ppdet.engine INFO: Epoch: [24] [ 200/1347] learning_rate: 0.000100 loss_xy: 2.912006 loss_wh: 0.311531 loss_obj: 0.893285 loss_cls: 0.018086 loss: 4.307564 eta: 5:23:42 batch_cost: 0.4295 data_cost: 0.0319 ips: 18.6243 images/s
- [11/25 16:02:39] ppdet.engine INFO: Epoch: [24] [ 400/1347] learning_rate: 0.000100 loss_xy: 2.907623 loss_wh: 0.316048 loss_obj: 0.877782 loss_cls: 0.020929 loss: 4.433521 eta: 5:21:36 batch_cost: 0.3874 data_cost: 0.0171 ips: 20.6519 images/s
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::BasicEngine::Execute()
- 1 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVariableWrapperMap const&, paddle::imperative::NameVariableWrapperMap const&, paddle::framework::AttributeMap const&, paddle::framework::AttributeMap const&)
- 2 phi::KernelImpl<void (*)(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*), &(void phi::ConvCudnnGradKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*))>::Compute(phi::KernelContext*)
- 3 void phi::ConvCudnnGradKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669363418 (unix time) try "date -d @1669363418" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e800000378) received by PID 969 (TID 0x7fa7ccf98700) from PID 888 ***]
-
- server not ready, wait 3 sec to retry...
- not ready endpoints:['127.0.0.1:34909', '127.0.0.1:44925']
- I1125 16:04:20.413393 9126 nccl_context.cc:83] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
- W1125 16:04:21.152788 9126 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1125 16:04:21.155948 9126 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
- [11/25 16:04:22] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- [11/25 16:04:22] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- [11/25 16:04:23] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- [11/25 16:04:23] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- [11/25 16:04:23] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- [11/25 16:04:23] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- [11/25 16:04:24] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- [11/25 16:04:27] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- [11/25 16:04:27] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- [11/25 16:04:28] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/25 16:04:30] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- [11/25 16:04:31] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- [11/25 16:04:31] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- [11/25 16:04:32] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- [11/25 16:04:33] ppdet.data.source.voc WARNING: Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- [11/25 16:04:36] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage/23.pdparams
- [11/25 16:04:39] ppdet.engine INFO: Epoch: [24] [ 0/1347] learning_rate: 0.000100 loss_xy: 3.239845 loss_wh: 0.233076 loss_obj: 2.280031 loss_cls: 0.053115 loss: 5.806068 eta: 1 day, 10:06:43 batch_cost: 2.5324 data_cost: 0.0002 ips: 3.1590 images/s
- [11/25 16:05:54] ppdet.engine INFO: Epoch: [24] [ 200/1347] learning_rate: 0.000050 loss_xy: 3.090260 loss_wh: 0.340126 loss_obj: 0.817104 loss_cls: 0.020165 loss: 4.348865 eta: 5:08:55 batch_cost: 0.3731 data_cost: 0.0177 ips: 21.4437 images/s
- [11/25 16:07:12] ppdet.engine INFO: Epoch: [24] [ 400/1347] learning_rate: 0.000050 loss_xy: 2.912152 loss_wh: 0.277910 loss_obj: 0.718560 loss_cls: 0.015679 loss: 3.983868 eta: 5:10:43 batch_cost: 0.3916 data_cost: 0.0168 ips: 20.4316 images/s
- [11/25 16:08:31] ppdet.engine INFO: Epoch: [24] [ 600/1347] learning_rate: 0.000050 loss_xy: 2.850488 loss_wh: 0.295521 loss_obj: 0.674263 loss_cls: 0.013474 loss: 4.002361 eta: 5:10:03 batch_cost: 0.3900 data_cost: 0.0256 ips: 20.5139 images/s
- [11/25 16:09:57] ppdet.engine INFO: Epoch: [24] [ 800/1347] learning_rate: 0.000050 loss_xy: 2.974627 loss_wh: 0.323181 loss_obj: 0.835872 loss_cls: 0.016079 loss: 4.172034 eta: 5:16:28 batch_cost: 0.4273 data_cost: 0.0654 ips: 18.7225 images/s
- [11/25 16:11:20] ppdet.engine INFO: Epoch: [24] [1000/1347] learning_rate: 0.000050 loss_xy: 2.938231 loss_wh: 0.317002 loss_obj: 0.732253 loss_cls: 0.012985 loss: 4.039285 eta: 5:17:58 batch_cost: 0.4160 data_cost: 0.0606 ips: 19.2295 images/s
- [11/25 16:12:39] ppdet.engine INFO: Epoch: [24] [1200/1347] learning_rate: 0.000050 loss_xy: 2.839763 loss_wh: 0.284455 loss_obj: 0.761194 loss_cls: 0.011604 loss: 3.977188 eta: 5:15:09 batch_cost: 0.3905 data_cost: 0.0278 ips: 20.4861 images/s
- [11/25 16:13:36] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/25 16:13:36] ppdet.engine INFO: Epoch: [25] [ 0/1347] learning_rate: 0.000050 loss_xy: 2.805353 loss_wh: 0.311254 loss_obj: 0.713233 loss_cls: 0.013513 loss: 4.043765 eta: 5:11:52 batch_cost: 0.3755 data_cost: 0.0397 ips: 21.3021 images/s
- [11/25 16:14:57] ppdet.engine INFO: Epoch: [25] [ 200/1347] learning_rate: 0.000050 loss_xy: 2.843819 loss_wh: 0.282333 loss_obj: 0.599953 loss_cls: 0.012162 loss: 3.881706 eta: 5:11:07 batch_cost: 0.4026 data_cost: 0.0633 ips: 19.8710 images/s
- [11/25 16:16:14] ppdet.engine INFO: Epoch: [25] [ 400/1347] learning_rate: 0.000050 loss_xy: 2.891647 loss_wh: 0.296907 loss_obj: 0.889312 loss_cls: 0.015048 loss: 4.193014 eta: 5:08:29 batch_cost: 0.3829 data_cost: 0.0278 ips: 20.8941 images/s
- [11/25 16:17:39] ppdet.engine INFO: Epoch: [25] [ 600/1347] learning_rate: 0.000050 loss_xy: 3.020533 loss_wh: 0.322534 loss_obj: 0.750925 loss_cls: 0.012160 loss: 4.069333 eta: 5:09:33 batch_cost: 0.4259 data_cost: 0.0366 ips: 18.7847 images/s
- [11/25 16:19:05] ppdet.engine INFO: Epoch: [25] [ 800/1347] learning_rate: 0.000050 loss_xy: 2.815755 loss_wh: 0.282608 loss_obj: 0.592035 loss_cls: 0.010677 loss: 3.798633 eta: 5:09:58 batch_cost: 0.4235 data_cost: 0.0106 ips: 18.8893 images/s
- [11/25 16:20:26] ppdet.engine INFO: Epoch: [25] [1000/1347] learning_rate: 0.000050 loss_xy: 2.655332 loss_wh: 0.282245 loss_obj: 0.684140 loss_cls: 0.014039 loss: 3.938884 eta: 5:08:48 batch_cost: 0.4037 data_cost: 0.0129 ips: 19.8153 images/s
- [11/25 16:21:47] ppdet.engine INFO: Epoch: [25] [1200/1347] learning_rate: 0.000050 loss_xy: 2.817552 loss_wh: 0.286047 loss_obj: 0.671819 loss_cls: 0.011767 loss: 3.749783 eta: 5:07:39 batch_cost: 0.4049 data_cost: 0.0409 ips: 19.7599 images/s
- [11/25 16:22:38] ppdet.utils.checkpoint INFO: Save checkpoint: output/11_AUG_10/yolov3_darknet53_270e_voc_garbage
- [11/25 16:22:39] ppdet.engine INFO: Epoch: [26] [ 0/1347] learning_rate: 0.000050 loss_xy: 2.843514 loss_wh: 0.298186 loss_obj: 0.668867 loss_cls: 0.013553 loss: 3.969399 eta: 5:03:55 batch_cost: 0.3406 data_cost: 0.0188 ips: 23.4893 images/s
- [11/25 16:24:12] ppdet.engine INFO: Epoch: [26] [ 200/1347] learning_rate: 0.000050 loss_xy: 2.979055 loss_wh: 0.323838 loss_obj: 0.933396 loss_cls: 0.013865 loss: 4.550645 eta: 5:06:03 batch_cost: 0.4639 data_cost: 0.0691 ips: 17.2450 images/s
- [11/25 16:25:31] ppdet.engine INFO: Epoch: [26] [ 400/1347] learning_rate: 0.000050 loss_xy: 2.983588 loss_wh: 0.311689 loss_obj: 0.858065 loss_cls: 0.015924 loss: 4.191109 eta: 5:04:07 batch_cost: 0.3909 data_cost: 0.0196 ips: 20.4647 images/s
- [11/25 16:26:50] ppdet.engine INFO: Epoch: [26] [ 600/1347] learning_rate: 0.000050 loss_xy: 2.848044 loss_wh: 0.311193 loss_obj: 0.858633 loss_cls: 0.014974 loss: 4.310790 eta: 5:02:30 batch_cost: 0.3957 data_cost: 0.0144 ips: 20.2188 images/s
- [11/25 16:28:09] ppdet.engine INFO: Epoch: [26] [ 800/1347] learning_rate: 0.000050 loss_xy: 3.016177 loss_wh: 0.330798 loss_obj: 0.809394 loss_cls: 0.010876 loss: 4.383578 eta: 5:00:52 batch_cost: 0.3946 data_cost: 0.0173 ips: 20.2716 images/s
- [11/25 16:29:28] ppdet.engine INFO: Epoch: [26] [1000/1347] learning_rate: 0.000050 loss_xy: 3.073762 loss_wh: 0.326041 loss_obj: 0.938508 loss_cls: 0.015052 loss: 4.499593 eta: 4:59:01 batch_cost: 0.3888 data_cost: 0.0448 ips: 20.5769 images/s
|