|
- I1121 19:24:12.187698 990 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:55217 successful.
- I1121 19:24:15.140249 990 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1121 19:24:15.958326 990 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1121 19:24:15.961112 990 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- I1121 19:24:46.742914 1439 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:38575 successful.
- I1121 19:24:49.578421 1439 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1121 19:24:50.337880 1439 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1121 19:24:50.345444 1439 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- I1121 19:25:49.457751 1921 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:36759 successful.
- I1121 19:25:52.423684 1921 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1121 19:25:53.235119 1921 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1121 19:25:53.240919 1921 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- I1121 19:27:01.113466 13209 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:33351 successful.
- I1121 19:27:04.088650 13209 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1121 19:27:04.979820 13209 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1121 19:27:04.984506 13209 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- I1121 20:37:21.240880 60151 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:44965 successful.
- I1121 20:37:24.178496 60151 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1121 20:37:25.021601 60151 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1121 20:37:25.027323 60151 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- I1121 21:51:27.985148 65229 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:47559 successful.
- I1121 21:51:30.994999 65229 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1121 21:51:31.818572 65229 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1121 21:51:31.824662 65229 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::BasicEngine::Execute()
- 1 paddle::imperative::EagerGradientAccumulator::SumGrad(std::shared_ptr<paddle::imperative::VariableWrapper>, unsigned long, bool)
- 2 paddle::imperative::VariableWrapperAdd(std::shared_ptr<paddle::imperative::VariableWrapper>, paddle::imperative::VariableWrapper*, bool)
- 3 void paddle::imperative::TensorAdd<paddle::framework::Variable>(paddle::framework::Variable const&, paddle::framework::Variable*)
- 4 paddle::imperative::TensorAddFunctor<float>::result_type paddle::platform::VisitPlace<paddle::imperative::TensorAddFunctor<float> >(phi::Place const&, paddle::imperative::TensorAddFunctor<float> const&)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669039373 (unix time) try "date -d @1669039373" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e80000fe6e) received by PID 65229 (TID 0x7ff29a90e700) from PID 65134 ***]
-
- I1122 12:32:08.695628 869 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:44743 successful.
- I1122 12:32:10.502156 869 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1122 12:32:11.154222 869 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1122 12:32:11.157335 869 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::BasicEngine::Execute()
- 1 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVariableWrapperMap const&, paddle::imperative::NameVariableWrapperMap const&, paddle::framework::AttributeMap const&, paddle::framework::AttributeMap const&)
- 2 phi::KernelImpl<void (*)(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*), &(void phi::ConvCudnnGradKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*))>::Compute(phi::KernelContext*)
- 3 void phi::ConvCudnnGradKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*)
- 4 paddle::operators::SearchAlgorithm<cudnnConvolutionBwdDataAlgoPerf_t>::GetWorkspaceSize(paddle::operators::ConvArgsBase<cudnnContext*, cudnnDataType_t> const&, cudnnConvolutionBwdDataAlgo_t)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669094669 (unix time) try "date -d @1669094669" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e8000002e6) received by PID 869 (TID 0x7f45e0179700) from PID 742 ***]
-
- I1122 13:25:53.735575 30909 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:39803 successful.
- I1122 13:25:56.588238 30909 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1122 13:25:57.256799 30909 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1122 13:25:57.259848 30909 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- I1122 14:39:57.001598 39598 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:36143 successful.
- I1122 14:39:57.016222 39598 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1122 14:39:57.779636 39598 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1122 14:39:57.782781 39598 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::BasicEngine::Execute()
- 1 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVariableWrapperMap const&, paddle::imperative::NameVariableWrapperMap const&, paddle::framework::AttributeMap const&, paddle::framework::AttributeMap const&)
- 2 phi::KernelImpl<void (*)(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*), &(void phi::ConvCudnnGradKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*))>::Compute(phi::KernelContext*)
- 3 std::vector<int, std::allocator<int> > const& phi::KernelContext::AttrAt<std::vector<int, std::allocator<int> > >(unsigned long) const
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669103620 (unix time) try "date -d @1669103620" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e800009a4e) received by PID 39598 (TID 0x7fa9e7269700) from PID 39502 ***]
-
- I1122 15:54:08.476439 49983 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:35961 successful.
- I1122 15:54:11.295094 49983 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1122 15:54:12.073060 49983 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1122 15:54:12.076112 49983 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- I1122 16:47:53.387939 31362 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:54671 successful.
- I1122 16:47:56.418866 31362 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1122 16:47:57.194705 31362 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1122 16:47:57.200351 31362 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Traceback (most recent call last):
- File "tools/train.py", line 172, in <module>
- main()
- File "tools/train.py", line 168, in main
- run(FLAGS, cfg)
- File "tools/train.py", line 127, in run
- trainer.resume_weights(FLAGS.resume)
- File "/home/aistudio/work/myDemo/ppdet/engine/trainer.py", line 392, in resume_weights
- self.ema if self.use_ema else None)
- File "/home/aistudio/work/myDemo/ppdet/utils/checkpoint.py", line 73, in load_weight
- "exists.".format(pdparam_path))
- ValueError: Model pretrain path output/AUG/yolov3_darknet53_270e_voc_garbage/38.pdparams does not exists.
- I1122 16:48:48.665015 31815 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:46501 successful.
- I1122 16:48:48.726775 31815 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1122 16:48:49.507046 31815 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1122 16:48:49.510296 31815 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_5.xml, x1: 249.0, y1: 567.0, x2: 249.0, y2: 602.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_3.xml, x1: 218.0, y1: 762.0, x2: 218.0, y2: 763.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_5.xml, x1: 793.0, y1: 747.0, x2: 793.0, y2: 748.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_6.xml, x1: 195.0, y1: 781.0, x2: 195.0, y2: 782.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_4.xml, x1: 815.0, y1: 766.0, x2: 815.0, y2: 767.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_3860_2.xml, x1: 240.0, y1: 577.0, x2: 240.0, y2: 615.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702.xml, x1: 846.0, y1: 789.0, x2: 846.0, y2: 790.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_7.xml, x1: 235.0, y1: 748.0, x2: 235.0, y2: 749.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_2.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 104.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_1.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_9.xml, x1: 821.0, y1: 771.0, x2: 821.0, y2: 771.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_0.xml, x1: 803.0, y1: 756.0, x2: 803.0, y2: 757.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_8.xml, x1: 197.0, y1: 780.0, x2: 197.0, y2: 780.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_702_1.xml, x1: 191.0, y1: 785.0, x2: 191.0, y2: 786.0.
- Found an invalid bbox in annotations: xml_file: dataset/AUG_10/./Annotations/img_918_0.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 105.0.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::BasicEngine::Execute()
- 1 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVariableWrapperMap const&, paddle::imperative::NameVariableWrapperMap const&, paddle::framework::AttributeMap const&, paddle::framework::AttributeMap const&)
- 2 void phi::LeakyReluGradKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, float, phi::DenseTensor*)
- 3 void phi::ActivationGradGPUImpl<float, phi::GPUContext, phi::funcs::CudaLeakyReluGradFunctor<float> >(phi::GPUContext const&, phi::DenseTensor const*, phi::DenseTensor const*, phi::DenseTensor const*, phi::DenseTensor*, phi::funcs::CudaLeakyReluGradFunctor<float> const&)
- 4 void phi::funcs::ElementwiseKernel<float, phi::funcs::CudaLeakyReluGradFunctor<float>, 1>(phi::GPUContext const&, std::vector<phi::DenseTensor const*, std::allocator<phi::DenseTensor const*> > const&, std::vector<phi::DenseTensor*, std::allocator<phi::DenseTensor*> >*, phi::funcs::CudaLeakyReluGradFunctor<float>)
- 5 void phi::funcs::ElementwiseCudaKernel<float, phi::funcs::CudaLeakyReluGradFunctor<float>, 2, 1, 4>(phi::GPUContext const&, std::vector<phi::DenseTensor const*, std::allocator<phi::DenseTensor const*> > const&, std::vector<phi::DenseTensor*, std::allocator<phi::DenseTensor*> >*, phi::funcs::CudaLeakyReluGradFunctor<float>)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669110483 (unix time) try "date -d @1669110483" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e800007be7) received by PID 31815 (TID 0x7fd8d955a700) from PID 31719 ***]
-
- I1124 13:46:57.600476 972 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:56687 successful.
- I1124 13:46:59.537802 972 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1124 13:47:00.209031 972 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 13:47:00.211637 972 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- No stack trace in paddle, may be caused by external reasons.
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669268941 (unix time) try "date -d @1669268941" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e80000034c) received by PID 972 (TID 0x7fd2f9545700) from PID 844 ***]
-
- I1124 13:49:08.869577 20582 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:51481 successful.
- I1124 13:49:11.833940 20582 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1124 13:49:12.546686 20582 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 13:49:12.549229 20582 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- No stack trace in paddle, may be caused by external reasons.
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669268956 (unix time) try "date -d @1669268956" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e800005004) received by PID 20582 (TID 0x7ffbc1e0a700) from PID 20484 ***]
-
- I1124 13:49:45.933806 20827 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:52063 successful.
- I1124 13:49:48.889227 20827 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1124 13:49:49.531194 20827 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 13:49:49.533941 20827 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- I1124 13:54:21.805327 55320 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:55231 successful.
- I1124 13:54:24.720923 55320 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1124 13:54:25.438038 55320 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 13:54:25.440677 55320 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::pybind::MultiDeviceFeedReader<paddle::operators::reader::LoDTensorBlockingQueue>::CheckNextStatus()
- 1 paddle::pybind::MultiDeviceFeedReader<paddle::operators::reader::LoDTensorBlockingQueue>::WaitFutures(std::__exception_ptr::exception_ptr*)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669270496 (unix time) try "date -d @1669270496" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e80000d7b7) received by PID 55320 (TID 0x7f872cf7c700) from PID 55223 ***]
-
- I1124 16:09:51.485803 1066 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:46869 successful.
- I1124 16:09:52.743580 1066 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1124 16:09:53.590251 1066 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 16:09:53.593056 1066 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::pybind::MultiDeviceFeedReader<paddle::operators::reader::LoDTensorBlockingQueue>::CheckNextStatus()
- 1 paddle::pybind::MultiDeviceFeedReader<paddle::operators::reader::LoDTensorBlockingQueue>::WaitFutures(std::__exception_ptr::exception_ptr*)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669280936 (unix time) try "date -d @1669280936" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e8000003ab) received by PID 1066 (TID 0x7fa18f190700) from PID 939 ***]
-
- I1124 17:09:53.431886 48704 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:43219 successful.
- I1124 17:09:56.237474 48704 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1124 17:09:57.133929 48704 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 17:09:57.136440 48704 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::pybind::MultiDeviceFeedReader<paddle::operators::reader::LoDTensorBlockingQueue>::CheckNextStatus()
- 1 paddle::pybind::MultiDeviceFeedReader<paddle::operators::reader::LoDTensorBlockingQueue>::WaitFutures(std::__exception_ptr::exception_ptr*)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669283284 (unix time) try "date -d @1669283284" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e80000bde1) received by PID 48704 (TID 0x7f916e7e9700) from PID 48609 ***]
-
- I1124 17:48:52.718082 20903 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:46561 successful.
- I1124 17:48:55.717111 20903 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1124 17:48:56.661314 20903 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 17:48:56.663872 20903 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::Tracer::TraceOp(std::string const&, paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap, std::map<std::string, std::string, std::less<std::string >, std::allocator<std::pair<std::string const, std::string > > > const&)
- 1 void paddle::imperative::Tracer::TraceOpImpl<paddle::imperative::VarBase>(std::string const&, paddle::imperative::details::NameVarMapTrait<paddle::imperative::VarBase>::Type const&, paddle::imperative::details::NameVarMapTrait<paddle::imperative::VarBase>::Type const&, paddle::framework::AttributeMap&, phi::Place const&, bool, std::map<std::string, std::string, std::less<std::string >, std::allocator<std::pair<std::string const, std::string > > > const&, paddle::framework::AttributeMap*, bool)
- 2 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap const&, paddle::framework::AttributeMap const&)
- 3 phi::KernelImpl<void (*)(phi::GPUContext const&, std::vector<phi::DenseTensor const*, std::allocator<phi::DenseTensor const*> > const&, int, phi::DenseTensor*), &(void phi::StackKernel<long, phi::GPUContext>(phi::GPUContext const&, std::vector<phi::DenseTensor const*, std::allocator<phi::DenseTensor const*> > const&, int, phi::DenseTensor*))>::Compute(phi::KernelContext*)
- 4 void phi::StackKernel<long, phi::GPUContext>(phi::GPUContext const&, std::vector<phi::DenseTensor const*, std::allocator<phi::DenseTensor const*> > const&, int, phi::DenseTensor*)
- 5 void paddle::memory::Copy<phi::Place, phi::CPUPlace>(phi::Place, void*, phi::CPUPlace, void const*, unsigned long, void*)
- 6 void paddle::memory::Copy<phi::Place, phi::Place>(phi::Place, void*, phi::Place, void const*, unsigned long, void*)
- 7 void paddle::memory::Copy<phi::GPUPlace, phi::CPUPlace>(phi::GPUPlace, void*, phi::CPUPlace, void const*, unsigned long, void*)
- 8 phi::backends::gpu::GpuMemcpyAsync(void*, void const*, unsigned long, cudaMemcpyKind, CUstream_st*)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669287849 (unix time) try "date -d @1669287849" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e800005148) received by PID 20903 (TID 0x7fa11074f700) from PID 20808 ***]
-
- I1124 19:14:43.711591 41925 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:47055 successful.
- I1124 19:14:46.530495 41925 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1124 19:14:47.345660 41925 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 19:14:47.349381 41925 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- I1124 19:15:44.399706 48533 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:41287 successful.
- I1124 19:15:47.317828 48533 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1124 19:15:48.227408 48533 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1124 19:15:48.232587 48533 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- I1125 10:33:18.089790 1047 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:39663 successful.
- I1125 10:33:18.385967 1047 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1125 10:33:19.046715 1047 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1125 10:33:19.049791 1047 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- Traceback (most recent call last):
- File "tools/train.py", line 172, in <module>
- main()
- File "tools/train.py", line 168, in main
- run(FLAGS, cfg)
- File "tools/train.py", line 127, in run
- trainer.resume_weights(FLAGS.resume)
- File "/home/aistudio/work/myDemo/ppdet/engine/trainer.py", line 392, in resume_weights
- self.ema if self.use_ema else None)
- File "/home/aistudio/work/myDemo/ppdet/utils/checkpoint.py", line 73, in load_weight
- "exists.".format(pdparam_path))
- ValueError: Model pretrain path output/11_AUG_10/yolov3_darknet53_270e_voc_garbage/.pdparams.pdparams does not exists.
- I1125 10:39:01.685369 2139 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:38879 successful.
- I1125 10:39:04.615003 2139 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1125 10:39:05.281702 2139 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1125 10:39:05.286049 2139 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_4.xml, x1: 282.0, y1: 74.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- The actual image height: 3120 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4208 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 3120 is not equal to the height: 4624.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4208 is not equal to the width: 3472.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 3472 is not equal to the height: 3120.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4624 is not equal to the width: 4208.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 4624 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 3472 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 3120 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4208 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 3472 is not equal to the height: 4624.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4624 is not equal to the width: 3472.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 3472 is not equal to the height: 3120.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4624 is not equal to the width: 4208.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 4624 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 3472 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 3472 is not equal to the height: 4624.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4624 is not equal to the width: 3472.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 3120 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4208 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 3120 is not equal to the height: 4624.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4208 is not equal to the width: 3472.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 3472 is not equal to the height: 4624.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4624 is not equal to the width: 3472.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 4624 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 3472 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 3120 is not equal to the height: 3472.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4208 is not equal to the width: 4624.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 3472 is not equal to the height: 3120.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4624 is not equal to the width: 4208.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 3120 is not equal to the height: 4624.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4208 is not equal to the width: 3472.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 3472 is not equal to the height: 3120.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 4624 is not equal to the width: 4208.0 in annotation, and update sample['w'] by actual image width.
- The actual image height: 4624 is not equal to the height: 3120.0 in annotation, and update sample['h'] by actual image height.
- The actual image width: 3472 is not equal to the width: 4208.0 in annotation, and update sample['w'] by actual image width.
- I1125 11:05:24.921324 1041 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:58983 successful.
- I1125 11:05:27.883283 1041 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1125 11:05:28.739238 1041 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1125 11:05:28.742666 1041 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::pybind::MultiDeviceFeedReader<paddle::operators::reader::LoDTensorBlockingQueue>::CheckNextStatus()
- 1 paddle::pybind::MultiDeviceFeedReader<paddle::operators::reader::LoDTensorBlockingQueue>::WaitFutures(std::__exception_ptr::exception_ptr*)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669347299 (unix time) try "date -d @1669347299" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e8000003b0) received by PID 1041 (TID 0x7fda76565700) from PID 944 ***]
-
- I1125 11:35:30.002033 48509 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:51959 successful.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::NCCLParallelContext::Init()
- 1 paddle::imperative::NCCLParallelContext::BcastNCCLId(std::vector<ncclUniqueId, std::allocator<ncclUniqueId> >&, int, int)
- 2 void paddle::platform::RecvBroadCastCommID<ncclUniqueId>(int, std::string, std::vector<ncclUniqueId, std::allocator<ncclUniqueId> >*, int)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669347330 (unix time) try "date -d @1669347330" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e80000bd1d) received by PID 48509 (TID 0x7f3603329700) from PID 48413 ***]
-
- I1125 11:35:38.227408 48658 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:33613 successful.
- I1125 11:35:41.189486 48658 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1125 11:35:42.108706 48658 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1125 11:35:42.111958 48658 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
- I1125 15:34:57.699628 1019 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:58863 successful.
- I1125 15:34:58.108000 1019 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1125 15:34:58.832461 1019 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1125 15:34:58.835040 1019 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
-
-
- --------------------------------------
- C++ Traceback (most recent call last):
- --------------------------------------
- 0 paddle::imperative::BasicEngine::Execute()
- 1 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVariableWrapperMap const&, paddle::imperative::NameVariableWrapperMap const&, paddle::framework::AttributeMap const&, paddle::framework::AttributeMap const&)
- 2 phi::KernelImpl<void (*)(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*), &(void phi::ConvCudnnGradKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*))>::Compute(phi::KernelContext*)
- 3 void phi::ConvCudnnGradKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::string const&, int, std::vector<int, std::allocator<int> > const&, std::string const&, bool, int, bool, phi::DenseTensor*, phi::DenseTensor*)
-
- ----------------------
- Error Message Summary:
- ----------------------
- FatalError: `Termination signal` is detected by the operating system.
- [TimeInfo: *** Aborted at 1669363418 (unix time) try "date -d @1669363418" if you are using GNU date ***]
- [SignalInfo: *** SIGTERM (@0x3e800000378) received by PID 1019 (TID 0x7f6295b46700) from PID 888 ***]
-
- I1125 16:04:17.540869 9141 gen_comm_id_helper.cc:205] Server listening on: 127.0.0.1:44925 successful.
- I1125 16:04:20.413388 9141 nccl_context.cc:83] init nccl context nranks: 4 local rank: 3 gpu id: 3 ring id: 0
- W1125 16:04:21.153676 9141 gpu_resources.cc:61] Please NOTE: device: 3, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
- W1125 16:04:21.158946 9141 gpu_resources.cc:91] device: 3, cuDNN Version: 8.2.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_898_8.xml, x1: 184.0, y1: 81.0, x2: 184.0, y2: 137.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_3.xml, x1: 272.0, y1: 82.0, x2: 272.0, y2: 288.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817.xml, x1: 282.0, y1: 50.0, x2: 281.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_6.xml, x1: 269.0, y1: 84.0, x2: 269.0, y2: 285.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_8.xml, x1: 99.0, y1: 67.0, x2: 99.0, y2: 105.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_5.xml, x1: 26.0, y1: 73.0, x2: 26.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_1.xml, x1: 274.0, y1: 81.0, x2: 274.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_7.xml, x1: 278.0, y1: 77.0, x2: 278.0, y2: 295.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_918_6.xml, x1: 100.0, y1: 67.0, x2: 99.0, y2: 106.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_8.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_1001_7.xml, x1: 273.0, y1: 193.0, x2: 320.0, y2: 193.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_9.xml, x1: 22.0, y1: 70.0, x2: 22.0, y2: 299.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/img_3860_7.xml, x1: 464.0, y1: 570.0, x2: 464.0, y2: 605.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_0.xml, x1: 37.0, y1: 81.0, x2: 37.0, y2: 290.0.
- Found an invalid bbox in annotations: xml_file: dataset/11_AUG_10/./Annotations/T00817_2.xml, x1: 29.0, y1: 75.0, x2: 29.0, y2: 299.0.
|