测试环境如下表所示:
GPU | 显存 | CPU | 内存 | |
---|---|---|---|---|
Serving端 | 4x Tesla P4-8GB | 7611MiB | Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz 48核 | 216G |
Client端 | 4x Tesla P4-8GB | 7611MiB | Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz 48核 | 216G |
使用单卡GPU,未开启TensorRT。
模型:ResNet_v2_50
通过测试,使用 Python Pipeline 模式通过多进程并发,充分利用 GPU 显卡,具有较好的吞吐性能。
测试数据如下:
model_name | thread_num | batch_size | CPU_util(%) | GPU_memory(mb) | GPU_util(%) | qps(samples/s) | total count | mean(ms) | median(ms) | 80 percent(ms) | 90 percent(ms) | 99 percent(ms) | total cost(s) | each cost(s) |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ResNet_v2_50 | 1 | 1 | 2.2 | 3327 | 17.25 | 17.633658869240787 | 355 | 56.428481238996476 | 38.646728515625 | 39.496826171875 | 39.98369140625 | 1273.1911083984373 | 20.131953477859497 | 20.033540725708008 |
ResNet_v2_50 | 1 | 4 | 2.7 | 3617 | 28.122 | 53.50748430453522 | 268 | 74.71539215543378 | 74.6181640625 | 75.3138671875 | 75.6051025390625 | 77.85322998046874 | 20.03458046913147 | 20.024930953979492 |
ResNet_v2_50 | 1 | 8 | 1.7 | 3877 | 25.7869 | 59.60582783086999 | 150 | 133.5897119140625 | 132.7503662109375 | 134.968310546875 | 136.470703125 | 140.79039062499996 | 20.132259607315063 | 20.03933620452881 |
ResNet_v2_50 | 1 | 16 | 7.0 | 4777 | 27.0175 | 63.2627646819339 | 80 | 252.30162048339844 | 251.8448486328125 | 253.046630859375 | 253.91142578125 | 263.361640625 | 20.233070850372314 | 20.18476152420044 |
ResNet_v2_50 | 1 | 32 | 7.5 | 6567 | 38.532 | 62.945314687348024 | 40 | 506.8969482421875 | 507.3531494140625 | 510.562353515625 | 511.421240234375 | 536.8068920898437 | 20.335111618041992 | 20.276386737823486 |
ResNet_v2_50 | 2 | 1 | 4.7 | 6567 | 49.4828 | 50.40600094376044 | 1010 | 39.63352195815285 | 39.5345458984375 | 40.452880859375 | 41.1375 | 42.940522460937494 | 20.037296772003174 | 20.01696753501892 |
ResNet_v2_50 | 2 | 4 | 2.7 | 6567 | 44.4744 | 83.4255836891382 | 420 | 95.38548002697172 | 95.7069091796875 | 97.599951171875 | 98.098583984375 | 102.39680908203125 | 20.137707471847534 | 20.03199553489685 |
ResNet_v2_50 | 2 | 8 | 2.2 | 6567 | 42.898 | 91.3727510505176 | 230 | 174.89108568274457 | 175.0452880859375 | 175.82001953125 | 176.7634033203125 | 178.64064453125002 | 20.13729453086853 | 20.1132071018219 |
ResNet_v2_50 | 2 | 16 | 2.2 | 6567 | 45 | 97.5591285698611 | 124 | 327.16720088835683 | 328.6126708984375 | 329.75185546875 | 330.386962890625 | 336.86397460937496 | 20.336385011672974 | 20.284939169883728 |
ResNet_v2_50 | 2 | 32 | 3.2 | 6567 | 59.5714 | 100.70765418116333 | 64 | 633.9812698364258 | 637.8568115234375 | 648.103515625 | 650.7439697265625 | 659.2212915039062 | 20.336090803146362 | 20.28787398338318 |
ResNet_v2_50 | 4 | 1 | 3.1 | 6567 | 64.3333 | 80.27845081929433 | 1617 | 49.56464230756223 | 49.4873046875 | 51.5537109375 | 52.693408203125 | 55.207568359374996 | 20.142391681671143 | 20.038144528865814 |
ResNet_v2_50 | 4 | 4 | 3.3 | 6567 | 70.4563 | 136.62061939701394 | 688 | 116.51574919944586 | 121.8629150390625 | 129.8181640625 | 133.384423828125 | 142.69500732421875 | 20.143372297286987 | 20.041599333286285 |
ResNet_v2_50 | 4 | 8 | 3.0 | 6567 | 70.896 | 158.46554975132275 | 399 | 201.30669079926378 | 210.69775390625 | 228.51748046875 | 236.427294921875 | 252.24822753906233 | 20.143179416656494 | 20.081032752990723 |
ResNet_v2_50 | 4 | 16 | 3.2 | 6567 | 66.3832 | 156.4935247130092 | 197 | 407.6668608224937 | 423.974609375 | 450.368212890625 | 464.45986328125 | 482.93658203125 | 20.141408443450928 | 20.078101694583893 |
ResNet_v2_50 | 4 | 32 | 3.3 | 6567 | 72.4791 | 162.01742190796557 | 104 | 785.5079204852765 | 813.0341796875 | 887.107958984375 | 909.6556640625 | 935.3334838867188 | 20.541000843048096 | 20.423666059970856 |
ResNet_v2_50 | 8 | 1 | 3.5 | 6567 | 93.977 | 115.9749228558386 | 2337 | 68.5580409078145 | 65.45849609375 | 76.13930664062501 | 83.542041015625 | 91.45666015624998 | 20.15090799331665 | 20.028797417879105 |
ResNet_v2_50 | 8 | 4 | 4.2 | 6567 | 90.0952 | 175.58748591910316 | 889 | 180.7330482920592 | 170.5810546875 | 218.99931640625 | 240.06337890625002 | 254.413759765625 | 20.252012729644775 | 20.084695398807526 |
ResNet_v2_50 | 8 | 8 | 2.6 | 6567 | 93.8693 | 206.76595246418208 | 526 | 306.52158695119414 | 303.043212890625 | 321.0791015625 | 350.5477294921875 | 400.32452392578125 | 20.351513147354126 | 20.15437400341034 |
ResNet_v2_50 | 8 | 16 | 3.2 | 6567 | 85.7273 | 205.31850043117367 | 265 | 614.1745522553066 | 552.372314453125 | 775.89169921875 | 802.022607421875 | 902.2763183593761 | 20.650842428207397 | 20.345011442899704 |
ResNet_v2_50 | 8 | 32 | 5.0 | 6567 | 89.8717 | 219.8410273718835 | 146 | 1138.4533474020761 | 1039.640869140625 | 1364.289794921875 | 1474.6744384765625 | 1788.2614379882834 | 21.251720190048218 | 20.777225106954575 |
ResNet_v2_50 | 12 | 1 | 5.0 | 6567 | 89.4762 | 110.00858327847862 | 2218 | 108.50048552943953 | 103.015625 | 121.09404296875003 | 137.1392333984375 | 151.80401123046872 | 20.162063121795654 | 20.055511037508648 |
ResNet_v2_50 | 12 | 4 | 4.1 | 6567 | 77.7619 | 153.7824464757549 | 779 | 309.68895575507463 | 285.585205078125 | 378.07421875 | 413.481640625 | 424.70853515625 | 20.262390613555908 | 20.104551911354065 |
ResNet_v2_50 | 12 | 8 | 3.6 | 6567 | 72.6977 | 165.36021780846013 | 425 | 571.1991590073529 | 510.995849609375 | 731.9383300781251 | 747.6568359375 | 757.304716796875 | 20.56117272377014 | 20.230452219645183 |
ResNet_v2_50 | 12 | 16 | 1.5 | 6567 | 76.2222 | 189.6414991568285 | 252 | 987.7153136238219 | 926.00390625 | 1080.99130859375 | 1249.4956298828126 | 1434.4802392578124 | 21.26116919517517 | 20.74245794614156 |
ResNet_v2_50 | 12 | 32 | 2.8 | 6567 | 84.25 | 203.868228281784 | 138 | 1811.640237559443 | 1764.2760009765625 | 1855.28046875 | 2023.56826171875 | 2586.8038134765625 | 21.66105055809021 | 20.834286351998646 |
ResNet_v2_50 | 16 | 1 | 4.8 | 6567 | 94.3333 | 116.34927733312234 | 2347 | 136.7957122373642 | 135.959716796875 | 144.1568359375 | 146.105517578125 | 175.05707519531248 | 20.172020435333252 | 20.067057371139526 |
ResNet_v2_50 | 16 | 4 | 15.4 | 6567 | 83.6364 | 160.59012047270738 | 822 | 393.3079394412447 | 396.446533203125 | 426.272216796875 | 429.777734375 | 564.1119360351562 | 20.47448492050171 | 20.206754431128502 |
ResNet_v2_50 | 16 | 8 | 6.8 | 6567 | 81.0233 | 169.95774070621547 | 437 | 741.5512622684854 | 751.521484375 | 763.199169921875 | 948.8041992187501 | 1001.156142578125 | 20.56981921195984 | 20.254074171185493 |
ResNet_v2_50 | 16 | 16 | 3.5 | 6567 | 77.8706 | 186.56600081516 | 248 | 1332.1007946383568 | 1365.2745361328125 | 1399.212255859375 | 1432.4037353515625 | 1771.4374853515626 | 21.26861262321472 | 20.64799252152443 |
ResNet_v2_50 | 16 | 32 | 4.3 | 6567 | 83.6371 | 201.1293408638195 | 140 | 2419.3400198800223 | 2561.09228515625 | 2616.081103515625 | 2642.0835205078124 | 2883.8197412109366 | 22.274224042892456 | 21.169659316539764 |
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》