Deleting a branch is permanent. It CANNOT be undone. Continue?
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》
问题描述
求问对于一个比较大的数据集(大约3G,每次训练和调试都需要加载并解压吗?且采用unzip命令解压的时候速度很慢的问题如何解决呢?
相关环境(GPU/NPU)
相关集群(启智/智算)
任务类型(调试/训练/推理)
任务名
日志说明或问题截图
期望的解决方案或建议
目前的资源调度逻辑是需要每次解压的。3个g的数据解压很慢是需要多久,具体是哪个数据集呢?
是我自己的数据集,目前我只是在调试任务中解压过,设备用的是T4,对于大概32W张的360*360的图片,我通过设置进度条查看,全解压完大概需要5个小时
建议调试任务只用小数据进行调试,在训练任务中使用这类数据集;数据集里的图片数量较多,这个解压速度在调试任务中是正常的。