#672 [Document and Code] Fix topk bugs, add hub and model_zoo, modify supportedlist

Merged
zoulq merged 14 commits from ziqi/MSAdapter:master into master 8 months ago
ziqi commented 8 months ago
- Fix Tensor.topk and torch.functional.topk bugs - Add hub and model_zoo - Modify supportedlist for hub and model_zoo
zoulq commented 8 months ago
Collaborator
CI用例失败:test_tensor.py执行出现core dumped
zoulq reviewed 8 months ago
msadapter/pytorch/functional.py
@@ -2091,3 +2091,3 @@
def topk(input, k, dim=None, largest=True, sorted=True, *, out=None):
input_x = cast_to_ms_tensor(input)
output = ms.ops.topk(input_x, k, dim, largest, sorted)
output = ([ms.ops.zeros([], dtype=input.dtype), ms.ops.zeros([], dtype=ms.int32)])
zoulq commented 8 months ago
如果只用k=0的场景会进这一句处理,就放到else分支里去,减少k!=0场景下的性能开销
zoulq reviewed 8 months ago
msadapter/pytorch/hub.py
@@ -3,3 +5,4 @@
import shutil
import sys
import tempfile
import torch
zoulq commented 8 months ago
这里不能依赖pytorch
zoulq reviewed 8 months ago
msadapter/pytorch/hub.py
@@ -13,3 +20,3 @@
except ImportError:
# fake tqdm if it's not installed
class tqdm(): # type: ignore[no-redef]
class tqdm(object): # type: ignore[no-redef]
zoulq commented 8 months ago
pylint应该会检查,python3.x之后不需要加object
ziqi commented 8 months ago
已修改
zoulq reviewed 8 months ago
msadapter/pytorch/hub.py
@@ -50,0 +56,4 @@
'download_url_to_file',
'get_dir',
'load_state_dict_from_url',
'set_dir',
zoulq commented 8 months ago
剩余3个可以实现吗?
ziqi commented 8 months ago
剩下的接口也未涉及torch源生接口调用,可以实现,但需调查常用场景并测试之后提上去
zoulq reviewed 8 months ago
msadapter/pytorch/hub.py
@@ -6,3 +12,4 @@
from urllib.request import urlopen, Request
from urllib.parse import urlparse # noqa: F401

try:
zoulq commented 8 months ago
这个文件和pytorch的源码重复度比较高,建议在开头注明源码链接说明
ziqi commented 8 months ago
已注明 代码来源
zoulq reviewed 8 months ago
@@ -105,0 +174,4 @@

# Hub used to support automatically extracts from zipfile manually compressed by users.
# The legacy zip format expects only one file from torch.save() < 1.6 in the zip.
# We should remove this support since zipfile is now default zipfile format for torch.save().
zoulq commented 8 months ago
这些注释根据实际含义确定是否保留
ziqi commented 8 months ago
已删除
zoulq reviewed 8 months ago
@@ -105,0 +229,4 @@
digits of the SHA256 hash of the contents of the file. The hash is used to
ensure unique names and to verify the contents of the file.
Default: False
file_name (string, optional): name for the downloaded file. Filename from ``url`` will be used if not set.
zoulq commented 8 months ago
同上
ziqi commented 8 months ago
属于对入参的解释,建议保留
zoulq reviewed 8 months ago
msadapter/pytorch/tensor.py
@@ -2057,3 +2057,3 @@
def topk(self, k, dim=None, largest=True, sorted=True):
input = cast_to_ms_tensor(self)
output = ms.ops.topk(input, k, dim, largest, sorted)
output = ([ms.ops.zeros([], dtype=self.dtype), ms.ops.zeros([], dtype=ms.int32)])
zoulq commented 8 months ago
统一修改
ziqi commented 8 months ago
已修改
zoulq reviewed 8 months ago
@@ -0,0 +1 @@
from msadapter.pytorch.hub import tqdm, load_state_dict_from_url as load_url # noqa: F401
zoulq commented 8 months ago
hub新增接口要看是否增加基本用例。至少能够看护接口的正确性
ziqi commented 8 months ago
已新增 test_hub.py 用例
Erpim reviewed 8 months ago
msadapter/pytorch/hub.py
@@ -7,1 +12,3 @@

from urllib.parse import urlparse
import msadapter.pytorch as torch # noqa: F401
# Part of the code was borrowed from https://github.com/pytorch/pytorch/blob/v1.12.1/torch/hub.py
Erpim commented 8 months ago
建议移到最顶上
ziqi commented 8 months ago
已修改
zoulq commented 8 months ago
Collaborator
CI报错:pylint检查失败 ************* Module msadapter.pytorch.cuda.amp.__init__ msadapter/pytorch/cuda/amp/__init__.py:1:0: R0401: Cyclic import (msadapter.pytorch -> msadapter.pytorch.hub) (cyclic-import)
Erpim commented 8 months ago
Collaborator
************* Module msadapter.pytorch.cuda.amp.__init__ msadapter/pytorch/cuda/amp/__init__.py:1:0: R0401: Cyclic import (msadapter.pytorch -> msadapter.pytorch.hub) (cyclic-import)
Erpim commented 8 months ago
Collaborator
用例超时了 testing/ut/pytorch/tensor/test_type.py ........ [ 98%] testing/ut/pytorch/torch/test_assert.py ...... [ 99%]
ziqi commented 8 months ago
Poster
> 用例超时了 > testing/ut/pytorch/tensor/test_type.py ........ [ 98%] > testing/ut/pytorch/torch/test_assert.py ...... [ 99%] 我换个小点的模型试试
zoulq merged commit 7a0b5196b1 into master 8 months ago
The pull request has been merged as 7a0b5196b1.
Sign in to join this conversation.
No reviewers
No Label
No Milestone
No Assignees
3 Participants
Notifications
Due Date

No due date set.

Dependencies

This pull request currently doesn't have any dependencies.

Loading…
There is no content yet.