测试环境选择Python3.7.5、MindSpore2.0.0Nightly、PyTorch1.12.0
我们选择使用最新版本的MindSpore开发,请使用2.0.0Nightly版本。安装指令可以参考MindSpore安装。
例如需要安装CPU版本的MindSpore2.0.0Nightly可以用如下指令:
pip install mindspore-dev -i https://pypi.tuna.tsinghua.edu.cn/simple
安装PyTorch是用于和开发的MSAdapter接口进行结果和功能比对,选择PyTorch1.12.0,安装可以参考PyTorch安装。
例如需要安装CPU版本PyTorch1.12.0可以用如下指令:
pip install torch==1.12.0+cpu torchvision==0.13.0+cpu torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cpu
接口代码开发主要涉及到nn的modules里面文件及nn.functional.py和functional.py;分别对应开发模型构建组件接口代码(对应torch.nn.Conv2d这类),nn函数接口代码(对应torch.nn.functional.conv2d这类)以及常用函数接口代码(对应torch.abs这些)。
代码测试用例编写目录结构如下,在开发完成接口后需要在对应的文件夹内创建测试代码
深度学习框架前端层次结构可以描述为如下表示。低阶接口包含Tensor、nn.functional.xx等;中阶接口是对低阶接口的封装,调用低阶接口实现用户友好的API;高阶接口又可以是中低阶接口的封装。因此我们开发中阶接口Layers(nn.Conv2d、nn.MaxPool2d)相关的API时,需要先开发对应的低级接口如nn.functional.xx(nn.functional.conv2d、nn.functional.max_pool2d)。
假设我们需要构建一个和PyTorch线性层(Linear)对应的接口。首先应该开发Linear所调用的低阶接口nn.functional.linear
其中低阶接口linear可以使用MindSpore的MatMul和bias_add算子,示例如下:
def linear(input, weight, bias=None):
input = cast_to_ms_tensor(input)
output = ms.ops.MatMul(transpose_b=True)(input, weight)
if bias is not None:
output = ms.ops.bias_add(output, bias)
output = cast_to_adapter_tensor(output)
return output
需要注意的是开发函数接口调用MindSpore算子时需要将tensor调用cast_to_ms_tensor转换成MindSpore所需tensor,输出调用cast_to_adapter_tensor转换成MSAdapter所需tensor。
接下来就可以使用nn.functional.linear提供的接口来开发更高阶的神经网络组件接口Linear了,示例如下:
class Linear(Module):
def __init__(self, in_features, out_features, bias=True, device=None, dtype=None):
super(Linear, self).__init__()
self.in_features = in_features
self.out_features = out_features
self.has_bias = False
self.bias = None
self.weight = Parameter(empty((self.out_features, self.in_features)), requires_grad=True)
if bias:
self.bias = Parameter(empty(self.out_features), requires_grad=True)
self.has_bias = True
self.reset_parameters()
unsupported_attr(device)
unsupported_attr(dtype)
def reset_parameters(self):
init.kaiming_uniform_(self.weight, a=math.sqrt(5))
if self.has_bias:
fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
bound = 1 / math.sqrt(fan_in) if fan_in > 0 else 0
init.uniform_(self.bias, -bound, bound)
def forward(self, input):
input = cast_to_ms_tensor(input)
x = linear(input, self.weight, self.bias)
return cast_to_adapter_tensor(x)
def extra_repr(self):
return 'in_features={}, out_features={}, bias={}'.format(
self.in_features, self.out_features, self.has_bias is not None
)
在__init__方法里进行属性的定义和参数初始化,在reset_parameters里进行参数的初始化方法定义,在forward里实现前向计算过程。extra_repr里返回属性信息。
此时Linear Layer和其nn.functioal接口都已经开发完成,需要编写测试用例,在testing/ut/pytorch/nn下新建测试文件test_linear.py。我们先简单测试一下训练参数属性能否和torch一样改变初始化方法。也需要测试相同输入和初始化方法输出是否和torch一样,具体可以参考test_conv.py。
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from ms_adapter.pytorch.nn import Module, Linear, Identity, Bilinear
from ms_adapter.pytorch import tensor
from mindspore import context
import numpy as np
import mindspore as ms
context.set_context(mode=ms.PYNATIVE_MODE)
def test_linear_model():
class LinearModel(Module):
def __init__(self):
super(LinearModel, self).__init__()
self.line1 = Linear(in_features=32, out_features=64)
self.line2 = Linear(in_features=64, out_features=128, bias=False)
self.line3 = Linear(in_features=128, out_features=10)
def forward(self, inputs):
x = self.line1(inputs)
x = self.line2(x)
x = self.line3(x)
return x
model = LinearModel()
model.train()
def weight_init(m):
if isinstance(m, Linear):
m.weight.data.normal_(0, 0.01)
if m.has_bias:
m.bias.data.zero_()
model.apply(weight_init)
inputs = tensor(np.ones(shape=(5, 32)))
output = model(inputs)
assert output.shape == (5, 10)
test_linear_model()
测试
用例编写完成,此时就开发完成了。我们可以提交代码到代码仓库。
首先从代码仓库clone代码到本地机器:
git clone https://openi.pcl.ac.cn/OpenI/MSAdapter.git
clone代码后,本地开发,我们新建一个分支进行开发:
git checkout -b {新分支名称}
一顿开发完成后,我们正式推送更新分支:
git add .
git status # 查看更新了什么代码,避免提交错了
git commit -m "你的commit标题,例如开发了Linear起名为:Add Linear Layer"
git push origin {新分支名称}
接下来就在启智平台进行操作
点击: 合并请求-->创建合并请求
我们继续检查一下开发内容是否正确,没问题后确认创建合并请求,等待审核入库。
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》