jiaqi
Loading Heatmap…

jiaqi pushed to master at qinsh/ResnetCifar

11 months ago

jiaqi pushed to master at qinsh/ResnetCifar

11 months ago

jiaqi pushed to master at qinsh/ResnetCifar

11 months ago

jiaqi pushed to master at qinsh/ResnetCifar

11 months ago

jiaqi pushed to master at qinsh/ResnetCifar

11 months ago

jiaqi pushed to master at qinsh/ResnetCifar

11 months ago

jiaqi pushed to master at qinsh/ResnetCifar

11 months ago

jiaqi pushed to master at qinsh/ResnetCifar

11 months ago

jiaqi commented on issue PCL-Platform.Inte.../AISynergy#5

添加模型参数稀疏化算法FedDropout,用于通信优化

后续相关进展内容转移至[issue 21](https://git.openi.org.cn/PCL-Platform.Intelligence/AISynergy/issues/21)

1 year ago

jiaqi closed issue PCL-Platform.Inte.../AISynergy#5

添加模型参数稀疏化算法FedDropout,用于通信优化

1 year ago

jiaqi commented on issue PCL-Platform.Inte.../AISynergy#21

基于参数稀疏化的个性化联邦算法

##### 研究进展 * 基于前期Fed-Dropout算法实验设计改进思路,融合AFD算法及Head Importance评估设计提出新算法; * 研究学习Freeze-Thaw, 个性化联邦等相关算法,改进模型参数稀疏化筛选过程; * 基于BERT模型进行前期调研实验。 ##### 当前问题 * 稀疏化模型参数筛选方法优化; * 稀疏化模型评估效率问题; * Server端维护模型规模及计算效率优化;

1 year ago

jiaqi opened issue PCL-Platform.Inte.../AISynergy#21

基于参数稀疏化的个性化联邦算法

1 year ago

jiaqi commented on issue PCL-Platform.Inte.../AISynergy#5

添加模型参数稀疏化算法FedDropout,用于通信优化

##### Head-Importance重要性剪枝: ###### 算法思路: 为避免随机选取子网络导致的收敛问题,采用计算网络头结构重要性的方法,评估模型结构分数,选取重要部分进行传输更新,以提高收敛速度和性能效率。 ###### 实验结果: 在Bert模型上实验根据重要性排序对头结构进行剪枝的效果 ``` Total acc: 0.835048395313296 18:05:19-INFO: Evaluating following pruning strategy 18:05:19-INFO: 12:12,8 11:2,6,7 8:8 6:2 18:06:22-INFO: ***** Running evaluation ***** 18:06:22-INFO: Num examples = 9815 18:06:22-INFO: Batch size = 32 18:06:22-INFO: ***** Pruning eval results ***** 18:06:22-INFO: 7 0.834538970962812 18:06:22-INFO: Evaluating following pruning strategy 18:06:22-INFO: 12:4,12,6,8 11:2,6,7 8:12,8 6:2 7:4 10:10,2 9:7 18:07:24-INFO: ***** Pruning eval results ***** 18:07:24-INFO: 14 0.8370860927152318 18:08:30-INFO: ***** Pruning eval results ***** 18:08:30-INFO: 21 0.8351502801833928 18:09:42-INFO: ***** Pruning eval results ***** 18:09:42-INFO: 28 0.8366785532348446 18:10:46-INFO: ***** Pruning eval results ***** 18:10:46-INFO: 36 0.8336220071319409 18:11:49-INFO: ***** Pruning eval results ***** 18:11:49-INFO: 43 0.8313805399898115 18:12:52-INFO: ***** Pruning eval results ***** 18:12:52-INFO: 50 0.8299541518084564 18:13:56-INFO: ***** Pruning eval results ***** 18:13:56-INFO: 57 0.8254712175241976 18:15:00-INFO: ***** Pruning eval results ***** 18:15:00-INFO: 64 0.8249617931737137 18:16:03-INFO: ***** Pruning eval results ***** 18:16:03-INFO: 72 0.8196637799286806 18:17:07-INFO: ***** Pruning eval results ***** 18:17:07-INFO: 79 0.8053998981151299 18:18:11-INFO: ***** Pruning eval results ***** 18:18:11-INFO: 86 0.7944982170147733 18:19:15-INFO: ***** Pruning eval results ***** 18:19:15-INFO: 93 0.7757514009169638 18:20:20-INFO: ***** Pruning eval results ***** 18:20:20-INFO: 100 0.7506877228731533 18:21:24-INFO: ***** Pruning eval results ***** 18:21:24-INFO: 108 0.6330106979113601 18:22:28-INFO: ***** Pruning eval results ***** 18:22:28-INFO: 115 0.5125827814569537 18:23:33-INFO: ***** Pruning eval results ***** 18:23:33-INFO: 122 0.5014773306164034 18:24:37-INFO: ***** Pruning eval results ***** 18:24:37-INFO: 129 0.3681100356597045 18:25:42-INFO: ***** Pruning eval results ***** 18:25:42-INFO: 136 0.36566479877738156 18:26:46-INFO: ***** Pruning eval results ***** 18:26:46-INFO: 144 0.31818644931227713 ``` 剪枝比例与精度损失结果:

1 year ago

jiaqi commented on issue PCL-Platform.Inte.../AISynergy#5

添加模型参数稀疏化算法FedDropout,用于通信优化

##### Fed-Dropout间隔轮数更新: ###### 改进策略: 间隔一定轮数发送Server端全部数据给Client进行一轮训练,同步Clients参数更新方向。 ###### 实验结果: Client 1: ![Result-1](https://git.openi.org.cn/PCL-Platform.Intelligence/AISynergy/src/branch/fedDropout/examples/Fed-MAE/mae/output/server-log-1.png) Client 2: ![Result-2](https://git.openi.org.cn/PCL-Platform.Intelligence/AISynergy/src/branch/fedDropout/examples/Fed-MAE/mae/output/server-log-2.png) ###### 结果分析 采用间隔轮数更新全局模型的方法可以提高Client端模型收敛精度,但收敛过程模型精度波动较大,且与Client本地数据分布有关。

1 year ago

jiaqi pushed to fedDropout at PCL-Platform.Inte.../AISynergy

  • bc0807f899 上传文件至 'examples/Fed-MAE/mae/output'

1 year ago

jiaqi commented on issue PCL-Platform.Inte.../AISynergy#5

添加模型参数稀疏化算法FedDropout,用于通信优化

##### Fed-Dropout测试实验: ###### 实验设置: Data Size:Client1: Training: 1750 Testing: 750; Client2: Training: 1712 Testing: 734 Dropout Rate: 20% Model:MAE-base ###### 实验结果 Figure 1: Fed-Drop实验结果,train loss为Client端使用Dropout后小模型训练每个epoch结果,test loss为Client利用本地数据对合并后完整Server模型进行测试结果 ![Result of Fed-Dropout](https://git.openi.org.cn/PCL-Platform.Intelligence/AISynergy/src/branch/fedDropout/examples/Fed-MAE/mae/output/Fed-Dropout.png) Figure 2:不使用Fed-Dropout聚合方式,Fed-Avg聚合的baseline对比结果 ![Result of Fed-Avg](https://git.openi.org.cn/PCL-Platform.Intelligence/AISynergy/src/branch/fedDropout/examples/Fed-MAE/mae/output/Fed-Avg.png) ###### 结果分析 Fed-Dropout聚合过程中,Client端训练精度稳定收敛,但Server端模型loss随着训练轮数的增加先降低后增长,并最终收敛至0.6左右,相比使用Fed-Avg策略的对比结果大幅下降。 ###### 改进计划 1. 尝试间隔一定轮数在Client端进行一次完整模型的更新同步; 2. 尝试不同的参数筛选策略,e.g. Head Importance; 3. 尝试从流水线并行等其他维度讨论参数稀疏化策略。

1 year ago

jiaqi pushed to fedDropout at PCL-Platform.Inte.../AISynergy

  • 151070ab63 上传文件至 'examples/Fed-MAE/mae/output'

1 year ago

jiaqi commented on issue PCL-Platform.Inte.../AISynergy#5

添加模型参数稀疏化算法FedDropout,用于通信优化

[MAE with FedDropout Readme](https://git.openi.org.cn/PCL-Platform.Intelligence/AISynergy/src/branch/fedDropout/examples/Fed-MAE/mae)

1 year ago

jiaqi pushed to fedDropout at PCL-Platform.Inte.../AISynergy

1 year ago

jiaqi pushed to fedDropout at PCL-Platform.Inte.../AISynergy

1 year ago