神经网络架构搜索（NAS）进阶指南
引言
神经网络架构搜索（Neural Architecture Search, NAS）旨在自动化设计神经网络架构，是AutoML的核心技术之一。
NAS搜索空间
定义搜索空间
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
| class NASearchSpace:
"""NAS搜索空间定义"""
def __init__(self):
self.operations = [
'conv_3x3',
'conv_5x5',
'max_pool_3x3',
'avg_pool_3x3',
'skip_connect',
'sep_conv_3x3',
'sep_conv_5x5',
'attention'
]
self.depth_range = (4, 12)
self.width_range = (0.5, 1.5)
self.resolution_range = (224, 384)
def sample_architecture(self):
depth = random.randint(*self.depth_range)
ops = random.choices(self.operations, k=depth)
return Architecture(ops=ops)
|
搜索策略
强化学习方法
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
| class RLNASAgent:
"""基于强化学习的NAS代理"""
def __init__(self, controller):
self.controller = controller
self.baseline = 0
def search(self, n_trials=1000):
for trial in range(n_trials):
arch = self.controller.sample()
accuracy = self.train_and_evaluate(arch)
reward = accuracy - self.baseline
self.controller.update(reward)
self.baseline = 0.9 * self.baseline + 0.1 * accuracy
return self.controller.get_best_arch()
|
DARTS：可微分搜索
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
| class DARTS:
"""可微分架构搜索"""
def __init__(self):
self.alpha = nn.Parameter(torch.randn(num_nodes, num_ops))
self.beta = nn.Parameter(torch.randn(num_nodes, num_ops))
def forward(self, x):
for i in range(num_nodes):
for j in range(i):
op = self.mixed_op(x[j], self.alpha[i][j])
x[i] = x[i] + op
return x
def mixed_op(self, x, alpha):
return sum(alpha[k] * ops[k](x) for k, op in enumerate(ops))
def search(self, epochs=50):
optimizer = torch.optim.Adam([self.alpha, self.beta], lr=3e-4)
for epoch in range(epochs):
optimizer.zero_grad()
loss = self.compute_loss()
loss.backward()
optimizer.step()
return self.get_best_architecture()
|
效率优化
权重共享
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| class WeightSharingSuperNet(nn.Module):
"""权重共享超网络"""
def __init__(self):
self.shared_weights = nn.ParameterDict()
self._build_shared_weights()
def _build_shared_weights(self):
self.shared_weights['conv'] = nn.Parameter(torch.randn(64, 64, 3, 3))
self.shared_weights['bn'] = nn.Parameter(torch.randn(64))
def forward(self, x, arch):
return self._execute_path(x, arch)
|
应用案例
NASNet搜索空间
| 搜索维度 |
搜索范围 |
| 卷积核大小 |
3x3, 5x5, 7x7 |
| 特征图通道 |
16-512 |
| 跳跃连接 |
是/否 |
| 重复次数 |
4-12 |
总结
NAS技术实现了神经网络设计的自动化，大幅提升了设计效率。
推荐阅读：《Neural Architecture Search》综述论文
