MgNet

The MgNet is an abstract and unified mathematical framework which simultaneously recovers some residual neural network (ResNet) type convolutional neural networks (CNNs) and multigrid methods for solving discretized partial differential equations (PDEs). As a CNN model, MgNet can be obtained by making some very minor modifications of a classic geometric multigrid method. Actually, connections between ResNet and classical multigrid methods were acknowledged in the original paper of ResNet from the viewpoint how residuals are applied in both methods. MgNet makes such a connection more direct and clear, and it makes it possible to directly obtain a class of efficient CNN models by simply making some very minor modification of a typical multigrid cycle but keeping the identically same algorithm structure.

Main structure and connections with ResNet

One core concept in MgNet, motivated by our research in algebraic multigrid methods, is the distinction between the so-called data and feature spaces (that are dual to each other). Based on this new concept, MgNet and a further research (Juncai He; Yuyan Chen; Jinchao Xu (2019). "Constrained Linear Data-feature Mapping for Image Classification". arXiv:1911.10428v1 [eess.IV].CS1 maint: Uses authors parameter (link)) proposes the constrained data-feature mapping model in every grid as

$A\ast u=f,$

where $f$ belongs to the data space and $u$ belongs to the feature space such that

$u\geq 0$.

The feature extraction process can then be obtained through an iterative procedure for solving the above system in each grids. For example, if the single step residual correction scheme is applied for the above system, it becomes

$u^{i}=u^{i-1}+\sigma \circ B^{i}\ast \sigma (f^{}-A\ast u^{i-1}),\quad i=1:\nu ,$

with $u\approx u^{\nu }$.

If the residual of the above iterative $r^{i}=f-A\ast u^{i}$is further considered, it becomes

$r^{i}=r^{i-1}-A\ast \sigma \circ B^{i}\ast \sigma (r^{i-1}),\quad i=1:\nu .$

This is almost the exact basic block scheme in Pre-act ResNet, which has the form

$r^{i}=r^{i-1}-A^{i}\ast \sigma \circ B^{i}\ast \sigma (r^{i-1}),\quad i=1:\nu .$

The next figure shows the pseudocode of MgNet:

One thing important to note is that the special MgNet Algorithm 1 is identical to a multigrid cycle if the boxed nonlinear operations are removed in the algorithm.

Summary

By revealing such a direct connection between CNN and multigrid method, this opens up a new door to the design and study of deep learning models from a more mathematical viewpoint and in particular the rich mathematical techniques developed for multigrid method can be applied in the study of deep learning.

References

This article "MgNet" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:MgNet. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.

📰 Article(s) of the same category(ies)

• Neuton (network)
• Broad Phase
• Kit Yates
• Data Amulet
• Omniscien Technologies
• The Number Decomposing Formula or GOF(Getal Ontledingsformule)
• Center for Mathematical Sciences, Huazhong University of Science and Technology

---