mindscience/MindElec/examples/physics_driven/incremental_learning

• Table of Contents
• • Maxwell's Equations
  • Solving Point Source Maxwell's Equations with AI
  • Datasets
  • Environmental Requirements
  • Script Description
  • • Script and Sample Code
    • Script Parameters
  • Model Description
  • Model Pre-training
  • Pre-training Performance and Accuracy
  • Solving New PDE with Fine-tuning
  • Fine-tuning Performance and Accuracy
  • Random Seed Setting
  • MindScience Home Page

Table of Contents

• Table of Contents
  • Maxwell's Equations
  • Solving Point Source Maxwell's Equations with AI
  • Datasets
  • Environmental Requirements
  • Script Description
    • Script and Sample Code
    • Script Parameters
  • Model Description
  • Model Pre-training
  • Pre-training Performance and Accuracy
  • Solving New PDE with Fine-tuning
  • Fine-tuning Performance and Accuracy
  • Random Seed Setting
  • MindScience Home Page

Maxwell's Equations

Maxwell's equations are a set of partial differential equations that describe how electric and magnetic fields are generated by charges and currents of the fields. The governing equations with excitation sources are described as follows:

$$
\nabla\times E=-\mu \dfrac{\partial H}{\partial t} + J(x, t),
$$
$$
\nabla\times H=\epsilon \dfrac{\partial E}{\partial t}
$$

The constants $\mu$ and $\epsilon$ are known as the permeability and permittivity of the free space, respectively.
$J(x, t)$ represents a known excitation source function. Common excitation sources include point source, line source, and area source. In this tutorial, point source is considered, of which mathematical description can be expressed as follows:

$$
J(x, t)=\delta(x - x_0)g(t)
$$

Solving Point Source Maxwell's Equations with AI

In the PINNs method, the neural network directly learns the mapping relationship between the solution function of the equation system and the independent variables of the equation. When the parameters of the equations change, the new equations need to be retrained. Therefore, the PINNs method does not have the generalization ability to solve the parametric PDEs.

In this model, we propose the Physics-Informed Auto-Decoder to overcome the drawbacks of the PINNs method. The method comprises two steps: pre-training and fine-tuning. The overall network architecture for AI solution to point source Maxwell equations is as follows:

In the preceding information, $\lambda$ is a parameter of the equation. Generally, the distribution of the variable parameter $\lambda$ forms a high-dimensional space. To reduce the model complexity and training costs, we first map the high-dimensional variable parameter space onto a low-dimensional manifold represented by a low-dimensional latent vector $Z$. Then, the characteristic parameter $Z$ of the manifold and the input $X$ of the equation are fused into the training of PINNs together with the input of the point source problem solving network, and thus the pre-training model can be obtained. For the newly given PDE, the solution of the new equation can be obtained by fine-tuning the pre-trained model.
According to our experience, in the fine-tuning phase, incremental training is performed on the network model and the randomly initialized implicit vector at the same time. Alternatively, only incremental training is performed on the implicit vectors by freezing the parameters of the network model. Compared with the PINNs method, the proposed method improves the efficiency of solving new problems by one order of magnitude.

Taking two-dimensional point source Maxwell equations as an example, the number of pre-trained equations is 9 $[\mu/\mu_0, \epsilon/\epsilon_0] = [1,3,5] \times [1,3,5]$.
9 latent vector $Z$ concatenate with the independent variable $\Omega=(x, y, t)\in [0,1]^3$ as network input.
The output is the solution of the equation $u=(E_x, E_y, H_z)$.
Based on the output of the network and the automatic differentiation function of MindSpore framework, the training loss function comes from four parts: control equation (PDE loss), initial condition (IC loss), boundary condition (BC loss), and regular term (reg loss) for latent vector.
The initial electromagnetic field is set to be zeros, and the strandard Mur's second-order absorbing boundary condition is utilized on the boundaries.
Due to the existence of the excitation source, we divide the PDE loss into two parts: the region near the excitation source $\Omega_0$ and the region without the excitation source $\Omega_1$.
Ultimately, our overall loss function can be expressed as follows:
$$L_{total} = \lambda_{src}L_{src} + \lambda_{src_ic}L_{src_ic} + \lambda_{no_src}L_{no_src} + \lambda_{no_src_ic}L_{no_src_ic} + \lambda_{bc}L_{bc} + \lambda_{reg}\left | Z \right | ^2 $$
In the preceding information, $\lambda$s indicates the weight of each loss function. In order to reduce the difficulty of weight selection, we adopt the adaptive weighting algorithm.

Datasets

• Pre-training data and fine-tuning data: Based on five loss functions, we randomly pick points for the source region, no source region, boundary, and initial condition, respectively, as the input of the network.
• Evaluation data: We generate high-precision electromagnetic fields based on traditional finite-difference time-domain algorithms.
  • Note: Data is processed in src/dataset.py.

Environmental Requirements

• Hardware (Ascend)
  • Prepare the Ascend AI Processor to set up the hardware environment.
• Framework
  • MindSpore Elec
• For more information, see the following resources:
  • MindSpore Elec Tutorial
  • MindSpore Elec Python API

Script Description

Script and Sample Code

.
└─Maxwell
  ├─README.md
  ├─docs                              # schematic diagram of README
  ├─src
    ├──dataset.py                     # dataset
    ├──maxwell.py                     # definition of point source Maxwell's equations
    ├──lr_scheduler.py                # lr scheduler
    ├──callback.py                    # callback function
    ├──sampling_config.py             # sampling configuration
    ├──utils.py                       # utility function
  ├─config
    ├──pretrain.json                  # pre-training configuration
    ├──reconstruct.json               # fine-tuning configuration
  ├──mad.py                          # run MAD

Script Parameters

The dataset sampling control parameters are set in the src/sampling_config.py file as follows:

src_sampling_config = edict({         # src sampling configuration
    'domain': edict({                 # src domain
        'random_sampling': True,      # random or not
        'size': 262144,               # total number of points in one epoch
        'sampler': 'uniform'          # random sampling method
    }),
    'IC': edict({                     # initial condition samlping configuration
        'random_sampling': True,      # random or not
        'size': 262144,               # total number of points in one epoch
        'sampler': 'uniform',         # random sampling method
    }),
    'time': edict({                   # time sampling configuration
        'random_sampling': True,      # random or not
        'size': 262144,               # total number of points in one epoch
        'sampler': 'uniform',         # random sampling method
    }),
})

no_src_sampling_config = edict({      # no src sampling configuration
    'domain': edict({                 # no src domain
        'random_sampling': True,      # random or not
        'size': 262144,               # total number of points in one epoch
        'sampler': 'uniform'          # random sampling method
    }),
    'IC': edict({                     # initial condition samlping configuration
        'random_sampling': True,      # random or not
        'size': 262144,               # total number of points in one epoch
        'sampler': 'uniform',         # random sampling method
    }),
    'time': edict({                   # time sampling configuration
        'random_sampling': True,      # random or not
        'size': 262144,               # total number of points in one epoch
        'sampler': 'uniform',         # random sampling method
    }),
})

bc_sampling_config = edict({          # boundary condition samlping configuration
    'BC': edict({                     # boundary condition spatial samlping configuration
        'random_sampling': True,      # random or not
        'size': 262144,               # total number of points in one epoch
        'sampler': 'uniform',         # random sampling method
        'with_normal': False          # return normal direction or not
    }),
    'time': edict({                   # time sampling configuration
        'random_sampling': True,      # random or not
        'size': 262144,               # total number of points in one epoch
        'sampler': 'uniform',         # random sampling method
    }),
})

The pre-model training and control parameters are configured in the config/pretrain.json file as follows:

{
    "Description" : ["PINNs for solve Maxwell's equations"],                # Case description
    "Case" : "2D_Mur_Src_Gauss_Mscale_MTL_Physical-Informed Auto Decoder",  # Case name
    "coord_min" : [0.0, 0.0],                                               # Minimum coordinates of the x and y axes of the rectangular computing domain
    "coord_max" : [1.0, 1.0],                                               # Maximum coordinates of the x and y axes of the rectangular computing domain
    "src_pos" : [0.4975, 0.4975],                                           # Point source location coordinates
    "SrcFrq": 1e+9,                                                         # Excitation source main frequency
    "range_t" : 4e-9,                                                       # Simulation Duration
    "input_center": [0.5, 0.5, 2.0e-9],                                     # Network input coordinate translation distance
    "input_scale": [2.0, 2.0, 5.0e+8],                                      # Scaling coefficient of network input coordinates
    "output_scale": [37.67303, 37.67303, 0.1],                              # Scaling coefficient of network output physical quantity
    "src_radius": 0.01,                                                     # Radius of the point source range after Gaussian smoothing
    "input_size" : 3,                                                       # Network input dimension
    "output_size" : 3,                                                      # Network output dimension
    "residual" : true,                                                      # Including ResBlock or not
    "num_scales" : 4,                                                       # Number of subnets in a multi-channel network
    "layers" : 7,                                                           # Number of fully-connected network layers (input, output, and hidden layers)
    "neurons" : 64,                                                         # Number of neurons
    "amp_factor" : 2,                                                       # Amplification factor of network input
    "scale_factor" : 2,                                                     # Subnet Amplification Coefficients
    "save_ckpt" : true,                                                     # save ckpt or not
    "load_ckpt" : false,                                                    # load ckpt or not
    "save_ckpt_path" : "./ckpt",                                            # save ckpt path
    "load_ckpt_path" : "",                                                  # load ckpt path
    "train_with_eval": false,                                               # train with eval
    "test_data_path" : "",                                                  # benchmark path
    "lr" : 0.001,                                                           # lr
    "milestones" : [2000],                                                  # lr milestones
    "lr_gamma" : 0.25,                                                      # lr decay rate
    "train_epoch" : 3000,                                                   # number of iterations
    "train_batch_size" : 1024,                                              # train batch size
    "test_batch_size" : 8192,                                               # test batch size
    "predict_interval" : 500,                                               # prediction interval during training
    "vision_path" : "./vision",                                             # result visualization path
    "summary_path" : "./summary",                                           # mindinsight summary path

    "EPS_candidates": [1, 3, 5],                                            # eps candidates
    "MU_candidates": [1, 3, 5],                                             # mu candidates
    "num_scenarios": 9,                                                     # number of pretrained scenarios
    "latent_vector_size": 16,                                               # latent vector size
    "latent_reg": 1.0,                                                      # latent vector regularization
    "latent_init_std": 1.0                                                  # std of init latent vector
}

The incremental training fine-tuning and control parameters are configured in the config/pretrain.json file as follows:

{
    "Description" : ["PINNs for solve Maxwell's equations"],                # Case description
    "Case" : "2D_Mur_Src_Gauss_Mscale_MTL_Physical-Informed Auto Decoder",  # Case name
    "coord_min" : [0, 0],                                                   # Minimum coordinates of the x and y axes of the rectangular computing domain
    "coord_max" : [1, 1],                                                   # Maximum coordinates of the x and y axes of the rectangular computing domain
    "src_pos" : [0.4975, 0.4975],                                           # Point source location coordinates
    "SrcFrq": 1e+9,                                                         # Excitation source main frequency
    "range_t" : 4e-9,                                                       # Simulation Duration
    "input_center": [0.5, 0.5, 2.0e-9],                                     # Network input coordinate translation distance
    "input_scale": [2.0, 2.0, 5.0e+8],                                      # Scaling coefficient of network input coordinates
    "output_scale": [37.67303, 37.67303, 0.1],                              # Scaling coefficient of network output physical quantity
    "src_radius": 0.01,                                                     # Radius of the point source range after Gaussian smoothing
    "input_size" : 3,                                                       # Network input dimension
    "output_size" : 3,                                                      # Network output dimension
    "residual" : true,                                                      # Including ResBlock or not
    "num_scales" : 4,                                                       # Number of subnets in a multi-channel network
    "layers" : 7,                                                           # Number of fully-connected network layers (input, output, and hidden layers)
    "neurons" : 64,                                                         # Number of neurons
    "amp_factor" : 2,                                                       # Amplification factor of network input
    "scale_factor" : 2,                                                     # Subnet Amplification Coefficients
    "save_ckpt" : true,                                                     # save ckpt or not
    "save_ckpt_path" : "./ckpt",                                            # save ckpt path
    "load_ckpt_path" : "",                                                  # load ckpt path
    "train_with_eval": true,                                                # train with eval
    "test_data_path" : "./benchmark/",                                      # benchmark path
    "lr" : 0.001,                                                           # lr
    "milestones" : [100],                                                   # lr milestones
    "lr_gamma" : 0.1,                                                       # lr decay rate
    "train_epoch" : 120,                                                    # train epochs
    "train_batch_size" : 8192,                                              # train batch size
    "test_batch_size" : 8192,                                               # test batch size
    "predict_interval" : 10,                                                # prediction intervals during training
    "vision_path" : "./vision",                                             # result visualization path
    "summary_path" : "./summary",                                           # mindinsight summary path

    "EPS_candidates": [2],                                                  # eps
    "MU_candidates": [2],                                                   # mu
    "num_scenarios": 1,                                                     # fine-tuning case
    "latent_vector_size": 16,                                               # latent vector size
    "latent_reg": 1.0,                                                      # latent regulzarization
    "latent_init_std": 1.0                                                  # std of latent init
    "finetune_model": true                                                  # fine-tune model with latent vector
    "enable_mtl" : true,                                                    # use uncertainty weighting algorithm or not
}

Model Description

In this tutorial, the network architecture of multi-channel residual network combined with Sin activation function is used.

Model Pre-training

You can use the mad.py script to train a parametric electromagnetic simulation model. During the training, the model parameters are automatically saved.

python mad.py --mode=pretrain

Pre-training Performance and Accuracy

The script provides the function of evaluation while training. The loss function, performance data, and precision evaluation result of network training are as follows:

epoch: 1 step: 28, loss is 4.332097
epoch time: 198849.624 ms, per step time: 7101.772 ms
epoch: 2 step: 28, loss is 4.150775
epoch time: 2697.448 ms, per step time: 96.337 ms
epoch: 3 step: 28, loss is 4.062408
epoch time: 2697.567 ms, per step time: 96.342 ms
epoch: 4 step: 28, loss is 3.942519
epoch time: 2695.202 ms, per step time: 96.257 ms
epoch: 5 step: 28, loss is 3.7573988
epoch time: 2684.708 ms, per step time: 95.882 ms
epoch: 6 step: 28, loss is 3.6898723
epoch time: 2688.102 ms, per step time: 96.004 ms
epoch: 7 step: 28, loss is 3.5447907
epoch time: 2685.811 ms, per step time: 95.922 ms
epoch: 8 step: 28, loss is 3.462135
epoch time: 2686.360 ms, per step time: 95.941 ms
epoch: 9 step: 28, loss is 3.33505
epoch time: 2678.746 ms, per step time: 95.669 ms
epoch: 10 step: 28, loss is 3.227073
epoch time: 2684.541 ms, per step time: 95.876 ms
......
epoch time: 2681.309 ms, per step time: 95.761 ms
epoch: 2991 step: 28, loss is 0.062745415
epoch time: 2680.290 ms, per step time: 95.725 ms
epoch: 2992 step: 28, loss is 0.0728458
epoch time: 2677.612 ms, per step time: 95.629 ms
epoch: 2993 step: 28, loss is 0.103519976
epoch time: 2680.193 ms, per step time: 95.721 ms
epoch: 2994 step: 28, loss is 0.086214304
epoch time: 2680.334 ms, per step time: 95.726 ms
epoch: 2995 step: 28, loss is 0.064058885
epoch time: 2681.828 ms, per step time: 95.780 ms
epoch: 2996 step: 28, loss is 0.06439945
epoch time: 2687.958 ms, per step time: 95.998 ms
epoch: 2997 step: 28, loss is 0.06600211
epoch time: 2684.817 ms, per step time: 95.886 ms
epoch: 2998 step: 28, loss is 0.08579312
epoch time: 2676.644 ms, per step time: 95.594 ms
epoch: 2999 step: 28, loss is 0.07677732
epoch time: 2677.484 ms, per step time: 95.624 ms
epoch: 3000 step: 28, loss is 0.061393328
epoch time: 2691.284 ms, per step time: 96.117 ms
==========================================================================================
l2_error, Ex:  0.06892983792636541 , Ey:  0.06803824510149464 , Hz:  0.07061244131423149
==========================================================================================

Solving New PDE with Fine-tuning

Given a new set of equation parameters, you can use the mad.py script to load the pre-trained model and fine-tune the test dataset to quickly obtain a solution to the new problem.

python mad.py --mode=reconstruct

Fine-tuning Performance and Accuracy

epoch: 1 step: 32, loss is 3.4485734
epoch time: 207.005 s, per step time: 6468.899 ms
epoch: 2 step: 32, loss is 3.2356246
epoch time: 2.859 s, per step time: 89.334 ms
epoch: 3 step: 32, loss is 3.0757806
epoch time: 2.873 s, per step time: 89.787 ms
epoch: 4 step: 32, loss is 2.9055781
epoch time: 2.864 s, per step time: 89.515 ms
epoch: 5 step: 32, loss is 2.7547212
epoch time: 2.865 s, per step time: 89.530 ms
epoch: 6 step: 32, loss is 2.5957384
epoch time: 2.849 s, per step time: 89.040 ms
epoch: 7 step: 32, loss is 2.4329371
epoch time: 2.848 s, per step time: 88.992 ms
epoch: 8 step: 32, loss is 2.3149633
epoch time: 2.845 s, per step time: 88.894 ms
epoch: 9 step: 32, loss is 2.153984
epoch time: 2.855 s, per step time: 89.222 ms
epoch: 10 step: 32, loss is 2.0002615
epoch time: 2.855 s, per step time: 89.224 ms
......
epoch: 120 step: 32, loss is 0.03491814
epoch time: 2.847 s, per step time: 88.966 ms
epoch: 121 step: 32, loss is 0.034486588
epoch time: 2.856 s, per step time: 89.254 ms
epoch: 122 step: 32, loss is 0.033784416
epoch time: 2.849 s, per step time: 89.044 ms
epoch: 123 step: 32, loss is 0.03252008
epoch time: 2.869 s, per step time: 89.657 ms
epoch: 124 step: 32, loss is 0.031876195
epoch time: 2.860 s, per step time: 89.364 ms
epoch: 125 step: 32, loss is 0.03133876
epoch time: 2.852 s, per step time: 89.128 ms
==================================================================================================
Prediction total time: 53.52936124801636 s
l2_error, Ex:  0.06008509896061373 , Ey:  0.06597097288551895 , Hz:  0.059188475323901625
==================================================================================================

Random Seed Setting

The random seed is set in mad.py, and the network input is randomly sampled through uniform distribution.

MindScience Home Page

Visit the official website home page.