Poisson test case with new API (AmgX_CSR)

[pledac]

Hello @mattmartineau , thanks for your new API,

I am playing with it and running the poisson test case where it is possible to update the coefficients of the matrix and not rebuilding it. But I am surprised that with AMG preconditioner, the setup is done again. Did i miss something ?

Thanks,

./poisson -caseName log -mode AmgX_CSR -cfgFileName ./AmgX_SolverOptions_Classical.info -Nx 10 -Ny 10 -Nz 10
========================================================================

---

Case Name: log
Nx: 10
Ny: 10
Nz: 10
Mode: AmgX_CSR
Config File: ./AmgX_SolverOptions_Classical.info
Number of Solves: 10
Output PETSc Log File ? false

========================================================================
AMGX version 2.1.0.131-opensource
Built on Mar 14 2021, 16:01:18
Compiled with CUDA Runtime 10.2, using CUDA driver 11.0
Cannot read file as JSON object, trying as AMGX config
Cannot read file as JSON object, trying as AMGX config
Converting config string to current config version
Parsing configuration string: exception_handling=1 ;
Using Normal MPI (Hostbuffer) communicator...
AMG Grid:
Number of Levels: 4
LVL ROWS NNZ SPRSTY Mem (GB)
--------------------------------------------------------------
0(D) 1000 6400 0.0064 0.000101
1(D) 500 7760 0.031 0.000193
2(D) 85 4245 0.588 9.82e-05
3(D) 10 100 1 2.55e-06
--------------------------------------------------------------
Grid Complexity: 1.595
Operator Complexity: 2.89141
Total Memory Usage: 0.000395462 GB
--------------------------------------------------------------
iter Mem Usage (GB) residual rate
--------------------------------------------------------------
Ini 0.762817 1.418968e+03
0 0.762817 4.049953e+02 0.2854
1 0.7628 2.272882e+01 0.0561
2 0.7628 2.135450e+00 0.0940
3 0.7628 1.660107e-01 0.0777
4 0.7628 1.927763e-02 0.1161
5 0.7628 2.591171e-03 0.1344
6 0.7628 2.747955e-04 0.1061
7 0.7628 2.890709e-05 0.1052
--------------------------------------------------------------
Total Iterations: 8
Avg Convergence Rate: 0.1093
Final Residual: 2.890709e-05
Total Reduction in Residual: 2.037191e-08
Maximum Memory Usage: 0.763 GB
--------------------------------------------------------------
Total Time: 0.0165809
setup: 0.0125757 s
solve: 0.00400515 s
solve(per iteration): 0.000500644 s
AMG Grid:
Number of Levels: 4
LVL ROWS NNZ SPRSTY Mem (GB)
--------------------------------------------------------------
0(D) 1000 6400 0.0064 0.000101
1(D) 500 7760 0.031 0.000193
2(D) 85 4245 0.588 9.82e-05
3(D) 10 100 1 2.55e-06
--------------------------------------------------------------
Grid Complexity: 1.595
Operator Complexity: 2.89141
Total Memory Usage: 0.000395462 GB
--------------------------------------------------------------
iter Mem Usage (GB) residual rate
--------------------------------------------------------------
Ini 0.762817 2.890709e-05
0 0.762817 4.493219e-06 0.1554
1 0.7628 5.017173e-07 0.1117
2 0.7628 5.233455e-08 0.1043
3 0.7628 5.699205e-09 0.1089
4 0.7628 6.339535e-10 0.1112
5 0.7628 7.636473e-11 0.1205
6 0.7628 8.610483e-12 0.1128
7 0.7628 8.893718e-13 0.1033
--------------------------------------------------------------
Total Iterations: 8
Avg Convergence Rate: 0.1151
Final Residual: 8.893718e-13
Total Reduction in Residual: 3.076656e-08
Maximum Memory Usage: 0.763 GB
--------------------------------------------------------------
Total Time: 0.0161896
setup: 0.0123392 s
solve: 0.00385043 s
solve(per iteration): 0.000481304 s
2-Norm: 0.986988
Max-Norm: 0.0577365
Iterations 9

========================================================================

End of log

========================================================================

[mattmartineau]

In this API we have the "updateA" call which replaces the matrix coefficients and then performs a lighter weight resetup (rather than the full setup). I am working internally to make this resetup even faster so in the future this should represent a substantial improvement in performance in the future.

It is of course possible to construct an API that would allow uploading coefficients without performing the resetup at all, and I have worked on some applications where this can be useful. It tends to be cases where you are willing to sacrifice some accuracy for the sake of avoiding the setup costs for some number of solve steps. Do you have a particular use case where this is applicable?

[pledac]

In this API we have the "updateA" call which replaces the matrix coefficients and then performs a lighter weight resetup (rather than the full setup).

Oh yes, I read it but forgot this... I will check on my largest test case how lighter is the resetup, thanks.

I am working internally to make this resetup even faster so in the future this should represent a substantial improvement in performance in the future.

Great.

It is of course possible to construct an API that would allow uploading coefficients without performing the resetup at all, and I have worked on some applications where this can be useful. It tends to be cases where you are willing to sacrifice some accuracy for the sake of avoiding the setup costs for some number of solve steps. Do you have a particular use case where this is applicable?

PETSc provides such a feature (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetReusePreconditioner.html), and a particular use case may be Newton's method (should test in my code) where keeping the setup of the first iteration could provide benefits despite the loss of some accuracy, as the Jacobian slightly changes.

Thanks for answering about the new CSR API.

[pledac]

I ran poisson test case on v100 card with on a bigger test case with no gain on re-setup unhappily.

./poisson -caseName log -mode AmgX_CSR -cfgFileName ./AmgX_SolverOptions_Classical.info -Nx 100 -Ny 100 -Nz 100

Case Name: AmgX_CSR_100x100x100
Nx: 100
Ny: 100
Nz: 100
Mode: AmgX_CSR
Config File: AmgX_SolverOptions_Classical.info
Number of Solves: 10
Output PETSc Log File ? false

========================================================================
AMGX version 2.1.0.131-opensource
Built on Apr 11 2021, 16:29:16
Compiled with CUDA Runtime 10.2, using CUDA driver 10.2
Cannot read file as JSON object, trying as AMGX config
Cannot read file as JSON object, trying as AMGX config
Converting config string to current config version
Parsing configuration string: exception_handling=1 ;
Using Normal MPI (Hostbuffer) communicator...
AMG Grid:
Number of Levels: 7
LVL ROWS NNZ SPRSTY Mem (GB)
--------------------------------------------------------------
0(D) 1000000 6940000 6.94e-06 0.107
1(D) 500000 9320600 3.73e-05 0.228
2(D) 83338 10681050 0.00154 0.242
3(D) 10337 3323825 0.0311 0.0747
4(D) 782 294852 0.482 0.00662
5(D) 69 4761 1 0.000109
6(D) 6 36 1 9.98e-07
--------------------------------------------------------------
Grid Complexity: 1.59453
Operator Complexity: 4.4042
Total Memory Usage: 0.658982 GB
--------------------------------------------------------------
iter Mem Usage (GB) residual rate
--------------------------------------------------------------
Ini 2.30548 7.468230e+04
0 2.30548 2.586563e+04 0.3463
1 2.3055 5.702930e+03 0.2205
2 2.3055 4.112957e+02 0.0721
3 2.3055 3.034172e+01 0.0738
4 2.3055 3.429337e+00 0.1130
5 2.3055 4.298335e-01 0.1253
6 2.3055 3.945168e-02 0.0918
7 2.3055 4.581627e-03 0.1161
--------------------------------------------------------------
Total Iterations: 8
Avg Convergence Rate: 0.1255
Final Residual: 4.581627e-03
Total Reduction in Residual: 6.134823e-08
Maximum Memory Usage: 2.305 GB
--------------------------------------------------------------
Total Time: 3.58589
setup: 3.56315 s
solve: 0.0227373 s
solve(per iteration): 0.00284217 s
AMG Grid:
Number of Levels: 7
LVL ROWS NNZ SPRSTY Mem (GB)
--------------------------------------------------------------
0(D) 1000000 6940000 6.94e-06 0.107
1(D) 500000 9320600 3.73e-05 0.228
2(D) 83338 10681050 0.00154 0.242
3(D) 10337 3323825 0.0311 0.0747
4(D) 782 294852 0.482 0.00662
5(D) 69 4761 1 0.000109
6(D) 6 36 1 9.98e-07
--------------------------------------------------------------
Grid Complexity: 1.59453
Operator Complexity: 4.4042
Total Memory Usage: 0.658982 GB
--------------------------------------------------------------
iter Mem Usage (GB) residual rate
--------------------------------------------------------------
Ini 2.30548 4.581627e-03
0 2.30548 8.070969e-04 0.1762
1 2.3055 7.323379e-05 0.0907
2 2.3055 8.564718e-06 0.1170
3 2.3055 8.627607e-07 0.1007
4 2.3055 9.406064e-08 0.1090
5 2.3055 1.163520e-08 0.1237
6 2.3055 1.156623e-09 0.0994
7 2.3055 1.194955e-10 0.1033
--------------------------------------------------------------
Total Iterations: 8
Avg Convergence Rate: 0.1127
Final Residual: 1.194955e-10
Total Reduction in Residual: 2.608145e-08
Maximum Memory Usage: 2.305 GB
--------------------------------------------------------------
Total Time: 3.53631
setup: 3.5141 s
solve: 0.022203 s
solve(per iteration): 0.00277538 s
2-Norm: 0.348553
Max-Norm: 0.00065713
Iterations 9

========================================================================

End of AmgX_CSR_100x100x100

========================================================================