Resolved: Why are there extra equations when using extra neurons in GEKKO?

In this post, we will see how to resolve Why are there extra equations when using extra neurons in GEKKO?

Question:

I’m a bit puzzled on why there are extra equations introduced in the optimization problem (using GEKKO) when increasing the amount of neurons in an ANN that e.g. is used within the objective function or in the constraints. I was hoping the find the answer in this paper, but I can’t seem to pinpoint the reason.
This is the log of a baseline example I made, using 2 Gekko_NN_SKlearn functions.

 ----------------------------------------------------------------
 APMonitor, Version 1.0.0
 APMonitor Optimization Suite
 ----------------------------------------------------------------
 
 Called files( 55 )
 files: overrides.dbs does not exist
 
 Run id   : 2022y11m03d15h47m50.642s
 
 COMMAND LINE ARGUMENTS
 coldstart:  0
 imode    :  3
 dbs_read :  T
 dbs_write:  T
 specs    :  T
 
 rto selected
 Called files( 35 )
 READ info FILE FOR VARIABLE DEFINITION: gk_model0.info
 SS MODEL INIT  0
 Parsing model file gk_model0.apm
 Read model file (sec): 1.1181
 Initialize constants (sec): 0.
 Determine model size (sec): 0.7627999999999999
 Allocate memory (sec): 0.0049000000000001265
 Parse and store model (sec): 0.7630000000000001
 
 --------- APM Model Size ------------
 Each time step contains
   Objects      :  247
   Constants    :  0
   Variables    :  752
   Intermediates:  249
   Connections  :  741
   Equations    :  745
   Residuals    :  496
 
 Error checking (sec): 0.2740999999999998
 Compile equations (sec): 2.8513
 Check for uninitialized intermediates (sec): 0.
 ------------------------------------------------------
 Total Parse Time (sec): 5.7744
 SS MODEL INIT  1
 SS MODEL INIT  2
 SS MODEL INIT  3
 SS MODEL INIT  4
 Called files( 31 )
 READ info FILE FOR PROBLEM DEFINITION: gk_model0.info
 Called files( 6 )
 Files(6): File Read rto.t0 F
 files: rto.t0 does not exist
 Called files( 51 )
 Read DBS File defaults.dbs
 files: defaults.dbs does not exist
 Called files( 51 )
 Read DBS File gk_model0.dbs
 files: gk_model0.dbs does not exist
 Called files( 51 )
 Read DBS File measurements.dbs
 Called files( 51 )
 Read DBS File overrides.dbs
 files: overrides.dbs does not exist
 Number of state variables:    1240
 Number of total equations: -  989
 Number of slack variables: -  0
 ---------------------------------------
 Degrees of freedom       :    251
 
 ----------------------------------------------
 Steady State Optimization with APOPT Solver
 ----------------------------------------------
Iter:     1 I:  0 Tm:      2.23 NLPi:   67 Dpth:    0 Lvs:    3 Obj:  6.96E-02 Gap:       NaN
--Integer Solution:   1.78E-01 Lowest Leaf:   6.96E-02 Gap:   1.08E-01
Iter:     2 I:  0 Tm:      0.13 NLPi:    6 Dpth:    1 Lvs:    2 Obj:  1.78E-01 Gap:  1.08E-01
Iter:     3 I:  0 Tm:      0.33 NLPi:    5 Dpth:    1 Lvs:    2 Obj:  1.74E-01 Gap:  1.08E-01
Iter:     4 I:  0 Tm:      0.52 NLPi:   11 Dpth:    1 Lvs:    3 Obj:  9.49E-02 Gap:  1.08E-01
--Integer Solution:   1.78E-01 Lowest Leaf:   9.49E-02 Gap:   8.27E-02
Iter:     5 I:  0 Tm:      0.28 NLPi:    5 Dpth:    2 Lvs:    2 Obj:  1.04E+00 Gap:  8.27E-02
--Integer Solution:   1.23E-01 Lowest Leaf:   1.23E-01 Gap:   0.00E+00
Iter:     6 I:  0 Tm:      0.30 NLPi:    5 Dpth:    2 Lvs:    2 Obj:  1.23E-01 Gap:  0.00E+00
 Successful solution
 
 ---------------------------------------------------
 Solver         :  APOPT (v1.0)
 Solution time  :  3.842599999999999 sec
 Objective      :  0.12267384658102941
 Successful solution
 ---------------------------------------------------
 
 Called files( 2 )
 Called files( 52 )
 WRITE dbs FILE
 Called files( 56 )
 WRITE json FILE
Timer #     1      10.41/       1 =      10.41 Total system time
Timer #     2       3.84/       1 =       3.84 Total solve time
Timer #     3       0.02/     191 =       0.00 Objective Calc: apm_p
Timer #     4       0.01/      99 =       0.00 Objective Grad: apm_g
Timer #     5       0.03/     191 =       0.00 Constraint Calc: apm_c
Timer #     6       0.00/       0 =       0.00 Sparsity: apm_s
Timer #     7       0.00/       0 =       0.00 1st Deriv #1: apm_a1
Timer #     8       0.00/      99 =       0.00 1st Deriv #2: apm_a2
Timer #     9       0.67/       1 =       0.67 Custom Init: apm_custom_init
Timer #    10       0.01/       1 =       0.01 Mode: apm_node_res::case 0
Timer #    11       0.00/       1 =       0.00 Mode: apm_node_res::case 1
Timer #    12       0.02/       1 =       0.02 Mode: apm_node_res::case 2
Timer #    13       0.00/       1 =       0.00 Mode: apm_node_res::case 3
Timer #    14       0.35/     387 =       0.00 Mode: apm_node_res::case 4
Timer #    15       1.51/     198 =       0.01 Mode: apm_node_res::case 5
Timer #    16       0.00/       0 =       0.00 Mode: apm_node_res::case 6
Timer #    17       0.01/      99 =       0.00 Base 1st Deriv: apm_jacobian
Timer #    18       0.00/      99 =       0.00 Base 1st Deriv: apm_condensed_jacobian
Timer #    19       0.00/       1 =       0.00 Non-zeros: apm_nnz
Timer #    20       0.00/       0 =       0.00 Count: Division by zero
Timer #    21       0.00/       0 =       0.00 Count: Argument of LOG10 negative
Timer #    22       0.00/       0 =       0.00 Count: Argument of LOG negative
Timer #    23       0.00/       0 =       0.00 Count: Argument of SQRT negative
Timer #    24       0.00/       0 =       0.00 Count: Argument of ASIN illegal
Timer #    25       0.00/       0 =       0.00 Count: Argument of ACOS illegal
Timer #    26       0.00/       1 =       0.00 Extract sparsity: apm_sparsity
Timer #    27       0.00/      13 =       0.00 Variable ordering: apm_var_order
Timer #    28       0.00/       1 =       0.00 Condensed sparsity
Timer #    29       0.00/       0 =       0.00 Hessian Non-zeros
Timer #    30       0.00/       1 =       0.00 Differentials
Timer #    31       0.00/       0 =       0.00 Hessian Calculation
Timer #    32       0.00/       0 =       0.00 Extract Hessian
Timer #    33       0.00/       1 =       0.00 Base 1st Deriv: apm_jac_order
Timer #    34       0.02/       1 =       0.02 Solver Setup
Timer #    35       1.87/       1 =       1.87 Solver Solution
Timer #    36       0.00/     202 =       0.00 Number of Variables
Timer #    37       0.00/     105 =       0.00 Number of Equations
Timer #    38       0.02/      14 =       0.00 File Read/Write
Timer #    39       0.00/       0 =       0.00 Dynamic Init A
Timer #    40       0.00/       0 =       0.00 Dynamic Init B
Timer #    41       0.00/       0 =       0.00 Dynamic Init C
Timer #    42       1.12/       1 =       1.12 Init: Read APM File
Timer #    43       0.00/       1 =       0.00 Init: Parse Constants
Timer #    44       0.76/       1 =       0.76 Init: Model Sizing
Timer #    45       0.00/       1 =       0.00 Init: Allocate Memory
Timer #    46       0.76/       1 =       0.76 Init: Parse Model
Timer #    47       0.27/       1 =       0.27 Init: Check for Duplicates
Timer #    48       2.85/       1 =       2.85 Init: Compile Equations
Timer #    49       0.00/       1 =       0.00 Init: Check Uninitialized
Timer #    50       0.00/    1257 =       0.00 Evaluate Expression Once
Timer #    51       0.00/       0 =       0.00 Sensitivity Analysis: LU Factorization
Timer #    52       0.00/       0 =       0.00 Sensitivity Analysis: Gauss Elimination
Timer #    53       0.00/       0 =       0.00 Sensitivity Analysis: Total Time

When I change the amount of neurons of 1 function from [25,20,20,10] to [50,40,40,40] I get the following log:

 ----------------------------------------------------------------
 APMonitor, Version 1.0.0
 APMonitor Optimization Suite
 ----------------------------------------------------------------
 
 Called files( 55 )
 files: overrides.dbs does not exist
 
 Run id   : 2023y02m04d11h08m15.999s
 
 COMMAND LINE ARGUMENTS
 coldstart:  0
 imode    :  3
 dbs_read :  T
 dbs_write:  T
 specs    :  T
 
 rto selected
 Called files( 35 )
 READ info FILE FOR VARIABLE DEFINITION: gk_model0.info
 SS MODEL INIT  0
 Parsing model file gk_model0.apm
 Read model file (sec): 1.4879
 Initialize constants (sec): 0.
 Determine model size (sec): 0.9460000000000002
 Allocate memory (sec): 0.01529999999999987
 Parse and store model (sec): 0.7256999999999998
 
 --------- APM Model Size ------------
 Each time step contains
   Objects      :  342
   Constants    :  0
   Variables    :  1037
   Intermediates:  344
   Connections  :  1026
   Equations    :  1030
   Residuals    :  686
 
 Error checking (sec): 0.2522000000000002
 Compile equations (sec): 2.8817999999999997
 Check for uninitialized intermediates (sec): 0.
 ------------------------------------------------------
 Total Parse Time (sec): 6.3089
 SS MODEL INIT  1
 SS MODEL INIT  2
 SS MODEL INIT  3
 SS MODEL INIT  4
 Called files( 31 )
 READ info FILE FOR PROBLEM DEFINITION: gk_model0.info
 Called files( 6 )
 Files(6): File Read rto.t0 F
 files: rto.t0 does not exist
 Called files( 51 )
 Read DBS File defaults.dbs
 files: defaults.dbs does not exist
 Called files( 51 )
 Read DBS File gk_model0.dbs
 files: gk_model0.dbs does not exist
 Called files( 51 )
 Read DBS File measurements.dbs
 Called files( 51 )
 Read DBS File overrides.dbs
 files: overrides.dbs does not exist
 Number of state variables:    1715
 Number of total equations: -  1369
 Number of slack variables: -  0
 ---------------------------------------
 Degrees of freedom       :    346
 
 ----------------------------------------------
 Steady State Optimization with APOPT Solver
 ----------------------------------------------
Iter:     1 I:  0 Tm:      2.84 NLPi:   53 Dpth:    0 Lvs:    3 Obj:  2.69E-01 Gap:       NaN
--Integer Solution:   4.43E-01 Lowest Leaf:   2.69E-01 Gap:   1.74E-01
Iter:     2 I:  0 Tm:      0.14 NLPi:    5 Dpth:    1 Lvs:    2 Obj:  4.43E-01 Gap:  1.74E-01
Iter:     3 I:  0 Tm:      0.55 NLPi:    6 Dpth:    1 Lvs:    2 Obj:  3.79E-01 Gap:  1.74E-01
Iter:     4 I:  0 Tm:      0.96 NLPi:   12 Dpth:    1 Lvs:    3 Obj:  4.17E-01 Gap:  1.74E-01
--Integer Solution:   4.43E-01 Lowest Leaf:   4.17E-01 Gap:   2.62E-02
Iter:     5 I:  0 Tm:      0.56 NLPi:    7 Dpth:    2 Lvs:    2 Obj:  1.18E+00 Gap:  2.62E-02
--Integer Solution:   4.43E-01 Lowest Leaf:   4.17E-01 Gap:   2.62E-02
Iter:     6 I:  0 Tm:      0.36 NLPi:    4 Dpth:    2 Lvs:    1 Obj:  1.04E+00 Gap:  2.62E-02
--Integer Solution:   4.43E-01 Lowest Leaf:   5.39E-01 Gap:  -9.53E-02
Iter:     7 I:  0 Tm:      0.36 NLPi:    4 Dpth:    2 Lvs:    1 Obj:  5.39E-01 Gap: -9.53E-02
 Successful solution
 
 ---------------------------------------------------
 Solver         :  APOPT (v1.0)
 Solution time  :  5.816599999999999 sec
 Objective      :  0.4433267264972657
 Successful solution
 ---------------------------------------------------
 
 Called files( 2 )
 Called files( 52 )
 WRITE dbs FILE
 Called files( 56 )
 WRITE json FILE
Timer #     1      13.21/       1 =      13.21 Total system time
Timer #     2       5.82/       1 =       5.82 Total solve time
Timer #     3       0.02/     189 =       0.00 Objective Calc: apm_p
Timer #     4       0.03/      91 =       0.00 Objective Grad: apm_g
Timer #     5       0.03/     189 =       0.00 Constraint Calc: apm_c
Timer #     6       0.00/       0 =       0.00 Sparsity: apm_s
Timer #     7       0.00/       0 =       0.00 1st Deriv #1: apm_a1
Timer #     8       0.00/      91 =       0.00 1st Deriv #2: apm_a2
Timer #     9       0.90/       1 =       0.90 Custom Init: apm_custom_init
Timer #    10       0.01/       1 =       0.01 Mode: apm_node_res::case 0
Timer #    11       0.01/       1 =       0.01 Mode: apm_node_res::case 1
Timer #    12       0.03/       1 =       0.03 Mode: apm_node_res::case 2
Timer #    13       0.00/       1 =       0.00 Mode: apm_node_res::case 3
Timer #    14       0.33/     383 =       0.00 Mode: apm_node_res::case 4
Timer #    15       2.01/     182 =       0.01 Mode: apm_node_res::case 5
Timer #    16       0.00/       0 =       0.00 Mode: apm_node_res::case 6
Timer #    17       0.06/      91 =       0.00 Base 1st Deriv: apm_jacobian
Timer #    18       0.02/      91 =       0.00 Base 1st Deriv: apm_condensed_jacobian
Timer #    19       0.00/       1 =       0.00 Non-zeros: apm_nnz
Timer #    20       0.00/       0 =       0.00 Count: Division by zero
Timer #    21       0.00/       0 =       0.00 Count: Argument of LOG10 negative
Timer #    22       0.00/       0 =       0.00 Count: Argument of LOG negative
Timer #    23       0.00/       0 =       0.00 Count: Argument of SQRT negative
Timer #    24       0.00/       0 =       0.00 Count: Argument of ASIN illegal
Timer #    25       0.00/       0 =       0.00 Count: Argument of ACOS illegal
Timer #    26       0.00/       1 =       0.00 Extract sparsity: apm_sparsity
Timer #    27       0.00/      13 =       0.00 Variable ordering: apm_var_order
Timer #    28       0.00/       1 =       0.00 Condensed sparsity
Timer #    29       0.00/       0 =       0.00 Hessian Non-zeros
Timer #    30       0.00/       1 =       0.00 Differentials
Timer #    31       0.00/       0 =       0.00 Hessian Calculation
Timer #    32       0.00/       0 =       0.00 Extract Hessian
Timer #    33       0.00/       1 =       0.00 Base 1st Deriv: apm_jac_order
Timer #    34       0.02/       1 =       0.02 Solver Setup
Timer #    35       3.26/       1 =       3.26 Solver Solution
Timer #    36       0.00/     200 =       0.00 Number of Variables
Timer #    37       0.02/      97 =       0.00 Number of Equations
Timer #    38       0.03/      14 =       0.00 File Read/Write
Timer #    39       0.00/       0 =       0.00 Dynamic Init A
Timer #    40       0.00/       0 =       0.00 Dynamic Init B
Timer #    41       0.00/       0 =       0.00 Dynamic Init C
Timer #    42       1.49/       1 =       1.49 Init: Read APM File
Timer #    43       0.00/       1 =       0.00 Init: Parse Constants
Timer #    44       0.95/       1 =       0.95 Init: Model Sizing
Timer #    45       0.02/       1 =       0.02 Init: Allocate Memory
Timer #    46       0.73/       1 =       0.73 Init: Parse Model
Timer #    47       0.25/       1 =       0.25 Init: Check for Duplicates
Timer #    48       2.88/       1 =       2.88 Init: Compile Equations
Timer #    49       0.00/       1 =       0.00 Init: Check Uninitialized
Timer #    50      -0.00/    1732 =      -0.00 Evaluate Expression Once
Timer #    51       0.00/       0 =       0.00 Sensitivity Analysis: LU Factorization
Timer #    52       0.00/       0 =       0.00 Sensitivity Analysis: Gauss Elimination
Timer #    53       0.00/       0 =       0.00 Sensitivity Analysis: Total Time

Hence, a significant amount of extra objects, variables, intermediates, connections, equations and residuals are introduced.
Many thank in advance for your replies!

Best Answer:

the paper talks about this a little in section 3.1.3.

The prediction functions used in both TensorFlow and Scikit-learn neural networks use linear algebra to relate the layers and neurons of the neural network to one another. Each neuron in an input layer has a specific weight that corresponds to an output neuron in the following layer. Each neuron in an output layer also has a corresponding bias value as shown in Figure 1.

Figure 1

For each neuron connection, there is a bias, weight, and an additional activation function. An additional activation equation, such as the linear rectifier or hyperbolic tangent functions, is generally used to normalize the activation between 0 and 1.

As you increase the number of neurons from [25,20,20,10] to [50,40,40,40], a new weight, bias, and activation function is being introduced for each new neuron connection. These objects are represented by an increase in the number of equations, variables, etc… in the optimization problem.
Hopefully this helps!

If you have better answer, please add a comment about this, thank you!

Source: Stackoverflow.com

Question:

Best Answer:

Related