.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/programmers/plot_optimization.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_programmers_plot_optimization.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_programmers_plot_optimization.py:


biogeme.optimization
====================

Examples of use of several functions.

This is designed for programmers who need examples of use of the
functions of the module. The examples are designed to illustrate the
syntax. They do not correspond to any meaningful model.

:author: Michel Bierlaire
:date: Thu Nov 23 22:25:13 2023

.. GENERATED FROM PYTHON SOURCE LINES 15-26

.. code-block:: default



    import pandas as pd
    from biogeme.version import getText
    import biogeme.biogeme as bio
    import biogeme.database as db
    from biogeme import models
    from biogeme.expressions import Beta, Variable
    import biogeme.biogeme_logging as blog









.. GENERATED FROM PYTHON SOURCE LINES 27-28

Version of Biogeme.

.. GENERATED FROM PYTHON SOURCE LINES 28-30

.. code-block:: default

    print(getText())





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    biogeme 3.2.13 [2023-12-23]
    Home page: http://biogeme.epfl.ch
    Submit questions to https://groups.google.com/d/forum/biogeme
    Michel Bierlaire, Transport and Mobility Laboratory, Ecole Polytechnique Fédérale de Lausanne (EPFL)





.. GENERATED FROM PYTHON SOURCE LINES 31-33

.. code-block:: default

    logger = blog.get_screen_logger(blog.INFO)








.. GENERATED FROM PYTHON SOURCE LINES 34-35

Data,

.. GENERATED FROM PYTHON SOURCE LINES 35-49

.. code-block:: default

    df = pd.DataFrame(
        {
            'Person': [1, 1, 1, 2, 2],
            'Exclude': [0, 0, 1, 0, 1],
            'Variable1': [1, 2, 3, 4, 5],
            'Variable2': [10, 20, 30, 40, 50],
            'Choice': [1, 2, 3, 1, 2],
            'Av1': [0, 1, 1, 1, 1],
            'Av2': [1, 1, 1, 1, 1],
            'Av3': [0, 1, 1, 1, 1],
        }
    )
    df






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Person</th>
          <th>Exclude</th>
          <th>Variable1</th>
          <th>Variable2</th>
          <th>Choice</th>
          <th>Av1</th>
          <th>Av2</th>
          <th>Av3</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>1</td>
          <td>0</td>
          <td>1</td>
          <td>10</td>
          <td>1</td>
          <td>0</td>
          <td>1</td>
          <td>0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1</td>
          <td>0</td>
          <td>2</td>
          <td>20</td>
          <td>2</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>2</th>
          <td>1</td>
          <td>1</td>
          <td>3</td>
          <td>30</td>
          <td>3</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>3</th>
          <td>2</td>
          <td>0</td>
          <td>4</td>
          <td>40</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>4</th>
          <td>2</td>
          <td>1</td>
          <td>5</td>
          <td>50</td>
          <td>2</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 50-52

.. code-block:: default

    my_data = db.Database('test', df)








.. GENERATED FROM PYTHON SOURCE LINES 53-54

Variables.

.. GENERATED FROM PYTHON SOURCE LINES 54-64

.. code-block:: default

    Choice = Variable('Choice')
    Variable1 = Variable('Variable1')
    Variable2 = Variable('Variable2')
    beta1 = Beta('beta1', 0, None, None, 0)
    beta2 = Beta('beta2', 0, None, None, 0)
    V1 = beta1 * Variable1
    V2 = beta2 * Variable2
    V3 = 0
    V = {1: V1, 2: V2, 3: V3}








.. GENERATED FROM PYTHON SOURCE LINES 65-73

.. code-block:: default

    likelihood = models.loglogit(V, av=None, i=Choice)
    my_biogeme = bio.BIOGEME(my_data, likelihood)
    my_biogeme.modelName = 'simpleExample'
    my_biogeme.save_iterations = False
    my_biogeme.generate_html = False
    my_biogeme.generate_pickle = False
    print(my_biogeme)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    File biogeme.toml has been parsed. 
    simpleExample: database [test]{'loglike': _bioLogLogitFullChoiceSet[choice=Choice]U=(1:(Beta('beta1', 0, -1.3407807929942596e+154, 1.3407807929942596e+154, 0) * Variable1), 2:(Beta('beta2', 0, -1.3407807929942596e+154, 1.3407807929942596e+154, 0) * Variable2), 3:`0.0`)av=(1:`1.0`, 2:`1.0`, 3:`1.0`)}




.. GENERATED FROM PYTHON SOURCE LINES 74-78

.. code-block:: default

    f, g, h, gdiff, hdiff = my_biogeme.checkDerivatives(
        beta=my_biogeme.beta_values_dict_to_list(), verbose=True
    )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    x               Gradient        FinDiff         Difference 
    beta1           +2.220446E-16   -6.039613E-07   +6.039613E-07 
    beta2           +2.000000E+01   +1.999994E+01   +6.110388E-05 
    Row             Col             Hessian FinDiff         Difference 
    beta1           beta1           -1.222222E+01   -1.222222E+01   +8.314151E-07 
    beta1           beta2           +6.111111E+01   +6.111115E+01   -4.166707E-05 
    beta2           beta1           +6.111111E+01   +6.111112E+01   -4.274759E-06 
    beta2           beta2           -1.222222E+03   -1.222223E+03   +8.332704E-04 




.. GENERATED FROM PYTHON SOURCE LINES 79-81

.. code-block:: default

    pd.DataFrame(gdiff)






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>6.039613e-07</td>
        </tr>
        <tr>
          <th>1</th>
          <td>6.110388e-05</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 82-84

.. code-block:: default

    pd.DataFrame(hdiff)






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
          <th>1</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>8.314151e-07</td>
          <td>-0.000042</td>
        </tr>
        <tr>
          <th>1</th>
          <td>-4.274759e-06</td>
          <td>0.000833</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 85-86

**scipy**: this is the optimization algorithm from scipy.

.. GENERATED FROM PYTHON SOURCE LINES 86-90

.. code-block:: default

    my_biogeme.algorithm_name = 'scipy'
    results = my_biogeme.estimate()
    results.getEstimatedParameters()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimization algorithm: scipy 
    Minimize with tol 1e-07 


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Value</th>
          <th>Rob. Std err</th>
          <th>Rob. t-test</th>
          <th>Rob. p-value</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>beta1</th>
          <td>0.144546</td>
          <td>0.366198</td>
          <td>0.394720</td>
          <td>0.693049</td>
        </tr>
        <tr>
          <th>beta2</th>
          <td>0.023502</td>
          <td>0.034280</td>
          <td>0.685574</td>
          <td>0.492982</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 91-94

.. code-block:: default

    for k, v in results.data.optimizationMessages.items():
        print(f'{k}:\t{v}')





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Algorithm:      scipy.optimize
    Cause of termination:   CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL
    Number of iterations:   13
    Number of function evaluations: 17
    Optimization time:      0:00:00.003132




.. GENERATED FROM PYTHON SOURCE LINES 95-96

**Newton with linesearch**

.. GENERATED FROM PYTHON SOURCE LINES 96-100

.. code-block:: default

    my_biogeme.algorithm_name = 'LS-newton'
    results = my_biogeme.estimate()
    results.getEstimatedParameters()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimization algorithm: Newton with line search [LS-newton] 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    ** Optimization: Newton with linesearch 


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Value</th>
          <th>Rob. Std err</th>
          <th>Rob. t-test</th>
          <th>Rob. p-value</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>beta1</th>
          <td>0.144545</td>
          <td>0.366198</td>
          <td>0.394720</td>
          <td>0.693050</td>
        </tr>
        <tr>
          <th>beta2</th>
          <td>0.023502</td>
          <td>0.034280</td>
          <td>0.685573</td>
          <td>0.492982</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 101-104

.. code-block:: default

    for k, v in results.data.optimizationMessages.items():
        print(f'{k}:\t{v}')





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Algorithm:      Unconstrained Newton with line search
    Relative gradient:      3.289367616755922e-07
    Cause of termination:   Relative gradient = 3.3e-07 <= 6.1e-06
    Number of function evaluations: 4
    Number of gradient evaluations: 4
    Number of hessian evaluations:  4
    Number of iterations:   3
    Optimization time:      0:00:00.002186




.. GENERATED FROM PYTHON SOURCE LINES 105-106

Changing the requested precision

.. GENERATED FROM PYTHON SOURCE LINES 106-110

.. code-block:: default

    my_biogeme.tolerance = 0.1
    results = my_biogeme.estimate()
    results.getEstimatedParameters()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimization algorithm: Newton with line search [LS-newton] 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    ** Optimization: Newton with linesearch 


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Value</th>
          <th>Rob. Std err</th>
          <th>Rob. t-test</th>
          <th>Rob. p-value</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>beta1</th>
          <td>0.144317</td>
          <td>0.365691</td>
          <td>0.394641</td>
          <td>0.693108</td>
        </tr>
        <tr>
          <th>beta2</th>
          <td>0.023428</td>
          <td>0.034256</td>
          <td>0.683887</td>
          <td>0.494047</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 111-114

.. code-block:: default

    for k, v in results.data.optimizationMessages.items():
        print(f'{k}:\t{v}')





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Algorithm:      Unconstrained Newton with line search
    Relative gradient:      0.014746524905601736
    Cause of termination:   Relative gradient = 0.015 <= 0.1
    Number of function evaluations: 3
    Number of gradient evaluations: 3
    Number of hessian evaluations:  3
    Number of iterations:   2
    Optimization time:      0:00:00.001344




.. GENERATED FROM PYTHON SOURCE LINES 115-116

**Newton with trust region**

.. GENERATED FROM PYTHON SOURCE LINES 116-120

.. code-block:: default

    my_biogeme.algorithm_name = 'TR-newton'
    results = my_biogeme.estimate()
    results.getEstimatedParameters()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimization algorithm: Newton with trust region [TR-newton] 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    ** Optimization: Newton with trust region 


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Value</th>
          <th>Rob. Std err</th>
          <th>Rob. t-test</th>
          <th>Rob. p-value</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>beta1</th>
          <td>0.144317</td>
          <td>0.365691</td>
          <td>0.394641</td>
          <td>0.693108</td>
        </tr>
        <tr>
          <th>beta2</th>
          <td>0.023428</td>
          <td>0.034256</td>
          <td>0.683887</td>
          <td>0.494047</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 121-124

.. code-block:: default

    for k, v in results.data.optimizationMessages.items():
        print(f'{k}:\t{v}')





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Relative gradient:      0.014746524905601736
    Cause of termination:   Relative gradient = 0.015 <= 0.1
    Number of function evaluations: 3
    Number of gradient evaluations: 3
    Number of hessian evaluations:  3
    Number of iterations:   2
    Optimization time:      0:00:00.001810




.. GENERATED FROM PYTHON SOURCE LINES 125-129

We illustrate the parameters. We use the truncated conjugate
gradient instead of dogleg for the trust region subproblem, starting
with a small trust region of radius 0.001, and a maximum of 3
iterations.

.. GENERATED FROM PYTHON SOURCE LINES 131-137

.. code-block:: default

    my_biogeme.dogleg = False
    my_biogeme.initial_radius = 0.001
    my_biogeme.maxiter = 3
    results = my_biogeme.estimate()
    results.getEstimatedParameters()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimization algorithm: Newton with trust region [TR-newton] 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    ** Optimization: Newton with trust region 
    It seems that the optimization algorithm did not converge. Therefore, the results may not correspond to the maximum likelihood estimator. Check the specification of the model, or the criteria for convergence of the algorithm. 


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Value</th>
          <th>Rob. Std err</th>
          <th>Rob. t-test</th>
          <th>Rob. p-value</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>beta1</th>
          <td>0.000053</td>
          <td>0.308910</td>
          <td>0.000170</td>
          <td>0.999864</td>
        </tr>
        <tr>
          <th>beta2</th>
          <td>0.007000</td>
          <td>0.030019</td>
          <td>0.233173</td>
          <td>0.815627</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 138-141

.. code-block:: default

    for k, v in results.data.optimizationMessages.items():
        print(f'{k}:\t{v}')





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Cause of termination:   Maximum number of iterations reached: 3
    Optimization time:      0:00:00.001556




.. GENERATED FROM PYTHON SOURCE LINES 142-143

Changing the requested precision

.. GENERATED FROM PYTHON SOURCE LINES 143-148

.. code-block:: default

    my_biogeme.tolerance = 0.1
    my_biogeme.maxiter = 1000
    results = my_biogeme.estimate()
    results.getEstimatedParameters()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimization algorithm: Newton with trust region [TR-newton] 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    ** Optimization: Newton with trust region 


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Value</th>
          <th>Rob. Std err</th>
          <th>Rob. t-test</th>
          <th>Rob. p-value</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>beta1</th>
          <td>0.111901</td>
          <td>0.357903</td>
          <td>0.312658</td>
          <td>0.754541</td>
        </tr>
        <tr>
          <th>beta2</th>
          <td>0.021615</td>
          <td>0.032531</td>
          <td>0.664435</td>
          <td>0.506412</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 149-152

.. code-block:: default

    for k, v in results.data.optimizationMessages.items():
        print(f'{k}:\t{v}')





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Relative gradient:      0.04203221663984542
    Cause of termination:   Relative gradient = 0.042 <= 0.1
    Number of function evaluations: 8
    Number of gradient evaluations: 8
    Number of hessian evaluations:  8
    Number of iterations:   7
    Optimization time:      0:00:00.002655




.. GENERATED FROM PYTHON SOURCE LINES 153-154

**BFGS with line search**

.. GENERATED FROM PYTHON SOURCE LINES 154-160

.. code-block:: default

    my_biogeme.algorithm_name = 'LS-BFGS'
    my_biogeme.tolerance = 1.0e-6
    my_biogeme.maxiter = 1000
    results = my_biogeme.estimate()
    results.getEstimatedParameters()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimization algorithm: BFGS with line search [LS-BFGS] 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    ** Optimization: BFGS with line search 


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Value</th>
          <th>Rob. Std err</th>
          <th>Rob. t-test</th>
          <th>Rob. p-value</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>beta1</th>
          <td>0.144546</td>
          <td>0.366198</td>
          <td>0.394720</td>
          <td>0.693049</td>
        </tr>
        <tr>
          <th>beta2</th>
          <td>0.023502</td>
          <td>0.034280</td>
          <td>0.685574</td>
          <td>0.492982</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 161-164

.. code-block:: default

    for k, v in results.data.optimizationMessages.items():
        print(f'{k}:\t{v}')





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Algorithm:      Inverse BFGS with line search
    Relative gradient:      5.5446416666399e-07
    Cause of termination:   Relative gradient = 5.5e-07 <= 1e-06
    Number of function evaluations: 19
    Number of gradient evaluations: 19
    Number of hessian evaluations:  0
    Number of iterations:   6
    Optimization time:      0:00:00.002813




.. GENERATED FROM PYTHON SOURCE LINES 165-166

**BFGS with trust region**

.. GENERATED FROM PYTHON SOURCE LINES 166-170

.. code-block:: default

    my_biogeme.algorithm_name = 'TR-BFGS'
    results = my_biogeme.estimate()
    results.getEstimatedParameters()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimization algorithm: BFGS with trust region [TR-BFGS] 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    This algorithm does not handle bound constraints. The bounds will be ignored. 
    ** Optimization: BFGS with trust region 


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Value</th>
          <th>Rob. Std err</th>
          <th>Rob. t-test</th>
          <th>Rob. p-value</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>beta1</th>
          <td>0.144546</td>
          <td>0.366198</td>
          <td>0.394720</td>
          <td>0.693049</td>
        </tr>
        <tr>
          <th>beta2</th>
          <td>0.023502</td>
          <td>0.034280</td>
          <td>0.685574</td>
          <td>0.492982</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 171-174

.. code-block:: default

    for k, v in results.data.optimizationMessages.items():
        print(f'{k}:\t{v}')





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Relative gradient:      9.951716866221394e-10
    Cause of termination:   Relative gradient = 1e-09 <= 1e-06
    Number of function evaluations: 13
    Number of gradient evaluations: 13
    Number of hessian evaluations:  0
    Number of iterations:   12
    Optimization time:      0:00:00.004345




.. GENERATED FROM PYTHON SOURCE LINES 175-176

**Newton/BFGS with trust region for simple bounds**

.. GENERATED FROM PYTHON SOURCE LINES 178-181

This is the default algorithm used by Biogeme. It is the
implementation of the algorithm proposed by `Conn et al. (1988)
<https://www.ams.org/journals/mcom/1988-50-182/S0025-5718-1988-0929544-3/S0025-5718-1988-0929544-3.pdf>`_.

.. GENERATED FROM PYTHON SOURCE LINES 181-185

.. code-block:: default

    my_biogeme.algorithm_name = 'simple_bounds'
    results = my_biogeme.estimate()
    results.getEstimatedParameters()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimization algorithm: hybrid Newton/BFGS with simple bounds [simple_bounds] 
    ** Optimization: Newton with trust region for simple bounds 
    Iter.           beta1           beta2     Function    Relgrad   Radius      Rho      
        0               0           0.001          5.5        3.4     0.01        1   ++ 
        1            0.01           0.011          5.3        1.2      0.1     0.99   ++ 
        2            0.11           0.021          5.3       0.11        1        1   ++ 
        3            0.14           0.023          5.3      0.013       10        1   ++ 
        4            0.14           0.024          5.3    8.4e-06    1e+02        1   ++ 
        5            0.14           0.024          5.3      2e-11    1e+02        1   ++ 


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Value</th>
          <th>Rob. Std err</th>
          <th>Rob. t-test</th>
          <th>Rob. p-value</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>beta1</th>
          <td>0.144546</td>
          <td>0.366198</td>
          <td>0.394720</td>
          <td>0.693049</td>
        </tr>
        <tr>
          <th>beta2</th>
          <td>0.023502</td>
          <td>0.034280</td>
          <td>0.685574</td>
          <td>0.492982</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 186-189

.. code-block:: default

    for k, v in results.data.optimizationMessages.items():
        print(f'{k}:\t{v}')





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Relative gradient:      2.0479777952529902e-11
    Cause of termination:   Relative gradient = 2e-11 <= 1e-06
    Number of function evaluations: 7
    Number of gradient evaluations: 7
    Number of hessian evaluations:  6
    Algorithm:      Newton with trust region for simple bound constraints
    Number of iterations:   6
    Proportion of Hessian calculation:      6/6 = 100.0%
    Optimization time:      0:00:00.004603




.. GENERATED FROM PYTHON SOURCE LINES 190-194

When the second derivatives are too computationally expensive to
calculate, it is possible to avoid calculating them at each
successful iteration. The parameter `second_derivatives` allows to
control that.

.. GENERATED FROM PYTHON SOURCE LINES 196-200

.. code-block:: default

    my_biogeme.second_derivatives = 0.5
    results = my_biogeme.estimate()
    results.getEstimatedParameters()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimization algorithm: hybrid Newton/BFGS with simple bounds [simple_bounds] 
    ** Optimization: Hybrid Newton 50.0%/BFGS with trust region for simple bounds 
    Iter.           beta1           beta2     Function    Relgrad   Radius      Rho      
        0               0           0.001          5.5        3.4     0.01        1   ++ 
        1            0.01           0.011          5.3        1.2      0.1     0.98   ++ 
        2            0.11           0.021          5.3       0.11        1        1   ++ 
        3            0.14           0.023          5.3      0.058       10      1.1   ++ 
        4            0.14           0.023          5.3    0.00012    1e+02        1   ++ 
        5            0.14           0.023          5.3    8.4e-07    1e+02        1   ++ 


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Value</th>
          <th>Rob. Std err</th>
          <th>Rob. t-test</th>
          <th>Rob. p-value</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>beta1</th>
          <td>0.144546</td>
          <td>0.366198</td>
          <td>0.394720</td>
          <td>0.693049</td>
        </tr>
        <tr>
          <th>beta2</th>
          <td>0.023502</td>
          <td>0.034280</td>
          <td>0.685573</td>
          <td>0.492982</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 201-204

.. code-block:: default

    for k, v in results.data.optimizationMessages.items():
        print(f'{k}:\t{v}')





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Relative gradient:      8.430721282926752e-07
    Cause of termination:   Relative gradient = 8.4e-07 <= 1e-06
    Number of function evaluations: 7
    Number of gradient evaluations: 7
    Number of hessian evaluations:  3
    Algorithm:      Hybrid Newton [50.0%] with trust region for simple bound constraints
    Number of iterations:   6
    Proportion of Hessian calculation:      3/6 = 50.0%
    Optimization time:      0:00:00.004522




.. GENERATED FROM PYTHON SOURCE LINES 205-207

If the parameter is set to zero, the second derivatives are not used
at all, and the algorithm relies only on the BFGS update.

.. GENERATED FROM PYTHON SOURCE LINES 209-213

.. code-block:: default

    my_biogeme.second_derivatives = 0.0
    results = my_biogeme.estimate()
    results.getEstimatedParameters()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimization algorithm: hybrid Newton/BFGS with simple bounds [simple_bounds] 
    ** Optimization: BFGS with trust region for simple bounds 
    Iter.           beta1           beta2     Function    Relgrad   Radius      Rho      
        0               0           0.001          5.5        3.4     0.01     0.97   ++ 
        1            0.01           0.011          5.3        1.2      0.1     0.98   ++ 
        2            0.11            0.02          5.3       0.21      0.1     0.73    + 
        3            0.14           0.023          5.3       0.21        1     0.98   ++ 
        4            0.14           0.023          5.3       0.21   0.0013     -1.2    - 
        5            0.14           0.024          5.3       0.14   0.0013     0.23    + 
        6            0.14           0.023          5.3    5.8e-05    0.013        1   ++ 
        7            0.14           0.024          5.3    2.2e-06     0.13        1   ++ 
        8            0.14           0.024          5.3    8.7e-07     0.13        1   ++ 


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Value</th>
          <th>Rob. Std err</th>
          <th>Rob. t-test</th>
          <th>Rob. p-value</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>beta1</th>
          <td>0.144545</td>
          <td>0.366198</td>
          <td>0.394719</td>
          <td>0.693050</td>
        </tr>
        <tr>
          <th>beta2</th>
          <td>0.023502</td>
          <td>0.034280</td>
          <td>0.685573</td>
          <td>0.492982</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 214-217

.. code-block:: default

    for k, v in results.data.optimizationMessages.items():
        print(f'{k}:\t{v}')





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Relative gradient:      8.717117167567343e-07
    Cause of termination:   Relative gradient = 8.7e-07 <= 1e-06
    Number of function evaluations: 10
    Number of gradient evaluations: 9
    Number of hessian evaluations:  0
    Algorithm:      BFGS with trust region for simple bound constraints
    Number of iterations:   9
    Proportion of Hessian calculation:      0/8 = 0.0%
    Optimization time:      0:00:00.006090




.. GENERATED FROM PYTHON SOURCE LINES 218-219

There are shortcuts to call the BFGS and the Newton versions

.. GENERATED FROM PYTHON SOURCE LINES 219-223

.. code-block:: default

    my_biogeme.algorithm_name = 'simple_bounds_newton'
    results = my_biogeme.estimate()
    results.getEstimatedParameters()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimization algorithm: Newton with simple bounds [simple_bounds_newton]. 
    Optimization algorithm: hybrid Newton/BFGS with simple bounds [simple_bounds] 
    ** Optimization: Newton with trust region for simple bounds 
    Iter.           beta1           beta2     Function    Relgrad   Radius      Rho      
        0               0           0.001          5.5        3.4     0.01        1   ++ 
        1            0.01           0.011          5.3        1.2      0.1     0.99   ++ 
        2            0.11           0.021          5.3       0.11        1        1   ++ 
        3            0.14           0.023          5.3      0.013       10        1   ++ 
        4            0.14           0.024          5.3    8.4e-06    1e+02        1   ++ 
        5            0.14           0.024          5.3      2e-11    1e+02        1   ++ 


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Value</th>
          <th>Rob. Std err</th>
          <th>Rob. t-test</th>
          <th>Rob. p-value</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>beta1</th>
          <td>0.144546</td>
          <td>0.366198</td>
          <td>0.394720</td>
          <td>0.693049</td>
        </tr>
        <tr>
          <th>beta2</th>
          <td>0.023502</td>
          <td>0.034280</td>
          <td>0.685574</td>
          <td>0.492982</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 224-227

.. code-block:: default

    for k, v in results.data.optimizationMessages.items():
        print(f'{k}:\t{v}')





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Relative gradient:      2.0479777952529902e-11
    Cause of termination:   Relative gradient = 2e-11 <= 1e-06
    Number of function evaluations: 7
    Number of gradient evaluations: 7
    Number of hessian evaluations:  6
    Algorithm:      Newton with trust region for simple bound constraints
    Number of iterations:   6
    Proportion of Hessian calculation:      6/6 = 100.0%
    Optimization time:      0:00:00.005003




.. GENERATED FROM PYTHON SOURCE LINES 228-232

.. code-block:: default

    my_biogeme.algorithm_name = 'simple_bounds_BFGS'
    results = my_biogeme.estimate()
    results.getEstimatedParameters()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimization algorithm: BFGS with simple bounds [simple_bounds_BFGS]. 
    Optimization algorithm: hybrid Newton/BFGS with simple bounds [simple_bounds] 
    ** Optimization: BFGS with trust region for simple bounds 
    Iter.           beta1           beta2     Function    Relgrad   Radius      Rho      
        0               0           0.001          5.5        3.4     0.01     0.97   ++ 
        1            0.01           0.011          5.3        1.2      0.1     0.98   ++ 
        2            0.11            0.02          5.3       0.21      0.1     0.73    + 
        3            0.14           0.023          5.3       0.21        1     0.98   ++ 
        4            0.14           0.023          5.3       0.21   0.0013     -1.2    - 
        5            0.14           0.024          5.3       0.14   0.0013     0.23    + 
        6            0.14           0.023          5.3    5.8e-05    0.013        1   ++ 
        7            0.14           0.024          5.3    2.2e-06     0.13        1   ++ 
        8            0.14           0.024          5.3    8.7e-07     0.13        1   ++ 


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Value</th>
          <th>Rob. Std err</th>
          <th>Rob. t-test</th>
          <th>Rob. p-value</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>beta1</th>
          <td>0.144545</td>
          <td>0.366198</td>
          <td>0.394719</td>
          <td>0.693050</td>
        </tr>
        <tr>
          <th>beta2</th>
          <td>0.023502</td>
          <td>0.034280</td>
          <td>0.685573</td>
          <td>0.492982</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 233-235

.. code-block:: default

    for k, v in results.data.optimizationMessages.items():
        print(f'{k}:\t{v}')




.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Relative gradient:      8.717117167567343e-07
    Cause of termination:   Relative gradient = 8.7e-07 <= 1e-06
    Number of function evaluations: 10
    Number of gradient evaluations: 9
    Number of hessian evaluations:  0
    Algorithm:      BFGS with trust region for simple bound constraints
    Number of iterations:   9
    Proportion of Hessian calculation:      0/8 = 0.0%
    Optimization time:      0:00:00.006053





.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.094 seconds)


.. _sphx_glr_download_auto_examples_programmers_plot_optimization.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example




    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_optimization.py <plot_optimization.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_optimization.ipynb <plot_optimization.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_