The NLMIXED Procedure

Finite-Difference Approximations of Derivatives

The FD= and FDHESSIAN= options specify the use of finite-difference approximations of the derivatives. The FD= option specifies that all derivatives are approximated using function evaluations, and the FDHESSIAN= option specifies that second-order derivatives are approximated using gradient evaluations.

Computing derivatives by finite-difference approximations can be very time-consuming, especially for second-order derivatives based only on values of the objective function (FD= option). If analytical derivatives are difficult to obtain (for example, if a function is computed by an iterative process), you might consider one of the optimization techniques that use first-order derivatives only (QUANEW, DBLDOG, or CONGRA). In the expressions that follow, bold-italic theta denotes the parameter vector, h Subscript i denotes the step size for the ith parameter, and bold e Subscript i is a vector of zeros with a 1 in the ith position.

Forward-Difference Approximations

The forward-difference derivative approximations consume less computer time, but they are usually not as precise as approximations that use central-difference formulas.

  • For first-order derivatives, n additional function calls are required:

    StartLayout 1st Row 1st Column g Subscript i 2nd Column equals StartFraction partial-differential f Over partial-differential theta Subscript i Baseline EndFraction almost-equals StartFraction f left-parenthesis bold-italic theta plus h Subscript i Baseline bold e Subscript i Baseline right-parenthesis minus f left-parenthesis bold-italic theta right-parenthesis Over h Subscript i Baseline EndFraction EndLayout

  • For second-order derivatives based on function calls only (Dennis and Schnabel 1983, p. 80), n plus n squared slash 2 additional function calls are required for dense Hessian:

    StartLayout 1st Row 1st Column StartFraction partial-differential squared f Over partial-differential theta Subscript i Baseline partial-differential theta Subscript j Baseline EndFraction 2nd Column almost-equals StartFraction f left-parenthesis bold-italic theta plus h Subscript i Baseline bold e Subscript i Baseline plus h Subscript j Baseline bold e Subscript j Baseline right-parenthesis minus f left-parenthesis bold-italic theta plus h Subscript i Baseline bold e Subscript i Baseline right-parenthesis minus f left-parenthesis bold-italic theta plus h Subscript j Baseline bold e Subscript j Baseline right-parenthesis plus f left-parenthesis bold-italic theta right-parenthesis Over h Subscript i Baseline h Subscript j Baseline EndFraction EndLayout

  • For second-order derivatives based on gradient calls (Dennis and Schnabel 1983, p. 103), n additional gradient calls are required:

    StartLayout 1st Row 1st Column StartFraction partial-differential squared f Over partial-differential theta Subscript i Baseline partial-differential theta Subscript j Baseline EndFraction 2nd Column almost-equals StartFraction g Subscript i Baseline left-parenthesis bold-italic theta plus h Subscript j Baseline bold e Subscript j Baseline right-parenthesis minus g Subscript i Baseline left-parenthesis bold-italic theta right-parenthesis Over 2 h Subscript j Baseline EndFraction plus StartFraction g Subscript j Baseline left-parenthesis bold-italic theta plus h Subscript i Baseline bold e Subscript i Baseline right-parenthesis minus g Subscript j Baseline left-parenthesis bold-italic theta right-parenthesis Over 2 h Subscript i Baseline EndFraction EndLayout

Central-Difference Approximations

Central-difference approximations are usually more precise, but they consume more computer time than approximations that use forward-difference derivative formulas.

  • For first-order derivatives, 2n additional function calls are required:

    StartLayout 1st Row 1st Column g Subscript i 2nd Column equals StartFraction partial-differential f Over partial-differential theta Subscript i Baseline EndFraction almost-equals StartFraction f left-parenthesis bold-italic theta plus h Subscript i Baseline bold e Subscript i Baseline right-parenthesis minus f left-parenthesis bold-italic theta minus h Subscript i Baseline bold e Subscript i Baseline right-parenthesis Over 2 h Subscript i Baseline EndFraction EndLayout

  • For second-order derivatives based on function calls only (Abramowitz and Stegun 1972, p. 884), 2 n plus 4 n squared slash 2 additional function calls are required.

    StartLayout 1st Row 1st Column StartFraction partial-differential squared f Over partial-differential theta Subscript i Superscript 2 Baseline EndFraction 2nd Column almost-equals StartFraction minus f left-parenthesis bold-italic theta plus 2 h Subscript i Baseline bold e Subscript i Baseline right-parenthesis plus 16 f left-parenthesis bold-italic theta plus h Subscript i Baseline bold e Subscript i Baseline right-parenthesis minus 30 f left-parenthesis bold-italic theta right-parenthesis plus 16 f left-parenthesis bold-italic theta minus h Subscript bold i Baseline e Subscript i Baseline right-parenthesis minus f left-parenthesis bold-italic theta minus 2 h Subscript i Baseline bold e Subscript i Baseline right-parenthesis Over 12 h Subscript i Superscript 2 Baseline EndFraction EndLayout
    StartLayout 1st Row 1st Column StartFraction partial-differential squared f Over partial-differential theta Subscript i Baseline partial-differential theta Subscript j Baseline EndFraction 2nd Column almost-equals StartFraction f left-parenthesis bold-italic theta plus h Subscript i Baseline bold e Subscript i Baseline plus h Subscript j Baseline bold e Subscript j Baseline right-parenthesis minus f left-parenthesis bold-italic theta plus h Subscript i Baseline bold e Subscript i Baseline minus h Subscript j Baseline bold e Subscript j Baseline right-parenthesis minus f left-parenthesis bold-italic theta minus h Subscript i Baseline bold e Subscript i Baseline plus h Subscript j Baseline bold e Subscript j Baseline right-parenthesis plus f left-parenthesis bold-italic theta minus h Subscript i Baseline bold e Subscript i Baseline minus h Subscript j Baseline bold e Subscript j Baseline right-parenthesis Over 4 h Subscript i Baseline h Subscript j Baseline EndFraction EndLayout

  • For second-order derivatives based on gradient calls, 2n additional gradient calls are required:

    StartLayout 1st Row 1st Column StartFraction partial-differential squared f Over partial-differential theta Subscript i Baseline partial-differential theta Subscript j Baseline EndFraction 2nd Column almost-equals StartFraction g Subscript i Baseline left-parenthesis bold-italic theta plus h Subscript j Baseline bold e Subscript j Baseline right-parenthesis minus g Subscript i Baseline left-parenthesis bold-italic theta minus h Subscript j Baseline bold e Subscript j Baseline right-parenthesis Over 4 h Subscript j Baseline EndFraction plus StartFraction g Subscript j Baseline left-parenthesis bold-italic theta plus h Subscript i Baseline bold e Subscript i Baseline right-parenthesis minus g Subscript j Baseline left-parenthesis bold-italic theta minus h Subscript i Baseline bold e Subscript i Baseline right-parenthesis Over 4 h Subscript i Baseline EndFraction EndLayout

You can use the FDIGITS= option to specify the number of accurate digits in the evaluation of the objective function. This specification is helpful in determining an appropriate interval size h to be used in the finite-difference formulas.

The step sizes h Subscript j, j equals 1 comma ellipsis comma n are defined as follows:

  • For the forward-difference approximation of first-order derivatives that use function calls and second-order derivatives that use gradient calls, h Subscript j Baseline equals RootIndex 2 StartRoot eta EndRoot left-parenthesis 1 plus StartAbsoluteValue theta Subscript j Baseline EndAbsoluteValue right-parenthesis.

  • For the forward-difference approximation of second-order derivatives that use only function calls and all central-difference formulas, h Subscript j Baseline equals RootIndex 3 StartRoot eta EndRoot left-parenthesis 1 plus StartAbsoluteValue theta Subscript j Baseline EndAbsoluteValue right-parenthesis.

The value of eta is defined by the FDIGITS= option:

  • If you specify the number of accurate digits by using FDIGITS=r, eta is set to 10 Superscript negative r.

  • If you do not specify the FDIGITS= option, eta is set to the machine precision epsilon.

Last updated: December 09, 2022