Overload List
Overload 1: function BFGS(Fun: TRealFunction; Grad: TGrad; var Pars: TDoubleArray; const Consts: TDoubleArray; const ObjConst: TObjectArray; out FMin: Double; const IHess: TMtx; out StopReason: TOptStopReason; const FloatPrecision: TMtxFloatPrecision; DFPAlgo: Boolean; SoftLineSearch: Boolean): Integer;
Minimizes the function of several variables by using the Quasi-Newton optimization method with no log.
| # | Name | Type | Description |
|---|---|---|---|
| 1 | Fun | TRealFunction | |
| 2 | Grad | TGrad | |
| 3 | Pars | TDoubleArray | |
| 4 | Consts | TDoubleArray | |
| 5 | ObjConst | TObjectArray | |
| 6 | FMin | Double | |
| 7 | IHess | TMtx | |
| 8 | StopReason | TOptStopReason | |
| 9 | FloatPrecision | TMtxFloatPrecision | |
| 10 | DFPAlgo | Boolean | |
| 11 | SoftLineSearch | Boolean |
Returns: Int32
Overload 2: function BFGS(Fun: TRealFunction; Grad: TGrad; var Pars: TDoubleArray; const Consts: TDoubleArray; const ObjConst: TObjectArray; out FMin: Double; const IHess: TMtx; out StopReason: TOptStopReason; const FloatPrecision: TMtxFloatPrecision; DFPAlgo: Boolean; SoftLineSearch: Boolean; MaxIter: Integer; Tol: Double; GradTol: Double; const Verbose: TStrings): Integer;
Minimizes the function of several variables by using the Quasi-Newton optimization algorithm.
| # | Name | Description |
|---|---|---|
| 1 | Fun | Real function (must be of TRealFunction type) to be minimized. |
| 2 | Grad | The gradient and Hessian procedure (must be of TGrad type), used for calculating the gradient. |
| 3 | Pars | Stores the initial estimates for parameters (minimum estimate). After the call to routine returns adjusted calculated values (minimum position). |
| 4 | Consts | Additional Fun constant parameteres (can be/is usually nil). |
| 5 | ObjConst | Additional Fun constant parameteres (can be/is usually nil). |
| 6 | FMin | Returns function value at minimum. |
| 7 | IHess | Returns inverse Hessian matrix. |
| 8 | StopReason | Returns reason why minimum search stopped (see TOptStopReason). |
| 9 | DFPAlgo | If True, BFGS procedure will use Davidon-Fletcher-Powell Hessian update scheme. If False, BFGS procedure will use Broyden-Fletcher-Goldberg-Shamo Hessian update scheme. |
| 10 | SoftLineSearch | Selects the line-search acceptance rule used along each quasi-Newton direction. It is a true/false parameter |
| 11 | it does not turn the gradient on or off. Both modes are gradient based - the internal line search evaluates the gradient at every trial step to form the directional derivative grad f(x+alpha d)^T d - so the flag only sets how strict the accepted step must be | * True (soft / inexact line search, the default): accept the first trial step that satisfies the Armijo sufficient-decrease condition f(x+alpha d) <= f(x) + rho alpha grad f^T d, with geometric step expansion. The curvature of the step is used only to decide whether to grow it, never to reject it. This is the robust choice when Grad is a numerical (finite-difference) approximation, whose directional-curvature measurement is too noisy for a strict test. * False (exact / Wolfe line search): in addition to sufficient decrease, the step is refined until the strong-Wolfe curvature condition |grad f(x+alpha d)^T d| <= beta |grad f^T d| also holds. Use this only with an exact (analytic) gradient, where the curvature test is meaningful. So "soft search off" still means a gradient-based (Wolfe) search - just a stricter one - not a gradient-free search. The gradient-free alternative is the Nelder-Mead Optimization.Simplex routine. |
| 12 | MaxIter | Maximum allowed numer of minimum search iterations. |
| 13 | Tol | Desired Pars - minimum position tolerance. |
| 14 | GradTol | Minimum allowed gradient C-Norm. |
| 15 | Verbose | If assigned, stores Fun, evaluated at each iteration step. Optionally, you can also pass TOptControl object to the Verbose parameter. This allows the optimization procedure to be interrupted from another thread and optionally also allows logging and iteration count monitoring. |
| 16 | FloatPrecision | Specifies the floating point precision to be used by the routine. |
Returns: Int32 - the number of iterations required to reach the solution(minimum) within given tolerance.
What it computes. Minimizes a smooth real function f of n
variables with the quasi-Newton (variable-metric) method. It is a gradient-based method - it requires the gradient via Grad (exact, or a finite-difference approximation). Each iteration: (1) forms the search direction d = -H grad f from the current inverse-Hessian estimate H (which starts as the identity); (2) runs a line search along d to pick a step length alpha - the line search itself is gradient based, and its acceptance rule (soft/Armijo vs exact/Wolfe) is selected by SoftLineSearch; (3) moves x <- x + alpha d; and (4) updates H from the parameter and gradient differences using the BFGS rank-2 formula (DFPAlgo=False) or the Davidon-Fletcher-Powell formula (DFPAlgo=True). The standard test objective is
f(x_0,x_1) = 100 (x_1 - x_0^2)^2 + (1 - x_0)^2
with the global minimum f=0 at and gradient
(d f)/(d x_0) = -400 (x_1 - x_0^2) x_0 - 2 (1 - x_0), (d f)/(d x_1) = 200 (x_1 - x_0^2)
Domain. Pars holds the n starting coordinates;
Grad must return the exact gradient (set SoftLineSearch=True instead when supplying a numerical gradient); MaxIter > 0; Tol,GradTol ≥ 0.
Defined behaviour. Returns the iteration count; Pars
holds the minimizer, FMin the minimum value and IHess the final inverse-Hessian estimate. StopReason reports why the search stopped: TOptStopReason.OptResConverged - the parameter step fell within Tol and the gradient has collapsed (its C-norm is below GradTol, or below 1% of its starting value), i.e. a genuine minimum; TOptStopReason.OptResSmallGrad - the gradient C-norm fell below GradTol; TOptStopReason.OptResSmallStep - the parameter step became negligible while the gradient is still large, i.e. the line search stalled short of a minimum (this is reported as such, not as a false convergence); or a non-convergence cause such as TOptStopReason.OptResMaxIter. A NaN from the objective propagates into FMin.
Uses MtxVec, Math387, Optimization, MtxIntDiff;
function Banana(const Pars: TVec; const Consts: TVec; const OConsts: Array of TObject): double;
begin
Banana := 100*Sqr(Pars[1]-Sqr(Pars[0]))+Sqr(1-Pars[0]);
end;
procedure GradBanana(Fun: TRealFunction; const Pars: TVec; const Consts: TVec; const ObjConsts: Array of TObject; const Grad: TVec);
begin
Grad[0] := -400*(Pars[1] - Sqr(Pars[0]))*Pars[0] - 2*(1 - Pars[0]);
Grad[1] := 200*(Pars[1] - Sqr(Pars[0]));
end;
procedure Example;
var Iters : integer;
Pars : Array [0..1] of double;
StopReason : TOptStopReason;
begin
// initial estimates for x1 and x2
Pars[0] := 0;
Pars[1] := 0;
Iters := BFGS(Banana,GradBanana,Pars,[],[],FMin,StopReason,mvDouble,IHess);
//stop if Iters > 500 or Tolerance < 1e-8
// Returns Pars = [1,1] and FMin = 0, meaning x1=1, x2=1 and minimum value is 0
end;Overload 3: function BFGS(Fun: TRealFunction; Grad: TGrad; var Pars: TDoubleArray; const Consts: TDoubleArray; const ObjConst: TObjectArray; out FMin: Double; const IHess: TMtx; out StopReason: TOptStopReason; const FloatPrecision: TMtxFloatPrecision; DFPAlgo: Boolean; MaxIter: Integer; Tol: Double; GradTol: Double): Integer;
Minimizes the function of several variables by using the Quasi-Newton optimization method with no log.
| # | Name | Type | Description |
|---|---|---|---|
| 1 | Fun | TRealFunction | |
| 2 | Grad | TGrad | |
| 3 | Pars | TDoubleArray | |
| 4 | Consts | TDoubleArray | |
| 5 | ObjConst | TObjectArray | |
| 6 | FMin | Double | |
| 7 | IHess | TMtx | |
| 8 | StopReason | TOptStopReason | |
| 9 | FloatPrecision | TMtxFloatPrecision | |
| 10 | DFPAlgo | Boolean | |
| 11 | MaxIter | Integer | |
| 12 | Tol | Double | scalar |
| 13 | GradTol | Double | scalar |
Returns: Int32
Overload 4: function BFGS(Fun: TRealFunction; Grad: TGrad; var Pars: TDoubleArray; const Consts: TDoubleArray; const ObjConst: TObjectArray; out FMin: Double; const FloatPrecision: TMtxFloatPrecision; DFPAlgo: Boolean): Integer;
Minimizes the function of several variables by using the Quasi-Newton optimization method with no log.
| # | Name | Type | Description |
|---|---|---|---|
| 1 | Fun | TRealFunction | |
| 2 | Grad | TGrad | |
| 3 | Pars | TDoubleArray | |
| 4 | Consts | TDoubleArray | |
| 5 | ObjConst | TObjectArray | |
| 6 | FMin | Double | |
| 7 | FloatPrecision | TMtxFloatPrecision | |
| 8 | DFPAlgo | Boolean |
Returns: Int32