optimize.optimize

Module Contents

Classes

MemoizeJac(self,fun) Decorator that caches the value gradient of function each time it
OptimizeResult() Represents the optimization result.
OptimizeWarning()
_LineSearchError()
Brent(self,func,args=tuple,tol=1.48e-08,maxiter=500,full_output=0)

Functions

_check_unknown_options(unknown_options)
is_array_scalar(x) Test whether x is either a scalar or an array scalar.
vecnorm(x,ord=2)
rosen(x) The Rosenbrock function.
rosen_der(x) The derivative (i.e. gradient) of the Rosenbrock function.
rosen_hess(x) The Hessian matrix of the Rosenbrock function.
rosen_hess_prod(x,p) Product of the Hessian matrix of the Rosenbrock function with a vector.
wrap_function(function,args)
fmin(func,x0,args=tuple,xtol=0.0001,ftol=0.0001,maxiter=None,maxfun=None,full_output=0,disp=1,retall=0,callback=None,initial_simplex=None) Minimize a function using the downhill simplex algorithm.
_minimize_neldermead(func,x0,args=tuple,callback=None,maxiter=None,maxfev=None,disp=False,return_all=False,initial_simplex=None,xatol=0.0001,fatol=0.0001,**unknown_options) Minimization of scalar function of one or more variables using the
_approx_fprime_helper(xk,f,epsilon,args=tuple,f0=None) See approx_fprime. An optional initial function value arg is added.
approx_fprime(xk,f,epsilon,*args) Finite-difference approximation of the gradient of a scalar function.
check_grad(func,grad,x0,*args,**kwargs) Check the correctness of a gradient function by comparing it against a
approx_fhess_p(x0,p,fprime,epsilon,*args)
_line_search_wolfe12(f,fprime,xk,pk,gfk,old_fval,old_old_fval,**kwargs) Same as line_search_wolfe1, but fall back to line_search_wolfe2 if
fmin_bfgs(f,x0,fprime=None,args=tuple,gtol=1e-05,norm=Inf,epsilon=_epsilon,maxiter=None,full_output=0,disp=1,retall=0,callback=None) Minimize a function using the BFGS algorithm.
_minimize_bfgs(fun,x0,args=tuple,jac=None,callback=None,gtol=1e-05,norm=Inf,eps=_epsilon,maxiter=None,disp=False,return_all=False,**unknown_options) Minimization of scalar function of one or more variables using the
fmin_cg(f,x0,fprime=None,args=tuple,gtol=1e-05,norm=Inf,epsilon=_epsilon,maxiter=None,full_output=0,disp=1,retall=0,callback=None) Minimize a function using a nonlinear conjugate gradient algorithm.
_minimize_cg(fun,x0,args=tuple,jac=None,callback=None,gtol=1e-05,norm=Inf,eps=_epsilon,maxiter=None,disp=False,return_all=False,**unknown_options) Minimization of scalar function of one or more variables using the
fmin_ncg(f,x0,fprime,fhess_p=None,fhess=None,args=tuple,avextol=1e-05,epsilon=_epsilon,maxiter=None,full_output=0,disp=1,retall=0,callback=None) Unconstrained minimization of a function using the Newton-CG method.
_minimize_newtoncg(fun,x0,args=tuple,jac=None,hess=None,hessp=None,callback=None,xtol=1e-05,eps=_epsilon,maxiter=None,disp=False,return_all=False,**unknown_options) Minimization of scalar function of one or more variables using the
fminbound(func,x1,x2,args=tuple,xtol=1e-05,maxfun=500,full_output=0,disp=1) Bounded minimization for scalar functions.
_minimize_scalar_bounded(func,bounds,args=tuple,xatol=1e-05,maxiter=500,disp=0,**unknown_options) Options
brent(func,args=tuple,brack=None,tol=1.48e-08,full_output=0,maxiter=500) Given a function of one-variable and a possible bracket, return
_minimize_scalar_brent(func,brack=None,args=tuple,xtol=1.48e-08,maxiter=500,**unknown_options) Options
golden(func,args=tuple,brack=None,tol=_epsilon,full_output=0,maxiter=5000) Return the minimum of a function of one variable using golden section
_minimize_scalar_golden(func,brack=None,args=tuple,xtol=_epsilon,maxiter=5000,**unknown_options) Options
bracket(func,xa=0.0,xb=1.0,args=tuple,grow_limit=110.0,maxiter=1000) Bracket the minimum of the function.
_linesearch_powell(func,p,xi,tol=0.001) Line-search algorithm using fminbound.
fmin_powell(func,x0,args=tuple,xtol=0.0001,ftol=0.0001,maxiter=None,maxfun=None,full_output=0,disp=1,retall=0,callback=None,direc=None) Minimize a function using modified Powell’s method. This method
_minimize_powell(func,x0,args=tuple,callback=None,xtol=0.0001,ftol=0.0001,maxiter=None,maxfev=None,disp=False,direc=None,return_all=False,**unknown_options) Minimization of scalar function of one or more variables using the
_endprint(x,flag,fval,maxfun,xtol,disp)
brute(func,ranges,args=tuple,Ns=20,full_output=0,finish=fmin,disp=False) Minimize a function over a given range by brute force.
show_options(solver=None,method=None,disp=True) Show documentation for additional options of optimization solvers.
main()
class MemoizeJac(fun)

Decorator that caches the value gradient of function each time it is called.

__init__(fun)
__call__(x, *args)
derivative(x, *args)
class OptimizeResult

Represents the optimization result.

x : ndarray
The solution of the optimization.
success : bool
Whether or not the optimizer exited successfully.
status : int
Termination status of the optimizer. Its value depends on the underlying solver. Refer to message for details.
message : str
Description of the cause of the termination.
fun, jac, hess: ndarray
Values of objective function, its Jacobian and its Hessian (if available). The Hessians may be approximations, see the documentation of the function in question.
hess_inv : object
Inverse of the objective function’s Hessian; may be an approximation. Not available for all solvers. The type of this attribute may be either np.ndarray or scipy.sparse.linalg.LinearOperator.
nfev, njev, nhev : int
Number of evaluations of the objective functions and of its Jacobian and Hessian.
nit : int
Number of iterations performed by the optimizer.
maxcv : float
The maximum constraint violation.

There may be additional attributes not listed above depending of the specific solver. Since this class is essentially a subclass of dict with attribute accessors, one can see which attributes are available using the keys() method.

__getattr__(name)
__repr__()
__dir__()
class OptimizeWarning
_check_unknown_options(unknown_options)
is_array_scalar(x)

Test whether x is either a scalar or an array scalar.

vecnorm(x, ord=2)
rosen(x)

The Rosenbrock function.

The function computed is:

sum(100.0*(x[1:] - x[:-1]**2.0)**2.0 + (1 - x[:-1])**2.0)
x : array_like
1-D array of points at which the Rosenbrock function is to be computed.
f : float
The value of the Rosenbrock function.

rosen_der, rosen_hess, rosen_hess_prod

rosen_der(x)

The derivative (i.e. gradient) of the Rosenbrock function.

x : array_like
1-D array of points at which the derivative is to be computed.
rosen_der : (N,) ndarray
The gradient of the Rosenbrock function at x.

rosen, rosen_hess, rosen_hess_prod

rosen_hess(x)

The Hessian matrix of the Rosenbrock function.

x : array_like
1-D array of points at which the Hessian matrix is to be computed.
rosen_hess : ndarray
The Hessian matrix of the Rosenbrock function at x.

rosen, rosen_der, rosen_hess_prod

rosen_hess_prod(x, p)

Product of the Hessian matrix of the Rosenbrock function with a vector.

x : array_like
1-D array of points at which the Hessian matrix is to be computed.
p : array_like
1-D array, the vector to be multiplied by the Hessian matrix.
rosen_hess_prod : ndarray
The Hessian matrix of the Rosenbrock function at x multiplied by the vector p.

rosen, rosen_der, rosen_hess

wrap_function(function, args)
fmin(func, x0, args=tuple, xtol=0.0001, ftol=0.0001, maxiter=None, maxfun=None, full_output=0, disp=1, retall=0, callback=None, initial_simplex=None)

Minimize a function using the downhill simplex algorithm.

This algorithm only uses function values, not derivatives or second derivatives.

func : callable func(x,*args)
The objective function to be minimized.
x0 : ndarray
Initial guess.
args : tuple, optional
Extra arguments passed to func, i.e. f(x,*args).
xtol : float, optional
Absolute error in xopt between iterations that is acceptable for convergence.
ftol : number, optional
Absolute error in func(xopt) between iterations that is acceptable for convergence.
maxiter : int, optional
Maximum number of iterations to perform.
maxfun : number, optional
Maximum number of function evaluations to make.
full_output : bool, optional
Set to True if fopt and warnflag outputs are desired.
disp : bool, optional
Set to True to print convergence messages.
retall : bool, optional
Set to True to return list of solutions at each iteration.
callback : callable, optional
Called after each iteration, as callback(xk), where xk is the current parameter vector.
initial_simplex : array_like of shape (N + 1, N), optional
Initial simplex. If given, overrides x0. initial_simplex[j,:] should contain the coordinates of the j-th vertex of the N+1 vertices in the simplex, where N is the dimension.
xopt : ndarray
Parameter that minimizes function.
fopt : float
Value of function at minimum: fopt = func(xopt).
iter : int
Number of iterations performed.
funcalls : int
Number of function calls made.
warnflag : int
1 : Maximum number of function evaluations made. 2 : Maximum number of iterations reached.
allvecs : list
Solution at each iteration.
minimize: Interface to minimization algorithms for multivariate
functions. See the ‘Nelder-Mead’ method in particular.

Uses a Nelder-Mead simplex algorithm to find the minimum of function of one or more variables.

This algorithm has a long history of successful use in applications. But it will usually be slower than an algorithm that uses first or second derivative information. In practice it can have poor performance in high-dimensional problems and is not robust to minimizing complicated functions. Additionally, there currently is no complete theory describing when the algorithm will successfully converge to the minimum, or how fast it will if it does. Both the ftol and xtol criteria must be met for convergence.

>>> def f(x):
...     return x**2
>>> from scipy import optimize
>>> minimum = optimize.fmin(f, 1)
Optimization terminated successfully.
         Current function value: 0.000000
         Iterations: 17
         Function evaluations: 34
>>> minimum[0]
-8.8817841970012523e-16
[1]Nelder, J.A. and Mead, R. (1965), “A simplex method for function minimization”, The Computer Journal, 7, pp. 308-313
[2]Wright, M.H. (1996), “Direct Search Methods: Once Scorned, Now Respectable”, in Numerical Analysis 1995, Proceedings of the 1995 Dundee Biennial Conference in Numerical Analysis, D.F. Griffiths and G.A. Watson (Eds.), Addison Wesley Longman, Harlow, UK, pp. 191-208.
_minimize_neldermead(func, x0, args=tuple, callback=None, maxiter=None, maxfev=None, disp=False, return_all=False, initial_simplex=None, xatol=0.0001, fatol=0.0001, **unknown_options)

Minimization of scalar function of one or more variables using the Nelder-Mead algorithm.

disp : bool
Set to True to print convergence messages.
maxiter, maxfev : int
Maximum allowed number of iterations and function evaluations. Will default to N*200, where N is the number of variables, if neither maxiter or maxfev is set. If both maxiter and maxfev are set, minimization will stop at the first reached.
initial_simplex : array_like of shape (N + 1, N)
Initial simplex. If given, overrides x0. initial_simplex[j,:] should contain the coordinates of the j-th vertex of the N+1 vertices in the simplex, where N is the dimension.
xatol : float, optional
Absolute error in xopt between iterations that is acceptable for convergence.
fatol : number, optional
Absolute error in func(xopt) between iterations that is acceptable for convergence.
_approx_fprime_helper(xk, f, epsilon, args=tuple, f0=None)

See approx_fprime. An optional initial function value arg is added.

approx_fprime(xk, f, epsilon, *args)

Finite-difference approximation of the gradient of a scalar function.

xk : array_like
The coordinate vector at which to determine the gradient of f.
f : callable
The function of which to determine the gradient (partial derivatives). Should take xk as first argument, other arguments to f can be supplied in *args. Should return a scalar, the value of the function at xk.
epsilon : array_like
Increment to xk to use for determining the function gradient. If a scalar, uses the same finite difference delta for all partial derivatives. If an array, should contain one value per element of xk.
\*args : args, optional
Any other arguments that are to be passed to f.
grad : ndarray
The partial derivatives of f to xk.

check_grad : Check correctness of gradient function against approx_fprime.

The function gradient is determined by the forward finite difference formula:

         f(xk[i] + epsilon[i]) - f(xk[i])
f'[i] = ---------------------------------
                    epsilon[i]

The main use of approx_fprime is in scalar function optimizers like fmin_bfgs, to determine numerically the Jacobian of a function.

>>> from scipy import optimize
>>> def func(x, c0, c1):
...     "Coordinate vector `x` should be an array of size two."
...     return c0 * x[0]**2 + c1*x[1]**2
>>> x = np.ones(2)
>>> c0, c1 = (1, 200)
>>> eps = np.sqrt(np.finfo(float).eps)
>>> optimize.approx_fprime(x, func, [eps, np.sqrt(200) * eps], c0, c1)
array([   2.        ,  400.00004198])
check_grad(func, grad, x0, *args, **kwargs)

Check the correctness of a gradient function by comparing it against a (forward) finite-difference approximation of the gradient.

func : callable func(x0, *args)
Function whose derivative is to be checked.
grad : callable grad(x0, *args)
Gradient of func.
x0 : ndarray
Points to check grad against forward difference approximation of grad using func.
args : \*args, optional
Extra arguments passed to func and grad.
epsilon : float, optional
Step size used for the finite difference approximation. It defaults to sqrt(numpy.finfo(float).eps), which is approximately 1.49e-08.
err : float
The square root of the sum of squares (i.e. the 2-norm) of the difference between grad(x0, *args) and the finite difference approximation of grad using func at the points x0.

approx_fprime

>>> def func(x):
...     return x[0]**2 - 0.5 * x[1]**3
>>> def grad(x):
...     return [2 * x[0], -1.5 * x[1]**2]
>>> from scipy.optimize import check_grad
>>> check_grad(func, grad, [1.5, -1.5])
2.9802322387695312e-08
approx_fhess_p(x0, p, fprime, epsilon, *args)
class _LineSearchError
_line_search_wolfe12(f, fprime, xk, pk, gfk, old_fval, old_old_fval, **kwargs)

Same as line_search_wolfe1, but fall back to line_search_wolfe2 if suitable step length is not found, and raise an exception if a suitable step length is not found.

_LineSearchError
If no suitable step size is found
fmin_bfgs(f, x0, fprime=None, args=tuple, gtol=1e-05, norm=Inf, epsilon=_epsilon, maxiter=None, full_output=0, disp=1, retall=0, callback=None)

Minimize a function using the BFGS algorithm.

f : callable f(x,*args)
Objective function to be minimized.
x0 : ndarray
Initial guess.
fprime : callable f’(x,*args), optional
Gradient of f.
args : tuple, optional
Extra arguments passed to f and fprime.
gtol : float, optional
Gradient norm must be less than gtol before successful termination.
norm : float, optional
Order of norm (Inf is max, -Inf is min)
epsilon : int or ndarray, optional
If fprime is approximated, use this value for the step size.
callback : callable, optional
An optional user-supplied function to call after each iteration. Called as callback(xk), where xk is the current parameter vector.
maxiter : int, optional
Maximum number of iterations to perform.
full_output : bool, optional
If True,return fopt, func_calls, grad_calls, and warnflag in addition to xopt.
disp : bool, optional
Print convergence message if True.
retall : bool, optional
Return a list of results at each iteration if True.
xopt : ndarray
Parameters which minimize f, i.e. f(xopt) == fopt.
fopt : float
Minimum value.
gopt : ndarray
Value of gradient at minimum, f’(xopt), which should be near 0.
Bopt : ndarray
Value of 1/f’‘(xopt), i.e. the inverse hessian matrix.
func_calls : int
Number of function_calls made.
grad_calls : int
Number of gradient calls made.
warnflag : integer
1 : Maximum number of iterations exceeded. 2 : Gradient and/or function calls not changing.
allvecs : list
OptimizeResult at each iteration. Only returned if retall is True.
minimize: Interface to minimization algorithms for multivariate
functions. See the ‘BFGS’ method in particular.

Optimize the function, f, whose gradient is given by fprime using the quasi-Newton method of Broyden, Fletcher, Goldfarb, and Shanno (BFGS)

Wright, and Nocedal ‘Numerical Optimization’, 1999, pg. 198.

_minimize_bfgs(fun, x0, args=tuple, jac=None, callback=None, gtol=1e-05, norm=Inf, eps=_epsilon, maxiter=None, disp=False, return_all=False, **unknown_options)

Minimization of scalar function of one or more variables using the BFGS algorithm.

disp : bool
Set to True to print convergence messages.
maxiter : int
Maximum number of iterations to perform.
gtol : float
Gradient norm must be less than gtol before successful termination.
norm : float
Order of norm (Inf is max, -Inf is min).
eps : float or ndarray
If jac is approximated, use this value for the step size.
fmin_cg(f, x0, fprime=None, args=tuple, gtol=1e-05, norm=Inf, epsilon=_epsilon, maxiter=None, full_output=0, disp=1, retall=0, callback=None)

Minimize a function using a nonlinear conjugate gradient algorithm.

f : callable, f(x, *args)
Objective function to be minimized. Here x must be a 1-D array of the variables that are to be changed in the search for a minimum, and args are the other (fixed) parameters of f.
x0 : ndarray
A user-supplied initial estimate of xopt, the optimal value of x. It must be a 1-D array of values.
fprime : callable, fprime(x, *args), optional
A function that returns the gradient of f at x. Here x and args are as described above for f. The returned value must be a 1-D array. Defaults to None, in which case the gradient is approximated numerically (see epsilon, below).
args : tuple, optional
Parameter values passed to f and fprime. Must be supplied whenever additional fixed parameters are needed to completely specify the functions f and fprime.
gtol : float, optional
Stop when the norm of the gradient is less than gtol.
norm : float, optional
Order to use for the norm of the gradient (-np.Inf is min, np.Inf is max).
epsilon : float or ndarray, optional
Step size(s) to use when fprime is approximated numerically. Can be a scalar or a 1-D array. Defaults to sqrt(eps), with eps the floating point machine precision. Usually sqrt(eps) is about 1.5e-8.
maxiter : int, optional
Maximum number of iterations to perform. Default is 200 * len(x0).
full_output : bool, optional
If True, return fopt, func_calls, grad_calls, and warnflag in addition to xopt. See the Returns section below for additional information on optional return values.
disp : bool, optional
If True, return a convergence message, followed by xopt.
retall : bool, optional
If True, add to the returned values the results of each iteration.
callback : callable, optional
An optional user-supplied function, called after each iteration. Called as callback(xk), where xk is the current value of x0.
xopt : ndarray
Parameters which minimize f, i.e. f(xopt) == fopt.
fopt : float, optional
Minimum value found, f(xopt). Only returned if full_output is True.
func_calls : int, optional
The number of function_calls made. Only returned if full_output is True.
grad_calls : int, optional
The number of gradient calls made. Only returned if full_output is True.
warnflag : int, optional

Integer value with warning status, only returned if full_output is True.

0 : Success.

1 : The maximum number of iterations was exceeded.

2 : Gradient and/or function calls were not changing. May indicate
that precision was lost, i.e., the routine did not converge.
allvecs : list of ndarray, optional
List of arrays, containing the results at each iteration. Only returned if retall is True.
minimize : common interface to all scipy.optimize algorithms for
unconstrained and constrained minimization of multivariate functions. It provides an alternative way to call fmin_cg, by specifying method='CG'.

This conjugate gradient algorithm is based on that of Polak and Ribiere [1]_.

Conjugate gradient methods tend to work better when:

  1. f has a unique global minimizing point, and no local minima or other stationary points,
  2. f is, at least locally, reasonably well approximated by a quadratic function of the variables,
  3. f is continuous and has a continuous gradient,
  4. fprime is not too large, e.g., has a norm less than 1000,
  5. The initial guess, x0, is reasonably close to f ‘s global minimizing point, xopt.
[1]Wright & Nocedal, “Numerical Optimization”, 1999, pp. 120-122.

Example 1: seek the minimum value of the expression a*u**2 + b*u*v + c*v**2 + d*u + e*v + f for given values of the parameters and an initial guess (u, v) = (0, 0).

>>> args = (2, 3, 7, 8, 9, 10)  # parameter values
>>> def f(x, *args):
...     u, v = x
...     a, b, c, d, e, f = args
...     return a*u**2 + b*u*v + c*v**2 + d*u + e*v + f
>>> def gradf(x, *args):
...     u, v = x
...     a, b, c, d, e, f = args
...     gu = 2*a*u + b*v + d     # u-component of the gradient
...     gv = b*u + 2*c*v + e     # v-component of the gradient
...     return np.asarray((gu, gv))
>>> x0 = np.asarray((0, 0))  # Initial guess.
>>> from scipy import optimize
>>> res1 = optimize.fmin_cg(f, x0, fprime=gradf, args=args)
Optimization terminated successfully.
         Current function value: 1.617021
         Iterations: 4
         Function evaluations: 8
         Gradient evaluations: 8
>>> res1
array([-1.80851064, -0.25531915])

Example 2: solve the same problem using the minimize function. (This myopts dictionary shows all of the available options, although in practice only non-default values would be needed. The returned value will be a dictionary.)

>>> opts = {'maxiter' : None,    # default value.
...         'disp' : True,    # non-default value.
...         'gtol' : 1e-5,    # default value.
...         'norm' : np.inf,  # default value.
...         'eps' : 1.4901161193847656e-08}  # default value.
>>> res2 = optimize.minimize(f, x0, jac=gradf, args=args,
...                          method='CG', options=opts)
Optimization terminated successfully.
        Current function value: 1.617021
        Iterations: 4
        Function evaluations: 8
        Gradient evaluations: 8
>>> res2.x  # minimum found
array([-1.80851064, -0.25531915])
_minimize_cg(fun, x0, args=tuple, jac=None, callback=None, gtol=1e-05, norm=Inf, eps=_epsilon, maxiter=None, disp=False, return_all=False, **unknown_options)

Minimization of scalar function of one or more variables using the conjugate gradient algorithm.

disp : bool
Set to True to print convergence messages.
maxiter : int
Maximum number of iterations to perform.
gtol : float
Gradient norm must be less than gtol before successful termination.
norm : float
Order of norm (Inf is max, -Inf is min).
eps : float or ndarray
If jac is approximated, use this value for the step size.
fmin_ncg(f, x0, fprime, fhess_p=None, fhess=None, args=tuple, avextol=1e-05, epsilon=_epsilon, maxiter=None, full_output=0, disp=1, retall=0, callback=None)

Unconstrained minimization of a function using the Newton-CG method.

f : callable f(x, *args)
Objective function to be minimized.
x0 : ndarray
Initial guess.
fprime : callable f'(x, *args)
Gradient of f.
fhess_p : callable fhess_p(x, p, *args), optional
Function which computes the Hessian of f times an arbitrary vector, p.
fhess : callable fhess(x, *args), optional
Function to compute the Hessian matrix of f.
args : tuple, optional
Extra arguments passed to f, fprime, fhess_p, and fhess (the same set of extra arguments is supplied to all of these functions).
epsilon : float or ndarray, optional
If fhess is approximated, use this value for the step size.
callback : callable, optional
An optional user-supplied function which is called after each iteration. Called as callback(xk), where xk is the current parameter vector.
avextol : float, optional
Convergence is assumed when the average relative error in the minimizer falls below this amount.
maxiter : int, optional
Maximum number of iterations to perform.
full_output : bool, optional
If True, return the optional outputs.
disp : bool, optional
If True, print convergence message.
retall : bool, optional
If True, return a list of results at each iteration.
xopt : ndarray
Parameters which minimize f, i.e. f(xopt) == fopt.
fopt : float
Value of the function at xopt, i.e. fopt = f(xopt).
fcalls : int
Number of function calls made.
gcalls : int
Number of gradient calls made.
hcalls : int
Number of hessian calls made.
warnflag : int
Warnings generated by the algorithm. 1 : Maximum number of iterations exceeded.
allvecs : list
The result at each iteration, if retall is True (see below).
minimize: Interface to minimization algorithms for multivariate
functions. See the ‘Newton-CG’ method in particular.

Only one of fhess_p or fhess need to be given. If fhess is provided, then fhess_p will be ignored. If neither fhess nor fhess_p is provided, then the hessian product will be approximated using finite differences on fprime. fhess_p must compute the hessian times an arbitrary vector. If it is not given, finite-differences on fprime are used to compute it.

Newton-CG methods are also called truncated Newton methods. This function differs from scipy.optimize.fmin_tnc because

  1. scipy.optimize.fmin_ncg is written purely in python using numpy
    and scipy while scipy.optimize.fmin_tnc calls a C function.
  2. scipy.optimize.fmin_ncg is only for unconstrained minimization
    while scipy.optimize.fmin_tnc is for unconstrained minimization or box constrained minimization. (Box constraints give lower and upper bounds for each variable separately.)

Wright & Nocedal, ‘Numerical Optimization’, 1999, pg. 140.

_minimize_newtoncg(fun, x0, args=tuple, jac=None, hess=None, hessp=None, callback=None, xtol=1e-05, eps=_epsilon, maxiter=None, disp=False, return_all=False, **unknown_options)

Minimization of scalar function of one or more variables using the Newton-CG algorithm.

Note that the jac parameter (Jacobian) is required.

disp : bool
Set to True to print convergence messages.
xtol : float
Average relative error in solution xopt acceptable for convergence.
maxiter : int
Maximum number of iterations to perform.
eps : float or ndarray
If jac is approximated, use this value for the step size.
fminbound(func, x1, x2, args=tuple, xtol=1e-05, maxfun=500, full_output=0, disp=1)

Bounded minimization for scalar functions.

func : callable f(x,*args)
Objective function to be minimized (must accept and return scalars).
x1, x2 : float or array scalar
The optimization bounds.
args : tuple, optional
Extra arguments passed to function.
xtol : float, optional
The convergence tolerance.
maxfun : int, optional
Maximum number of function evaluations allowed.
full_output : bool, optional
If True, return optional outputs.
disp : int, optional
If non-zero, print messages.
0 : no message printing. 1 : non-convergence notification messages only. 2 : print a message on convergence too. 3 : print iteration results.
xopt : ndarray
Parameters (over given interval) which minimize the objective function.
fval : number
The function value at the minimum point.
ierr : int
An error flag (0 if converged, 1 if maximum number of function calls reached).
numfunc : int
The number of function calls made.
minimize_scalar: Interface to minimization algorithms for scalar
univariate functions. See the ‘Bounded’ method in particular.

Finds a local minimizer of the scalar function func in the interval x1 < xopt < x2 using Brent’s method. (See brent for auto-bracketing).

fminbound finds the minimum of the function in the given range. The following examples illustrate the same

>>> def f(x):
...     return x**2
>>> from scipy import optimize
>>> minimum = optimize.fminbound(f, -1, 2)
>>> minimum
0.0
>>> minimum = optimize.fminbound(f, 1, 2)
>>> minimum
1.0000059608609866
_minimize_scalar_bounded(func, bounds, args=tuple, xatol=1e-05, maxiter=500, disp=0, **unknown_options)
maxiter : int
Maximum number of iterations to perform.
disp : bool
Set to True to print convergence messages.
xatol : float
Absolute error in solution xopt acceptable for convergence.
class Brent(func, args=tuple, tol=1.48e-08, maxiter=500, full_output=0)
__init__(func, args=tuple, tol=1.48e-08, maxiter=500, full_output=0)
set_bracket(brack=None)
get_bracket_info()
optimize()
get_result(full_output=False)
brent(func, args=tuple, brack=None, tol=1.48e-08, full_output=0, maxiter=500)

Given a function of one-variable and a possible bracket, return the local minimum of the function isolated to a fractional precision of tol.

func : callable f(x,*args)
Objective function.
args : tuple, optional
Additional arguments (if present).
brack : tuple, optional
Either a triple (xa,xb,xc) where xa<xb<xc and func(xb) < func(xa), func(xc) or a pair (xa,xb) which are used as a starting interval for a downhill bracket search (see bracket). Providing the pair (xa,xb) does not always mean the obtained solution will satisfy xa<=x<=xb.
tol : float, optional
Stop if between iteration change is less than tol.
full_output : bool, optional
If True, return all output args (xmin, fval, iter, funcalls).
maxiter : int, optional
Maximum number of iterations in solution.
xmin : ndarray
Optimum point.
fval : float
Optimum value.
iter : int
Number of iterations.
funcalls : int
Number of objective function evaluations made.
minimize_scalar: Interface to minimization algorithms for scalar
univariate functions. See the ‘Brent’ method in particular.

Uses inverse parabolic interpolation when possible to speed up convergence of golden section method.

Does not ensure that the minimum lies in the range specified by brack. See fminbound.

We illustrate the behaviour of the function when brack is of size 2 and 3 respectively. In the case where brack is of the form (xa,xb), we can see for the given values, the output need not necessarily lie in the range (xa,xb).

>>> def f(x):
...     return x**2
>>> from scipy import optimize
>>> minimum = optimize.brent(f,brack=(1,2))
>>> minimum
0.0
>>> minimum = optimize.brent(f,brack=(-1,0.5,2))
>>> minimum
-2.7755575615628914e-17
_minimize_scalar_brent(func, brack=None, args=tuple, xtol=1.48e-08, maxiter=500, **unknown_options)
maxiter : int
Maximum number of iterations to perform.
xtol : float
Relative error in solution xopt acceptable for convergence.

Uses inverse parabolic interpolation when possible to speed up convergence of golden section method.

golden(func, args=tuple, brack=None, tol=_epsilon, full_output=0, maxiter=5000)

Return the minimum of a function of one variable using golden section method.

Given a function of one variable and a possible bracketing interval, return the minimum of the function isolated to a fractional precision of tol.

func : callable func(x,*args)
Objective function to minimize.
args : tuple, optional
Additional arguments (if present), passed to func.
brack : tuple, optional
Triple (a,b,c), where (a<b<c) and func(b) < func(a),func(c). If bracket consists of two numbers (a, c), then they are assumed to be a starting interval for a downhill bracket search (see bracket); it doesn’t always mean that obtained solution will satisfy a<=x<=c.
tol : float, optional
x tolerance stop criterion
full_output : bool, optional
If True, return optional outputs.
maxiter : int
Maximum number of iterations to perform.
minimize_scalar: Interface to minimization algorithms for scalar
univariate functions. See the ‘Golden’ method in particular.

Uses analog of bisection method to decrease the bracketed interval.

We illustrate the behaviour of the function when brack is of size 2 and 3 respectively. In the case where brack is of the form (xa,xb), we can see for the given values, the output need not necessarily lie in the range (xa, xb).

>>> def f(x):
...     return x**2
>>> from scipy import optimize
>>> minimum = optimize.golden(f, brack=(1, 2))
>>> minimum
1.5717277788484873e-162
>>> minimum = optimize.golden(f, brack=(-1, 0.5, 2))
>>> minimum
-1.5717277788484873e-162
_minimize_scalar_golden(func, brack=None, args=tuple, xtol=_epsilon, maxiter=5000, **unknown_options)
maxiter : int
Maximum number of iterations to perform.
xtol : float
Relative error in solution xopt acceptable for convergence.
bracket(func, xa=0.0, xb=1.0, args=tuple, grow_limit=110.0, maxiter=1000)

Bracket the minimum of the function.

Given a function and distinct initial points, search in the downhill direction (as defined by the initital points) and return new points xa, xb, xc that bracket the minimum of the function f(xa) > f(xb) < f(xc). It doesn’t always mean that obtained solution will satisfy xa<=x<=xb

func : callable f(x,*args)
Objective function to minimize.
xa, xb : float, optional
Bracketing interval. Defaults xa to 0.0, and xb to 1.0.
args : tuple, optional
Additional arguments (if present), passed to func.
grow_limit : float, optional
Maximum grow limit. Defaults to 110.0
maxiter : int, optional
Maximum number of iterations to perform. Defaults to 1000.
xa, xb, xc : float
Bracket.
fa, fb, fc : float
Objective function values in bracket.
funcalls : int
Number of function evaluations made.
_linesearch_powell(func, p, xi, tol=0.001)

Line-search algorithm using fminbound.

Find the minimium of the function func(x0+ alpha*direc).

fmin_powell(func, x0, args=tuple, xtol=0.0001, ftol=0.0001, maxiter=None, maxfun=None, full_output=0, disp=1, retall=0, callback=None, direc=None)

Minimize a function using modified Powell’s method. This method only uses function values, not derivatives.

func : callable f(x,*args)
Objective function to be minimized.
x0 : ndarray
Initial guess.
args : tuple, optional
Extra arguments passed to func.
callback : callable, optional
An optional user-supplied function, called after each iteration. Called as callback(xk), where xk is the current parameter vector.
direc : ndarray, optional
Initial direction set.
xtol : float, optional
Line-search error tolerance.
ftol : float, optional
Relative error in func(xopt) acceptable for convergence.
maxiter : int, optional
Maximum number of iterations to perform.
maxfun : int, optional
Maximum number of function evaluations to make.
full_output : bool, optional
If True, fopt, xi, direc, iter, funcalls, and warnflag are returned.
disp : bool, optional
If True, print convergence messages.
retall : bool, optional
If True, return a list of the solution at each iteration.
xopt : ndarray
Parameter which minimizes func.
fopt : number
Value of function at minimum: fopt = func(xopt).
direc : ndarray
Current direction set.
iter : int
Number of iterations.
funcalls : int
Number of function calls made.
warnflag : int
Integer warning flag:
1 : Maximum number of function evaluations. 2 : Maximum number of iterations.
allvecs : list
List of solutions at each iteration.
minimize: Interface to unconstrained minimization algorithms for
multivariate functions. See the ‘Powell’ method in particular.

Uses a modification of Powell’s method to find the minimum of a function of N variables. Powell’s method is a conjugate direction method.

The algorithm has two loops. The outer loop merely iterates over the inner loop. The inner loop minimizes over each current direction in the direction set. At the end of the inner loop, if certain conditions are met, the direction that gave the largest decrease is dropped and replaced with the difference between the current estimated x and the estimated x from the beginning of the inner-loop.

The technical conditions for replacing the direction of greatest increase amount to checking that

  1. No further gain can be made along the direction of greatest increase from that iteration.
  2. The direction of greatest increase accounted for a large sufficient fraction of the decrease in the function value from that iteration of the inner loop.
>>> def f(x):
...     return x**2
>>> from scipy import optimize
>>> minimum = optimize.fmin_powell(f, -1)
Optimization terminated successfully.
         Current function value: 0.000000
         Iterations: 2
         Function evaluations: 18
>>> minimum
array(0.0)

Powell M.J.D. (1964) An efficient method for finding the minimum of a function of several variables without calculating derivatives, Computer Journal, 7 (2):155-162.

Press W., Teukolsky S.A., Vetterling W.T., and Flannery B.P.: Numerical Recipes (any edition), Cambridge University Press

_minimize_powell(func, x0, args=tuple, callback=None, xtol=0.0001, ftol=0.0001, maxiter=None, maxfev=None, disp=False, direc=None, return_all=False, **unknown_options)

Minimization of scalar function of one or more variables using the modified Powell algorithm.

disp : bool
Set to True to print convergence messages.
xtol : float
Relative error in solution xopt acceptable for convergence.
ftol : float
Relative error in fun(xopt) acceptable for convergence.
maxiter, maxfev : int
Maximum allowed number of iterations and function evaluations. Will default to N*1000, where N is the number of variables, if neither maxiter or maxfev is set. If both maxiter and maxfev are set, minimization will stop at the first reached.
direc : ndarray
Initial set of direction vectors for the Powell method.
_endprint(x, flag, fval, maxfun, xtol, disp)
brute(func, ranges, args=tuple, Ns=20, full_output=0, finish=fmin, disp=False)

Minimize a function over a given range by brute force.

Uses the “brute force” method, i.e. computes the function’s value at each point of a multidimensional grid of points, to find the global minimum of the function.

The function is evaluated everywhere in the range with the datatype of the first call to the function, as enforced by the vectorize NumPy function. The value and type of the function evaluation returned when full_output=True are affected in addition by the finish argument (see Notes).

func : callable
The objective function to be minimized. Must be in the form f(x, *args), where x is the argument in the form of a 1-D array and args is a tuple of any additional fixed parameters needed to completely specify the function.
ranges : tuple
Each component of the ranges tuple must be either a “slice object” or a range tuple of the form (low, high). The program uses these to create the grid of points on which the objective function will be computed. See Note 2 for more detail.
args : tuple, optional
Any additional fixed parameters needed to completely specify the function.
Ns : int, optional
Number of grid points along the axes, if not otherwise specified. See Note2.
full_output : bool, optional
If True, return the evaluation grid and the objective function’s values on it.
finish : callable, optional
An optimization function that is called with the result of brute force minimization as initial guess. finish should take func and the initial guess as positional arguments, and take args as keyword arguments. It may additionally take full_output and/or disp as keyword arguments. Use None if no “polishing” function is to be used. See Notes for more details.
disp : bool, optional
Set to True to print convergence messages.
x0 : ndarray
A 1-D array containing the coordinates of a point at which the objective function had its minimum value. (See Note 1 for which point is returned.)
fval : float
Function value at the point x0. (Returned when full_output is True.)
grid : tuple
Representation of the evaluation grid. It has the same length as x0. (Returned when full_output is True.)
Jout : ndarray
Function values at each point of the evaluation grid, i.e., Jout = func(*grid). (Returned when full_output is True.)

basinhopping, differential_evolution

Note 1: The program finds the gridpoint at which the lowest value of the objective function occurs. If finish is None, that is the point returned. When the global minimum occurs within (or not very far outside) the grid’s boundaries, and the grid is fine enough, that point will be in the neighborhood of the global minimum.

However, users often employ some other optimization program to “polish” the gridpoint values, i.e., to seek a more precise (local) minimum near brute’s best gridpoint. The brute function’s finish option provides a convenient way to do that. Any polishing program used must take brute’s output as its initial guess as a positional argument, and take brute’s input values for args as keyword arguments, otherwise an error will be raised. It may additionally take full_output and/or disp as keyword arguments.

brute assumes that the finish function returns either an OptimizeResult object or a tuple in the form: (xmin, Jmin, ... , statuscode), where xmin is the minimizing value of the argument, Jmin is the minimum value of the objective function, “…” may be some other returned values (which are not used by brute), and statuscode is the status code of the finish program.

Note that when finish is not None, the values returned are those of the finish program, not the gridpoint ones. Consequently, while brute confines its search to the input grid points, the finish program’s results usually will not coincide with any gridpoint, and may fall outside the grid’s boundary. Thus, if a minimum only needs to be found over the provided grid points, make sure to pass in finish=None.

Note 2: The grid of points is a numpy.mgrid object. For brute the ranges and Ns inputs have the following effect. Each component of the ranges tuple can be either a slice object or a two-tuple giving a range of values, such as (0, 5). If the component is a slice object, brute uses it directly. If the component is a two-tuple range, brute internally converts it to a slice object that interpolates Ns points from its low-value to its high-value, inclusive.

We illustrate the use of brute to seek the global minimum of a function of two variables that is given as the sum of a positive-definite quadratic and two deep “Gaussian-shaped” craters. Specifically, define the objective function f as the sum of three other functions, f = f1 + f2 + f3. We suppose each of these has a signature (z, *params), where z = (x, y), and params and the functions are as defined below.

>>> params = (2, 3, 7, 8, 9, 10, 44, -1, 2, 26, 1, -2, 0.5)
>>> def f1(z, *params):
...     x, y = z
...     a, b, c, d, e, f, g, h, i, j, k, l, scale = params
...     return (a * x**2 + b * x * y + c * y**2 + d*x + e*y + f)
>>> def f2(z, *params):
...     x, y = z
...     a, b, c, d, e, f, g, h, i, j, k, l, scale = params
...     return (-g*np.exp(-((x-h)**2 + (y-i)**2) / scale))
>>> def f3(z, *params):
...     x, y = z
...     a, b, c, d, e, f, g, h, i, j, k, l, scale = params
...     return (-j*np.exp(-((x-k)**2 + (y-l)**2) / scale))
>>> def f(z, *params):
...     return f1(z, *params) + f2(z, *params) + f3(z, *params)

Thus, the objective function may have local minima near the minimum of each of the three functions of which it is composed. To use fmin to polish its gridpoint result, we may then continue as follows:

>>> rranges = (slice(-4, 4, 0.25), slice(-4, 4, 0.25))
>>> from scipy import optimize
>>> resbrute = optimize.brute(f, rranges, args=params, full_output=True,
...                           finish=optimize.fmin)
>>> resbrute[0]  # global minimum
array([-1.05665192,  1.80834843])
>>> resbrute[1]  # function value at global minimum
-3.4085818767

Note that if finish had been set to None, we would have gotten the gridpoint [-1.0 1.75] where the rounded function value is -2.892.

show_options(solver=None, method=None, disp=True)

Show documentation for additional options of optimization solvers.

These are method-specific options that can be supplied through the options dict.

solver : str
Type of optimization solver. One of ‘minimize’, ‘minimize_scalar’, ‘root’, or ‘linprog’.
method : str, optional
If not given, shows all methods of the specified solver. Otherwise, show only the options for the specified method. Valid values corresponds to methods’ names of respective solver (e.g. ‘BFGS’ for ‘minimize’).
disp : bool, optional
Whether to print the result rather than returning it.
text
Either None (for disp=False) or the text string (disp=True)

The solver-specific methods are:

scipy.optimize.minimize

  • Nelder-Mead
  • Powell
  • CG
  • BFGS
  • Newton-CG
  • L-BFGS-B
  • TNC
  • COBYLA
  • SLSQP
  • dogleg
  • trust-ncg

scipy.optimize.root

  • hybr
  • lm
  • broyden1
  • broyden2
  • anderson
  • linearmixing
  • diagbroyden
  • excitingmixing
  • krylov
  • df-sane

scipy.optimize.minimize_scalar

  • brent
  • golden
  • bounded

scipy.optimize.linprog

  • simplex
  • interior-point
main()