#include <itkAdaptiveStochasticVarianceReducedGradientOptimizer.h>

Detailed Description

This class implements a gradient descent optimizer with adaptive gain.

If $C(x)$ is a cost function that has to be minimized, the following iterative algorithm is used to find the optimal parameters $x$ :

$x(k+1) = x(k) - a(t_k) dC/dx$

The gain $a(t_k)$ at each iteration $k$ is defined by:

$a(t_k) = a / (A + t_k + 1)^alpha$

.

And the time $t_k$ is updated according to:

$t_{k+1} = [ t_k + sigmoid( -g_k^T g_{k-1} ) ]^+$

where $g_k$ equals $dC/dx$ at iteration $k$ . For $t_0$ the InitialTime is used, which is defined in the the superclass (StandardGradientDescentOptimizer). Whereas in the superclass this parameter is superfluous, in this class it makes sense.

This method is described in the following references:

[1] P. Cruz, "Almost sure convergence and asymptotical normality of a generalization of Kesten's stochastic approximation algorithm for multidimensional case." Technical Report, 2005. http://hdl.handle.net/2052/74

[2] S. Klein, J.P.W. Pluim, and M. Staring, M.A. Viergever, "Adaptive stochastic gradient descent optimisation for image registration," International Journal of Computer Vision, vol. 81, no. 3, pp. 227-239, 2009. http://dx.doi.org/10.1007/s11263-008-0168-y It is very suitable to be used in combination with a stochastic estimate of the gradient $dC/dx$ . For example, in image registration problems it is often advantageous to compute the metric derivative ( $dC/dx$ ) on a new set of randomly selected image samples in each iteration. You may set the parameter NewSamplesEveryIteration to "true" to achieve this effect. For more information on this strategy, you may have a look at:

See also: AdaptiveStochasticVarianceReducedGradient, StandardGradientDescentOptimizer

If $C(x)$ is a costfunction that has to be minimised, the following iterative algorithm is used to find the optimal parameters $x$ :

$x(k+1) = x(k) - a(t_k) dC/dx$

The gain $a(t_k)$ at each iteration $k$ is defined by:

$a(t_k) = a / (A + t_k + 1)^alpha$

.

And the time $t_k$ is updated according to:

$t_{k+1} = [ t_k + sigmoid( -g_k^T g_{k-1} ) ]^+$

where $g_k$ equals $dC/dx$ at iteration $k$ . For $t_0$ the InitialTime is used, which is defined in the the superclass (StandardGradientDescentOptimizer). Whereas in the superclass this parameter is superfluous, in this class it makes sense.

This method is described in the following references:

[1] P. Cruz, "Almost sure convergence and asymptotical normality of a generalization of Kesten's stochastic approximation algorithm for multidimensional case." Technical Report, 2005. http://hdl.handle.net/2052/74

[2] S. Klein, J.P.W. Pluim, and M. Staring, M.A. Viergever, "Adaptive stochastic gradient descent optimisation for image registration," International Journal of Computer Vision, vol. 81, no. 3, pp. 227-239, 2009. http://dx.doi.org/10.1007/s11263-008-0168-y

It is very suitable to be used in combination with a stochastic estimate of the gradient $dC/dx$ . For example, in image registration problems it is often advantageous to compute the metric derivative ( $dC/dx$ ) on a new set of randomly selected image samples in each iteration. You may set the parameter NewSamplesEveryIteration to "true" to achieve this effect. For more information on this strategy, you may have a look at:

See also: AdaptiveStochasticVarianceReducedGradient, StandardGradientDescentOptimizer

Definition at line 70 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

Inheritance diagram for itk::AdaptiveStochasticVarianceReducedGradientOptimizer:

[legend]

Public Types
using	ConstPointer = SmartPointer<const Self>

using	Pointer = SmartPointer<Self>

using	Self = AdaptiveStochasticVarianceReducedGradientOptimizer

using	Superclass = StandardStochasticVarianceReducedGradientOptimizer

Public Types inherited from itk::StandardStochasticVarianceReducedGradientOptimizer
using	ConstPointer = SmartPointer<const Self>

using	Pointer = SmartPointer<Self>

using	Self = StandardStochasticVarianceReducedGradientOptimizer

enum	StopConditionType

using	Superclass = StochasticVarianceReducedGradientDescentOptimizer

Public Types inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
using	ConstPointer = SmartPointer<const Self>

using	Pointer = SmartPointer<Self>

using	ScaledCostFunctionPointer

using	ScaledCostFunctionType

using	ScalesType

using	Self = StochasticVarianceReducedGradientDescentOptimizer

enum	StopConditionType { MaximumNumberOfIterations , MetricError , MinimumStepSize , InvalidDiagonalMatrix , GradientMagnitudeTolerance , LineSearchError }

using	Superclass = ScaledSingleValuedNonLinearOptimizer

Public Types inherited from itk::ScaledSingleValuedNonLinearOptimizer
using	ConstPointer = SmartPointer<const Self>

using	Pointer = SmartPointer<Self>

using	ScaledCostFunctionPointer = ScaledCostFunctionType::Pointer

using	ScaledCostFunctionType = ScaledSingleValuedCostFunction

using	ScalesType = NonLinearOptimizer::ScalesType

using	Self = ScaledSingleValuedNonLinearOptimizer

using	Superclass = SingleValuedNonLinearOptimizer

Public Member Functions
virtual const char *	GetClassName () const

virtual double	GetSigmoidMax () const

virtual double	GetSigmoidMin () const

virtual double	GetSigmoidScale () const

virtual bool	GetUseAdaptiveStepSizes () const

	ITK_DISALLOW_COPY_AND_MOVE (AdaptiveStochasticVarianceReducedGradientOptimizer)

virtual void	SetSigmoidMax (double _arg)

virtual void	SetSigmoidMin (double _arg)

virtual void	SetSigmoidScale (double _arg)

virtual void	SetUseAdaptiveStepSizes (bool _arg)

Public Member Functions inherited from itk::StandardStochasticVarianceReducedGradientOptimizer
void	AdvanceOneStep () override

virtual double	GetCurrentTime () const

virtual double	GetInitialTime () const

virtual double	GetParam_A () const

virtual double	GetParam_a () const

virtual double	GetParam_alpha () const

virtual double	GetParam_beta () const

	ITK_DISALLOW_COPY_AND_MOVE (StandardStochasticVarianceReducedGradientOptimizer)

virtual void	ResetCurrentTimeToInitialTime ()

virtual void	SetInitialTime (double _arg)

virtual void	SetParam_A (double _arg)

virtual void	SetParam_a (double _arg)

virtual void	SetParam_alpha (double _arg)

virtual void	SetParam_beta (double _arg)

void	StartOptimization () override

Public Member Functions inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
virtual unsigned int	GetCurrentInnerIteration () const

virtual unsigned int	GetCurrentIteration () const

virtual const DerivativeType &	GetGradient ()

virtual unsigned int	GetLBFGSMemory () const

virtual const double &	GetLearningRate ()

virtual const unsigned long &	GetNumberOfInnerIterations ()

virtual const unsigned long &	GetNumberOfIterations ()

virtual const DerivativeType &	GetPreviousGradient ()

virtual const ParametersType &	GetPreviousPosition ()

virtual const DerivativeType &	GetSearchDir ()

virtual const StopConditionType &	GetStopCondition ()

virtual const double &	GetValue ()

	ITK_DISALLOW_COPY_AND_MOVE (StochasticVarianceReducedGradientDescentOptimizer)

virtual void	MetricErrorResponse (ExceptionObject &err)

virtual void	ResumeOptimization ()

virtual void	SetLearningRate (double _arg)

virtual void	SetNumberOfIterations (unsigned long _arg)

void	SetNumberOfWorkUnits (ThreadIdType numberOfThreads)

virtual void	SetPreviousGradient (DerivativeType _arg)

virtual void	SetPreviousPosition (ParametersType _arg)

virtual void	SetUseEigen (bool _arg)

virtual void	SetUseMultiThread (bool _arg)

void	StartOptimization () override

virtual void	StopOptimization ()

Public Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
const ParametersType &	GetCurrentPosition () const override

virtual bool	GetMaximize () const

virtual const ScaledCostFunctionType *	GetScaledCostFunction ()

virtual const ParametersType &	GetScaledCurrentPosition ()

bool	GetUseScales () const

virtual void	InitializeScales ()

	ITK_DISALLOW_COPY_AND_MOVE (ScaledSingleValuedNonLinearOptimizer)

virtual void	MaximizeOff ()

virtual void	MaximizeOn ()

void	SetCostFunction (CostFunctionType *costFunction) override

virtual void	SetMaximize (bool _arg)

virtual void	SetUseScales (bool arg)

Static Public Member Functions
static Pointer	New ()

Static Public Member Functions inherited from itk::StandardStochasticVarianceReducedGradientOptimizer
static Pointer	New ()

Static Public Member Functions inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
static Pointer	New ()

Static Public Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
static Pointer	New ()

Protected Member Functions
	AdaptiveStochasticVarianceReducedGradientOptimizer ()

void	UpdateCurrentTime () override

	~AdaptiveStochasticVarianceReducedGradientOptimizer () override=default

Protected Member Functions inherited from itk::StandardStochasticVarianceReducedGradientOptimizer
virtual double	Compute_a (double k) const

virtual double	Compute_beta (double k) const

	StandardStochasticVarianceReducedGradientOptimizer ()

	~StandardStochasticVarianceReducedGradientOptimizer () override=default

Protected Member Functions inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
void	PrintSelf (std::ostream &os, Indent indent) const override

	StochasticVarianceReducedGradientDescentOptimizer ()

	~StochasticVarianceReducedGradientDescentOptimizer () override=default

Protected Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
virtual void	GetScaledDerivative (const ParametersType &parameters, DerivativeType &derivative) const

virtual MeasureType	GetScaledValue (const ParametersType &parameters) const

virtual void	GetScaledValueAndDerivative (const ParametersType &parameters, MeasureType &value, DerivativeType &derivative) const

void	PrintSelf (std::ostream &os, Indent indent) const override

	ScaledSingleValuedNonLinearOptimizer ()

void	SetCurrentPosition (const ParametersType &param) override

virtual void	SetScaledCurrentPosition (const ParametersType &parameters)

	~ScaledSingleValuedNonLinearOptimizer () override=default

Protected Attributes
DerivativeType	m_PreviousGradient {}

Protected Attributes inherited from itk::StandardStochasticVarianceReducedGradientOptimizer
double	m_CurrentTime { 0.0 }

bool	m_UseConstantStep {}

Protected Attributes inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
unsigned long	m_CurrentInnerIteration {}

unsigned long	m_CurrentIteration { 0 }

DerivativeType	m_Gradient {}

unsigned long	m_LBFGSMemory { 0 }

double	m_LearningRate { 1.0 }

ParametersType	m_MeanSearchDir {}

unsigned long	m_NumberOfInnerIterations {}

unsigned long	m_NumberOfIterations { 100 }

DerivativeType	m_PreviousGradient {}

ParametersType	m_PreviousPosition {}

ParametersType	m_PreviousSearchDir {}

ParametersType	m_SearchDir {}

bool	m_Stop { false }

StopConditionType	m_StopCondition { MaximumNumberOfIterations }

MultiThreaderBase::Pointer	m_Threader { MultiThreaderBase::New() }

double	m_Value { 0.0 }

Protected Attributes inherited from itk::ScaledSingleValuedNonLinearOptimizer
ScaledCostFunctionPointer	m_ScaledCostFunction {}

ParametersType	m_ScaledCurrentPosition {}

Private Attributes
double	m_SigmoidMax { 1.0 }

double	m_SigmoidMin { -0.8 }

double	m_SigmoidScale { 1e-8 }

bool	m_UseAdaptiveStepSizes { true }

Additional Inherited Members
Protected Types inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
using	ThreadInfoType = MultiThreaderBase::WorkUnitInfo

Member Typedef Documentation

◆ ConstPointer

using itk::AdaptiveStochasticVarianceReducedGradientOptimizer::ConstPointer = SmartPointer<const Self>

Definition at line 80 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ Pointer

using itk::AdaptiveStochasticVarianceReducedGradientOptimizer::Pointer = SmartPointer<Self>

Definition at line 79 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ Self

using itk::AdaptiveStochasticVarianceReducedGradientOptimizer::Self = AdaptiveStochasticVarianceReducedGradientOptimizer

Standard ITK.

Definition at line 76 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ Superclass

using itk::AdaptiveStochasticVarianceReducedGradientOptimizer::Superclass = StandardStochasticVarianceReducedGradientOptimizer

Definition at line 77 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

Constructor & Destructor Documentation

◆ AdaptiveStochasticVarianceReducedGradientOptimizer()

itk::AdaptiveStochasticVarianceReducedGradientOptimizer::AdaptiveStochasticVarianceReducedGradientOptimizer ( )

protected

◆ ~AdaptiveStochasticVarianceReducedGradientOptimizer()

itk::AdaptiveStochasticVarianceReducedGradientOptimizer::~AdaptiveStochasticVarianceReducedGradientOptimizer ( )

overrideprotecteddefault

Member Function Documentation

◆ GetClassName()

virtual const char * itk::AdaptiveStochasticVarianceReducedGradientOptimizer::GetClassName ( ) const

virtual

Run-time type information (and related methods).

Reimplemented from itk::StandardStochasticVarianceReducedGradientOptimizer.

Reimplemented in elastix::AdaptiveStochasticVarianceReducedGradient< TElastix >.

◆ GetSigmoidMax()

virtual double itk::AdaptiveStochasticVarianceReducedGradientOptimizer::GetSigmoidMax ( ) const

virtual

◆ GetSigmoidMin()

virtual double itk::AdaptiveStochasticVarianceReducedGradientOptimizer::GetSigmoidMin ( ) const

virtual

◆ GetSigmoidScale()

virtual double itk::AdaptiveStochasticVarianceReducedGradientOptimizer::GetSigmoidScale ( ) const

virtual

◆ GetUseAdaptiveStepSizes()

virtual bool itk::AdaptiveStochasticVarianceReducedGradientOptimizer::GetUseAdaptiveStepSizes ( ) const

virtual

◆ ITK_DISALLOW_COPY_AND_MOVE()

itk::AdaptiveStochasticVarianceReducedGradientOptimizer::ITK_DISALLOW_COPY_AND_MOVE ( AdaptiveStochasticVarianceReducedGradientOptimizer )

◆ New()

static Pointer itk::AdaptiveStochasticVarianceReducedGradientOptimizer::New ( )

static

Method for creation through the object factory.

◆ SetSigmoidMax()

virtual void itk::AdaptiveStochasticVarianceReducedGradientOptimizer::SetSigmoidMax ( double _arg )

virtual

Set/Get the maximum of the sigmoid. Should be >0. Default: 1.0

◆ SetSigmoidMin()

virtual void itk::AdaptiveStochasticVarianceReducedGradientOptimizer::SetSigmoidMin ( double _arg )

virtual

Set/Get the maximum of the sigmoid. Should be <0. Default: -0.8

◆ SetSigmoidScale()

virtual void itk::AdaptiveStochasticVarianceReducedGradientOptimizer::SetSigmoidScale ( double _arg )

virtual

Set/Get the scaling of the sigmoid width. Large values cause a more wide sigmoid. Default: 1e-8. Should be >0.

◆ SetUseAdaptiveStepSizes()

virtual void itk::AdaptiveStochasticVarianceReducedGradientOptimizer::SetUseAdaptiveStepSizes ( bool _arg )

virtual

Set/Get whether the adaptive step size mechanism is desired. Default: true

◆ UpdateCurrentTime()

void itk::AdaptiveStochasticVarianceReducedGradientOptimizer::UpdateCurrentTime ( )

overrideprotectedvirtual

Function to update the current time If UseAdaptiveStepSizes is false this function just increments the CurrentTime by $E_0 = (sigmoid_{max} + sigmoid_{min})/2$ . Else, the CurrentTime is updated according to:
time = max[ 0, time + sigmoid( -gradient*previousgradient) ]
In that case, also the m_PreviousGradient is updated.

Reimplemented from itk::StandardStochasticVarianceReducedGradientOptimizer.

Field Documentation

◆ m_PreviousGradient

DerivativeType itk::AdaptiveStochasticVarianceReducedGradientOptimizer::m_PreviousGradient {}

protected

The PreviousGradient, necessary for the CruzAcceleration

Definition at line 132 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ m_SigmoidMax

double itk::AdaptiveStochasticVarianceReducedGradientOptimizer::m_SigmoidMax { 1.0 }

private

Definition at line 137 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ m_SigmoidMin

double itk::AdaptiveStochasticVarianceReducedGradientOptimizer::m_SigmoidMin { -0.8 }

private

Definition at line 138 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ m_SigmoidScale

double itk::AdaptiveStochasticVarianceReducedGradientOptimizer::m_SigmoidScale { 1e-8 }

private

Definition at line 139 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ m_UseAdaptiveStepSizes

bool itk::AdaptiveStochasticVarianceReducedGradientOptimizer::m_UseAdaptiveStepSizes { true }

private

Settings

Definition at line 136 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

Detailed Description

Public Types

Public Member Functions

Static Public Member Functions

Protected Member Functions

Protected Attributes

Private Attributes

Additional Inherited Members

Member Typedef Documentation

◆ ConstPointer

◆ Pointer

◆ Self

◆ Superclass

Constructor & Destructor Documentation

◆ AdaptiveStochasticVarianceReducedGradientOptimizer()

◆ ~AdaptiveStochasticVarianceReducedGradientOptimizer()

Member Function Documentation

◆ GetClassName()

◆ GetSigmoidMax()

◆ GetSigmoidMin()

◆ GetSigmoidScale()

◆ GetUseAdaptiveStepSizes()

◆ ITK_DISALLOW_COPY_AND_MOVE()

◆ New()

◆ SetSigmoidMax()

◆ SetSigmoidMin()

◆ SetSigmoidScale()

◆ SetUseAdaptiveStepSizes()

◆ UpdateCurrentTime()

Field Documentation

◆ m_PreviousGradient

◆ m_SigmoidMax

◆ m_SigmoidMin

◆ m_SigmoidScale

◆ m_UseAdaptiveStepSizes