My Explorations Into Deep Learning: A random walk into ideas for Machine Learning optimization

Just a summary of terms used in different context. Details are not covered.

Step sizing, modelling, approximation, statistical randomized vibration/motion, stability vs convergence rate, correlation/anti-correlation, peer pressures/influences/interaction.

minibatching, Boltzmann distribution, sampling based optimization, entropy optimization, Monte Carlo approximation,

Features identification/approximation, attention mechanism,

How does attention mechanism/strategy help to find the optimal path towards solution?

How can "divide and conquer", "tasklist breakdown" heuristics helped in optimization?

Finding the optimal solution is proportional to the number of combinations that have been attempted or generated, is there any correlation?

How to generate changes incrementally, so that each of these generated candidates can be used for optimization criterion evaluation?

How does applying the optimization heuristics early vs applying it late help towards optimal solutions?

Using statistical + variance minization for optimization:

Bayesian approximation + statistics

Reversibility of decomposition, dimensionality transformation: orthogonality and inverse operation, recovering of subfeatures, features of features, higher abstraction features of data etc.

Weighted sum and activation function. Reversibility of operations.

Reinforced learning: incremental knowledge to help veering towards the optimal solutions.

Greedy layer wise optimization.

Adaptive strategies. Momentum or energy-based optimization.

Crowd/popularity-based optimization.

Heuristics search by optimized strategies for optimization heuristics.