Why Initialize a Neural Network with Random Weights?

Last Updated on March 26, 2020

The weights of artificial neural networks must be initialized to small random numbers.

This is because this is an expectation of the stochastic optimization algorithm used to train the model, called stochastic gradient descent.

To understand this approach to problem solving, you must first understand the role of nondeterministic and randomized algorithms as well as the need for stochastic optimization algorithms to harness randomness in their search process.

In this post, you will discover the full background as to why neural network weights must be randomly initialized.

After reading this post, you will know:

• About the need for nondeterministic and randomized algorithms for challenging problems.
• The use of randomness during initialization and search in stochastic optimization algorithms.
• That stochastic gradient descent is a stochastic optimization algorithm and requires the random initialization of network weights.

Kick-start your project with my new book Deep Learning With Python, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.