The following experiment highlights the fundamental concept of model uncertainty in neural networks, where different parameter initializations can lead to equally valid solutions in the weight space, all capable of correctly classifying unseen data.
Understanding this variability is crucial for developing robust AI systems, as it reminds us that there is rarely a single ‘correct’ neural network for a given problem, but rather a family of solutions with similar performance characteristics.
To illustrate this, we use the Iris dataset, which contains 150 data points representing three different flower species. Each sample includes four numerical features: sepal length, sepal width, petal length, and petal width, as shown in the picture.
The three Iris species. Measurements are shown only for Iris Versicolor.
We exclude one sample from the dataset, which is:
\[
\mathbf{x} = [6.4, 3.2, 4.5, 1.5]^T.
\]
This sample is belong to versicolor flowers, i.e.,
\[
\text{The original sample class} = \text{2 (versicolor)}.
\]
Thus, we have 149 samples in total. If we split the data into 80% training and 20% testing (inference), we get approximately (120, 30).
We consider a one-layer neural network architecture, as shown below. This network has four inputs and three outputs. Therefore, we train a single-layer network using 120 data points.
A single-layer neural network with 4 inputs and 3 outputs.
We test the network on the sample we held out earlier to demonstrate that different initializations lead to different trained networks. How do we know the trained networks are different? They each have a matrix of weights and a vector of biases, and as you can see, all of them differ across runs.
