Lack of convergence based on initial weights

Sometimes, the neural network does not converge to the correct results. This is caused by the randomly generated initial weights.
There were some improvements after changing min/ max of initial values to -1/ +1 instead of -10/ +10 but the problem is still present.
