Después de de esta prolongada recopilación de datos pudimos resolver esta cuestión que presentan muchos los lectores. Te dejamos la solución y esperamos que te resulte de mucha ayuda.
Solución:
En primer lugar, como se ve en la documentación numpy.random.randn genera muestras a partir de la distribución normal, mientras que numpy.random.rand de una distribución uniforme (en el rango [0,1)).
Second, why uniform distribution didn’t work? The main reason in this is activation function, especially in your case where you use sigmoid function. The plot of the sigmoid looks like following:
So you can see that if your input is away from 0, the slope of the function decreases quite fast and as a result you get a tiny gradient and tiny weight update. And if you have many layers – those gradients get multiplied many times in the back pass, so even “proper” gradients after multiplications become small and stop making any influence. So if you have a lot of weights which bring your input to those regions you network is hardly trainable. That’s why it is a usual practice to initialize network variables around zero value. This is done to ensure that you get reasonable gradients (close to 1) to train your net.
However, uniform distribution is not something completely undesirable, you just need to make the range smaller and closer to zero. As one of good practices is using Xavier initialization. In this approach you can initialize your weights with:
-
Normal distribution. Where mean is 0 and
var = sqrt(2. / (in + out)), where in – is the number of inputs to the neurons and out – number of outputs. -
Uniform distribution in range
[-sqrt(6. / (in + out)), +sqrt(6. / (in + out))]
np.random.randes para distribución uniforme (en el intervalo semiabierto[0.0, 1.0))np.random.randnes para la distribución normal estándar (también conocida como gaussiana) (media 0 y varianza 1)
Puede explorar visualmente las diferencias entre estos dos muy fácilmente:
import numpy as np
import matplotlib.pyplot as plt
sample_size = 100000
uniform = np.random.rand(sample_size)
normal = np.random.randn(sample_size)
pdf, bins, patches = plt.hist(uniform, bins=20, range=(0, 1), density=True)
plt.title('rand: uniform')
plt.show()
pdf, bins, patches = plt.hist(normal, bins=20, range=(-4, 4), density=True)
plt.title('randn: normal')
plt.show()
que producen:
y
Valoraciones y comentarios
Tienes la opción de añadir valor a nuestra información añadiendo tu veteranía en las aclaraciones.