I want to create a very simple neural network, but I’ve never created a neural network before. I have the conceptual design down, but I don’t know how to implement. Hoping someone here can point me in the right direction.
Allow me to explain it:
- start with a fully connected RNN.
- arrange the neurons in layers (preferably in an autoencoder pattern).
- remove all connections to other cells in the same layer, including the self-connection.
- create a time delay for the connections based on how many layers away the target is.
- when a layer is activated force a constant, small percentage of the cells to be on. In other words, the activation function must take the entire layer into account so that the top ~5% of activated cells really become active.
- lastly, when adjusting the weights (applying error) I think the further away a cell is predicting the more gradual the learning rate should be, I’m really not sure how to implement this part, but I think a cell from layer 0 should have more confidence about its predictions on layer 1, in the very next timestep than it does for predictions made on the last layer, 4 timesteps in the future. So for that further-away layer its weights should be smaller so it has less of an effect than nearby layers, and it should readjust those weights more slowly. I’m not sure how to balance that with the rest of the group - maybe you have some ideas?
Anyway, thanks for any help or direction you can offer me!
PS. If you’re curious as to the motivation behind this design, I’m going to attempt to use it in a sensorimotor context. It’s organized into layers so there’s a 2-way directionality to the information flow in it. One side, perhaps the left side, can be thought of as sensory input data, the other side can be thought of as motor output data. So it’s really as if there are two feed-forward networks readjusting the weights for each other: the sensory network and the motor network. The sensory network is saying, “given that I get this input, what do I predict my motor behaviors will ultimately be?” and the motor network is saying, “given that this is the motor activation, what do I expect my sensory input to be as a result.” Of course, this is the absolute simplest implementation of the idea, so even if it works it won’t be very powerful, but its a start. Thanks for your help!