I have been working on a neural network regression problem. The problem is in time series, so the training data precede the test data in time. In some cases, over the range of the traiing data, the model generates a very good representation of the time series, except it is biased by a nearly constant value. I thought this might be insufficient training time, but I tested that and get nearly identical results even with much longer training. Any thoughts on what causes this and how to correct it?
This may be due to a number of factors. If the training time is not the issue, you can try different architectures. It may be due to the shallowness of the model. You could also try scaling your dataset or try different optimizers.
I know this old but thought I would close it out as I found the solution was not converging properly; I think the main issue was something like vanishing gradients. I used a different optimization method and obtained acceptable results.