How to improve use of data augmentation



Hi I am training a model to detect arrowheads from drone aerial view as shown in the image below. I had a very small dataset of just 50 images. So I performed data augmentation by putting arrow at different positions in random backgrounds.

After training a convnet classifier to detect presence of arrow , I got 0.98 of validation f score and 0.96 of train f score. But I am getting many false positives in test set.

I noticed in few predictions of test set that anything white or bright is getting high confidence. As shown in below images. 4

As data is fully augmented , I increased percentage of negative samples in training set, but that didn’t help. I have used batch normalisation , dropout and model works well enough on train set so I am not able to understand the problem. Any tips are appreciated.

First image is sample image. Second is Wrong prediction with confidence 0.98. Third is Wrong prediction with confidence 0.88