Very good question! I would recommend understanding the evaluation metric in any competition as early as possible. Why? Because, if you are competing in any competition, ranks usually go out for a few points. In these scenarios, you can improve your score by understanding how the metric works.
For example, if a competition uses RMSE as a parameter to judge the competition, you can be very significantly affected by the outliers. So, you should be extra careful to make sure you are not making gross assumptions about outliers. Similarly, there are a few metrics, which get affected by balancing your data, but others which remain unaffected. This would directly impact the way you build your models.
As per my experience, you get get into top 30% of a competition by applying generic techniques and not having a deep understanding of evaluation metric. In order to move above that, you would need to understand how it works.
@srk and @tavish_srivastava an add more perspective on this
Hope this helps!