At a baby and you course I went to, the lady there (from SPARC I think) said that the reason they recommend children don't watch TV is cos their eyes get used to focusing in one place, not going back and forth - like they do when you read. So they reckon that if kids/babies watch too much TV then when they get to school and learn to read, it's difficult for them cos their eyes are 'unfit' and aren't used to tracking across the page.
But then she also said it's unrealistic that they don't watch any TV at all, but you should try limit it. And do heaps of tracking exercises to help their eyes (watching bubbles, toys etc).
Personally I don't think there's anything wrong with it in moderation