One of the reasons for this is the need to keep people interested, and to stop them zapping onto another channel if they are not stimulated in the next 15 seconds. So, imagine a world without the remote control - it's an interesting way to think of the effects of technology on our lives.
The inventor of the remote control just passed away, and he could reasonably claim to have been a major contributor to the current short attention span-friendly TV programming, with his invention of, as his patent application put it "a system to regulate the receiver operation without requiring the observer to leave the normal viewing position" (Rosen)
On the other hand, and in a good example of the need to avoid plain technodeterminism (i.e. the argument that technologies cause social change), it is instructive that one of the selling points of the ‘Flash-Matic' was its ability to "tune out annoying commercials" (by reducing the volume, Rosen), and the ability to avoid advertisements is still a motivating factor for most TV viewers.
Toddlers want control, they need it. Which is why they keep saying ‘No'. It's an important part of how they develop a sense of their own individuality in the world. So it doesn't surprise me that they (apparently) pick up the use of a remote very quickly, and one research reported that
One subject, three-year-old Jimmy, was incapable of articulate conversation and could neither recognize numbers nor tell time, but he "had mastered the basics of RCD use." He "primarily used the RCD to change channels on the TV in order to watch his favorite programs," and when told the time, clever Jimmy "knows if his program should be airing." (Rosen)
Frankly, I can't help thinking that if poor Jimmy hadn't been given free reign of the remote he might have learnt more words, numbers, and the time. Let's hope that he eventually managed to learn to read.