Depends on what units are attacking I think. If it's a BlaM, then no use microing the tower that's already hitting the one unit attacking. Though if it's a full on army, that's another story. Most times it's an issue of get your ass to your main to save it, who cares what the Frost tower is hitting.
Personally I always thought frost tower was used just for early game, IMO anyway.