In the tai chi classics, it says something like, "The eyes and the hands must follow each diverse." However, this does not suggest that your eyes must exactly follow the movement of the hands.
Your intention leads the movements of their time.