I have one thing to say, one of Silver's excuses could easily be it is uncomfortable. Puffy sleeves with collar like ends feel really tight and can often inhibit one's ability to reach up high, like when Silver was lifting his hands up high when he was using his psychokinesis to put dishes away. So this dress may have a way of restricting his abilities.... Another theory is he can just say he is uncomfortable to Amy and say he really does appreciate her clothing, and respects her even more for wearing such uncomfortable things for appeal...but that could end up with him getting slapped for looking. But of course we would prefer he keeps wearing the outfit, so just ignore those theories.
I have one thing to say, one of Silver's excuses could easily be it is uncomfortable. Puffy sleeves