It is true, if you go during off-peak periods, the parks tend to close earlier, rides (and monorail) cut back for refurbs, parades and fireworks happen less frequently, and there is less to do overall. Resort restaurants might actually be busier in the evenings if MK closes early, I would think.
On the other hand, you can go on rides over and over.
I enjoy watching and dealing with the crowds during Presidents week the last 15 years, and we even went Christmas Week one year. But if I were to go without kids, after Labor Day would be best for the weather (opps, forgot about the hurricaines) and right after Thanksgiving for the Christmas decor.