Im an AP holder and we go 2-3 days a week. Sunday is the best day for Potter. Smaller crowds Forbidden Journey is usually around 45-60 min wait, Dragon Challege is around 15 mins, Hippogriff around 20 mins. #0 min lines to get into Honeydukes.
The only thing that it doesnt matter what day you go you will have a horribly long wait is the Wand Experience at Ollivanders. Its never under a 3 hour wait. No matter the day or the time of day.
Also, If your going to do both parks or even just IOA, I would suggest skipping Potter first thing in the morning and doing everything else and saving Potter for last. In the morning people rush to Potter Island and you have these horrible lines to just get into the island its self, THEN have to wait in those lines.
But if you wait until around 6ish to do Potter the crowd usually has thinned out ALOT and oyu dont have to wait in line to get onto the island. And not to mention being around dusk and into the night time it gives the area a more realistic" feel.