It's not very far from Burlington to Montreal. We are a bit further from YUL (granted, no international border in the way) and don't think twice about flying from Montreal if the price and/or schedule works out better than our local airport in Ottawa.
Customs into Canada is unlikely to be much of an issue, assuming your paperwork is in order and you don't have some prior history of trouble at international borders! Customs at the airport will add a little time, anywhere from 5 minutes to maybe an hour at very busy times, but I'd usually assume somewhere around 15-30 minutes. Probably won't be unusually busy in February. Local "spring break" is in March, including most of the universities.
I am not a huge fan of Rouge; it's Air Canada's "vacation" brand and IMO the service and spacing is a little less than ideal, but it's not really worse than most mainstream airlines these days, and certainly not worse than most discount airlines. I might not choose them first if I had another option at the same price, but if the savings was significant, I would do it. I don't think they're any less reliable than any other airline.
One thing you should check, if you haven't already, is the cost to park in Montreal. Parking at Canadian airports tends to be notably more expensive than US airports, in my experience. But it sounds like you've already factored that in with your hotel.
I looked into a similar option once and was informed I could not drive to Montreal and then fly an international flight back into the US. I wasn't allowed to book it.
That's interesting. And a little bizarre. I'm not doubting your experience, but I don't know who would enforce that, or how. You tell the Canadian customs officer that you're visiting Montreal overnight and returning to the US the next day (completely true). They won't ask if you're flying back, and I don't know why they would care anyway. Likewise, as an American returning to the US, I can't see why a US pre-clearance officer at the airport would give you any trouble. As for actually booking the flight - how would the airline even know where you were coming from? Non-residents book flights across international borders all the time. I've done it myself. Never heard of it being a problem.