Yes, the pool and hot tub are only for those staying at Fort Wilderness.
I would come over in the afternoon and perhaps walk around the loops near the Trails End/Settlement Trading Post area (100/200/400/700). It's not a hike - more like an amble. If mom gets grumpy, leave her to enjoy a Trails End rocking chair on the porch while the rest of you explore.
I would get dinner at the Trails End takeout counter (known as P&J's Southern Takeout) and get the Giddy Up and Go dinner with 8 pieces of fried chicken, choice of two sides, and cornbread for $25. If you have refillable mugs at CBR you can fill them at P&J's or buy drinks. If you ask at the register, they will give you paper plates, KFS, and there are napkins/condiments available - everything you need - to eat dinner. Picnic tables to eat at are on the side porch of Trails End or if they are full you can go to the tables down by the dock.
Then hop an internal bus and get off for the campfire singalong (ask the driver where to get off). The Food Truck at the Campfire sells basics for roasting marshmallows (sticks and said mallows) or a S'mores kit for $10. The movie schedule is posted on the Camping Board monthly. Get your picture made with Chip and/or Dale.
Watch the movie if you want. Go down to the beach and watch the MK fireworks if you want.
It's easy to kill a couple hours enjoying the Fort.
Bama Ed