Banff will give you the street and shops etc to stroll around. The interior resorts won’t. They are purpose built resorts, not towns or really even villages. Big White has a lot of activities but not much of a village feel IMO. Silver Star is cute but the ‘village’ is really small. Sun Peaks (haven’t been there yet) is supposed to have more shops and restaurants than the other two. Panorama is lovely but also has a very limited on mountain village (but a great outdoor swimming pool). It does have better scenery than the other three (or at least SS and BW) too. But if you want to fly in and out of Calgary, Panorama would be a good option for a week or so.
The strength of these 4 resorts is that they make a ski holiday with kids very easy. That’s their calling card. But no resort is all things to all people.
And none of them are similar to Whistler, which is big and buzzing and busy. Mid January mid week though, Whistler is certainly manageable. It’s orders of magnitude bigger in all respects than the others.
With any resort, you need to have your eyes open to its strengths and weaknesses.
I think you mentioned Breckenridge earlier. Breck is one of the classic American ski towns. Very attractive town with ski lifts right into town. You also get this at Aspen, Telluride, Park City. These American ski towns offer a different experience again from the purpose built Canadian resorts. But you can certainly expect more people at the big American resorts, particularly those on the Epic Pass (Telluride not so much). My own preference is the town/resort combo. That said, I may very well be giving Sun Peaks a try in January.
As far as spending time in Vancouver or Calgary, Vancouver is much more appealing for a couple of days than Calgary.