I wouldn't recommend the distant campgrounds. They aren't campgrounds in any reasonable sense of the word. They haven't been maintained in a decade or two and are a mess. I went in from the east end to Lm22
in 2010, and we barely found one vaguely flattish spot to call a "tent pad", and an outhouse long since reclaimed by the spiders. In June, the ticks will have calmed down a little (it was just *swarming* in April), but the "campgrounds" will still be pretty awful.
There is also a wolf pack that lives around the east end, two of which approached us at twilight, which was not enjoyable.
Bears are an issue, but it's more that they like it back in the Ghost and tend to come through Aylmer as their access to the Bow Valley. But mid-June, I wouldn't rule out the area for that. I'd just be on alert. You say "we", so it sounds like you're not alone, which is good.
Personally, I would suggest that you carry your packs out to the junction, hike up Aylmer Pass (with your backpacking packs on) if the snow conditions seem appropriate, or Aylmer Lookout if the snow is lingering, and then go back down, continue east, and camp at Lm9, or even better, Lm11. It's a little further out of the main area, avoiding the issues at Lm8 and Lm20-22, but is still a proper campground,
And doing that you'll get a solid day with elevation in your packs, even if it's via a scenic detour.