Hi Jared,
I've panned in quite a few spots along the Fraser and in the Southern Chilcotin west of Lillooet. The best places accessible by road in SW BC are probably the Cayoosh Creek and the Bralorne/Gold Bridge areas. Tons of placer gold was mined from the lower Cayoosh creek area durring the 1880s-90s. The area can be reached via the Duffy lake road a short drive south of Lillooet. The southern Chilcotin area is by far my favorite spot for rock hounding, fossil hunting and gold panning. The best spot is probably Cadwallader creek (near Bralorne) downstream from the old Pioneer mine, once the biggest producing gold mine in western Canada. Much of the area has been worked over but the historic rains and floods we had last fall and the washouts they caused probably brought new gold down into places once panned out. The upper Anderson river area has had five small producing gold operations throughout history, the Kawkawa mine near the Coquihalla Hwy being one of them. You can get to the Anderson river along a logging road just north of the hwy1 bridge over the Fraser R. north of Yale.
If you're interested in prospecting I would recommend becoming familiar with the BC gov't Minfile system (found on the link in one of my previous posts). Minfile is a database of all publically known mines, mineral deposits and mineral/metal showings, mined or unmined in all of BC.
I used to be a mine geologist (lived Logan Lake for a while) and most of my prospecting was for industrial metals like copper, zinc, lead and molybdenum so I'm not that good a source of info if you're looking for semi precious gems & crystals etc in southern BC. I did a fair amount of panning & gold prospecting in northern BC but most of those areas can only be reached by air. I could recommend a book or two for SW BC though. I'll have to dig them up so I can give you the correct titles later.
Andrew