Actually, all things being equal and in a studio controlled environment, anyone who wants to can clearly observe the very obvious difference in image quality between various levels of lenses.
Pick two lenses to compare and the camera body of your choice... [u]Lens Comparison</u>
I have owned mid range glass that, as a walk around lens, was adequate but only until I became more familiar with photography and processing RAW images in photoshop. Eventually the resolving detail and performance just became too much of an issue for me so I upgraded to all f2.8 Ultra-Low Dispersion glass and I have never looked back or regretted ever since.
Also, and to add another point to Splitboader post, not all the good lenses are expensive or even marked "L" by Canon for that matter. I would absolutely agree that the 50mm f1.2 is a fantastic lens in so far as it's sharpness and usability. Indeed one of the sharpest zooms Canon has ever produced in the wide angle range is not an "L" lens either, and only mid range in price.
Bottom line is that the glass you use will make a noticeable difference in the final image quality, and that is observably undeniable.