Capped 29 times by Scotland he scored 13 tries in 27 European appearances for Glasgow Warriors, Sale Sharks and Toulon after making his Heineken Cup debut for the Warriors against Bath in 2005.
"I'm sad to be retiring from professional rugby but I've been struggling with injury for some time now and it's clear I won't be able to get back to the level I was playing at before," said the 30-year-old.
"Despite a tough time personally it's been a privilege to see the boys do so well and make such good progress this season.
"It's fantastic to have been at the club during this exciting time and I wish them all the very best for the remaining games this season."
Glasgow head coach Gregor Townsend added: "Rory has had a frustrating time with injury and everyone at the club feels for him not being able to get back to fitness.
"We were all hoping he'd make a full recovery and show what he could do in a Warriors jersey. I'd like to thank him for his contribution to the club over his two spells here and he goes with our very best wishes for the future."