The English club went into the match at Parc Municipal des Sports with a series-winning 33-point advantage from last week's first leg, and seemed to be just going through the motions until a burst of three Brive tries either side of the break woke them up.
An 18-point haul from Brive scrum-half Valentin Courrent rattled Quins, but it wasn't enough in the final reckoning.
A blindside side move from a lineout set up Harlequins' first try to Samoan centre George Harder, and the trusty boot of former Ireland international Paul Burke kept the scoreboard ticking over with three penalty goals.
Brive hit back with two late tries to take a 22-14 lead at halftime and give the home fans a glimmer of hope of an unlikely 34-point victory to steal the two-leg series.
Another try to speedy Brive winger Yves Donguy immediately after the resumption forced a rethink in the Quins' game plan and two more Burke penalty goals managed to settle the nerves and end Brive's charge.
Tempers frayed in the closing minutes and in the 77th-minute Brive's Australian lock John Welborn saw a red card for over-vigorous rucking.
In the other match of the day, Saracens bowed out the of the competition despite beating Montferrand 18-14 at Vicargae road.
The English side were trying to haul back the 16 point deficit form the first leg, but were unable to do so.
Nicky little kicked all the points for his side with six penalty goals.