But Northampton did at least take a losing bonus point back to the East Midlands and it could so easily have been even more had they turned pressure into points and made the most of a dominant setpiece.
The result came on the back of a 19-19 draw at bitter rivals Leicester Tigers the previous week and Hartley has no qualms with the way the Saints are playing ahead of next Sunday's crucial clash with the Ospreys at Franklin's Gardens.
"We've had a couple of difficult weeks with a couple of away games against quality opposition and we feel like we could have taken four points away from each of those games," said Hartley.
"We played away against the English champs last week and should have won that game. We came close in France.
"We'll take that losing bonus point: in the Heineken Cup you've got to take points from every game, whether that's a win or a loss.
"It's about collecting points and winning your home games. We'll have to play well at home next week and take the victory.
"We've been on the road quite a bit recently and we're looking forward to being back on our home ground, stepping it up a notch against another quality side. But we are playing some good rugby at the moment and we're confident as a group."
Hartley was impressed with what he saw from Castres as well, with Kockott and co suggesting they have enough quality in their side to shock triple Heineken Cup kings Leinster in Dublin next weekend.
"Castres really are a team out there. We saw glimpses of what they can do, in both aspects of the game," added Hartley.
"We saw some offloading, free flowing rugby and some French flair. We also saw the hardness of the pack holding us out at the end and the physicality. They're a good side.
"But saying that, we were still hopeful coming there, and if you take away that runaway try at the beginning, I think we were well in that game."