The victory puts Saracens in pole position after four rounds of action, whilst the Ospreys find themselves in third with just two rounds of the pool stage left.
The defeat also ended the Ospreys' impressive home record of 17 victories in the tournament and whilst admitting that the defeat hurt, Holley was adamant his team will bounce back.
"We can't give up," he said.
"We're very disappointed because we've given up a six-year home record and that hurts.
"We have to do what we can in the last two games and then hope Biarritz and Treviso (Saracens' next opponents) can still have a big say in the pool.
"But we won't give up without a fight.
"We dominated the game, territory and possession but errors at crucial times cost us.
"Saracens also had an outstanding defence and we found that difficult to breakdown.
"Perhaps we were guilty of playing too much rugby. We just needed to tidy up our accuracy."