when you flee the floor to heal. when you return, the monsters are still there, but you don't get credit.
I'd like to add that even escaping the zoo, closing the door, running, whatever and leaving it for a while can also break the zoo counter. Not even having to leave the floor your on.
Ohh you don't even have to do ANYTHING for a zoo to break, once you get down to where teleporting mobs are prevalent zoos become a Russian Roulette of "Will I get all of them this time or will I be left with 1 monster left in the zoo for the entire level?"
Really? I've got that a few times, I just put it down to some mob teleporting somewhere where I couldn't reach them.
