from my experence, monsters can see invisible much like the player could see inivible monsters IE right next to you.
Did you die on the very next turn? This is happening to invisibility potions, too. Monsters have aggro on you the turn you active an 'invisibility' effect and lose it the next turn.
Yup. Gotta have at least 2 squares of space for invisibility to be useful, otherwise they just smack you.
There seems to be a bug though or maybe some of them DO see invisible heroes: I kept being the target of long range attacks from Wights when invisible. And I was a good 5 or 6 square away.
It seems ALL critters with a ranged attack can ignore invisibility at will - yes, this DOES mean skipping Zoos unless you have spammable AoE (AKA mage)might be a good idea, sine you'll have 15-20 ranged critters hiding in the back who WILL constant-blast you as soon as you come close enough to trigger them.
It's gotten bad enough that I've been saving bolts of squid for AFTER the zoo's been cleared out, as opposed to before.