We initiate a systematic study of key-avoidance on alternating sign matrices (ASMs) defined via pattern-avoidance on an associated permutation called the \emph{key} of an ASM. We enumerate alternating sign matrices whose key avoids a given set of permutation patterns in several instances. We show that ASMs whose key avoids $231$ are permutations, thus any known enumeration for a set of permutation patterns including $231$ extends to ASMs. We furthermore enumerate by the Catalan numbers ASMs whose key avoids both $312$ and $321$. We also show ASMs whose key avoids $312$ are in bijection with the gapless monotone triangles of [Ayyer, Cori, Gouyou-Beauchamps 2011]. Thus key-avoidance generalizes the notion of $312$-avoidance studied there. Finally, we enumerate ASMs with a given key avoiding $312$ and $321$ using a connection to Schubert polynomials, thereby deriving an interesting Catalan identity.