This thesis addresses the surprising features of zero-temperature statics and dynamics of several spin glass models, including correlations between soft spins that arise spontaneously during avalanches, and the discovery of localized states that involve the presence of two-level systems. It also presents the only detailed historiographical research on the spin glass theory. Despite the extreme simplicity of their definition, spin glasses display a wide variety of non-trivial behaviors that are not yet fully understood. In this thesis the author sheds light on some of these, focusing on both the search for phase transitions under perturbations of Hamiltonians and the zero-temperature properties and responses to external stimuli.