Artificial Intelligence
This paper presents the Slingshot Effect, a phenomenon in neural network optimization occurring in late training stages. It involves cyclic phase transitions between stable and unstable training regimes, demonstrated by cyclic behavior of the last layer’s weight norm. The effect can be replicated in various settings, but its implications remain unexplored.