New theoretical analysis frames deep neural networks as geometric systems, revealing why ReLU's preservation of spatial distance information gives it a decisive edge over sigmoid for deep inference. The geometric perspective offers a principled framework for understanding activation function choices and their real-world cost implications.
