It is generally accepted that the mental representation of numerical magnitude consists of a spatial “mental number line” (MNL) with smaller quantities on the left and larger quantities on the right. However, the amount of dissociations between tasks that were believed to tap onto this representational medium is accumulating, questioning the universality of this model. The aim of the present study was to unravel the functional relationship between the different tasks and effects that are typically used as evidence for the MNL. For this purpose, a group of right brain damaged patients (with and without neglect) and healthy controls were subjected to physical line bisection, number interval bisection, parity judgment, and magnitude comparison. Using principal component analysis, different orthogonal components were extracted. We discuss how this component structure captures the dissociations reported in the literature and how it can be considered as a first step toward a new unitary framework for understanding the relation between numbers and space.