Water displays a vast array of unique properties, known as water's anomalies, whose origin remains subject to hot debate. Our aim in this article is to provide a unified microscopic physical picture of water's anomalies in terms of locally favored structures, encompassing both thermodynamic and dynamic anomalies, which are often attributed to different origins. We first identify locally favored structures via a microscopic structural descriptor that measures local translational order and provide direct evidence that they have a hierarchical impact on the anomalies. At each state point, the strength of thermodynamic anomalies is directly proportional to the amount of locally favored structures, while the dynamic properties of each molecule depend on the local structure surrounding both itself and its nearest neighbors. To incorporate this, we develop a novel hierarchical two-state model. We show by extensive simulations of two popular water models that both thermodynamic and kinetic anomalies can be almost perfectly explained by the temperature and pressure dependence of these local and non-local versions of the same structural descriptor, respectively. Moreover, our scenario makes three unique predictions in supercooled water, setting it apart from other scenarios: (1) Presence of an "Arrhenius-to-Arrhenius" crossover upon cooling, as the origin of the apparent "fragile-to-strong" transition; (2) maximum of dynamic heterogeneity around 20 K below the Widom line and far above the glass transition; (3) violation of the Stokes-Einstein-Debye relation at ∼2Tg, rather than 1.2Tg typical of normal glass-formers. These predictions are verified by recent measurement of water's diffusion at very low temperatures (point 1) and discoveries from our extensive simulations (points 2-3). We suggest that the same scenario may generally apply to water-like anomalies in liquids tending to form locally favored structures, including not only other important tetrahedral liquids such as silicon, germanium, and silica, but also metallic and chalcogenide liquids.