High impact physical activity (PA) is thought to benefit bone. We examined associations of lifetime walking and weight bearing exercise with accelerometer-measured high impact and overall PA in later life. Data were from 848 participants (66.2% female, mean age = 72.4 years) from the Cohort for Skeletal Health in Bristol and Avon, Hertfordshire Cohort Study and MRC National Survey of Health and Development. Acceleration peaks from seven-day hip-worn accelerometer recordings were used to derive counts of high impact and overall PA. Walking and weight bearing exercise up to age 18, between 18–29, 30–49 and since age 50 were recalled using questionnaires. Responses in each age category were dichotomised and cumulative scores derived. Linear regression was used for analysis. Greater lifetime walking was related to higher overall, but not high impact PA, whereas greater lifetime weight bearing exercise was related to higher overall and high impact PA. For example, fully-adjusted differences in log-overall and log-high impact PA respectively for highest versus lowest lifetime scores were: walking [0.224 (0.087, 0.362) and 0.239 (− 0.058, 0.536)], and weight bearing exercise [0.754 (0.432, 1.076) and 0.587 (0.270, 0.904)]. For both walking and weight bearing exercise, associations were strongest in the ‘since age 50’ category. Those reporting the most walking and weight bearing exercise since age 50 had highest overall and high impact PA, e.g. fully-adjusted difference in log-high impact PA versus least walking and weight bearing exercise = 0.588 (0.226, 0.951). Promoting walking and weight bearing exercise from midlife may help increase potentially osteogenic PA levels in later life.