Background: The increasing incidence of cognitive impairment and dementia in an aging population poses a significant burden on healthcare. Consequently, identifying modifiable physiological factors which may influence the onset of cognitive decline are becoming increasingly important. Previous studies have suggested an association between levels of insulin-like growth factors and cognitive function. Objective: To investigate whether low IGF-I, IGF-II, and IGF molar ratio is associated with greater cognitive decline and increased risk of dementia. Methods: We examined prospective associations between IGF-I, IGF-II, and IGFBP-3 and cognitive function in the Caerphilly Prospective Study (CaPS) (n = 746 men) from samples obtained around 1986, with assessment in around 2003 for clinical diagnosis of cognitive impairment but no dementia (CIND) or dementia, as well as with CAMCOG scores at three phases. Results: IGF-II was associated with a reduced odds ratio for CIND (0.76, 95% CI 0.60, 0.96) which hardly altered after further adjustment for confounders. A one standard deviation increase in IGFBP-3 among participants without dementia or CIND was associated with greater decline in cognition (p = 0.002) equivalent to 2.4 years difference in age. All the associations between IGF-I and our outcomes were consistent with chance. Conclusion: In this study of men, we found that both IGF-II and IGFBP-3 are associated with both normal age-related cognitive decline and clinical pathology associated with CIND, but we failed to replicate previous associations with IGF-I. Assuming these findings are replicated, they may provide new insights into potential biological mechanisms that underlie age-related cognitive changes and development of dementia.