Objective: Cognitive impairment and depression significantly reduce quality of life in the aging population. This study aimed to investigate the prevalence of depressive symptoms and cognitive impairment and explore its relationship in the elderly. Methods: A total of 1645 elderly people in nursing homes and 4703 elderly people in the community were enrolled in the survey. The Patient Health Questionnaire-9 and Ascertain Dementia-8 were employed to evaluate depressive symptoms and cognitive impairment. Results: The overall prevalence of cognitive impairment was 12.5% in the community-dwelling group and 52.2% in the nursing home group. The prevalence of cognitive impairment in nursing homes was significantly higher than that in community-dwelling groups for the same age group (p < 0.001). The overall prevalence of depressive symptoms was 3.9% in the community-dwelling group and 2.0% in the nursing home group. The prevalence of depressive symptoms increased with age in the community-dwelling group (p < 0.001). The binary logistic regression results showed that the type of care mode affected the prevalence of cognitive impairment, and the elderly in nursing homes had a high risk of cognitive impairment (odds ratio [OR] = 3.528, 95% confidence interval [CI]: 2.209–5.635, p < 0.001); depressive symptoms had a significant positive correlation with the odds of cognitive impairment (OR = 1.854, 95% CI: 1.052– 3.266, p < 0.05); and the cognitive impairment rate increased with age (OR = 1.412, 95% CI: 1.044–1.910, p < 0.05). Conclusions: There was an increased prevalence in cognitive impairment as well as depressive symptoms in the aging population in Zhongshan city. Population-based mental health strategies need to be urgently implemented for the aging.