The cognitive demands of summary writing are dependent upon the type of summary to be produced. This paper reports part of a larger study in which 157 Chinese undergraduates were asked to write summaries of extended English texts in both English and Chinese. It examines the differential effects of the use of the two languages on summarization as a measure of reading comprehension, drawing upon data from students' actual test performances as well as their perceptions of such differential effects, as evidenced in the post-summarization questionnaire and interviews. It was found that the use of the different languages had significant effects on both summarization processes and products. Students wrote significantly longer Chinese summaries but these were rated consistently of poorer quality than English summaries. However, Chinese summarization was found to be a better measure of students' English reading abilities. The implications of these findings are discussed with specific reference to summarization task design, in particular, how the use of the first and target languages may affect test takers' performance as well as rater behaviours and the potential of using first language in large-scale international language testing programmes incorporating integrated reading/writing tasks as a measure of reading comprehension.