Report from the sub-committee on “mining” NSSE data (S2009)
The sub-committee consists of Arne Kildegaard (convener), Julia Dabbs, Matthew Privratski and Michael O’Reilly.
The sub-committee has met three times. Formal minutes were not kept. The charge to the committee was to investigate the possibility of gaining useful information by detailed analysis of NSSE data over the last years. One initial target was to seek aspects related to academic challenge.
The committee obtained the services of Tom Vail, a nontraditional student who held a MSAF grant under the Scholastic Committee and was released for this work with the ASLC. Tom has expertise in multivariate analysis, the required background for this analysis. He was mentored in the work by Engin Sungur.
Throughout the process, the committee members realized that such an analysis was complicated and to find useful information and interpretation required expert guidance.
The first meeting was a general discussion involving Tom aimed at determining what might be possible. Tom set the first objective as doing a principal component analysis of the whole data (2002 – 2008) and then looking at what aspects related to academic challenge and separately to student study hours.
At the second meeting Tom related the tentative progress made. The component analysis was not valuable because of the large number of variables (questions asked and replies received). He planned to restrict the responses to a subset of the survey questions that represented the overall set. He presented an analysis of aspects related to academic challenge in tree that separated students that rated the challenge as high from low. The first significant factor was each group gave a similar rating for the amount of synthesis they were asked to do. When these subcategories were further subdivided, the next most important separating factor was the amount of study hours.
For the third meeting Tom presented an analysis of clustering of selected representative responses. For example, the responses to the analyzing question and the synthesizing question were very close to each other in each of 2004, 2006, 2008. He also presented more decision trees analyzing related to three other aspects.
Having seen samples of what might be obtained, the committee began some general discussion as to what might be feasible with such analysis and what specific questions would prove most useful. The usefulness (and cost) of the NSSE survey was questioned. It did give comparisons with other institution and gave a time trend within the institution. It would seem to be more useful if administered every three years. For it to be useful as a developmental tool, we would have to find ways of identifying and extracting information relevant to our program. That is, we must learn to ask the statistical program the right questions.
The problem is difficult and the committee has not yet come to a conclusion on this issue but is interested in continuing the process on a trial basis.
(Report composed by M. O’Reilly on behalf of A. Kildegaard).