TOWARDS AN ASSESSMENT AND EVALUATION
FOR MENTORING PROCESSES AND RESULTS
Diola Bagayoko, Ph.D.
Professor of Physics and Chancellor's Fellow
Director, the Timbuktu Academy
Louisiana Alliance for Minority Participation (LAMP)
Southern University and A&M College
Baton Rouge, Louisiana 70813
The assessment and evaluation activities for mentoring are as intricate as mentoring itself. While a thorough discussion of this critical topic may require tens of pages, the following outline is intended to provide a framework that captures some essential features of emerging assessment and evaluation standards for mentoring. The actual implementation of this framework will naturally vary with specific mentoring relationships or programs. The essential point, however, resides in the fact that some features are common to most mentoring scenarios. The importance of valid, reliable, comprehensive assessments and evaluations that address value-added is as great as that of mentoring itself.
This framework for the assessment and evaluation of mentoring processes and results is supported by a summative overview of mentoring [1-7] and by the National Science Education Standards (NSES). NSES provides six distinct and complementary standards which, together, constitute a dynamic synthesis of major K-12 educational reform efforts. (Except for the Content Standards, the remaining five standards are directly transferable to the college level!) The support by the NSES stems from the inclusion, in the NSES, of science education program standards and of educational system standards along with the usual content, teacher training, professional development, and assessment and evaluation standards. It turns out that for mentoring, particularly systemic mentoring, the environment where the mentors operate, are funded, tolerated, supported, or valued could be critical. In addition to NSES, this framework is in agreement with some key research findings on undergraduate retention [1-3], a major report on the impact of the college environment in fostering scholastic excellence and personal and professional growth , an establishment of the scientific basis for high expectations for all students , a recent attempt to define mentoring , and with extensive insights on mentoring as provided by the Proceedings of the 1996 Symposium for the US Presidential Awards for Excellence in Science, Mathematics, and Engineering Mentoring [PAESMEM, 7, i.e., NSF 97-124]. Page 57 of this later document leads directly to some major features of this framework.
The framework is first characterized by four (4) important criteria deemed necessary for assessment and evaluation processes and instruments as they apply to large scale or systemic mentoring activities and programs. These criteria are (a) validity, i.e., the actual measurement of what is intended to be measured, including the attainment of specified quantitative and qualitative objectives, (b) reliability, i.e., the ability of the process and instrument to provide similar outcomes when applied to similar or comparable situations, (c) comprehensiveness, i.e., for the need to capture the program in its systemic nature as opposed to selected aspects or components, and (d) intervention-outcome relationships (or value-added), with emphasis on relating--to the extent possible-- the causes to the effects or the interventions to the outcomes, taking into account some lag times and external factors. The last two criteria are generally not included in classic designs and developments of assessment and evaluation instruments and processes. For the purpose of mentoring, however, they are as pivotal as the first two. Validity and reliability are only "accurate in part" if comprehensiveness is not reached or at least approached. The entire assessment and evaluation, from a policy making stand-point, may mean little in the absence of "intervention-outcome" or value added information. The latter information is a key factor in project inception or support decision-making. The relationship between dividends and investments is indeed a profound motivation to invest.
For the above reasons, standards of assessment and evaluation for mentoring activities should address not only outcomes but also processes. According to the integrated or compound  law of human performance, a discussion of behavioral, scholastic, or research outcomes in the absence of the pertinent processes and interventions, or vice versa, has little meaning. Two (2) related kinds of processes should be distinguished: the first kind pertains to the organization, resources, personnel, operation, etc. of a mentoring relationship, project, or program and the second kind deals with the scope, duration, and quality of actual learning, mentoring, or research activities that engage each individual student or mentee. It is the second kind that has to be necessarily addressed along with mentoring outcomes. For this reason, processes of the second kind are partly results: we call them process-results! We refer to While the outcomes are generally quantitative (i.e., grade point averages, number of precollege mentees to take college preparatory curricula, ACT/SAT scores, GRE scores, number of mentees to enter and succeed in graduate school, etc.), the processes are often described in qualitative terms. As the latter greatly determines the former, a focus on students' grade point averages or standardized test scores, for example, without mention of the supportive and challenging environment in which the students are immersed (or the lack of same) could easily lead to a blaming of the students and to some other serious inferences that could be totally erroneous. When this occurs, then problems are actually exacerbated by the indirect exoneration of programs, processes, or individuals that actually failed. The above processes or outcomes, for mentoring, could be complicated by the existence of lag times between the administration of some interventions and the actual attainment of behavioral and performance outcomes by the mentees. Hence, follow-up, for the affected groups and of control groups, is often necessary.
A key feature of the framework and of the emerging standards consists of clear provisions for "peer-review." Given the possibilities of subjective views serving as substitutes to pertinent objective realities, the explicit inclusion of peer-review elements in assessment and evaluation processes and instruments is critically needed. These peer-review elements can be as varied as the mentoring activities and contexts. There are, however, some general elements of "peer-review" that can be utilized by most mentoring relationships, projects, and programs. Comparisons of the scholastic and research performance of the mentees with that of appropriate control groups provide "peer-reviewed" features of the assessment and evaluation. A second element, associated with a lag time, stems from the performance of the mentees, as compared to that of others, in contexts or situations not controlled by the assessed mentoring program. For instance, the summer internship performance of our scholars at federal, industrial, and university research sites around the country provides an external evaluation (peer-review) element. The scholastic performance of the mentees on regional or national standardized tests is a very important "peer-review" element. Of course, the refereed publications of the mentees pass peer-review in the classic sense of the expression. One of several peer-review validations utilized by the summer mentoring and academic enrichment programs of the Timbuktu Academy consists of comparisons of scores on standardized tests taken before and after the programs. Needless to add that objective value-added information is obtained in the same process.
Attention should be paid to a particular non-cognitive and non-numeric objective of mentoring. It consists of the development and enhancement of a set of behavioral traits that are essential in guiding and overseeing the execution of various learning and research tasks for competitiveness in an ever-changing environment. Terms like judgment, character, self-discipline, self-esteem (rooted in efforts and related incremental results), ethics, drive to contribute, sense of self-worth, and personal and professional growth fall under this critical category that is quite difficult to grasp thoroughly. It entails behavioral empowerment as a quintessential objective of holistic mentoring. This objective is perhaps best understood through an analogy with the development and enhancement of a "special operating system" for each mentee. Unlike in the case of current computers, this operating system not only enables the execution of other "good programs" but also assists in its own upgrading! We hasten to assert that no social engineering is intimated here; we are merely stating the fact, established by the social and behavioral sciences for decades, that favorable behavioral attributes or habits are also learned -- very often in an experiential fashion as ideally provided by a mentoring relationship, project, or program! It should be noted that the attainment of this somewhat amorphous objective is partly gauged through the actual efforts and the quantified learning and research outcomes, at least in the long-run. The role of mentoring is to engage the mentee in this self-guidance and self-drive mode which, like other skills, is acquired through practice . In the tableau of the highly perceptive and profound painting of a mentor-mentee relationship by [the student's name from Dr. Ward], this non-cognitive objective consists not only of having the mentor serve as a lighthouse to the ship of the mentee, but also in assisting this ship to become a source of light--for itself and hopefully for others--for safe and productive journeys on the ever-challenging seas.
The above guiding principles basically define our proposed framework that can be easily adapted to conduct thorough assessments and evaluations of high school and college student mentoring processes and results. It should be underscored that the quantitative and qualitative data or information are both needed and that one set (i.e., qualitative set) cannot serve as a substitute to the other. Attitudinal and generally non-cognitive information cannot be replaced by test scores. It addresses, among other factors, the internalization of the locus of control, emotions and related motivations, and provides key indicators of personal and professional growth and of mental, emotional, and intellectual empowerment. Key features of the mentoring processes or activities, on one hand, and outcomes, on the other hand, are both indispensable, particularly if one is to meet the criteria of comprehensiveness and of intervention-outcome relationships. Essential elements of the assessment and evaluation of mentoring at the Timbuktu Academy, as partly derived from the above framework, are listed at its web site (http://www.phys.subr.edu). A certain amount of overlap or redundancy between these assessment and evaluation items is by design! The Timbuktu Academy is a national mentoring program funded by the Department of the Navy, Office of Naval Research (ONR) and by the National Science Foundation (NSF). The latter funding is through the Louisiana Alliance for Minority Participation (LAMP).
This essay, it is hoped, will stimulate further thoughts and efforts
towards the development of assessment and evaluation standards for mentoring
activities and results. This framework certainly advocates the same level
of care and of thoroughness as we bring to bear in our research endeavors!
A rationale for this seriousness simply stems from the fact that the mentoring
of "new" scientists and engineers is at least as important as
the production of "new" knowledge . In fact, many successful
researchers remember how they have been so well trained and understand
that quality teaching/mentoring and research are really two poles of the
same magnet. Where some may see incompatibility, there really are similarities
and synergy. This assertion becomes particularly apparent in light of the
process by which human civilization is transmitted from one generation
to the next. The detailed implementation of the framework to the assessment
and evaluation of mentoring at the Timbuktu Academy has been as critical
for our success as any other activity. Several aspects of it provided the
"peer-review" equivalent our practice of science [physics] demands
that we seek and pass. Further, this implementation allowed us to pass
site visits and audits (of activities, finances, results as noted above
among the quantitative objectives, publications, etc.) very successfully.
1. Tinto, V. (1975). "Dropout from Higher Education: A Theoretical
Synthesis of Recent Research." Review of Educational Research, Vol.
45, No. 1, 89-125.
2. Bean, J. P. (1982). "Conceptual Model of Student Attrition: How Theory can Help the Institutional Researcher." Studying Student Attrition, Edited by E. T. Pascarella. San Francisco: Jossey-Bass, 17-33.
3. Cabrera, A. F., Castaneda, M. B., Nora A., and Hengstler D. (1992). "The Convergence Between Two Theories of College Persistence." Journal of Higher Education 63, No. 2, 144-163.
4. Undergraduate Science Education: The Impact of Different College Environments on the Educational Pipeline in the Sciences [and Engineering], Higher Education Research Institute (HERI), Graduate School of Education, UCLA, 405 Hilgrad Ave., Los Angeles, CA 90024.
5. Bagayoko, D. and Kelley, E. L. (1994). "The Dynamics of Student Retention." Education Vol. 115, No. 1, 31-39.
6. D. Bagayoko, "Mentoring: A Strategy for Increasing Minority Participation," Proceedings, National Conference to Explore the Role of Social and Behavioral Science Careers, Radisson Hotel, New Orleans, Louisiana, April, 1997. Published Southern University and A&M College for the Louisiana Alliance for Minority Participation (LAMP), Edited by Dr. Emmanuel Osagie, Ph.D., Economics, Associate Vice Chancellor for Research and Strategic Initiatives.
7. Proceedings, Symposium on Mentoring for the 21st Century, 1996 Presidential Awards for Excellence in Science, Mathematics, and Engineering Mentoring (PAESMEM), National Science Foundation (NSF), Publication No. NSF 97-124.
Assessment and Evaluation Elements of the Timbuktu Academy
For precollege and undergraduate mentoring activities
(the overlap or redundancy between some elements is by design)
1. Data/information on time on learning tasks as dictated by the power
law of human performance [5,6] and the scope and depth of covered materials
- opportunity to participate in study groups (social and academic integration);
- adequacy of access to faculty members, tutors, mentors for learning or research purpose (academic and social integration);
- access to computers, Internet, CD-ROMS, and other pertinent resources for for studying or research purposes.
2. Extent and nature of the supportive and challenging environment to
effect the academic integration and social integration[1-3] as per the
student retention models;
- explicit nature of expectations as enunciated by a mentor, teacher, or professor as opposed operating on assumptions (for mentees or mentors);
- explicit and thorough nature of the discussions/reviews of all requirements for progress to the next level (of study or research) and for graduation; the earlier this occurs the better it is;
- explicit nature of discussions/reviews of the expectations of the next level in the educational continuum (next grade, college, graduate school, the competitive job market); the earlier this occurs the better it is; - adequacy of the overall support, including the financial one when applicable;
- guidance for communication and computer skill development and other non-cognitive enhancement not generally attainable in a course, including preparation for the Graduate Record Examination (GRE) for undergraduates;
- frequency and nature of monitoring, including the keeping of a portfolio;
- opportunity for integration into the academic life of a department, unit, or group as through the participation in seminars, picnics, workshops, study groups, conferences, and, when applicable, the opportunity to be a tutor, to have some responsibility (and hence to be trusted and to belong further)!
3. Quantitative data on the scholastic achievements of the mentees:
- the grade point averages in relation to actual SEM courses taken, or, for high school students, college-SEM preparatory courses (mathematics, science, English, etc.);
- standardized test scores (i.e., ACT/SAT, GRE, etc.); for summer academic enrichment programs for high school students, pre- and post ACT/SAT scores are critically needed.
4. Data/information on exposure to and participation in research--when applicable-- to include the availability of research mentors/supervisors on and off-campus and the extent (scope and depth) of research involvement.
5. Quantitative data on research performance: reports, refereed and other publications, presentations.
6. Extent of personal and professional growth -- as through the enhancement
of sound judgment and self-discipline, hard-work, sense of ethics, temperament;
- enhancement of attitudinal traits conducive to working well with others;
- internalization of the locus of control with the subsequent decrease of negative effects of peer pressure, if any. Measuring these items is challenging indeed!
7. Extent of the guidance and assistance for moving to the next level
(i.e., next grade or college for high school students, graduate school
or job market for undergraduate and graduate students: knowledge of opportunities,
substantive and informed nature of letters of recommendation;
- introduction to or immersion into supportive networks conducive to continued growth and life long learning.
8. Comparison of the above outcome-results with those of control groups,
whenever possible; the following comparisons and others are essential;
- the percentage of precollege students taking college preparatory curricula as compared to the percentage for an appropriate control group;
- undergraduate retention and graduation rates of Academy Scholars as compared to those for control groups;
- percentages of scholars enrolling and succeeding in SEM graduate programs, particularly for the Ph.D. degree, as compared to those for control groups;
- research (or learning to learn) accomplishments of mentees as compared to that of others.
9. Follow-up on mentees; this could be extensive and very long-term, depending of the nature of the mentoring activities or programs.
10. Nature and scope of institutional support, external funding, and the participation of partners. The incentive structure of the institution and its financial support for and services rendered to mentors and mentoring programs can be critical indeed.