The discussion document Towards a Framework for the Monitoring and Evaluation of Higher Education, published by the Council on Higher Education (CHE) in 2004, explained the relationships between monitoring, evaluation and research as different methodologies used to understand higher education at the systemic and institutional levels. Monitoring, the document argued, depends on the data it gathers from management information systems that operate at various levels of the higher education system, which it uses to help to identify trends. Evaluation in turn depends to a large extent on the data provided by monitoring systems, which it uses to assess the achievements of policies and interventions against set goals or benchmarks. Research is independent of both monitoring and evaluation: it does not depend on the existence of management information systems and it does not have to assess what an intervention has achieved. Neither does it depend on the existence of evaluative studies, although these studies can provide useful and interesting insights into a problem. Research as a specific type of intellectual enquiry can construct its own data sources according to its purposes, and does not have to come to a formative or summative judgment. Nevertheless, social science research can analyse and interpret different types of data in order to highlight problem areas, establish explanatory links, identify information gaps, suggest possible conclusions and make recommendations in order to solve a particular problem.
The CHE suggested in 2004 that the Monitoring and Evaluation Directorate was going to use all three methodologies – monitoring, evaluation and research – to deal with different aspects of the higher education system depending on the nature of the issue it was trying to address. This third issue of the Higher Education Monitor is a good example of the differing relations between monitoring, evaluation and research at different junctures of policy implementation. It also shows how, in a context of weak monitoring systems and unfinished policy implementation, monitoring and evaluation give way to research as a more suitable tool to understand institutional and individual responses to a policy issue.
In 1997 the White Paper on an Integrated National Disability Strategy remarked on the dire lack of data on disability in South Africa, data which would allow government and relevant organisations to design, plan and implement strategies for disabled persons as well as to measure their impact. To a large extent, the lack of data on disability reflects the ineffective role that management information systems have had up to now, both at different levels of the state agencies and departments and at the level of the institutions and organisations that deal with disability. This situation can be explained through the absence of management information systems altogether or, more often, by the fact that these systems do not collect data on disability, or by the fact that confusions in the definition of disability undermine accurate data collection. In the case of the Higher Education Management System (HEMIS) of the Department of Education (DoE), although the system includes disability as a field of collectable data, it has not yet been implemented. Higher education institutions (HEIs) are not yet obliged to provide data on disabled students as part of their submissions to the DoE. Yet, as will be seen in the report, this is not solely a HEMIS problem. HEIs have very uneven capacity to systematically collect reliable data on disability in their own campuses.
As a consequence of this, no systematic central monitoring of disability in higher education has been put in place. This, in turn, undermines most attempts at evaluating policy implementation in relation to disabled students. Thus, rather than proceeding from the analysis of monitoring data to the evaluation of policy implementation to the setting up of research projects to look into specific issues suggested by monitoring and evaluation, it has been necessary to take a different route. Qualitative and quantitative research, done with different social sciences methodologies, became the point of departure to stress the need to monitor and to evaluate social and political interventions in the terrain of disability among students enrolled at HEIs.
In this regard, this study constitutes, as far as we are aware, the first systematic attempt at obtaining baseline data on disability among higher education students directly from HEIs. The study was undertaken as a collaborative project between the CHE, the Centre for the Study of Higher Education (CSHE) at the University of the Western Cape, the Inclusive Education Directorate of the Department of Education, and the South African Federal Council on Disability (SAFCD).1
Despite its attempt to collect baseline data through questionnaires, this study is not focused on the quantification of the prevalence of disability among higher education students enrolled at public HEIs. On the contrary, the study is very aware of the limitations of the quantitative information yielded. The research project was designed to generate knowledge about the different ways in which public HEIs work to broaden access for disabled students. The investigation probed institutions’ familiarity with policies, and it examined instances of the creation and resourcing of specific structures to deal with the special needs of disabled students.
The lines of enquiry pursued in this research stem from a conceptual engagement with different local and international understandings of disability and their implications for policy making and the modification of social practices as well as the manner in which these debates have influenced South African policies on disability. The South African disability movement and the South African government approach disability from a social model. This model sees the circumstances of people with disabilities and the discrimination they face as a socially created phenomenon which is not related to the impairments of disabled people. Consistent with this, the response to disability in the social model is the restructuring of society for it to be able to deal appropriately with people with impairments. Unsurprisingly then, this research focused on the ways in which HEIs respond to the special needs of disabled students.
The methodology used in the study, a combination of surveys and interviews, allowed the researchers to identify a range of practices in relation to disabilities and to analyse how these relate to the enabling or constraining circumstances found at institutions. The work highlights the roles of structures and individuals in both overcoming and stressing institutional constraints and shows that while adequate financial resources are a key element in creating an enabling teaching and learning environment for disabled students, personal attitudes play an even greater part in facilitating access and asserting equity.
In undertaking this project the CHE acknowledged that disability constitutes an important, and often overlooked, aspect of the definition of equity of access to higher education. Bringing disability to a more prominent place in our analysis of policy implementation and policy impact has a number of important implications for the CHE as well as for other higher education stakeholders. The findings of this study suggest areas for medium- and long-term interventions for different bodies and organisations.
For the CHE Monitoring and Evaluation Directorate the results of the study highlight the need to start a process of development of indicators to monitor access and equity in relation to disability. An important part of this process would be to approach the DoE to officially request HEIs to provide information on disabled students in their submissions to HEMIS. The development of indicators on disability will also require further research on international practices in monitoring disability in higher education.
From the point of view of the HEQC the research highlights that the assessment of the quality of students’ teaching and learning experiences needs to include a specific focus on both the concrete experience of students with disabilities at HEIs and the training of academic staff to deal with disabled students in a supportive manner.
From the point of view of HEIs themselves the research findings suggest a number of areas for reflection and intervention. First, and foremost, is the need to develop internal systems to identify disabled students and their profiles, to understand their needs and to monitor the extent to which these needs are met at individual institutions. The development of support mechanisms for disabled students and academic staff in order to facilitate teaching and learning processes seems to be the second area for institutional intervention. From the perspective of the DoE’s Inclusive Education Directorate, the research points out that there is often weak and incomplete awareness at HEIs’ dedicated disability offices or equivalent of government policies directly or indirectly related to disability. Which strategies can and should be used to achieve a clearer understanding of the ways in which White Paper 3, the Higher Education Act, White Paper 6 on disability and the National Plan on Higher Education relate to each other in the specific area of equity and disability?
From the point of view of the Higher Education Branch of the DoE, the study suggests that mergers are impacting differently on the ways in which disabled students are dealt with at each newly-merged institution. This seems to depend on the levels of awareness, resourcing and the kind of historical practices vis-à-vis disability predominant at each merging institution. This suggests the need to, on the one hand, find mechanisms to monitor the implementation of equity goals at newly-merged institutions, and on the other, suggests the difficulties in funding equivalent provision of services and education for disabled students at all the campuses of merged and incorporated institutions.
None of these issues can be tackled and solved immediately. On the contrary, time, resources and careful reflection are required for the analysis of the specific policies and the establishment of adequate monitoring systems with a view to evaluate impact and to better understand existing practices, including in this process the voices of disabled students themselves.
As these elements take shape and different stakeholders find their pace and space in this process, the CHE will use this issue of the Higher Education Monitor to generate opportunities for public discussion and engagement with this research report.
Dr Lis Lange Director: Monitoring and Evaluation September 2005
- The project was initiated in May 2003 with the setting up of a project management and research team within the CSHE. The final draft report was submitted to the CHE in March 2005. The following people made up the research team and contributed in different ways to the success of the project: Ms Colleen Howell, project coordinator and primary researcher (CSHE, UWC); Mr Raji Matshedisho, primary researcher (UCT); Prof Sandy Lazarus, research advisor (Education Faculty, UWC); Prof David Cooper, research advisor (Department of Sociology, UCT & Education Faculty, UWC); and Ms Petronella Linders (SAFCD).