Managing public examinations in a regional collaboration construct

By Susan Giles, Senior Assistant Registrar – Examinations Administration & Security

Caribbean Examinations Council

Extract from a Paper presented at the 37th International Association for Educational Assessment Conference, 23 – October 23-28, 2011, Manila, Philippines

Background

The Caribbean Examinations Council (CXC) is a regionally funded non-profit examining board established in 1972.  CXC receives subventions from 16 Participating Governments – its primary stakeholders but its main operating budget comes from the examination fees from candidates of all the countries involved.  CXC was established to set examinations regionally at all levels and nationally as requested by any government and to maintain standards that ensure regional and international acceptance of its certification.  CXC’s first task therefore, was to replace the traditional Ordinary and Advanced level examinations offered by the British Boards with examinations that were more relevant to the needs of the region while ensuring that high standards were maintained. In 2005 CXC expanded this mandate to include the award of Associate Degrees.

In 1979 CXC offered, for the first time, the Caribbean Secondary Education Certificate (CSEC), examinations to the 16+ cohort and has become the market leader in the English-Speaking Caribbean for this level of examination.  In 1998 CXC implemented its mandate for the 18+ cohort by introducing the first examinations for the Caribbean Advanced Proficiency Examination (CAPE) which is offered in two units, and now dominates at that level of examination in the region.

Since 1977 CXC has developed over 60 syllabuses for CSEC examinations, 46 for CAPE Units 1 & 2, and 5 syllabuses for the Caribbean Certificate of Secondary Level Competence (CCSLC) the first examinations for which were written in 2007.  CXC also offers an Associate Degree for candidates writing certain combinations of the CAPE examinations and may be the first degree-granting examination body. In 2006 CXC also introduced the Caribbean Vocational Qualification (CVQ) to its suite of technical vocational offerings.

In 2011 CXC examined over 250,000 candidates at two sittings with 5000 teachers involved in the script marking.  National examination services at the primary level are also provided for four regional governments.

2.1  Structure of the Council

Council and its sub-Committees the Administrative and Finance Committee have responsibility for inter alia, overall policy and management and matters such as Finances, staffing, and accommodation.  The Schools Examinations Committee (SEC) and its sub-Committees SUBSEC and FAC are responsible for, inter alia, the policy and technical matters related to the examinations, e.g. grading, the appointment of Examining Committees, Moderators, Examiners and the syllabus development activity.  The Headquarters office, which also subsumes the Administrative Centre for the Eastern Caribbean, is located in Barbados. The Jamaica office is the Administrative Centre for the Western Zone area which includes Belize, Jamaica, Cayman Islands and the Turks and Caicos Islands.  The office there has full responsibility for syllabus development and from 2009 the name was changed to Syllabus and Curriculum Development to reflect the new thinking in the CXC transformation process.  CXC depends heavily on regional resource persons/consultants to supervise the question paper setting and script marking activities and to provide direction on the subject panels. They, together with the highly trained and qualified full time specialist staff maintains the high quality of its product.

There is a Local Registrar in each Ministry of Education who acts as the CXC agent in the territory and is responsible for the administration of the examinations, the registration of candidates, the local security of the question papers, appointment of invigilators, collection of fees and forwarding same to the CXC.  Local Registrars, who are employees of the regional governments, receive no fee or stipend from CXC, nevertheless operate as an integral part of the CXC structure.

3.0  Early Challenges

In 1979 when the first CSEC examinations were written, the 13 regional governments which offered candidates directed that all eligible students in government controlled secondary schools should write the examinations.  This was 90% of the regional school population. This directive provided CXC with a captive market and made its operations viable.  However, while Governments directed their schools to write the new examination, they also mandated CXC to prepare the examination timetable in such a way as to allow candidates to also register and write the examinations offered by the Overseas Boards.

It was not until 1981 that the first five subjects introduced in 1979 could be offered concurrently with those offered by the Overseas Boards.  This was seen as a way of appeasing those schools and their administrators that had objected to the caribbeanisation of regional education and who preferred to hold on to things British. It could also be regarded as an insurance policy for those governments who were uncertain about the success of the new Body. This situation reflected the continuing interest of the Overseas Boards in the Caribbean market and a failure by CXC to mount an early and aggressive public information campaign on its objectives and offerings.  That effort began in 1979 only months before the start of the new examinations.

3.1  Recognition

Recognition of the CXC examinations also proved a considerable challenge. In several of the territories, concern was demonstrated about the acceptance of CXC qualifications internationally.  It had been reported to CXC that students who had gone to Canada after writing CXC examinations had found that the qualification was not accepted by certain Canadian High Schools or Universities. Five years after the first examinations and during the evaluation of the USAID teacher training project in 1984, the Evaluators reported that they found, among students, significant pockets of concern about the acceptance of the CXC qualification.  They recommended that CXC target the high schools and universities in a high profile public relations campaign, and negotiate agreements with North American institutions to ensure full acceptance of the qualifications. (Final Report USAID 1984)

It is evident that CXC followed the recommendations made by the Evaluators. Subsequent issues of the CXC annual report and CXC publications such as the CXC News (now the Examiner) show an ongoing campaign to ensure international acceptance. There is now very little issue with respect to recognition of all of the CXC qualifications in the traditional markets for Caribbean students. In 1999 the National Academic Records Information Centre (NARIC), a UK clearing house on qualifications, reported that they were impressed by the content and structure of the Advanced Proficiency examinations (CAPE) and in particular the
flexibility they offered. “In examining the syllabus content and duration, we have concluded that the qualification resulting from two subject courses are comparable to the GCE Advanced level.” (NARIC July 1999).   CXC must now address the recognition of its qualifications in non-traditional markets like the European and Asian countries which may attract Caribbean students in the future.

The support from governments when they unreservedly made their full 16+ populations available to CXC was not apparent in 1998 when the first CAPE examinations were introduced. Over the years regional governments had from time to time found it necessary to remind the organization that it should commence plans to replace the Advanced A Level examinations offered by the Overseas Boards. The decision to implement CAPE was taken in the full knowledge that the market for this product was contracting and that other regional sub-degree programmes were increasing their portion of market share. When CAPE syllabuses were finally introduced, the response from regional governments, the primary stakeholders, was lukewarm.  In the first year of the examination only six of the sixteen Participating Territories had agreed to pilot the examinations.  More importantly, two of the major territories with significant A level populations initially declined to participate.  This sent confusing signals to CXC which believed that since the governments had mandated the examination they would participate when the examinations were available.

Similarly, the introduction in 2007 of a new competency-based test, the Caribbean Certificate of Secondary level competence (CCSLC), has not been fully endorsed by regional governments even though they were intimately involved in the development of the new examination.

An analysis of these responses by the Governments, would suggest that other factors were at work since the introduction of the new examinations in 1979.  In 1998 the introduction of CAPE had been affected by the lengthy time-to-market process and the emergence of several alternative government sponsored programmes.  Additionally, although Caribbean Governments continue to spend significant sums on education, the sector is in competition with all others for scarce resources and regional organizations like CXC must seek to ensure a larger measure of self-sufficiency. CXC realised that it was no longer business as usual.

3.1   Syllabus Development and Review

At the time of the establishment of the CXC there was no structured curriculum at the secondary level in several territories.  In preparation for the external examinations offered by the Overseas Boards, the territories would receive the list of topics to be covered prior to the examination, and schools taught to that list. (Walter 2011).  In the years preceding its first examinations CXC had to develop syllabuses based on non-existent curriculum.

Mrs. Irene Walter, the first Pro Registrar and Head of the Jamaica Office from 1974-1994 has suggested that this was one of the most exciting but challenging periods in the life of the new body. She gives full praise to those early resource persons who headed the subject panels charged with developing syllabuses for English A (English Language), Caribbean History, Mathematics, Integrated Science and Geography to be examined for the first time in 1979.

To facilitate this process, National Committees were established in each of the 16 Participating Territories.  Their responsibility was two-fold, first to feed information back to the Panels which would assist with the syllabus development process and secondly to assist in “selling” the concept of a regional examination to the public.  There were many challenges in this process, as the islands were all at different levels of educational development and universal secondary education was not pervasive.  What the establishment of the CXC did for the region, was to ensure the democratizing of the Caribbean education system.  It was as a result of this new body, that the territories paid greater attention to the development of curricula that would allow their students to perform optimally in the new examinations. 30 years later, the statistics available to Governments on the performance of their candidates in the examinations are used for planning and developmental purposes.  One of the major features of the examination system has been the feedback mechanism which CXC established with regional schools.  Teachers and principals may comment on the syllabuses, make recommendations about the content or provide information on the difficulties schools experienced in implementing any aspect of a particular syllabus.  This facility is also in place for the examination papers, and teachers over the years, have recognized that their comments are valued and are considered either at syllabus review or during the grading activities.

In recent years, CXC has established its website, facebook and twitter accounts which allows for real-time interaction with candidates, schools and the general public, and in 2011, for the first time, candidates and schools could access their results online.

A lesson learned over the years has been the importance of providing support materials for teachers related to the delivery of the syllabus and preparation for the examinations and others for students to provide them with examples of how assessment criteria have been applied to various pieces of work. Today, CXC through its website provides teachers and students with exemplars of good responses and guidance on how their work may be improved.  Further, CXC in 2011  embarked on a strategic partnership with Notesmaster Ltd of the United Kingdom to provide students and teachers with an interactive learning portal to support teaching and learning for CXC examinations.

The syllabus development process that has evolved ensures that what is included in CXC’s syllabuses is accurate, relevant, and appropriate and that students’ are exposed to a knowledge pool that is rapidly changing in a knowledge-based society.  Regional Governments have signed on to a value system aimed at creating the Ideal Caribbean person and this together with the tenets outlined in the UNESCO Pillars of Learning has informed the new syllabus development initiatives.   CXC has also incorporated into its syllabuses sound international theories of learning as well as the 21st century skills of critical thinking, problem solving and decision making. CXC continues to benchmark its syllabuses against international standards as the focus is not only about preparing candidates to live and work in the region, but also to assist them in functioning effectively in the international arena.

3.2  Structure of the Examinations

Most of what was proposed by CXC over the years was new to the region and therefore questioned. Administrators, teachers, students, parents and the Caribbean public had to grapple with new concepts such as: objective tests, school based assessment, profiles, criterion- referencing, multidisciplinary subjects, proficiencies and competency based tests.

Objective tests (multiple choice) created early concerns as many territories believed that candidates could not be fairly assessed using this format even though Multiple Choice (MC) was only one mode of assessment in an examination that normally comprised MC, essay and coursework (SBA). 30 years later, there are some who still feel this way.

The Multiple choice component of the examination was and still is a confidential document because the CXC item bank is not at optimum size and needs to be expanded. This fact has added to the mystery surrounding the MC examination.

The School Based Assessment (SBA) was conceptualized as a way of helping candidates, especially those that did not perform well in the examination environment by allowing them to enter the test with scores obtained from class work and to develop their research skills. This was a new mode of assessment for regional teachers and despite initial challenges they were favourably disposed to its implementation. (Miller1983).

It was CXC’s intention to have SBA or continuous assessment treated as home-work and to discourage teachers from viewing it as additional work. (AFC Dec 1985). However, the feeling persists among both teachers and students that the workload involved is excessive and should be reviewed.  So much so that at a meeting in 2010 with the Caribbean Union of Teachers (CUT), the present CXC management was asked to review an earlier decision and consider compensation to teachers for this ‘additional’ work.

The challenges experienced in getting schools to comply with the regulations with respect to SBA returns in 1981 (See SUBSEC Report 1981) has improved over the years.  However in 2011 there were still a total of 923 candidates (0.6% of the CSEC entry) who did not receive results because the SBA samples requested had not been submitted by the published deadline. (FAC Aug 2011).

The five point grading system was another innovation.  CXC was proud to report that it was not a pass/fail norm-referenced system but that each grade described a different level of competence. However, CXC’s confidence in its grading was tempered by the need to secure external recognition and to satisfy its own regional university entrance requirements.  It accepted the proposal that Grades 1 and 2 be regarded as the equivalent of British O levels A-C.  When CXC conceded this point, the grading system was fatally compromised. (Miller 1983).  The territories expressed concern about having three failing grades and felt that the Grade 3 was comparable with GCE and Cambridge.  In 1998 the current six-point grading scheme was introduced and Grades 1-3 have since been considered ‘acceptable’ grades.

CXC also reported results using profile grades to complement and elaborate on the overall grades.  Profiles are descriptors of the skills to be tested in an examination, and in the CXC syllabuses are defined in terms of cognitive skills or in terms of content areas.   For example, in CSEC Geography students are expected to develop skills under three profile dimensions: practical skills, knowledge and comprehension and use of knowledge, while in Agricultural Science the profiles are:  the business of farming, crop production, and animal production. The profile system was considered by nearly all educators in the region as an excellent concept despite the initial confusion with the profile to grade relationship.

Two main difficulties emerged for the public with the profile reporting system. Firstly, there were three profile letter grades A-C and five overall grades 1 – 5 which meant that candidates could obtain 3 A’s and get a grade 1 or 2 or 3 B’s and get either a grade 2 or 3.  Secondly, in the Caribbean context C was not usually regarded as below average and B as average. CXC made considerable effort to simplify the relationship between the profiles and grades.  In 1998 when the six point grading scheme was introduced the matter was resolved.

The criterion-referenced approach to grading was a departure from the norm-referenced approach which had been used by the examining boards operating in the region. This form of grading allowed the examiners to measure candidates’ performance against pre-defined standards. Even though CXC keeps the cut scores confidential, criterion-referenced testing is regarded as more transparent, a fairer system of assessment, and one that allows the users of CXC’s certificates to make sound inferences about candidates’ mastery of the domains tested while also facilitating the equating of standards from year to year.

CXC introduced multidisciplinary offerings like Prin-ciples of Business and Social Studies – areas that were not previously available to candidates through the Overseas Boards.  Candidates who wanted to write these subjects had to seek out the Overseas Boards that offered them. CXC’s Integrated Science Double Award, (i.e. counted as two subjects) was an excellent multidisciplinary offering that failed.  Mrs Walter explained that the subject which was the combination of Chemistry, Biology and Physics into one subject was a new concept which proved difficult for schools to timetable.  It also proved too challenging for teachers who were probably very familiar with one or two areas but who found three disciplines and the in depth teaching required for a Double Award, too great a challenge. The examination was discontinued in 2000.

3.3  Schemes of Examination

Two schemes of examination were introduced.  The Basic proficiency was comparable to the CSE in Britain; the standard of General is comparable to GCE O level.  Both examinations tested achievement at the completion of secondary education which, in terms of years and academic requirements is the same for all candidates. CXC described the examinations as follows: “Basic proficiency connoted subject activity designed to complete a secondary school course in the subject and General connoted subject activity designed not only to complete a secondary school course in the specific subject, but also to provide a foundation for further studies in the specific subject area beyond the fifth year of secondary schooling.”(Broomes et al 1998)

The introduction of two different schemes of examination was new to the school system and the message from CXC with respect to the Basic scheme was not always clear.  On occasion, the public was informed that the Basic proficiency was designed for persons entering the world of work as well as for the high achiever who wanted to write an examination in an area outside of their specialty.  This message did not resonate with teachers many of whom simply considered it a low level test and discouraged parents from registering their children for the examination.  Parents, it was reported, “preferred CXC General Proficiency (O level examinations) over the CXC Basic.  Many admitted that they were influenced by the school’s perspective on the worth of the different examinations”. (Broomes et al 1998)

It was evident that there was some general confusion about the way in which the two levels were defined.  Teachers felt that the overlap between the two levels in the syllabuses, the common questions that appeared on bot papers, and the use by employers and others of the Certificates in such a way that the Basic Grade 1 was equated to the General Grade 2 were the major problems.  As far back as 1983 those who supported the two levels felt that they should have been distinctly different i.e. that the Basic level should have a completely different syllabus and clearly different examination.  In other words, the Basic should be an examination for fourth formers, whereas the General should be one for fifth formers.  As things stood, the Basic was regarded as a “watered down” version of the General.  (Miller 1983)

The greatest area of concern was the question of the recognition of the Basic proficiency for employment purposes.  In all the Participating Territories, there was widespread variation in the manner in which employers, including the governments treated the Basic.  In many territories, the governments did not accept the Basic proficiency for entry into the public service, a fact that was not missed by the private sector.  As a result, the entries declined and the final Basic Proficiency examination was offered in 2007.

3.4   Competency Based Testing

The Caribbean Certificate of Secondary Level Competence (CCSLC) is a competency-based test.  When first introduced in 2007, it was embraced by one government as a certification programme as they perceived the need to have more of their candidates certified on leaving school.  However, CXC very quickly recognized that another school-leaving certification programme was not required by the region, and with the history of the confusion surrounding the Basic proficiency, moved quickly to revise its syllabuses and rebrand the examination as a foundation Competency-Based Education Training and Assessment (CBETA) qualification in the secondary system.  It now has been clearly defined as a foundation programme to assist candidates in the lower school to master concepts in the higher secondary school.  A very important feature of this new certification is that it “departs from simply testing knowledge of the learner, by taking an innovative and creative approach to learning through its emphasis on the use of real-world scenarios to arrive at learning outcomes that will build the competencies necessary for life-long learning and acting.” (Jules 2011) Many more governments have signed on to this programme as it encompasses qualities critical to the global competitiveness and sustainable development in the region.

The Caribbean Vocational Qualification (CVQ), introduced in 2007 is a competency based approach to training, assessment and certification. Candidates are expected to demonstrate competence in attaining occupational standards developed by practitioners, industry experts and employers.  The CVQ is a portable qualification which facilitates freedom of movement in the Caribbean and is geared to the development of the Ideal Caribbean Worker.  It also enhances the investment attractiveness of the work/labour force of CARICOM states and would harmonise the TVET systems across the region. The CVQ levels 1 and 2 are accessible to persons already in the workforce as well as students in secondary schools across the region.

3.4  Teacher Training and Orientation

In the early years, and as late as 1984, CXC debated whether it should be involved in this area since teacher-training was not the business of an examining body. (SUBSEC 1984).  However, when the first syllabuses were introduced to the region, as was seen earlier, CXC recognised the need to assist teachers by embarking on substantial teacher training and material development programmes.  Some of the new syllabuses contained School Based Assessment (SBA) a concept that was unfamiliar to the regional teaching fraternity, so SBA had to be ‘sold’ to every teacher/educator in the region.

The issue too of the volume of work required by candidates and teachers had to be addressed.  Miller 1983 posits that “the general impression given by teachers, students and principals is that the CXC syllabuses are all very ambitious but that there is need for some adjustments especially when the curriculum as a whole is taken into consideration”.  Teacher-training and teacher orientation therefore became part of the business of the Council with CXC emphasizing that it was doing so in partnership with the regional ministries and tertiary institutions.

This was manifested in a major USAID-funded teacher training project for teachers at secondary schools that trained 4500 teachers in 14 territories from 1980-83.

At the end of the project the external evaluators noted: “The CXC/USAID project has clearly demonstrated the viability and desirability of utilizing a regional approach to educational development in the Caribbean……..At the same time CXC has demonstrated its own leadership not only for the Project, but for future endeavours directed toward region-wide improvement of secondary education”. (Final Report on CXC/USAID Project 1984)

With the introduction of each new syllabus, CXC has continued teacher-training/orientation and materials production.  Since the introduction of CAPE in 1998, 3711 teachers in 16 territories in 24 subject areas have been trained.  Similarly, with the introduction of the CCSLC in 2007 there were 1497 teachers trained in 16 territories.  In 2009 CXC signed a Memorandum of Understanding (MOU) with the University of the West Indies Open Campus to further assist with the orientation of teachers to the new syllabuses. It is expected that through mixed mode distance education, all subject teachers across the region can be exposed to systematic training in the mastery of syllabus content and its effective delivery.

The training and orientation workshops ensure that the syllabuses are embraced as teachers become comfortable and confident to deliver the instruction required for the new or revised syllabuses.  CXC recognizes that it is in its best interest to work with governments in this area, but will not lose sight of the fact that teacher training is ultimately the responsibility of the individual territories.

3.5  Script Marking

CXC also had to make some decisions on the type of script marking to be employed.  Consideration was given to residential marking and also to cottage type marking, the latter being the choice of the Cambridge consultants who were instrumental in training the early script markers.  Residential marking saw teachers being brought from several territories to one location to engage in the process of first standardizing before marking the scripts which were then sent to one central location from across the region for final processing.  The script marking was different from that employed by Cambridge as CXC examiners marked by table and by question; thus no one person was responsible for an entire script. The marking process proved to be a major teacher training activity in itself and was critical to the development of the teaching fraternity at a time when there was no indigenous tradition of their involvement in public examinations.

Script marking has had the single greatest impact on the regional teaching profession.  In 2011 at two marking sessions in four countries over 5,000 teachers have participated in a process that allows teachers to interact with their counterparts from Guyana on the South American continent through the archipelago to Belize in Central America.  There is no other forum of this magnitude available to teachers in the Caribbean to interact and to exchange ideas in an intense learning environment.

CXC has not undertaken a formal study to ascertain the impact of the script marking activity on teaching.   In a documentary produced for the 30th Anniversary celebrations, teachers spoke glowingly of what the script marking has done for them, in particular the exposure to CXC’s system of marking which has improved their classroom instruction. Based on this feedback and the improved performance of candidates in the examinations, there is little doubt that the script marking experience has strengthened teaching and assessment skills.

It has been recognized that not all teachers can participate in the script marking and CXC encourages ministries to regard the script marking as a training opportunity. Thus teachers who attend the marking are expected to return home and participate in workshops organized by the ministry to expose their colleagues to script marking – the CXC Way.

The CXC will introduce electronic marking (e-marking) in the very near future. Three e-marking pilots have been completed and further research is being undertaken. This is necessary because the cost of centrally marking over a million scripts and the provision of air transport and hotel accommodation for 2000 teachers in four territories is unsustainable. There are some challenges facing CXC in pursuance of an e-marking solution.  E-marking is an expensive solution, so cost will be a consideration.  More importantly, the availability of bandwidth and the level of internet penetration in the region are issues that also have to be addressed.

E-marking is estimated to eliminate about 70% of the costs associated with the script movement, data capture, travel and accommodation expenses.  It will also reduce the processing time, automate the collection and analysis of data and provide real-time management information throughout the script marking activity.

4.0   CONCLUSION

There can be no doubt that CXC in its 32 years of offering examinations has redirected, restructured and remodeled the public examination system in the region.  The eventual development of curricula to support the new syllabuses has been a signal achievement for a region which had no history of teaching to a specific curriculum.  The establishment of the subject panels charged with developing new syllabuses, created a cadre of persons who were empowered and who through their innovations created syllabuses and systems which have maintained high quality examinations in the region.

Qualified resource persons recruited to the Examining Committees with responsibility for the question paper development received training in psychometrics and measurement. For the first time in the region, personnel were exposed to the development of question papers based on objectives defined by the various syllabuses.  This training has been of significant benefit not only to CXC but to the major tertiary institutions in the region, where these resource persons are employed.

The introduction of the school based assessment (coursework) corrected many of the poor practices evident in the teaching in schools and ensured that candidates had a better opportunity for improved performance in the examinations. One of the trends in assessment in the world is the focus on the importance of practical assessment and the development of skills rather than on delivery of content. There is also an increasing focus on the acquisition of practical skills and the recognition that practical skills are acquired through practice or action (interaction) rather than through theory or speculation. There is also evidence that these skills are achieved through School-Based Assessment that focuses on practical assessment (Cambridge Report 2011).  It is of concern to the CXC that dishonest practices on the part of both teachers and candidates, have invaded the administration of this mode of assessment, and more forensic auditing of SBAs is now routinely undertaken.

The Basic proficiency examination was a sound concept that was poorly implemented. The elitist nature of Caribbean education ensured that teachers did not encourage parents to register their children for the examination.  More importantly, there were some essential issues with respect to student needs that should have been resolved. For example, did the test cater to a distinctly different range of intellectual competence, a different level of intellectual sophistication or a different ability range from the General proficiency?   It is commendable that in the development of the CCSLC, CXC recognized early the candidate needs that should be addressed and moved swiftly to rebrand the examination.

CXC has achieved significant success over the past 30 years.  This success does not only relate to the examinations it has introduced, or its syllabuses and curriculum development activities, but in the support and leadership it has provided to the development of education in the region. There is considerable work still to be done and “ultimately, the guarantee of CXC’s future lies, not just in exemplary execution of its core functions, but in how far ahead of the curve it is, in adding value to education in the region.” (Jules 2011)