1. Introduction

1Evaluation process in distance education is fundamentally different from traditional education. Moore (1999) in his editorial in American Journal of Distance Education has identified some key features of an effective monitoring and evaluation process in a distance education system. Among the key features identified by him were: specifying “good” learning objectives, the efficient handling of assignments, and implementing a data gathering and reporting system. Assignments are a source of feedback to educational authorities on different aspects of student achievement. Moore also called for designing interesting and suitable assignments that provide evidence of “what” the student has actually learned as compared to “what” exactly is required in learning. Here the instructor has two major responsibilities to fulfil: (1) to respond to the student regularly (as per the assignment submission schedule); and (2) to collate the results of assignments evaluated (i.e., scores or grades). A review of such data enables the administration to identify weak and strong points of their system, such as weaknesses related to the instructors’ misinterpretation of evaluation criteria, non-receipt of learning packages, or even something as specific as an incorrect explanation made by a given tutor when teaching a concept during a tutorial meeting. Other more specific weaknesses might include late receipt of assignments, or a missed training session, use of inappropriate teaching materials, unattainable learning objectives, or ineffective learning measurements. Moore suggested designing the assignment to test exactly “no more, no less” than what is desired from the programme of study. He also stressed that at all times educators need to be aware of their roles in the monitoring system.

2. Evaluation of Assignments

2At the Indira Gandhi National Open University (IGNOU), evaluation of student performance takes place at two levels: (1) during continuous learning evaluations and (2) during end-of-term examinations. Continuous evaluation has as its major component “assignment evaluation,” intended to check students’ progress.

3Feedback from such assignments helps educators analyse the levels of successful learning among students, as well as the pedagogical effectiveness of self-instructional material. From the students’ perspective, assignments also play an important role because they convey students’ individual levels of learning achievements, which, in turn, can help them iron-out the negative aspects of the learning process while cementing the more positive aspects. This process is facilitated through tutors’ comments on the quality of assignments. Walker (1987) highlighted the significance of tutor marked assignments as the main vehicle for correspondence teaching, through which tutors may infuse independent learning among learners. Tutors achieve this ideal by providing students with constructive criticism, support, and encouragement via marking and grading assignments. Pradhan (2002) recommended that grading could be used in assessing students’ answers on both extended and restricted response type questions. He observed that grading is more reliable than marking, because grading helps to minimize tutor subjectivity and maximize objectivity.

4Ronan (1997) explored the role played by tutors in resolving the learning difficulties of distance learners registered for a Master’s degree in training programme at a major British university. He found that feedback on the marked assignments was comprehensive, but he also found that often tutors failed to elaborate on what needed to be done by students to obtain a better grade. Ronan also pointed out the occurrence of negative emotions when students did not realize their grade expectations. He reported, “The mentor endeavoured to provide a rational basis for the grade earned, but his task would have been greatly assisted through the provision of suggested answers” (p.63).

5It is essential that ”assignment evaluation and commenting” must be properly monitored. A paper titled ”More than Merely Monitoring” by Edwards and Metcalfe (1988) raises a pertinent question: Who monitors the monitors? They pointed out that although the teaching comments are important, students’ main consideration nonetheless remained their grade;, students are more concerned about “fair reflection of the ‘worth’ of their script” (p.17). Edwards and Metcalfe also referred to different ways of enhancing correspondence teaching skills and feedback to tutorial staff through some form of official monitoring system. Various researchers have suggested the use of different feedback mechanisms; for example, Orton (1978) reported on the use of telephones for feedback, and Evans (1984) reported on the use of audiocassettes for this purpose. Hussain and Sahoo (1994) undertook a study to assess the type of comments given by tutors on students’ distance learning assignments. This assessment was quantitative and qualitative, and designed to suggest ways to improve upon the quality of tutor comments. During their study, Hussain and Sahoo discovered that only 22 per cent of the assignments reviewed by them contained “positive” reinforcements, 18 per cent had general, and 28 per cent contained negative comments. They also found that subject related comments were given on ten percent of the assignments; step-by-step instructional comments on two percent; whereas ten per cent of the assignments contained holistic comments. They concluded that an adequate number of comments should be given on each assignment, and that these should be more pertinent to the subject under study. Reflecting on the importance of monitoring assignment evaluation, Satyanarayanan and Sesharatnam (1992) suggested that pertinent goals of evaluation monitoring include uniformity of marking standards, feedback to course teams concerning students’ progress on their work, rescheduling assignments, reducing/increasing the number of course assignments, modifying advice and guidance given to learners on how to answer questions, and changing tutor notes.

6The assignment model of IGNOU is a three way interactive model. Assignments are dispatched to students from the University. Students subsequently complete and submit their assignments to tutors for marking, after which the marked and graded assignments are returned to students. The tutor also sends his feedback to the University.

3. Tutor’s Comments on Assignments

7Tutor comments form the tangible basis for impressing upon students the desirable aspects of their learning process. In analysing the significance of tutor comments, Morgan and O’Reilly (1999, p. 74) explained that written feedback on assessment items may be the only dialogue that occurs between distance learners and their teachers. To enhance continuous and purposive learning, constructive dialogue is desirable between the student and the teacher (Grugeon 1977, Harrison 1980, Bulkeley 1981, Roberts 1984 and Cole et al. 1986). Lee and Yuen (1993) state, “the turnaround time of tutor marking, and the quality of marking and the comments on the written assignments will certainly reflect the integrity of the tutors.” Cole et al., (1987) also strongly favour an immediate turn around of monitoring feedback, which enable learners to put their tutors’ advice and direction into practice. Jarvis (1978), Baath (1980), Freeman (1983), Mullett (1983) and Rickwood (1992) also support the early return of feedback to students.

8Tutors’ comments can be categorized broadly into (1) teaching type comments and (2) non-teaching type comments. By offering concrete suggestions for improvement, teaching type comments point out the strengths and weaknesses of students’ academic progress.

9One of the important abilities of a good evaluator is to provide useful and constructive feedback to the learner about their work (Morgan and O’Reilly, 1999, p. 74). Positive comments also play an important role in encouraging students to achieve better learning out comes, just as the lack of positive comments may result in students’ increasing a sense of disenchantment with the learning process. Further, the use of positive comments encourages the student to respond positively to the learning process, a dynamic that can ultimately act as encouragement to the tutor as well, thus leading to an immense sense of satisfaction for tutors and students alike. As Lee and Yuen (1993) state: “Learner dropout rates of certain tutors can also explain the extent of encouragement given to the learners by the tutors.”

10Non-teaching type comments, on the other hand, act to either actively discourage or signify nothing whatsoever to students. Under the IGNOU system (IGNOU, 2001), Teaching comments are classified under four major heads. These are:

1. Positive comments

2. Constructive comments

3. Personal comments

4. Global comments

11Positive comments indicate that answers given by a student learning at a distance is either excellent, up to the mark, or that in spite of certain flaws, the answer is original. A comment such as “Your analysis of the causes of depression is acceptable and quite appropriate in this context,” fits the bill. Studies conducted by Mackenzie (1974), Baath (1980) and Mullett (1983) also report that students desire encouraging and supporting comments.

12Cole et al., (1987) emphasised that monitoring comments must be positive, constructive and supportive if they are to enhance learner performance. In addition to academic content, they also recommended linking of comments to tutors’ didactic skills.

13For Cole et al., constructive comments go a long way in addressing these concerns, in that they neither negate nor approve of what the learner has written, but instead offer constructive suggestions as to how students may improve their answer. Morgan and O’Reilly (1999, p. 74) acknowledged that constructive feedback creates dialogue between teacher and students, which then helps students to identify areas for improvement, acquire new skills; and develop reflective and critical self-evaluative thinking skills. A comment such as “You could have enriched your argument by including the following points, a. ------, b. ----, c. ------- etc.,” is constructive in nature.

14Personal comments are intended to mitigate the feelings of isolation often faced by distance learners. At this juncture, the study efforts of Cole et al. (1986, 1987) have to be brought in relation to the efforts of other researchers. Since distance education students operate in isolation, often detached from their fellow distance-learning counterparts engaged in the same course of distance study, it is important to make individual students realize that they are not the only ones working on specific tasks. In short, their efforts can and must be brought into relative comparison with the efforts of other students studying in the same program. In short, the individual learner must be made to realize that the problems and effort undertaken are something that they are not facing alone, but instead are faced by a host of fellow distance education students. Such comments play a vital role in making students realize as to where they stand in relation to other students engaged in the same course of study. This vital function is performed using personal comments like: “This is what most students think about the problem, but you have substantiated your arguments much better than the others.”

15Global comments are intended to cover the entire assignment with reference to the various aspects mentioned in the answer. They are most often used to explain the grading system.

16In simple terms, the use of Global Comments provides students with reasons of why certain grades are given.

17Non teaching comments can be sub-classified under:

• Harmful comments

• Hollow comments

• Misleading comments

• Null comments

• Negative comments

18Harmful comments put off the learner. Learners receiving harmful comments may feel so dejected, that they may dropout from their studies entirely. In short, harmful comments fail to build any rapport with the student on any level. For example, comments such as “Your analysis is absolutely wrong,” can have this effect.

19Hollow comments make it impossible for the student to make any tangible deduction from the comment. A comment such as, “You can improve the structuring of the answer,” looks meaningful on the surface, but is actually hollow in that no meaning can be drawn from it.

20Misleading comments mislead learners to adopt untenable perceptions about themselves, their study environment, and method of study, etc. Comments such as “Read the lesson again” (without pointing out the weaknesses) will lead students to a perception that having gone through the lesson again will enable them to gain academically, whereas in reality they might not have.

21Null comments are basically nonverbal in nature in that they scarcely question, illustrate, or explain. Null Comments include question marks, underlinings, and side brackets to name a few. These sorts of comments are not helpful to the learner.

22Negative comments always indicate that the answer is either incomplete or wrong, but do nothing to explain or indicate as to how to make the answer complete or correct.

4. A Previous IGNOU Study of Tutor Comments

23Distance tutors must make all efforts to write teaching comments and avoid using non-teaching comments. However, it has been our general experience that teachers in the IGNOU system, commonly resort to non teaching comments and are often not well inclined to write teaching comments.

24A report published by IGNOU’s Post Graduate Diploma in Distance Education Programme (IGNOU, 2001, booklet no.3, ES-313, pp. 36-37) provides details of a previous study of tutors’ comments on students’ assignments. Tutors participating in this study were asked to write comments and, more importantly, they were specifically instructed not to write non-teaching type comments. The total number of tutor trainees present for the study was 48.

25They provided a total of 208 comments of which 52(25%) were Hollow comments, 128(61.5%) were Negative Comments and 11(5.3%) were Constructive Comments.

26The term “tutor trainees” refers to the number of trainee teachers who were briefed on various aspects of assignment evaluations in distance education settings. Percentages are based on responses rather than the number of cases, meaning a single trainee teacher is at liberty to give more than one comment. In spite of clear instructions to the contrary, results showed that the teacher trainees engaged in this study, tended to employ non-teaching type comments.

5. Objectives of the Current study

27To study this phenomenon further, the IGNOU Karnal Regional Centre conducted a survey at its study centres. In this survey, tutors were asked to evaluate a sample assignment and write comments on it.

28Our study was carried out with the following objectives:

• To study the pattern of student assignment grading by tutors

• To gauge any gender variation based on tutors’ evaluation of a common assignment

• To determine if there is a differentiation in scaling pattern (as indicated by their comments on assignments) between male and female tutors

• To examine the effectiveness of the briefing on assignment evaluation

• To study variations in grading patterns on a programme by programme basis

• To observe retention levels of tutorial skills acquired by tutors during orientation sessions

• To discover and suggest areas for improvement

29Based on the above objectives and the authors’ experience in the field of distance education, the following set of hypotheses was arrived at:

• There is a certain degree of uniformity in the grading pattern of a given answer, irrespective of the demographic variation of tutors

• Based on norm patterns established by experts, there is a certain degree of uniformity in the grading pattern of tutors

• Tutors rarely give teaching type comments and are more likely to give non-teaching type comments even after being briefed on the salient aspects of commenting on assignments

• The skills acquired by trainee tutors during training programme diminish over time

30Table 1 indicates the population for the study, i.e., the academic counsellors (tutors) working at the 14 study centres that constitute the network of IGNOU’s Karnal Regional Centre. It must be noted that when and where it made sense, tutors working in adjacent study centres were invited to attend orientation sessions at a mutually convenient location.

31Further, it must also be noted that not all tutors who participated in these orientation sessions resorted to grading or commenting referred to in this article as “scaling.”

32Indeed, citing personal reasons, some counsellors chose not take part in this research exercise. In total, of the 296 tutors who took part in the orientation sessions, 115 agreed to participate.

Sampling Technique

33The sample technique selected for our study was carefully chosen so as to provide adequate representation to all the study centres operating under Karnal Regional Centre’s umbrella. Care was taken to ensure that participants were representative of the population with respect to different criteria like age, gender, courses of study, etc.

Data Collection Tools

34For data collecting purposes, we used a “standard” assignment written by a student. Tutors from all study disciplines were asked to analyse this “standard” assignment, based on such evaluative criteria as:

1. Margin comments: (i.e., those of positive, negative, null, constructive nature, etc.)

2. Grading the assignment on a scale from “A” to “E”

3. Scaling (adding comments to) the assignment

4. Providing Global comments

Procedure

35This study was conducted during orientation sessions, where tutors received instruction on aspects of distance education pedagogy.

36Table 1 provides details of the dates and locations where the orientations sessions took place and the number of tutors involved. The resource personnel charged with presenting these orientation sessions consisted of IGNOU Karnal Regional Centre senior academic staff. For this study, tutors were invited to participate in a daylong orientation session, which took place between September and December 2001 at the IGNOU study centre locations listed in Table 1. During each orientation session, tutors were provided with details outlining the functional areas related to distance education in operation at IGNOU, including the functions of the IGNOU study centres and regional centres. Tutors were also provided details on the acquisition of “good” academic counselling skills, along with tips on how to organize and conduct effective counselling sessions, and examples on how to write more effective comments on student assignments.

37Our study, which was designed to research tutor “assignment evaluation,” was part of the day-long orientation session. During this “assignment evaluation” segment, tutors were supplied with the same standard assignment, and invited to evaluate, comment, scale, and grade this assignment.

Rationale for Selecting a Particular Assignment for this Study

38One problem the Karnal Regional Centre faced was that the academic counsellors participating in this study came from different academic disciplines and programmes of study. This meant that as researchers, we were faced with the necessity of administering a “standard” assignment that everyone could feel

39comfortable with. To overcome this problem, a more general assignment was selected for our tutors to assess, grade, scale and comment on. The subject matter of our “standard” assignment was distance education, a field of study we felt tutors should be familiar with given that they worked for IGNOU, a major distance education institution.

40But before attempting such an evaluation, tutors were briefed about the various types of comments available for use to them.

41During these orientation sessions, it was impressed upon the tutors to avoid writing non-teaching comments. The comments given by the tutors were then divided into the various categories (outlined above). The tutors were also instructed to write “Global Comments” on the sheet provided for the purpose. The data for scaling responses were derived from the scaling responses of the study participants. Tutors were requested to assess the quality of the standard assignment using a six-point scale. Responses in the first two points on our six-point scale were grouped under the category “excellent.” Responses in the next two points were grouped under the category “satisfactory.” Finally the responses in the final two points were categorized under the “much to be desired” (MBD). Ultimately, data for each sub-analysis varied, because of the existence or absence of evaluative criteria captured on the response sheets.

6. Findings

42The study revealed interesting results. The facts related to distribution of teaching comments according to gender are provided in Table 2

43The percentages for each category of comments were calculated based on the number of comments divided by that of gender. For example, for the category of ”No Comments” for female tutors, the percentage 5.12 was arrived at by the calculation, (2/39)*100= 5.12. The most important finding that has emerged from our study is that the overall substantive nature of the comments. Approximately, 50 per cent of the respondents gave both positive and constructive comments. So although over 10 per cent of the tutors gave null comments, and over 16 per cent gave negative comments, the fact remains that the underlying “attitude” of the tutors remained positive. The sections that follow discuss the various parameters associated with “assignment evaluation” as based on the research objectives of our study.

Nature of Comments

44Of the total sample size of 115 tutors, 56 (48.6 per cent of the sample) offered positive comments. A total of 60 tutors (52 per cent) gave constructive comments. And 97 per cent of female tutors and 95 per cent of the male tutors gave some kind of global comment in the space allotted for this purpose.

45Nineteen per cent gave null comments. Only a small percentage (0.86 per cent) gave negative comments. Twenty-two per cent wrote hollow comments. And 1.73 per cent each of the tutors gave harmful and personal comments respectively.

Scaling Pattern

46When measuring the use of “scaling” (types of comments) by tutors, it is interesting to note that of the 76 male and 39 female tutors, 51.3 per cent did not use add comments to the assignment. What is more interesting, is that 29 (74.4 per cent) of the 39 female tutors engaged in our study, did not use scaling when assessing the standard assignment. Of the 47 tutors who used scaling, 33 (70.2 per cent) considered the quality of the assignment as highly accurate; 27 (57.44 per cent) as being excellent in coverage; 25 (53.19 per cent) as conceptually clear; 25 (53.19 per cent) as excellent or well planned; 22 (46.8 per cent) as being of ideal length; and finally, 25 (53.19 per cent) as being clear in expression. The parameter listings do not follow any specific sequence, but is instead based on the order of importance.

47Nineteen per cent gave null comments. Only a small percentage (0.86 per cent) gave negative comments. Twenty-two per cent wrote hollow comments. And 1.73 per cent each of the tutors gave harmful and personal comments respectively.

Scaling Pattern

48When measuring the use of “scaling” (types of comments) by tutors, it is interesting to note that of the 76 male and 39 female tutors, 51.3 per cent did not use add comments to the assignment. What is more interesting, is that 29 (74.4 per cent) of the 39 female tutors engaged in our study, did not use scaling when assessing the standard assignment. Of the 47 tutors who used scaling, 33 (70.2 per cent) considered the quality of the assignment as highly accurate; 27 (57.44 per cent) as being excellent in coverage; 25 (53.19 per cent) as conceptually clear; 25 (53.19 per cent) as excellent or well planned; 22 (46.8 per cent) as being of ideal length; and finally, 25 (53.19 per cent) as being clear in expression. The parameter listings do not follow any specific sequence, but is instead based on the order of importance.

49The most notable result of our study is that overall there is a general disinclination among tutors to use scaling. However, the disinclination to use scaling appears more prevalent among female tutors, than that of their male counterparts. In total, 48.68 per cent of male counsellors used scaling, as compared to only 25.64 per cent of female tutors. This finding essentially confirms our hypothesis that males are more inclined to scale than females.

Chi-Square Test on the Extent of Resemblance in Scaling pattern

50Each scaling pattern parameter (accuracy, coverage, conceptual clarity, planning, length, and clarity of expression) was subjected to rigorous Chi-Square testing. This test was intended to test whether there is uniformity in the scaling pattern between male and female counsellors. The values have been based on degrees of freedom as 2 [calculated (3-1) (2-1)= 2 X 1= 2], derived from a 3*2 table. The degree of freedom at 0.5 level of significance was 5.991. In the succeeding tables, observed values have been shown as ‘O’ and expected values as ‘E.’

Accuracy

51The parameter of Accuracy was put to Chi-Square testing. The expected values for the “excellent” scaling parameter were 31.7 for males [(42*37)/49] and 5.3 for females [37-31.7]. The “satisfactory” scaling parameter was 7.7 for males [(42*9)/49] and 1.3 for females [9-7.7]. The results for “much to be desired (MTD)” was 2.6 for males [(42*3)/49] and 0.4 for females {3-2.6]. The calculated value of Chi-Square for the distribution is 4.32, which falls in the acceptance range of 5.991. This means that no significant difference was found in the scaling pattern on the parameter accuracy between males and females.

Coverage

52On putting the parameter of Coverage to Chi-Square testing, the following aspects were noticed.

53The expected values for “Excellent” scaling parameter were 22.7 for males [(32*27)/38] and 4.3 for females [27-22.7]. A similar pattern emerged for the “satisfactory” parameter, which showed 6.7 for males [(32*8)/38] and 1.3 for females [8-6.7]. Similarly, the results for the MTD was 2.5 for males [(32*3)/38] and 0.5 for females {3-2.5]. The Chi-Square test result of 1.556 falls within the acceptance limit and therefore confirms our hypothesis that there is a great deal of uniformity between the scaling pattern of male and female counsellors.

Conceptual Clarity

54The parameter for Conceptual Clarity when put to Chi Square testing revealed the following results.

55The expected values for “excellent” scaling parameter were 21.5 for males [(31*25)/36] and 3.5 for females [25-21.5]. The same pattern emerged for the “satisfactory” parameter, which was 8.6 for males [(31*10)/36] and 1.4 for females [10-8.6]. Similarly, the results for MTD was 0.86 for males [(31*1)/36] and 0.14 for females {1-0.86]. The Chi-Square test result of 4.474 falls within the acceptance limit and therefore confirms our hypothesis.

Planning

56The Chi-Square test on the parameter of planning revealed the following facts. The expected values for “excellent” scaling parameter were 19.8 for males [(34*25)/43] and 5.2 for females [25-19.8]. When calculated for “satisfactory” parameter a similar pattern emerged with 6.3 for males [(34*8)/43] and 1.7 for females [8-6.3]. Similarly, the results for “MTD” were 7.9 for males [(34*10)/43] and 2.1 for females [10-7.9]. Again the Chi-Square result of 1.6102 falls with in the acceptance range.

Length

57The expected values for “excellent” scaling parameter were 19.6 for males [(33*22)/37] and 2.4 for females [22-19.6]. When calculated for “satisfactory” parameter, a similar pattern emerged with 6.25 for males [(33*7)/37] and 0.75 for females [7-6.25]. The results for “MTD” were 7.13 for males [(33*8)/37] and 0.87 for females [8-7.13]. Here too the Chi-Square value of 3.3023 falls within the acceptance range.

Clarity of Expression

58The expected values for “excellent” scaling parameter were 20.7 for males [(34*25)/41] and 4.3 for females [25-20.7]. When calculated for the “satisfactory” parameter, a similar pattern emerged with 9.12 for males (34*11)/41] and 1.86 for females [11-9.12]. The results for “MTD” was 4.14 for males (34*5)/41] and 0.86 for females [5-4.14]. The result for Chi-Square test was 1.346, which falls with in the accepted range at 0.5 level of significance, which was 5.991.

7. Discussion

59When the parameter of accuracy was put to test at two degrees of freedom (as obtained from the number of columns and data rows and at 5 per cent level of significance) the result obtained for Chi-Square test was 4.32, whereas the table value for the accuracy parameter was 5.991. The result falls within the acceptance range and therefore, it can be concluded that there exists a general uniformity in the scaling pattern between male and female tutors.

60When the parameter of coverage was put to the same test at the same level of significance, the Chi-Square test revealed a result of 1.566, which also falls within the acceptance limit of 5.991. Here, too, our hypothesis of a general familiarity between the scaling patterns between male and female tutors is authenticated.

61The Chi-Square test on the parameter of conceptual clarity brought a result of 4.474, which again falls with in the acceptance limit set by 5 per cent level of significance at 2 degrees of freedom. Hence, the pattern of scaling uniformity prevails in this parameter as well.

62The variable of planning was then subjected to Chi-Square testing. At 5 per cent Level of Significance and at 2 degrees of freedom, it returned a test result of 1.6102, which falls with in the acceptance limit. Therefore the hypothesis of uniformity of scaling also holds true for the parameter of planning.

63The Variable of length returned with a result of 3.3023, which again falls with in the acceptance range of 5.991 and therefore holds true the test hypothesis of uniformity in scaling for this parameter.

64The parameter pertaining to clarity of expression also confirmed the hypothesis at 2 degrees of freedom and 5 per cent Level of Significance. This parameter returned a test result of 1.346, which again falls with in the acceptance range.

65Thus, on all parameters studied, the test results indicated that a great deal of similarity exists in the scaling pattern between male and female tutors.

Analysis of Comments as Based on Age Demographic Factor

66The demographic division of comments based on age has revealed interesting results. Of the 44 comments given by tutors in the 20-30 years age group, 12 were positive comments, 17 were constructive, four were null, one was negative, and two misleading. Of the 48 comments belonging to the 31 - -40 year age group, 17 were positive, and 15 constructive. Nine of the 26 comments from the 41 - -50 year age group were positive, and five constructive.

67This finding can be compared to the 51-60 age group, where the number of positive and constructive comments was nine and nine respectively (the sample size here is different from other samples presented in the study because the age of some of the participants could not be ascertained). Table 17 provides a clear representation of the division of comments according to tutor demographics, information that is not easily comprehensible in earlier tables.

68Our study reveals that the dispersal of hollow comments is approximately uniform across all age groups. The incidence of positive and constructive comments is high among younger age groups (ages 20 to 30 and 31 to 40 years). This fact is borne out in that of the 48 positive comments given, 29 were generated from the 20 to 30 and 31 to 40 age groups. Similarly, for constructive comments, these two age groups accounted for 32 out of 48 comments. Therefore the overall effectiveness of the orientation sessions appears to be the highest in these two age groups.

Analysis of Comments as Based on Programmes of Study

69Our data also revealed interesting results when the analysis of comments was compared across programmes. As a researcher, it was satisfying to find that in all subject group categories studied, the number of positive and constructive comments remained high. The results are shown in Table 5.

70Another positive fact pertaining to the grading pattern of tutors engaged in all programmes of study, is that the proportion of positive and constructive comments is also high. However, concern remains in that the use of non- teaching comments is found to be higher among the bachelor’s degree programme tutors than found in other programmes of study.

Award of Grades

71With regard to grading, 24 tutors awarded ‘A’ grades, while 30 awarded ‘B’ grades, and one a ‘C’ grade (see Table 6).

72With regard to the allotment of grades for the assessment of the standard assignment, there was considerable variation in the grading patterns of female and male tutors. Of the total ‘A’ grades given, while 75 per cent were awarded by males, 25 per cent were awarded by females. This finding reveals that males are significantly more liberal than females in awarding higher grades. Nonetheless, our study shows that overall the degree of concentration for ‘A’ and ‘B’ grades among both male and female tutors is high.

Perfection in Grading

73To analyse the degree of grading proficiency among tutors, we subjected the data to a test to determine the degree to which the tutors actually have awarded students with the grade they deserved. To make this determination, we used a ”dispersion technique.” But first, to make this technique more clear, we have attempted to clarify some terminology.

Norm

74Norm refers to the grade awarded to a response by the best judgment available for the purpose. A panel of four experts with the objective to determine a “perfect grade.” The panel was then assigned the task of arriving at a median grade, which we considered the “norm” for the purposes of this study. For the current “standard” assignment, all the four experts arrived at the grade of “B” as the Norm.

Dispersion

75Dispersion refers to the scattering of grades away from the Norm. A grade above the norm will elicit a positive “dispersion value” and vice-versa. With the norm in the current study determined as a “B,” a grade of “A” elicits a positive dispersion value of +1, while a “C” grade will elicit a negative dispersion value of -1

n= number of tutors who assigned a particular grade

d= distance of the academic counsellor’s grade assignment from the norm

76As shown in Table 7, the range of dispersion is from +24 to –1, a range of 25 points. An interesting revelation is that although the modal number stayed within the Norm, a sizeable number (24 responses) deviated above the Norm.

77As shown in Table 7, the range of dispersion is from +24 to –1, a range of 25 points. An interesting revelation is that although the modal number stayed within the Norm, a sizeable number (24 responses) deviated above the Norm.

Results of the standard deviation Test

78The data derived from our study were subjected to statistical analysis, which yielded some interesting results. To obtain a comparable guidance value, the data from the previous IGNOU study were also subjected to the same test.

79The results are presented in Tables 8 to 11. In the following table the intermediate steps for the manual calculation of standard deviation (a measure of dispersion by using the deviation from Actual Mean) are given.

80This observation holds true for all Tables 8 to Table No. 11. The standard deviation is the value of 46.58, and the coefficient of variation is 89.59, as derived from Table 8.

Variation by Age

81When the age related data were subjected to the same test criterion, the results produced elements of class uniformity. Let us first see what the analysis of the data for different age group yields.

(x-x)= Deviation of the frequencies from arithmetic mean

(x-x)²= Square of deviation of the frequencies from arithmetic mean

82As derived from Table 9, for the 20 to 30 age group, the standard deviation was 5.70, and the coefficient of variation was 78.16. For the 31-40 age group the standard deviation was 6.38, and the coefficient of variation was 81.89. The standard deviation was 2.92, and the coefficient of variation was 67.47 for the 41-50 age group.

83Finally, for the 51-60 age group, the standard deviation was 3.4, and the coefficient of variation was 61.88.

Degree of Variation in Grading Patterns on the Basis of Gender

84The analysis of tutors’ grading preferences by gender was also analysed using the same technique. The results are captured in Table 10.

(x-x)= Deviation of the frequencies from arithmetic mean

(x-x)²= Square of deviation of the frequencies from arithmetic mean

85As shown in Table 10, the standard deviation is 9.10 for male tutors, and the coefficient of variation is 66.94. The standard deviation for female tutors is 3.4, and the coefficient of variation is 73.91.

Degree of Variation of Tutor Comments Across Programmes of Study

86When the data for distribution of comments across different programmes of study were put to statistical test, the following results were revealed.

(x-x)= Deviation of the frequencies from arithmetic mean

(x-x)²= Square of deviation of the frequencies from arithmetic mean

87As shown in Table 11, the standard deviation for Bachelor’s degree programme is 12.74, and the coefficient of variation is 74.96. For Other Courses and Diplomas, the standard deviation is 1.85, and the coefficient of variation is 77.2. For Computing Science, the standard deviation is 2.86, and the coefficient of variation is 62.25. For Management, the standard deviation is 4.67, and the coefficient of variation is 85.18.

Retention of the Concepts

88A random sample of 20 tutors who participated in the orientation sessions were selected on September 28, 2002 and asked to evaluate and comment on a second “standard” assignment.

89Results of this follow-up exercise suggested that many of the concepts taught to the tutors during the orientation sessions had been retained. The results are outline in Table 12.

90The tutors, who participated in follow-up exercise, produced a total of 31 comments. Of these 31 comments, nine (29 per cent) were positive in nature, 15 (48 per cent) were constructive, two (6.4 per cent) were null, three (9.67 per cent) were hollow, and two (6.4%) were personal. The results of this follow-up exercise, suggests that tutors retained a considerable degree of the skills taught during the previous orientation sessions.

8. Conclusions

91The study of the grading pattern of tutors produced interesting results. The relatively high coefficient of variation exhibited by management programme tutors is based on their preference to use constructive comments, which is a positive sign.

92However, the high percentage of null comments given by the tutors of bachelor’s degree programme reveals an area of concern. This exercise has confirmed our hypothesis that a certain degree of uniformity exists among tutors irrespective of differentiation based on gender, programme of studies taught, and age.

93Our study also shows that the predominant form of comment given by the tutors was generally of the ”teaching” type. A study by Edwards and Metcalfe (1988) also suggests the use of uniform grading and commenting standards for students. However, their contention that counsellors are more likely to give non-teaching type comments has largely been refuted by our study of IGNOU’s programme.

94Indeed, our study did not reveal no major differences in tutor evaluation patterns based on gender.

95This finding is supported by the evidence, which shows that approximately 50 per cent of male and female tutors generated positive and constructive teaching comments.

96One conclusion we can safely derive from our application of chi-square test on gender-based data, is that on all parameters of analysis related to scaling, a general degree of uniformity exists between male and female tutors. In sum, this finding indicates that there is least degree of difference between the male and female counsellors when evaluating the same “standard” assignment.

97The study further revealed that future orientation sessions should ideally focus more on averting negative comments (i.e., null and hollow). The key to success on this front is to repeat the briefing process often. Indeed, previous experience has shown that repeating this training has reduced the incidence of non-teaching

98comments (IGNOU, 2001). Attention should also be given to the area of scaling.

99Indeed, our study, which reveals a general disinclination of many tutors, especially females, towards the use of scaling is a matter of concern. To address this problem, future orientation sessions may stress the positive aspects of scaling.

100Our study has also highlighted the need for ongoing efforts to help broaden the assessment skills of tutors. This can be achieved by reinforcing tutors’ familiarity with the issue areas to which they have already been exposed. To build familiarity, Walker (1987) suggested provision of training exercises for new tutors in the form of evaluating real or simulated student assignments, which are subsequently to be evaluated by experienced tutors. The benefit of such a familiarity-building exercise would acquaint new tutors with a variety of practical techniques in student assignment assessment.

101The test of dispersion also allowed us to arrive at some important conclusions. The fact that the majority of respondents confirmed the norm grade indicated to us that more tutors have acquired the skills to provide a “perfect” grade and therefore, this could be an offshoot of the orientation sessions. Again the tendency to weigh-in on the higher side of the norm could indicate that tutors are more willing to encourage the students by giving them a “better grade” than they deserved.” Thus the hypothesis is confirmed in that that there is a certain degree of concurrence with the norm in regards to the grading pattern of the tutors after the training.

102Another important conclusion that emerged from this study is that the effectiveness of the orientations sessions seems to be highest in the 20 - 30 and 31 – 40 age groups. Future orientation sessions should aim at tapping the potential offered by this age segment. What is more significant, however, is the fact that with proper training, the bias of tutors can be changed from an inclination towards the use of non-teaching comments, towards an inclination in favour of using teaching comments.

103The results of the statistical study are also

104significant. When compared with the previous IGNOU (2001) study, the group of tutors involved in under the Karnal Regional Centre study exhibited a slightly lesser degree of variation. The coefficient of variation was least for the 51-60 age group, which suggests that with increasing experience, distance educators may adopt more rigidly-constructed paradigms that make it difficult for training programmes to introduce newer concepts into this age

105group. The statistical analysis also revealed that the degree of consistency among male tutors (age 51-60) tends to be more than among female tutors. However, there was a high degree of uniformity of dispersion among the various comments. Ideally, there should be a preponderance of constructive and positive comments. Among the data presented so far, we can see that in the age 51-60 group, the lower coefficient of variation is due to the presence of a significant percentage of null and hollow comments corresponding to the percentage of positive and constructive comments.

106In case of statistical data based on programmes of study, the high coefficient of variation of the tutors of the management programme is a source of satisfaction, since the slant is also in favour of teaching type comments. However, the high percentage of non-teaching comments among bachelor degree programme tutors reveals an area of concern.

107To determine the actual retention levels of the evaluative skills taught to tutors during the orientation sessions, we suggest that this exercise be repeated periodically. To this end, we conducted a follow-up test on a random sample of 20 tutors who had taken part in the orientation sessions. The results were encouraging, as they indicated a high level of retention of the assessment skills taught to tutors. This enables us to reject the hypothesis that trainee tutors retain little of the skills acquired during training over time.

108This finding is noteworthy; as it indicates to us that given proper face-to-face orientation, significant improvement may be observed in the quality of comments tutors provide students to assess and to help them improve their performance. It also attests the validity of conducting periodic orientation sessions to upgrade and reinforce tutors’ assessment skills.

Asian Journal of Distance Education

http://www.AsianJDE.org

© 2007 The Asian Society of Open and Distance Education ISSN 1347-9008 Asian J D E 2007 vol 5, no 3, pp 10 - 27

http://www.asianjde.org/​2007v5.3.Sharma.Abstract.html

For copyright / reproducing permission details, email: Office@AsianJDE.org

109.