Continuous Assessment in the Evolution of a CS1 Course: The Pass Rate/Workload Ratio

Abstract

The first programming course (Programming-1, CS1) in the Informatics Engineering Degree of the Facultat d’Informàtica de Barcelona was completely redesigned in 2006 in order to reinforce the learn-by-doing methodology. Along the following eight years several pedagogical measures —mostly related with continous assessment— were introduced with the aim of increasing the pass rate of the course without lowering its high quality standards. This paper analyzes to what extent the added workload on faculty entailed by these measures affects the pass rate. We use a classical marginal cost-benefit approach —from Economics— to compare these two values along time. This process allows us to relate the evolution of the pass rate of students with the workload of the faculty through a productivity curve, as well as to assess the impact of each pedagogical measure. We conclude that, for this course, continuous assessment is expensive. In fact, abstracting from short term oscillations, the slope of the productivity curve is close to zero.

Authors are partially supported by funds from the European Union (FEDER funds), from the Spanish Ministry for Economy and Competitiveness (MINECO) under grants TIN2013-46181-C2-1-R and TIN2012-37930-C02-02, and also from AGAUR of the Generalitat de Catalunya under grant SGR 2014:1034 (ALBCOM).

A Technical Details

In this appendix we calculate the amount of working hours per task at each stage of our course based on the 14 answers that we obtained by surveying current instructors and extrapolating these values to previous timestamps. Note that most measures, once taken, remain in force, thus the workloads involved are accumulated.

• $t_0$ Kick-off (2006–2007): The course started with two kind of lectures, theory and practical, of 3h per week each. Theory lectures were given to groups of 60 students, and the survey says that in average it takes 1.5h to prepare one hour of theory lectures. This results in a total of 2.5h of work (preparation + lecturing). Since the course is 15 weeks long, we have:

  $$\begin{aligned}&T_{t_0}\; \text {hours}=\\&\;\; (1+ 1.5)\frac{\text {hours}}{\text {1h}\; \text {theory}}\times 3\frac{\text {1h}\;\text {theory}}{\text {week}\times \text {group}}\\&\;\;\times 15 \; \text {week}\times \frac{N_{t_0}}{60}\; \text {group} \approx 1.7 \times N_{t_0} \; \text {hours} \end{aligned}$$

  where $N_t$ is the total number of enrolled students at time t. Proceeding similarly for practical sessions and considering that the size of the laboratory groups is of 20 students, and that the preparation of each hour of practical sessions takes 1h, we have that $L_{t_0}=2\times 3\times 15\times \frac{N_{t_0}}{20}$. There were two exams of 2 problems each. There were 2 turns of exams (morning and afternoon), all the students that have morning classes are examined with the same exam which is different from the exam of the afternoon students. So 4 problems should be prepared (2 per turn). Since each exam lasted for 2h and the students were distributed in laboratory rooms with 20 computers we have that $V_{t_0} = 2\times 2\times \frac{N_{t_0}}{20}$. We estimate that the preparation of each problem takes in average 3h of work (this include writing the statement, implementing the solution and designing the tests that the system requires to judge the submissions). Therefore, $E_{t_0} = 24$. Only the solutions of students that obtained a green verdict for a problem were graded by hand, and this was, approximately, a third of the students, so $G_{t_0} = 2\times 2\times 0.2\times \frac{N_{t_0}}{3}$ h, considering that grading one problem takes 12 min. The coordination of the whole course has two parts. A fixed cost, estimated as 1h per week giving 15h, that is $k_0=15$. Another part depending on the number of students of each course. We estimate this last amount in one half hour per group of 10 students, then $C_{t_0} = k_0 + 0.5\times \frac{N_{t_0}}{10}$. Finally, we are estimating that the software maintenance takes 4h a day yielding to $S_{t_0} = 4\times 5\times 15$ per course. As this period corresponds to a start-up, there is no redesign and therefore, $R_0 =0$.

• $t_1$ Introduction of Quizzes (2007–2008): In this period 4 small mid-term exams were introduced in addition to the two original exams and they were applied also in two turns. Considering that the time required to prepare each small exam was 1.5h and that the time required to grade the small exam of one student was 6 min, this measure increased $E_{t}$ and $G_t$ to $E_{t_1} = E_{t_0} + 4\times 2\times 1.5$ and $G_{t_1} = 4\times 0.3\times \frac{N_{t_1}}{3}$. The workload of all the other tasks remained the same. There is no redesign, $R_1=0$. We also take $k_1=k_0$

• $t_2$ Grading Red Verdicts (2008–2009): When all the submissions (and not only the green labelled ones) have to be graded $G_t$ was triplicated. $G_{t_2} = 4\times 0.3\times N_{t_2}$. We take $k_2=k_0$ and $R_t=0$.

• $t_3$ Hand-written Final Exam (2009–2010): At this point the final exam was changed to be a written exam of 3 problems. The exam was organized in only one turn applied to all the students (same exam for all students). The time to prepare a problem for a written exam is estimated in 2h (1h less than the time of a practical exam). Therefore $E_{t_3} = E_{t_1} -2\times 2\times 3 + 3\times 2$. The written exam lasts for 3h. Since rooms with place for 40 students where used for a written exam, $V_t$ decreased to $V_{t_3} = 2\times \frac{N_{t_3}}{20}+ 3\times \frac{N_{t_3}}{40}$. As before $k_3=k_0$ and $R_3=0$.

• $t_4$ New Degree (2010–2011): The fix cost due to bureaucratic duty in increases from $k_0$ in an extra half an hour per week plus 2h of exam coordination giving $k_4= k_0+ \frac{1}{2}\times 15 + 2= 24.5$. With the new degree the hours of theory lectures per week decrease from 3 to 2 per group yielding $T_{t_4} = 2.5\times 2\times 15\times \frac{N_{t_4}}{60}$. The course redesign $R_4$ increases. As the 15 sessions shrink in 1h, a main redesign of the contents is needed. The estimated cost is 3h per session giving $R_4=3\times 15= 45$. The evaluation system also changed. The big mid term exam disappeared. The four small mid term exams became formal exams of one problem each. Thus, $E_{t_4}= 4\times 3\times 2 + 2\times 3$. The first 3 mid term practical exams lasts 1.5h each while the last one for 2.5h. The final exam still lasts 3h, thus $V_{t_4}= (3\times 2\times 1.5 + 2\times 2.5)\times \frac{N_{t_4}}{20} + 3\times \frac{N_{t4}}{40}$.

• $t_5$ Lists of problems to hand-in (2011–2012): A list of problems per exam to be delivered by the students before the exam was introduced. The problems of the exams were chosen from the problems of the list. The preparation of each laboratory problem decreased to 1.5h. Hence $E_{t_5}= 4\times 2\times 1.5 + 3\times 2=18$. $C_{t_5} = k_4+ 0.5\frac{N_{t_5}}{10}$. Finally, the preparation of the lists set up $R_5$ to 10h, so that $R_5=10$.

• $t_6$ Re-evaluation Course (2012–2013): The $k_5$ increases by 2 coordination hours needed to manage of the re-evaluation, $k_6=k_5+2$. The $R_6$ is increased by 6h because 3 new large lists plus the corresponding sessions has to be generated, $R_t=16$. In this case $E_6= E_5+2\times 2+3=7$. Moreover we need to add 20h to $T_t$, 60h to $L_t$, 15h to $G_t$ and 3h to $V_t$.

• $t_7$ Mid-term Exams with New Exercises (2013–2014): This measure involved the creation of new problems for the mid-term practical exams, instead of taking them from the lists. This increased the time for preparing each problem from 1.5 to 3h. Thus $E_{t_5} = 4\times 2\times 3 + 3\times 2=30$.

• $t_8$ Course Diversification In this case $k_8=k_7$. $C_8$ is computed as in the previous step. Other cases are similar. The big difference is in the course redesign. First of all, there is a fix part cost due to list redesign. New 15 list has to be build, with a cost of half an hour per list. The lab training of the new list, depends on the number of groups $N_8/10$, on the number of list and is also 1 / 2h. Therefore

  $$R_8 = 1.5\frac{\text {hours}}{\text {list}}\times 15 \;\text {list}+ \frac{1}{2}\;\frac{\text {hours}}{\text {list}\times \text {grup}}\times 15\;\text {list}\times \frac{N_8}{10}\; \text {grup} = 358.5\;\text {h}$$

The quantification of this model was provided in Tables 3 and 4.