Skip to main content

Assessment

The same educational issues arise for Computer Science as for all other subjects, but the unique flavour of our subject does give them a different emphasis. This can perhaps be summed up by two simple facts.

The first is that computers can be used for almost anything. So, although Computer Science is largely a technical subject, it is also very broad, with an emphasis on a wide variety of non-technical skills as well as technical ones. All subjects take general transferable skills seriously, of course, but it goes beyond that for us. For example, students aiming for the film animation industry need graphic design and drama skills as much as they need direct computer skills. This leads to a wide variety of different types of activity and assessment.

The second fact is that the subject moves very quickly. Everyone knows that there has been a meteoric rise in computer technology in the last few decades, and there is little sign of it abating. So self-motivated life-long learning is not a desirable extra for Computer Science students, but is essential to their survival in their later careers. This causes tension in teaching between the desire to give maximum help to students in developing the right skills, and the desire to avoid spoon-feeding and help students develop self-reliance.

As well as colouring our courses and teaching, these two essential features of our subject also deeply affect our departmental ethos. We change things frequently and quickly, and we experiment a great deal. Of course, we are in a unique position to try out new computer-based ideas, and we are also interested in seeing to what extent these ideas can be exported to other groups in the university.

The Marking Scale

Every assessment is on the standard UK university 40-70 scale. It is not linear, with the low marks being easy to get, and higher marks being increasingly harder to get. Here is a rough explanation of the levels, and a guideline correspondence to European ECTS and American GPA grades. There is a more detailed official description of the meanings of marks.

Mark Description ECTS GPA
0 to 29 Fail F 0.0 to 1.4
30 to 39 Fail, but possibly compensatable FX 1.5 to 1.9
40 to 49 Third class (years 1-3) or Fail (4/MSc) E 2.0 to 2.4
50 to 59 Lower second class (1-4) or Pass (MSc) C or D 2.5 to 2.9
60 to 69 Upper second class (1-4) or Merit (MSc) B 3.0 to 3.4
70 to 79 First class (1-4) or Distinction (MSc) A 3.5 up
80 to 89 represents work above and beyond the call of duty
90 to 99 represents work of publishable quality
100 represents the absolute limit of human achievement

At the masters level (4th year of four year programmes, and MSc programmes) the pass mark is higher. That is not a different scale, just an expectation of better performance on the same scale. For example, there is no third class MEng degree - students with less than 50% are given a BSc based on their first three years instead.

There is a faculty wide guideline that the average for a unit should be between 50 and 65 (or 55 and 68 for M level units) unless there is good justification. Group work and major projects, for example, may justifiably have higher averages (on the grounds of a missing tail in the distribution). There is a lesser convention that each assessment should have an average roughly within the guidelines, unless it is clear that different assessments are going to balance each other out. This is strongly encouraged by our online marks display system, which displays the distribution of marks and points out discrepancies to a marker before the marks are released.

There is a tendency for coursework averages to be higher than exam averages. However, since we have some all-coursework and some all-exam units, for fair treatment of students (especially if the units are optional), the guideline applies to all units.

Of course, there is no requirement for a distribution to be a perfect bell curve, but we do look out for these problems:

graph showing narrow range of marks graph showing no dip at the top end graph showing two peaks

The first is a narrow curve, which may mean the marking criteria are too woolly so that everybody gets similar marks. The second is a top-heavy curve, which may indicate something which is too easy, so it doesn't stretch the top students. The third is a bimodal curve, which may indicate something which is too hard, so that weak students can't get anywhere with it.

The scale is not arbitrary. It is based on national averages and percentiles for university students, with 50% representing the mean (though we are an above average university, so our mean is above 50%). It is not norm-referenced (i.e. we don't scale marks so that a fixed percentage of students get firsts etc.) but is instead criterion-referenced (i.e. based on aims, objectives and teaching outcomes) and regulated by a system of external examiners to ensure national uniformity.

Types of Assessment

All of our units choose a method of assessment which is suited to the subject matter being taught. Many units are assessed 50% by coursework and 50% by exam, but some units have other weightings, including 100% coursework or 100% exam. This reflects a belief in variety, and that too much reliance on any one system is risky.

Exams are mostly conventional written ones, but they increasingly include lab exams and online exams. In lab exams, students do practical tasks like programming under supervised exam conditions. In online exams, students answer conventional questions, but type the answers into a computer rather than writing them by hand.

Some units, particularly in the third or fourth years, are project units. These may be individual projects or group projects, and they have their own individual style of assessment. Although they count as 100% coursework units, the final mark may not always be released until after examiners meetings are held.

Other coursework includes a wide variety of activities, including programming, designing computer systems, writing reports or CVs or business plans, graphic design, and making real or animated films.

Paper Exams

Most exams are held in May/June, in about week 6 or 7 of the Summer term; some are held in January. There are also resit exams in early September for first and second year units, which some students may be allowed to take if they fail at the first attempt.

Exams are held in a variety of places around the University, and some exams are split across two or more rooms, so it is essential to check your personal exam timetable to find out when and where each of your exams is being held.

In the exams themselves, make sure you know your exam number, and sit at the desk with that number on it. You need to switch off mobile phones etc., and leave such personal belongings at the front of the room, not by your desk. If you want to take a calculator, you need to get it approved. Your first priority is then to fill in the front of the answer booklet. The most important parts are your name in the top right corner and your exam number in the large box at the top left; these allow your exam paper to be marked anonymously. In some exams, you will be asked to fill in separate booklets for separate sections, so check the front of the question paper; this is to speed up marking.

Rough work should be done in the back of the answer booklet. Your answers to the parts of any one question should be kept together, so space your answers out, starting each question at least on a new page. (The cover of the booklet says to leave at least two lines between questions, but this is aimed at essay subjects rather than technical ones.) If there are blank pages left between answers, you may wish to cross them through at the end of the exam to ensure that markers do not miss later pages.

You will not normally be allowed to sit the exam if you are more than 30 minutes late, nor can you leave the exam in the first 30 minutes (this is for security reasons). Also, you cannot leave in the last 15 minutes (to avoid disturbing others). At the end of the exam, moisten and stick down the top right corner of your booklet and leave it on the desk; don't take it away with you by mistake.

If, for any reason, you are more than 30 minutes late for an exam, miss an exam completely, or have an exam clash, contact the exams officer in the department as soon as possible. The sooner you get in touch, the easier it is to do something about it.

In some circumstances, for example if you have been diagnosed as dyslexic, you may be allowed extra time, and your exams bay be in a different room. Exam information and old exam papers are available online at http://www.cs.bris.ac.uk/Teaching/Exams/

A few of your exams may be lab exams, taken in the department labs. You will be informed about any special arrangements for those.

Online Exams

A typical written exam in Computer Science lasts two hours, and has four questions, of which three must be answered, although there are other formats. A question typically has about half bookwork, i.e. testing of knowledge from lectures, and about half problem solving, i.e. testing skills or application of the knowledge.

With online exams, different structures may be suitable. For example, one structure which seems successful involves 50 short-answer questions, with no choice. The questions include both bookwork and problem solving, and get gradually harder, though they are all worth one mark each.

Feedback from students and markers is very positive. An online exam is less stressful, and there is less time pressure. Having so many small questions with no choice means that they cover the syllabus evenly. If a student revises half the course, they can answer half the questions, rather than luck determining whether they can answer everything or nothing. The inability to answer one question does not cause panic.

Having no choice also means that the questions do not have to be of equal difficulty. Weak students can concentrate on the early questions, while strong students are stretched by more demanding later questions. There have been experiments with conventional written exams in difficult technical topics where there is a compulsory section with smaller and easier questions, and a later section where students choose from harder and longer questions. The online scheme seems to be able to include the benefits of this arrangement, without the complications.

The short answers also prevent students from waffling, hoping to include the right buzz phrase by mentioning everything they know. This makes for more thinking and less typing.

It is natural to worry about whether the right things are being tested. It takes a little thought to break down a conventional essay or argumentation question into a series of small questions, without actually giving away the answer in the questions. However, in the subjects where online exams have been used so far, it seems that the online exam does test the right things, and is generally of better quality overall than a conventional exam, if only because it doesn't test handwriting, writing speed, or reaction to pressure.

There is a lot more to do in this area. We want to adapt the software to cope with more subjects by allowing long answers, answers involving line drawings, and answers involving mathematical symbols. It would also be interesting to investigate whether we can produce an authoring tool which reduces the technical knowledge needed to set and mark an online exam.

Projects

Projects are our larger coursework assessments, lasting for half a year or a whole year. A project usually forms a unit on its own, or sometimes two units. We used to deal with projects in a similar way to other technical departments. We became dissatisfied with this, and have evolved a different style, which we now promote around the university as an example of good practice.

The conventional approach taken in other departments and subjects, at its worst, goes like this. A small number of project topics are invented, and each student chooses one. A supervisor is allocated. The student carries out the project and writes a report on it. The supervisor and one other marker mark the report, and the two marks are averaged. Here's a summary of the principles of our approach, which make it different from this standard approach:

Here's a more detailed description of how it works.

Staff give short presentations, one from each research group in the department, describing the sort of things the groups, and individual staff, are interested in. This information is backed up with web pages, including some specific suggested topics. Students are encouraged to pick or invent a topic within the interests of one of the groups, or from their own interests. The students visit potential supervisors with their ideas, until they negotiate an agreement with one.

There is a marking panel involving a handful of staff members for each type of project. We have sufficient types that all staff are on one or two panels. Although the panels change from year to year, there is sufficient continuity to build up an expertise in that particular type of project.

The student writes a half-page proposal. Two members of the relevant marking panel are allocated as the main markers, not including the supervisor. For a group project, all the panel members are markers. The markers vet the proposal to make sure it seems viable, and give any preliminary advice.

Some sort of intermediate presentation and/or report is required from the students, and looked at by the panel. This varies depending on the type of project, and may or may not be marked. It is the formative feedback which is important.

At the end, each student gives a short presentation to the markers, sometimes in front of an A1 poster, and gives a short demonstration of the project itself, and hands in the final report, both online and printed. For group projects, the group also submits an agreed statement about what percentage of the work was done by each person.

Each person on the panel receives the reports for which they have been assigned as a marker. Each pile of reports is for one type of project only, making it easier to avoid confusion, and to compare and rank the submissions. Although each project type has a checklist of things to look for, there is no rigid marking scheme, because of the large variety of project topics with different emphases (e.g. self-contained program development, or working for a commercial client, or carrying out research experiments). Each marker independently produces a mark, and some measure of confidence in that mark (e.g. "plus or minus 5%"). The supervisor reports any unusual circumstances, and gives an estimated mark which is not included in the assessment.

There is then a panel meeting. If the two main markers agree to within one or two percent (which happens surprisingly frequently) and there is no serious discrepancy with the supervisor's estimate, or other uncertainty, there is no problem. If there is a disagreement (which can occasionally be surprisingly large) it has to be resolved. Averaging is banned, because it tends to produce an unnaturally narrow spread of marks. Instead, the panel tries to understand the reason for the disagreement. If there is no quick resolution, or any uneasiness remains, further panel members read the report and get involved, until agreement is reached.

The panel then ranks all the projects, making sure that the order they are in is the correct order. Hence, if two students get the same mark for their project then this is done by a definite decision of the panel and not by some statistical fluke. This means panel meetings can take a long time, but it means that the final result is robust and easily seen to be justified, to both students and outside evaluators such as external examiners.

Using a custom program, all the individual marks, agreed marks, and marks changes made during the meeting are recorded, along with a textual record of the proceedings. This is done during the meeting and projected onto the wall. It is used to look at the results for each panel member to see if there are discrepancies in their average marks or rankings, to check the ranking of the projects as a whole, to investigate the borderlines and firsts and fails, and so on. It is also used after the meeting to produce automatically the minutes, the student feedback, and the report for the external examiner.

Group Projects

Group projects are marked in a similar way to individual projects, the main difference being how your final mark is computed. You will be asked to decide as a group the contribution you have all made to a project. Ideally, all members of a group will agree their mark distribution and submit only one team report. If the distribution cannot be agreed let us know well in advance, so vivas can be arranged (in this case we'll also advise on what documentation is required, typically a report per student is necessary).

The marking panel will take both your group mark and your individual contribution to determine your individual mark. This mark is further moderated to fit it onto the Faculty scale. This reflects the fact that if, for a traditional piece of coursework you were to get a mark of 60%, a colleague who'd spent twice as long as you would not get a mark of 120%. All our marking is done using a non-linear scale.

Other factors will also be taken into account by the panel include the size of the group (with respect to others on the same unit) and problems with facilities or software availability. Problems such as health that have affected the group are dealt with by our Special Circumstances Committee and Examinations Board.

Coursework

Most units have coursework associated with them, which counts for a certain percentage of the unit mark, typically 30%, 50% or 100%. Each coursework assignment has a start date and a deadline, and is intended to be done during that period. The relative weight of each assignment in a unit normally matches the length of time allocated for it, e.g. a 2-week assignment has twice the weight of a 1-week assignment. As far as possible, no assignments are set outside of the unit's 12-week teaching block, and coursework deadlines are not altered once they have been set.

Almost all coursework is handed in using the department's online submission system, which automatically enforces deadlines. Deadlines are published on the web pages for the relevant unit, and summarised on year pages. The way in which the system handles late submission is changing. The old system was to have an explicit three-day extension at your own risk with no penalty. The new system, bringing the department in line with faculty and university policies, will have an implicit period at your own risk with no penalty, then a one day extension with a 10% penalty, then a further six day extension capped at the pass mark. Please take deadlines seriously, and be advised that when asking for a reference, future employers normally want to know about your time management skills. The only instrument we have to assess this is your ability to stick to deadlines.

We have to be very strict about deadlines, especially for small weekly exercises, because we want to provide some feedback such as a model answer to everybody straight afterwards. We can only afford to have very exceptional cases of students continuing to work on an assignment after the model answer has been published. Individual exceptions are only accepted where there are serious problems.

So please remember that it is better to hand in whatever you have done early, even if you intend to improve it later. If you do nothing until too close to the deadline, and then you have a cold or the computer system goes down, then that is your problem. Also, all deadlines are at midnight, and you should avoid submitting at the last moment. This is because (a) a large submission may take a little while to process and the time of submission is taken to be at the end of processing, (b) other people are likely to be doing the same so submission may be slow and (c) your watch and the central system clock may not agree exactly.

Making Backups

If you do your coursework on your own computer, it is up to you to make backups. Even with the high level of reliability of modern computers, a hard disk will occasionally crash, or a memory stick will become unreadable, and everything will be lost. So, it is absolutely vital that your work should not be stored in just one place - you need to copy it frequently to some other place. Excuses for failing to hand in coursework such as "my hard disk crashed and I lost my program" are not accepted. Computer scientists are expected to know better.

The department servers are backed up regularly. So a good strategy is to copy your work (via the Internet or a memory stick or whatever) from your own computer onto your department account. The department will then look after your work.

If you work in the computer labs, then you have less to worry about, because you are working directly on a system which is backed up regularly. However, you still need to guard against accidentally deleting or over-writing your own work, so you may still want to make an occasional snapshot copy of your work in a separate directory.

Use of Time

In our department, we don't provide as rigid a timetable for students as in some other technical subjects. For most coursework, students do it 'in their own time'. In a few cases, we provide a non-compulsory timetabled lab session with expert supervision.

However, there is a notional amount of time allocated for coursework and other study, namely 5 hours per week for a 10 credit unit, or 10 hours per week for a 20 credit unit. This is based on a 40-hour working week, subtracting the fixed hours for lectures etc., and on the fact that 60 credits are typically taken at any one time.

Each assignment has a start date as well as a deadline, and thus has a notional number of hours attached to it, according to how long it is supposed to last. The weight of an assignment (relative to the total coursework weight for the unit) is also based on how long it lasts so, for example, a two-week assignment normally has twice the weight of a one-week assignment. There is no explicit difference between units with a low overall coursework weighting and a high one. The same allowance (5 or 10 hours per week) is made for coursework and other study, though notionally you might expect the other study to take a greater fraction of the time for units with a low coursework weighting, in order to be well prepared for the high-weighted exam.

If we get the workload for an assignment wrong, we don't regard it as a big problem, but as more-or-less self-correcting. The situation is the same for everyone on the unit, we can adjust the marking a little to take account of it, and note the need for change the following year. In any case, many assignments are deliberately quite open ended to allow students to show their initiative, and in that case you are expected simply to do whatever you can within the time available.

Deadlines inevitably clash sometimes, especially at the end of a semester. In principle, deadline clashes are of no importance, because if you work steadily, and don't leave everything to the last minute, the work can always be done within the notional time allowance. On the other hand, we realise that deadline clashes are unpopular, and we try to coordinate them via year tutors where we can. In general, we have found that moving a deadline causes as many complaints (e.g. from those who finished early) as leaving it alone. It is better just to accept that everyone is in the same boat.

Mitigating Circumstances

If you become seriously ill or have any other personal problem which prevents you from meeting deadlines you should contact your personal tutor as soon as possible.

Your tutor will record the circumstances in the personal filestore on your web page. This will document the problem and allow us to take action if necessary. The information will be treated in strict confidence and will be accessible only to you and to staff who need it to make decisions on, for example, deadline extensions or progression to the next year. If you wish, you can instead put details of the case in a sealed envelope, marked "to be opened only in case of XXX", where XXX might for example be issues of progression to the next year or borderline degree classification. This should be handed in to the Department office.

For problems which affect a few coursework assignments (e.g. problems lasting a couple of weeks) the year tutor may set give you an individual extension or waive coursework. The year tutor is the only person who can do this. If coursework is waived you must check that a `W' appears in place of the coursework mark.

Problems which affect an entire unit or year must also be recorded, but will be dealt with at the end of the year by a special committee.

With few exceptions you will have to produce appropriate evidence for the problem. In most cases this will be a note from your doctor stating the extent to which the problem is likely to affect or has affected your ability to study. If it is impossible to produce documentary evidence, you need to submit a written personal statement. If you miss an exam for medical reasons you need to obtain a note from a doctor the same day or as soon as possible and pass this to the exams officer or department administrator.

You should check that the details regarding your mitigating circumstances in your personal filestore are correct. It is also important that any changes in your circumstances are promptly and accurately recorded in the filestore, including how and when matters are resolved. It is your responsibility to ensure that your personal filestore is kept up to date - incomplete information on mitigating circumstances, particularly the duration of the problems, may mean that we are not able to make allowances.

On the Faculty web site, there is a self-certification form for documenting short illness problems without needing a doctor's note. This is used primarily to explain absences from fixed lab sessions etc, and so is less relevant in the Computer Science department. A self-certification form will eventually be copied to your department personal filestore, but this is likely to be less effective than approaching your tutor directly.

Automation

We are interested in getting computers to help with assessment, as you might expect. A small selection of the software we have developed is listed here.

Our general approach to computer-based assessment is based on two ideas. The first is that the greatest gains come from getting the computer to do simple but boring and repetitive stuff, and leaving cleverer tasks to humans. The second is that commercial or centralised or commissioned software is too rigid. Instead, we use software which we write ourselves, and evolve to suit our needs. This is done by staff or project students or students on short-term, low-cost contracts. Here are some of the approaches that we do and don't use:

Blackboard Blackboard is the university's "standard" package for course materials. In the past, it has hardly been used at all in Computer Science. Criticisms from students include "you can only submit once, and not repeatedly", "there is no flexibility on the deadline" and "you can't download your work again afterwards". Criticisms from staff include "you have to provide lists of students, instead of the software knowing which students are on which units", "you can't see the student view of the materials you put up", and "you are forced to do things one way, instead of the way you would like". However, recent improvements may make us take another look.

Online Submission: There is a faculty web site where we have our own secure document store for students to submit work. This allows us to be flexible about how we handle deadlines and various other issues. It is described in more detail later.

Online marks display: The same faculty system stores marks for each assignment and displays them. The marks can be entered by hand or via one of the marking programs described below. One of the advantages of this is that markers are forced to review the average and distribution of the marks for an assignment and consider scaling them before publishing them. This helps to keep everything to the standard marking scale, to keep coursework marks and exam marks in line with each other, and to avoid later scaling issues. Students can see their own marks, and they see all the other marks anonymously, so they can assess where they stand in the class.

Online exams: The idea is that students take the exam in a computer lab, under exam conditions, and type in their answers online. Up to now, we have exclusively used short-answer questions where the answer must fit on one line. The marking is done by hand by ticking boxes, but because the answers are short and the computer brings all the answers to one question together, an exam of 100 students can be marked in 2 or 3 hours instead of 2 or 3 days. Of course, we are rather fortunate in having a large computer lab to do these exams in. As well as end of year exams, the software can also be used for progress tests.

We don't use multiple choice questions much, because although they make marking fully automatic, it usually takes too much effort preparing them to make sure they effectively test the right things. If this increases preparation time by more than the few hours that could be saved, it definitely isn't worth it. Other approaches to online tests, such as building up a big database of questions and giving randomly chosen ones to each student, are interesting, but don't fit our style of using minimum effort systems with maximum benefit.

Online tutorials:Generally, developing interactive tutorial material has a high cost, so we do very little of it. However, there are one or two subject areas such as beginner's programming which benefit from it. Each tutorial is developed as an independent programming project. We haven't (yet) tried to build authoring tools to allow non-programmers to author tutorials.

Online Forum: We use a home-grown forum system in the form of a series of bulletin boards on the Web, with (roughly) one forum for each unit. It is driven by our database of unit registrations. People can be notified of postings by e-mail or RSS, if they wish.

Tick box feedback: Imagine that you are marking any kind of assignment, let's say a pile of essays. One program we have developed displays a list of one-line comments, each with a tick box by it. For each essay, the marker ticks the boxes of the comments that apply. At any stage, they can add more comments or change the text of existing ones to make them more generally applicable. When they have finished, the program generates feedback for each student in the form of the comments that they ticked. The difference between this and a pro-forma marking sheet is that the comments are specific to each assignment, and they evolve as one does the marking.

Tick box marking: The same program has been extended to aid marking. There is a box where a marker can enter a manual mark for each student. Also, each comment can optionally have a mark associated with it, which is added to the manual mark if that comment is ticked. This can be a positive mark for something achieved, or a negative mark for a mistake. A group of comments can act like a menu, where the marker can choose just one out of the group to generate a mark for one aspect of the work. For some assignments, one can generate the entire mark from the comments, and not use the manual box at all.

Auto-marking: We also have an automatic marking program where the idea is to run a battery of tests over each student's work, and award a mark based on which tests are passed. This is useful for tightly specified technical assignments, and for some minor aspects of report writing such as checking for spelling errors or for exceeding a maximum word count. This program is somewhat difficult to use because both the assignment and the tests have to be designed very carefully. However, it has proven invaluable in coping with first year weekly programming exercises for large classes where the marking load would otherwise be completely impossible.

Building Documents: We have a mechanism for using data from our database, or from a single, original, authoritative, editable version of a document, and translating it into other formats automatically. For example, we can produce web pages or transcripts, or reproduce many University and Faculty forms in a printable (pdf) format.

Faculty Spreadsheets: The Engineering Faculty uses custom spreadsheet software for recording unit mark details. We don't use this software during the year, because our web marks display system does the same job. However, we do automatically convert unit details into the faculty format for final processing. The faculty also has spreadsheet software for presenting end-of-year results for each stage of each programme. This software has almost all the faculty rules and conventions built into it, and is maintained from year to year as the conventions change.

The Online Submission System

We have been using online submission for a long time, starting with e-mail (in the mid-1980s, before the public had heard of it) and now (since 2000) using a custom submission system via our web site into a document store.

As soon as web-based submission was introduced experimentally, staff and students alike clamoured to make it compulsory for all coursework submission. It is now the backbone of our coursework handling processes.

Immediate advantages are that coursework doesn't get lost, and there is no argument about when coursework was handed in (because of the timestamps on the files). Students can submit from home, and staff can do marking from home, using password access to the web site. Deadlines are handled automatically, and are always at midnight. Checks for cheating can be automated. External examiners have a single place of reference for looking at student coursework.

Perhaps the most important aspect of the submission system is the psychological effect on deadlines. The automated handling helps to make deadlines absolute.

The widely publicised and emphasised rule is that the extension is meant to cover minor problems such as colds at the student end, and minor problems such as service interruptions at the department end. No guarantees are made about services or support during the extension, which is often over a weekend, and students use the extension entirely at their own risk. Students are strongly urged to submit what they have early, and then re-submit something better later if they wish.

This allows us to make deadlines absolute. The idea is to make the number of exceptions truly small, allowing us to publish some feedback such as model answers immediately after the deadline. Exceptions must be serious and must be supported by written documentary evidence such as a medical certificate. They can be dealt with by a further extension or by waiving that assignment. Importantly, though, one exception to the rule about documentary evidence is where a student has lost time management or motivation to the point that they are continually missing deadlines. We would rather do a deal with them and design a catching up strategy than have to deal with a later unit failure.

Assessing Skills

The story of teaching the fundamental skill of programming is an interesting one with a happy ending. Many other technical and non-technical skills have similar stories.

This is a skill which we have, of course, always grappled with. We used to regard it as something which we partly teach, and which students partly pick up for themselves during their time with us. During one course redesign, we lamented the fact that a few students still reached their final year individual project without sufficient competence or confidence in this skill. As a result, we made it a learning outcome of the first year of our programmes that all students should be competent in this skill by the end of the year.

We identified a couple of potential stumbling blocks. One is students not getting enough practice, and the other is a tendency for students to think they can program when they can't. We needed a combination of techniques.

First, we switched to having many small exercises, one per week. There is a current fashion for accusing educators of over-assessing, and we feel it is something that we need to watch out for, but in this case we felt that increasing the assessments was essential. The marking load for this was initially extremely high, until we later introduced automatic marking.

Second, we put a lot of effort into making the exercises varied and interesting. Some are tightly specified (play a game of mastermind), some are open ended (draw a scene for possible inclusion in an art gallery), and some are just fun (write a tank program to battle with other students' tanks).

Third, we carefully reduced the amount of support the students get from postgraduate lab supervisors while doing the coursework, to improve independence. Students end up having to go from the lab to a different floor in our building to get help, which makes them think twice.

Fourth, we instituted lab exams, where students have to write a program on their own under exam conditions. This deals very effectively with the problem of students thinking they can program when they can't. However, it leads to many marks which are zero or very low, so it has to be treated like a driving test. Students can have several goes at it during the year, replacing previous poor marks.

Setting Assessments

The university advice on assessment says that assessments shouldn't be set by just one person. We don't follow this explicitly, but what we do that has the same effect is to make sure that many of our units have two lecturers. This encourages cooperation and discussion, and provides continuity when one of the lecturers moves on.

It almost goes without saying that the description of an assignment should include the expected outcomes, and the marking criteria or a marking scheme, and that the marking and feedback should be based on these. However, how you set an assignment can also have a great effect on student perceptions and how easy it is to mark.

For example, for a technical exercise, a good plan is to describe what is required in stages. These can increase in difficulty or scope or sophistication or open endedness. That way, weak students have achievable targets and can produce something they are proud of, rather than just doing something ambitious badly.

Feedback

Feedback on coursework is vital. No matter how good the quality of the feedback produced, it is almost useless if it is given to students after they have lost interest in it. It is also almost useless if a lot of effort is put into producing detailed individual feedback which most students then ignore. Minimum feedback given quickly is far more effective.

So here is what we promise. Each assignment will either be marked before the deadline for the next assignment (or within four weeks for the last assignment), or students will be kept informed about the marking progress. For every assignment, there will be at least a minimum of feedback, either online or in a subsequent lecture. The way in which feedback is to be given will be included with the assignment description. In addition to the mark, there will be, for example, a model answer (written by the setter or marker) or sample answers (anonymised from the submitted work of this year's or last year's students) accompanied by a brief explanation. If necessary, this feedback will be given before marking is complete.

In general, we do not promise to produce explicit detailed individual feedback as a matter of course, simply because it is too time-consuming. Instead, we rely on students to seek further feedback when they need it. As well as explicit feedback, students can obtain a wide variety of implicit feedback via face-to-face contact with staff (who have an 'open door' policy), lab supervisors, the help desk, fellow students in labs, emails, progress pages and forums etc. on the department web site, and so on.

We don't use double marking, except on projects as described later, so there is a fairness issue. The main way we deal with this is to ensure that only one person marks each assignment. If several markers are needed, they rotate, taking one assignment each. If two markers are really needed for one assignment, we still arrange for one marker to mark all students for one part of the assessment. Also, marking is done by username, which provides reasonable "voluntary anonymity", i.e. markers don't normally know who each student is unless they make some explicit effort to find out.

Some people in the university bemoan the fact that students seem obsessed by marks. We tend to take the opposite view - that we can use marked assessment to mould student behaviour. Unmarked work or assessment is unpopular with students, and experiments we have done with it via personal tutor groups have had patchy results, at best. So, on the whole, everything important that students do is assessed with a mark. We try to make sure that the importance we place on a particular kind of work is reflected in the nominal time that students spend on it and consequently on the weight given to the marks for it.

The result is that we rely almost exclusively on straightforward weighted averages of marks for overall assessment. There are two main exceptions to this. First, faculty conventions and accreditation bodies place greater weight on final year individual projects than other units, so projects have to be passed to get a degree. Second, a student's final degree grade is based on a weighted average of their year marks, and these exclude the first year. Our various 3 and 4 year undergraduate degrees, some with a year abroad, are weighted 0:20:80 or 0:30:70 or 0:10:20:70 or 0:10:30:60 or 0:10:40:50.

The important things seem to be to keep the marking load as low as possible, to make sure marking is as painless as possible, and to avoid over-marking (e.g. if an assignment is worth 5% of a unit, mark it out of 5 or 10, not out of 100).

For first year weekly programming exercises, and for a few other technical assignments, automatic marking is essential. There can be problems, for example, a student can get zero by making one small formatting mistake which makes all the tests fail. To help with this, we try to provide some tests in advance, similar to the first few we are actually going to use. That also makes the first few marks easy to get, which is good because it turns out that students often find it harder to get high marks when the marking is fully automatic.

With automatic marking, there is also an obvious question about whether we are testing the right things. In the case of basic programming, the idea of precision, and the idea that programs which don't work are worth nothing, are valuable lessons. What is more, the tests don't just deal with what the programs do as black boxes. They also look inside at some elementary aspects of programming style, such as consistent layout. Some aspects of automatic marking are still a bit crude, but results are generally better than trying to mark large quantities of work by hand at high speed.

The quality of the feedback is somewhat mediocre with automatic marking. The program produces a report which tells the student which tests failed, and the student has to work out from this, and a general marking report, what mistakes they made or which technical issues they misunderstood. However, the automatic feedback is produced so quickly that staff have time to offer further individual feedback, on request, by meeting with students.

Assignments in later years are not so suitable for automatic treatment (although sometimes some aspects can be automated). For these, the most widely applicable and effective technique to speed up marking seems to be self marking. This is used in a variety of different ways.

If an assignment is staged, each student can be asked for a brief statement about which stages were achieved. If an assignment is open ended, a bullet point statement of what achievements students feel they are proud of has a similar effect. The marker only has to check these achievements, not deduce them with a lot of detective work.

For some assignments, we give students the detailed marking scheme and ask them to estimate their own mark. We then check this swiftly, with a penalty if their estimate is out by much in either direction. In one experiment of this kind, the final checking was done by interview, on the basis that a ten minute interview per student is less painful, more accurate, and provides better interactive feedback, than ten minutes of conventional marking per student. The experiment was successful, except for the logistical problems of getting a lot of students to appear at 10 minute intervals without fail.

Feedback on Exams

Here's how we deal with feedback on exam scripts, to try to reduce the potential effort, while acknowledging that students have a right of challenge. No comments are written on the scripts themselves, and no feedback is normally given. If a student challenges a result, the mechanical checks are repeated, i.e. whether the signature on the exam script is right, whether everything in the script was marked, and whether the marks were added up and transcribed correctly.

We run some exams in January. This is for units in the first teaching block taken by fourth year students. The faculty permits us to follow an early release process, even though the results are not official until the summer examiners meetings, as follows. There is a preliminary internal examiners meeting after the January exams to finalise unit marks and deal with any preliminary scaling issues. Then, we release the marks, rounded down to grades, i.e. 70+ becomes A, 60-69 becomes B, and so on. These grades are flagged as provisional, subject to moderation in the summer (though in practice we try to make sure no changes are made).

Conclusions

Here are a few points which have emerged during this chapter: