<< 2011-2 >>
Department of
Computer Science
 

Assessment

The same educational issues arise for Computer Science as for all other subjects, but the unique flavour of our subject does give them a different emphasis. This can perhaps be summed up by two simple facts.

The first is that computers can be used for almost anything. So, although Computer Science is largely a technical subject, it is also very broad, with an emphasis on a wide variety of non-technical skills as well as technical ones. All subjects take general transferable skills seriously, of course, but it goes beyond that for us. For example, students aiming for the film animation industry need graphic design and drama skills as much as they need direct computer skills. This leads to a wide variety of different types of activity and assessment.

The second fact is that the subject moves very quickly. Everyone knows that there has been a meteoric rise in computer technology in the last few decades, and there is little sign of it abating. So self-motivated life-long learning is not a desirable extra for Computer Science students, but is essential to their survival in their later careers. This causes tension in teaching between the desire to give maximum help to students in developing the right skills, and the desire to avoid spoon-feeding and help students develop self-reliance.

As well as colouring our courses and teaching, these two essential features of our subject also deeply affect our departmental ethos. We change things frequently and quickly, and we experiment a great deal. Of course, we are in a unique position to try out new computer-based ideas, and we are also interested in seeing to what extent these ideas can be exported to other groups in the university.

Types of Assessment

All of our units choose a method of assessment which is suited to the subject matter being taught. Some are 100% coursework, some 50% coursework and 50% exam, whilst some are 100% exam. This reflects a belief in variety, and that too much reliance on any one system is risky. Some points in favour of exams are:

On the other hand, some points against exams are:

Coursework includes a wide variety of activities, including programming, designing computer systems, writing reports or CVs or business plans, graphic design, making real or animated films, and doing individual or group projects.

Exams are mostly conventional written ones, but they increasingly include lab exams and online exams. In lab exams, students do practical tasks like programming under supervised exam conditions. In online exams, students answer conventional questions, but type the answers into a computer rather than writing them by hand.

Automation

We are interested in getting computers to help with assessment, as you might expect. A small selection of the software we have developed is listed here.

Our general approach to computer-based assessment is based on two ideas. The first is that the greatest gains come from getting the computer to do simple but boring and repetitive stuff, and leaving cleverer tasks to humans. The second is that commercial or centralised or commissioned software is too rigid. Instead, we use software which we write ourselves, and evolve to suit our needs. This is done by staff or project students or students on short-term, low-cost contracts. Here are some of the approaches that we do and don't take.

Blackboard etc. Packages like Blackboard and WEB-CT, used elsewhere in the university, try to do too much, and end up not doing what you want. It seems that "expensive" implies "inflexible". Criticisms from students include "you can only submit once, and not repeatedly", "there is no flexibility on the deadline" and "you can't download your work again afterwards". Criticisms from staff include "you have to provide lists of students, instead of the software knowing which students are on which units", "you can't see the student view of the materials you put up", and "you are forced to do things one way, instead of the way you would like". As a result, hardly anybody in the department uses these.

Online Submission: We run our own web site with our own secure document store for students to submit work. This allows us to be flexible about how we handle deadlines and various other issues. It is described in more detail later.

Online marks display: We have a system for storing marks for each assignment and displaying them on our web site. The marks can be entered by hand or via one of the marking programs described below. One of the advantages of this is that markers are forced to review the average and distribution of the marks for an assignment and consider scaling them before publishing them. This helps to keep everything to the standard marking scale, to keep coursework marks and exam marks in line with each other, and to avoid later scaling issues. Students can see their own marks, and they see all the other marks anonymously, so they can assess where they stand in the class.

Online exams: The idea is that students take the exam in a computer lab, under exam conditions, and type in their answers online. Up to now, we have exclusively used short-answer questions where the answer must fit on one line (100 characters). The marking is done by hand by ticking boxes, but because the answers are short and the computer brings all the answers to one question together, an exam of 100 students can be marked in 2 or 3 hours instead of 2 or 3 days. Of course, we are rather fortunate in having a large computer lab to do these exams in. As well as end of year exams, the software can also be used for progress tests.

We don't use multiple choice questions, because although they make marking fully automatic, it usually takes too much effort preparing them to make sure they effectively test the right things. If this increases preparation time by more than the 2 or 3 hours that could be saved, it definitely isn't worth it. Other approaches to online tests, such as building up a big database of questions and giving randomly chosen ones to each student, are interesting, but don't fit our style of writing simple programs with maximum benefit.

Online tutorials: Generally, developing interactive tutorial material has a high cost, so we do very little of it. However, there are one or two subject areas such as beginner's programming which benefit from it. Each tutorial is developed as an independent programming project. We haven't (yet) tried to build authoring tools to allow non-programmers to author tutorials.

Online Forum: We use a home-grown forum system in the form of a series of bulletin boards on the Web, with (roughly) one forum for each unit. It is driven by our database of unit registrations. People can be notified of postings by email, if they wish.

Tick box feedback: Imagine that you are marking any kind of assignment, let's say a pile of essays. One program we have developed displays a list of one-line comments, each with a tick box by it. For each essay, the marker ticks the boxes of the comments that apply. At any stage, they can add more comments or change the text of existing ones to make them more generally applicable. When they have finished, the program generates feedback for each student in the form of the comments that they ticked. The difference between this and a pro-forma marking sheet is that the comments are specific to each assignment, and they evolve as one does the marking.

Tick box marking: The same program has been extended to aid marking. There is a box where a marker can enter a manual mark for each student. Also, each comment can optionally have a mark associated with it, which is added to the manual mark if that comment is ticked. This can be a positive mark for something achieved, or a negative mark for a mistake. A group of comments can act like a menu, where the marker can choose just one out of the group to generate a mark for one aspect of the work. For some assignments, one can generate the entire mark from the comments, and not use the manual box at all.

Auto-marking: We also have an automatic marking program where the idea is to run a battery of tests over each student's work, and award a mark based on which tests are passed. This is useful for tightly specified technical assignments, and for some minor aspects of report writing such as checking for spelling errors or for exceeding a maximum word count. This program is somewhat difficult to use because both the assignment and the tests have to be designed very carefully. However, it has proven invaluable in coping with first year weekly programming exercises for large classes where the marking load would otherwise be completely impossible.

Building Documents: We have a mechanism for using data from our database, or from a single, original, authoritative, editable version of a document, and translating it into other formats automatically. For example, we can produce web pages or transcripts, or reproduce many University and Faculty forms in a printable (pdf) format. Each construction of a new form currently requires some programming, though.

Faculty Spreadsheets: The Engineering Faculty uses custom spreadsheet software for recording unit mark details. We don't use this software during the year, because our web marks display system does the same job. However, we do automatically convert unit details into the faculty format for final processing. The faculty also has spreadsheet software for presenting end-of-year results for each stage of each programme. This software has almost all the faculty rules and conventions built into it, and is maintained from year to year as the conventions change.

The Marking Scale

Every assessment is on the standard UK university 40-70 scale. It is not linear, with the low marks being easy to get, and higher marks being increasingly harder to get. Here is a rough explanation of the levels:

0 to 39 Fail
40 to 49 Third class (BSc/MEng1-3) or Fail (MEng4/MSc)
50 to 59 Lower second class (BSc/MEng) or Pass (MSc)
60 to 69 Upper second class (BSc) or Pass (MSc)
70 to 79 First class (BSc/MEng) or Distinction (MSc)
80 to 89 represents work above and beyond the call of duty
90 to 99 represents work of publishable quality
100 represents the absolute limit of human achievement

At the masters level (4th year of four year programmes, and MSc programmes) the pass mark is higher. That is not a different scale, just an expectation of better performance on the same scale.

There is a faculty wide guideline that the average for a unit should be between 50 and 65 (or 55 and 68 for M level units) and outside that range, marks should be scaled or justified. There is, however, a general consensus that group work and major projects may justifiably have higher averages (on the grounds of a missing tail to the distribution).

There is a tendency for coursework averages to be higher than exam averages. However, since we have some all-coursework and some all-exam units, for fair treatment of students (especially if the units are optional), the guideline applies to all units.

To avoid unpleasant scaling issues later on, we have a further department convention that each assessment should have an average roughly within the guidelines. This is strongly encouraged by our online marks display system, which displays the distribution of marks and points out discrepancies to a marker before the marker is allowed to release the marks.

Of course, there is no requirement for a distribution to be a perfect bell curve, but we do look out for these problems:

graph showing narrow range of marks graph showing no dip at the top end graph showing two peaks

The first is a narrow curve, which may mean the marking criteria are too woolly. The second is a top-heavy curve, which may indicate something which is too easy, so it doesn't stretch the top students. The third is a bimodal curve, which may indicate something which is too hard, so that weak students can't get anywhere with it.

By the way, the scale is not arbitrary. Suppose, for each assessment, you imagine ranking all the students, and then assigning a percentage according to how many others in the class have lower marks. The top student gets 100% and the bottom student gets 0%. Now imagine averaging these percentages over a large number of students, and a large number of assignments for each student. The result is almost exactly the scale discussed here, with the lower end of the scale being easy to attain, the top end difficult, 40% being passable, 70% being excellent, and with a mean of 50%. The only difference is that the scale we use is based on all the universities in this country, and we are an above average university, so we allow the mean to be above 50%.

Bureaucracy

Nobody likes unnecessary paperwork or external audits. We have evolved a three-part plan for dealing with this. First, we do the "Right Thing", i.e. make sure that the real quality of our activities is high. Second, we record what we do continually, with as much automation as possible. Third, when QA audits or subject reviews or accreditation visits or research assessment exercises or whatever come along, we provide copies of our handbook, and secure access to our web site, but otherwise try to refuse to duplicate this available information in any other form.

Another example is filling in forms with aims, objectives, learning outcomes and so on. This often feels as if it is no more than putting together platitudes to satisfy other people that we are doing our jobs properly. However, there are two rays of sunshine.

One is that there is not actually much effort involved in writing, say, a unit description and then maintaining it. Most of the irritation comes from unnecessary copying. So, our preference is for a single, authoritative, electronic version of the information, with automated ways of translating it into other formats for other people.

The other is that sometimes, interpreting terminology in our own way and being sufficiently blunt when filling these forms in can sometimes trigger important improvements. This happened the last time we radically redesigned our programmes, as described next.

Assessing Skills

The story of teaching the fundamental skill of programming is an interesting one with a happy ending. Many other technical and non-technical skills have similar stories.

This is a skill which we have, of course, always grappled with. We used to regard it as something which we partly teach, and which students partly pick up for themselves during their time with us. During one course redesign, we lamented the fact that a few students still reached their final year individual project without sufficient competence or confidence in this skill. As a result, we made it a learning outcome of the first year of our programmes that all students should be competent in this skill by the end of the year.

We identified a couple of potential stumbling blocks. One is students not getting enough practice, and the other is a tendency for students to think they can program when they can't. We needed a combination of techniques.

First, we switched to having many small exercises, one per week. There is a current fashion for accusing educators of over-assessing, and we feel it is something that we need to watch out for, but in this case we felt that increasing the assessments was essential. The marking load for this was initially extremely high, until we later introduced automatic marking.

Second, we put a lot of effort into making the exercises varied and interesting. Some are tightly specified (play a game of mastermind), some are open ended (draw a scene for possible inclusion in an art gallery), and some are just fun (write a tank program to battle with other students' tanks).

Third, we carefully reduced the amount of support the students get from postgraduate lab supervisors while doing the coursework, to improve independence. Students end up having to go from the lab to a different floor in our building to get help, which makes them think twice.

Fourth, we instituted lab exams, where students have to write a program on their own under exam conditions. This deals very effectively with the problem of students thinking they can program when they can't. However, it leads to many marks which are zero or very low, so it has to be treated like a driving test. Students can have several goes at it during the year, replacing previous poor marks.

Paper Exams

Most exams are held in May/June, in about week 6 or 7 of the Summer term; some are held in January. There are also resit exams in early September which some students may be allowed to take if they fail at the first attempt.

Exams are held in a variety of places around the University, and some exams are split across two or more rooms, so it is essential to check your personal exam timetable to find out when and where each of your exams is being held.

In the exams themselves, make sure you know your exam number, and sit at the desk with that number on it. You need to switch off mobile phones etc., and leave such personal belongings at the front of the room, not by your desk. If want to take a calculator, you need to get it approved. Your first priority is then to fill in the front of the answer booklet. The most important parts are your name in the top right corner and your exam number in the large box at the top left; these allow your exam paper to be marked anonymously. In some exams, you will be asked to fill in separate booklets for separate sections, so check the front of the question paper; this is to speed up marking.

Rough work should be done in the back of the answer booklet. Your answers to the parts of any one question should be kept together, so space your answers out, starting each question at least on a new page. (The cover of the booklet says to leave at least two lines between questions, but this is aimed at essay subjects rather than technical ones.) If there are blank pages left between answers, you may wish to cross them through at the end of the exam to ensure that markers do not miss later pages.

You will not normally be allowed to sit the exam if you are more than 30 minutes late, nor can you leave the exam in the first 30 minutes (this is for security reasons). Also, you cannot leave in the last 15 minutes (to avoid disturbing others). At the end of the exam, moisten and stick down the top right corner of your booklet and leave it on the desk; don't take it away with you by mistake.

If, for any reason, you are more than 30 minutes late for an exam, miss an exam completely, or have an exam clash, contact the exams officer in the department as soon as possible. The sooner you get in touch, the easier it is to do something about it.

In some circumstances, for example if you have been diagnosed as dyslexic, you may be allowed extra time, and your exams bay be in a different room. Exam information and old exam papers are available online at http://www.cs.bris.ac.uk/Teaching/Exams/

A few of your exams may be lab exams, taken in the department labs. You will be informed about any special arrangements for those.

Online Exams

The original intention was to duplicate conventional written exams, but a different structure has emerged.

A conventional exam in Computer Science lasts two hours, and has four questions, of which three must be answered, although there are other formats. A question typically has about half bookwork, i.e. testing of knowledge from lectures, and about half problem solving, i.e. testing skills or application of the knowledge.

The structure which has emerged from experiments so far is one with 50 short-answer questions, with no choice. These include both bookwork and problem solving, and get gradually harder, though they are all worth one mark each.

Feedback from students and markers is very positive. An online exam is less stressful, and there is less time pressure. Having so many small questions with no choice means that they cover the syllabus evenly. If a student revises half the course, they can answer half the questions, rather than luck determining whether they can answer everything or nothing. The inability to answer one question does not cause panic.

Having no choice also means that the questions do not have to be of equal difficulty. Weak students can concentrate on the early questions, while strong students are stretched by more demanding later questions. There have been experiments with conventional written exams in difficult technical topics where there is a compulsory section with smaller and easier questions, and a later section where students choose from harder and longer questions. The online scheme seems to include the benefits of this arrangement, without the complications.

The short answers also prevent students from waffling, hoping to include the right buzz phrase by mentioning everything they know. This makes for more thinking and less typing.

It is natural to worry about whether the right things are being tested. It takes a little thought to break down a conventional essay or argumentation question into a series of small questions, without actually giving away the answer in the questions. However, in the few subjects where experiments have been done so far, it seems that the online exam does test the right things, and is generally of better quality overall than a conventional exam, if only because it doesn't test handwriting, writing speed, or reaction to pressure. The distribution of marks produced seems fine.

There is a lot more to do in this area. We want to adapt the software to cope with more subjects by allowing long answers, answers involving line drawings, and answers involving mathematical symbols. It would also be interesting to investigate whether we can produce an authoring tool which reduces the technical knowledge needed to set and mark an online exam.

Setting Assessments

The university advice on assessment says that assessments shouldn't be set by just one person. We don't follow this explicitly, but what we do that has the same effect is to make sure that most of our units have two lecturers. This encourages cooperation and discussion, and provides continuity when one of the lecturers moves on.

It almost goes without saying that the description of an assignment should include the expected outcomes, and the marking criteria or a marking scheme, and that the marking and feedback should be based on these. However, how you set an assignment can also have a great effect on student perceptions and how easy it is to mark.

For example, for a technical exercise, a good plan is to describe what is required in stages. These should increase in difficulty or scope or sophistication or open endedness. That way, weak students have achievable targets and can produce something they are proud of, rather than just doing something ambitious badly.

Another issue is student load. In our department, we don't provide as rigid a timetable for students as in some technical subjects. For most coursework, students do it in their own time. In a few cases, we provide a non-compulsory timetabled lab session with expert supervision.

However, each assignment has a notional number of hours attached to it. This works out at 5 hours per week for a 10 credit unit, or 10 hours per week for a 20 credit unit. This is based on a 39 hour week so, for example, a 20 credit unit in one teaching block gets a total of 13 hours per week. It is then just a case of subtracting the lecture hours and having a start date as well as end date for each assignment. If you take revision and exams into account, this agrees with University norms. If we get the loading for an assignment wrong, we don't regard it as a big problem, but as more-or-less self-correcting. We adjust the marking a little to take account of it, and note the need for change the following year.

Deadline clashes are dealt with partly by trying to coordinate them, but mostly by insisting that students should be working steadily, and ensuring that everything a student needs to begin an assignment is ready by the start date. In general, we have found that moving a deadline causes as many complaints (e.g. from those who finished early) as leaving it alone. It is better to state that everyone is in the same boat and that whatever the problem is, it will be taken into account in some way in the marking.

Coursework Submission

We have been using online submission for a long time, starting with email (in the mid-1980s, before the public had heard of it) and now (since 2000) using a custom submission system via our web site into a document store.

As soon as web-based submission was introduced experimentally, staff and students alike clamoured to make it compulsory for all coursework submission. It is now the backbone of our coursework handling processes.

Immediate advantages are that coursework doesn't get lost, and there is no argument about when coursework was handed in (because of the timestamps on the files). Students can submit from home, and staff can do marking from home, using password access to the web site. Deadlines are handled automatically, and are always at midnight. Checks for cheating can be automated. External examiners have a single place of reference for looking at student coursework.

Perhaps the most important aspect of the submission system is the psychological effect on deadlines. The way it works is that a unit organiser sets an official deadline day for an assignment. The system allows an automatic 3-day extension beyond that. The only penalty for using the extension is a red mark on a student's progress web page. If these build up, the student's tutor starts asking questions about time management.

The widely publicised and emphasised rule is that the extension is meant to cover minor problems such as colds at the student end, and minor problems such as service interruptions at the department end. No guarantees are made about services or support during the extension, which is often over a weekend, and students use the extension entirely at their own risk. Students are strongly urged to submit what they have by the first deadline, and then re-submit something better during the extension if they wish.

This allows us to make the extended deadline absolute, the penalty for missing it being the inability to submit, and a zero mark. The idea is to make the number of exceptions truly small, allowing us to publish some feedback such as model answers immediately after the late deadline. Exceptions must be serious and must be supported by written documentary evidence such as a medical certificate. They cab be dealt with by a further extension or by waiving that assignment. Importantly, though, one exception to the rule about documentary evidence is where a student has lost time management or motivation to the point that they are continually missing deadlines. We would rather do a deal with them and design a catching up strategy than have to deal with a later unit failure.

Feedback

Feedback on coursework is vital. It can be reduced over the years, or more accurately it should be at a higher level, but it is always needed. We don't use double marking, except on projects as described later, so there is a fairness issue. The main way we deal with this is to ensure that only one person marks each assignment. If several markers are needed, they rotate, taking one assignment each: a marker usually prefers one gruelling marking session followed by several weeks off, anyway. If two markers are really needed for one assignment, we still arrange for one marker to mark all students for one part of the assessment. Also, marking is done by username, which provides reasonable "voluntary anonymity", i.e. markers don't normally know who each student is unless they make some explicit effort to find out.

Some people in the university bemoan the fact that students seem obsessed by marks. We tend to take the opposite view - that we can use marked assessment to mould student behaviour. Unmarked work or assessment is unpopular with students, and experiments we have done with it via personal tutor groups have had patchy results, at best. So, on the whole, everything important that students do is assessed with a mark. We try to make sure that the importance we place on a particular kind of work is reflected in the the nominal time that students spend on it and consequently on the weight given to the marks for it.

The result is that we rely almost exclusively on straightforward weighted averages of marks for overall assessment. There are two main exceptions to this. First, faculty conventions and accreditation bodies place greater weight on final year individual projects than other units, so projects have to be passed to get any kind of degree. Second, a student's final degree grade is based on a weighted average of their year marks, and these exclude the first year. Our various 3 and 4 year undergraduate degrees, some with a year abroad, in two faculties, are weighted 0:20:80 or 0:30:70 or 0:10:20:70 or 0:10:30:60 or 0:10:40:50.

No matter how good the quality of the feedback produced, it is almost useless if it is given to students after they have lost interest in it. Minimum feedback given quickly is far more effective. We used to promise in our handbook to mark every assignment, and produce at least minimal feedback, within four weeks, however this is becoming increasingly hard to meet, due to various issues both on the student and the staff side.

On the one hand, this four week promise is much too long for weekly assignments, because students want feedback from an assignment before the next one is over. Also, the best quality assurance mechanism for assessment is to let students see the marks and feedback as soon as possible, and to be as open as possible with them about the processes used.

On the other hand, with some assignments, we still fail to meet the four week target. This is a continuing long term problem. Our next step to attempt to deal with it is simply to get our online system to record the date of release of marks by markers, so we can monitor the situation, or even "name and shame". We want to reach a situation where we either release marks by a given deadline, or explain why and give students an estimate of when.

We now guarantee to provide group feedback to the whole group before the deadline for the next assignment in the case that individual feedback needs to be delayed (for example because we are investigating possible plagiarism), or because we are carefully marking work and so it is taking longer than expected.

The important things seem to be to keep the marking load as low as possible, to make sure marking is as painless as possible, and to avoid over-marking (e.g. if an assignment is worth 5% of a unit, mark it out of 5, not out of 100). Types of feedback we give in various circumstances are:

For first year weekly programming exercises, and for a few other technical assignments, automatic marking works really well, though we did have some teething troubles. For example, students sometimes got zero by making one small formatting mistake which made all the tests fail. To solve this, we give students some tests in advance, similar to the first few we are actually going to use. That also makes the first few marks easy for them to get, which is good because it turns out that students often find it harder to get high marks when the marking is fully automatic.

With automatic marking, there is also an obvious question about whether we are testing the right things. In the case of basic programming, the idea of precision, and the idea that programs which don't work are worth nothing, are valuable lessons. What is more, the tests don't just deal with what the programs do as black boxes. They also look inside at some elementary aspects of programming style, such as consistent layout and dividing the program into manageable pieces. Some aspects of the marking are still a bit crude, but generally better than a marker who is tired or bored or has left the marking too late.

The quality of the feedback is somewhat mediocre with automatic marking. The program produces a report which tells the student which tests failed, and the student has to work out from this, and a general marking report, what mistakes they made or which technical issues they misunderstood. However, the automatic feedback is produced so quickly that staff have time to offer further individual feedback, on request, by meeting with students.

Not all the weekly exercises can be automatically marked, for example, the one about creating a picture of a scene in Java in the first year. However, marking the submissions and picking the best for an art gallery, is good fun because of the variety, so there is no problem in getting it done quickly.

Assignments in later years are not so suitable for automatic treatment (although sometimes some aspects can be automated). For these, the most widely applicable and effective technique to speed up marking seems to be self marking. This is used in a variety of different ways.

If an assignment is staged, each student can be asked for a brief statement about which stages were achieved. If an assignment is open ended, a bullet point statement of what achievements students feel they are proud of has a similar effect. The marker only has to check these achievements, not deduce them with a lot of detective work.

For some assignments, we give students the detailed marking scheme and ask them to estimate their own mark. We then check this swiftly, with a penalty if their estimate is out by much in either direction. In one experiment of this kind, the final checking was done by interview, on the basis that a ten minute interview per student is less painful, more accurate, and provides better interactive feedback, than ten minutes of conventional marking per student. The experiment was successful, except for the logistical problems of getting a lot of students to appear at 10 minute intervals without fail.

Feedback on Exams

Here's how we deal with feedback on exam scripts, to try to reduce the potential effort, while acknowledging that students have a right of challenge. No comments are written on the scripts themselves, and no feedback is normally given. If a student challenges a result, the mechanical checks are repeated, i.e. whether the signature on the exam script is right, whether everything in the script was marked, and whether the marks were added up and transcribed correctly.

On the subject of exams, for some years we have run some exams in January. This is for units in the first teaching block taken by fourth year students. There have been continual vociferous and understandable complaints about not getting the marks for these until the summer. Until a couple of years ago, our faculty rules would not allow early release of these marks. However, we obtained permission to use an early release process. The idea is to have a preliminary internal examiner's meeting after the January exams to finalise unit marks and deal with any preliminary scaling issues. Then, we release the marks, rounded down to grades, i.e. 70+ becomes A, 60-69 becomes B, and so on. These grades are flagged as provisional, subject to moderation in the summer (though we try to make sure no changes are made).

Projects

Projects are our larger coursework assessments, lasting for half a year or a whole year. A project usually forms a unit on its own, or sometimes two units. We used to deal with projects in a similar way to other technical departments. We became dissatisfied with this, and have evolved a different style, which we now promote around the university as an example of good practice.

The conventional approach taken in other departments and subjects, at its worst, goes like this. A small number of project topics are invented, and each student chooses one. A supervisor is allocated. The student carries out the project and writes a report on it. The supervisor and one other marker mark the report, and the two marks are averaged. Here's a summary of the principles of our approach, which make it different from this standard approach:

Here's a more detailed description of how it works.

Staff give short presentations, one from each research group in the department, describing the sort of things the groups, and individual staff, are interested in. This information is backed up with web pages, including some specific suggested topics. Students are encouraged to pick or invent a topic within the interests of one of the groups, or from their own interests. The students visit potential supervisors with their ideas, until they negotiate an agreement with one.

There is a marking panel involving a handful of staff members for each type of project. We have sufficient types that all staff are on one or two panels. Although the panels change from year to year, there is sufficient continuity to build up an expertise in that particular type of project.

The student writes a half-page proposal. Two members of the relevant marking panel are allocated as the main markers, not including the supervisor. For a group project, all the panel members are markers. The markers vet the proposal to make sure it seems viable, and give any preliminary advice.

Some sort of intermediate presentation and/or report is required from the students, and looked at by the panel. This varies depending on the type of project, and may or may not be marked. It is the formative feedback which is important.

At the end, each student gives a short presentation to the markers, sometimes in front of an A1 poster, and gives a short demonstration of the project itself, and hands in the final report, both online and printed. For group projects, the group also submits an agreed statement about what percentage of the work was done by each person.

Each person on the panel receives the reports for which they have been assigned as a marker. Each pile of reports is for one type of project only, making it easier to avoid confusion, and to compare and rank the submissions. Although each project type has a checklist of things to look for, there is no rigid marking scheme, because of the large variety of project topics with different emphases (e.g. self-contained program development, or working for a commercial client, or carrying out research experiments). Each marker independently produces a mark, and some measure of confidence in that mark (e.g. "plus or minus 5%"). The supervisor reports any unusual circumstances, and gives an estimated mark which is not included in the assessment.

There is then a panel meeting. If the two main markers agree to within one or two percent (which happens surprisingly frequently) and there is no serious discrepancy with the supervisor's estimate, or other uncertainty, there is no problem. If there is a disagreement (which can occasionally be surprisingly large) it has to be resolved. Averaging is banned, because it tends to produce an unnaturally narrow spread of marks. Instead, the panel tries to understand the reason for the disagreement. If there is no quick resolution, or any uneasiness remains, further panel members read the report and get involved, until agreement is reached.

The panel then ranks all the projects, making sure that the order they are in is the correct order. Hence, if two students get the same mark for their project then this is done by a definite decision of the panel and not by some statistical fluke. This means panel meetings can take a long time, but it means that the final result is robust and easily seen to be justified, to both students and outside evaluators such as external examiners.

Using a custom program, all the individual marks, agreed marks, and marks changes made during the meeting are recorded, along with a textual record of the proceedings. This is done during the meeting and projected onto the wall. It is used to look at the results for each panel member to see if there are discrepancies in their average marks or rankings, to check the ranking of the projects as a whole, to investigate the borderlines and firsts and fails, and so on. It is also used after the meeting to produce automatically the minutes, the student feedback, and the report for the external examiner.

Group Projects

Group projects are marked in a similar way to individual projects, the main difference being how your final mark is computed. You will be asked to decide as a group the contribution you have all made to a project. Ideally, all members of a group will agree their mark distribution and submit only one team report. If the distribution cannot be agreed let us know well in advance, so vivas can be arranged (in this case we'll also advise on what documentation is required, typically a report per student is necessary).

The marking panel will take both your group mark and your individual contribution to determine your individual mark. This mark is further moderated to fit it onto the Faculty scale. This reflects the fact that if, for a traditional piece of coursework you were to get a mark of 60%, a colleague who'd spent twice as long as you would not get a mark of 120%. All our marking is done using a non-linear scale.

Other factors will also be taken into account by the panel include the size of the group (with respect to others on the same unit) and problems with facilities or software availability. Problems such as health that have affected the group are dealt with by our Special Circumstances Committee and Examinations Board.

Conclusions

Here are a few points which have emerged during this chapter:

Contact Us  |  News 25th Anniversary
© 1995-2012 University of Bristol  |  Terms and Conditions
About this Page