Quality enhancement meetings

An annual Quality enhancement meeting (QEM) is held to discuss audit experiences, share good practice, and consider current issues of academic equability. The meeting is jointly sponsored by the New Zealand Universities Academic Audit Unit and the universities. Participants are delegates from all New Zealand universities, and for many years, invited representatives from the wider New Zealand tertiary sector. In most years, a guest speaker from an international university or quality audit agency has been invited as a keynote speaker and commentator on international practices.

The tenth QEM was held at the University of Auckland, 15-16 September, 2005. Documents available here are:

• Summary of proceedings. A linked web version is given below. The summary can also be down-loaded as a Word document (click here).
• National Centre for Tertiary Teaching Excellence - presentation by Chris Heath on the 2005 Teaching Matters Forum and proposals for the Centre. Click here to down-load the power point slides from the presentation.
• Quality Assurance of Teaching and Learning - Can lessons be learned from external experience? This invited keynote presentation was given by Dotre Kristoffersen, Audit Director, Australian Universities Quality Agency. Click here to down-load the power point slides from the presentation.
• Preparing a teaching portfolio. Workshop paper by Neil Haigh - click here to down-load background document (in Word format).

Summary of the 10th Quality enhancement meeting (QEM10), September 2005

Click on the linked titles; the 'Index' buttons at the end of each section return to this index

Next meeting

QEM11: The 2006 Quality Enhancement Meeting will be hosted by the Victoria University of Wellington, 28-29 September, 2006
Venue: Museum Hotel, 90 Cable Street, Wellington.

QEM 10: Summary of Proceedings

The University of Auckland, 15-16 September 2005

Day 1

Session 1- Recent Government Policy Initiatives in Teaching and Learning Quality: Implications for Practice and Internal Quality Assurance/Evaluation. 

Chair: Rob Allen, Auckland University of Technology

Panel:

John Jennings/NZUAAU; Colleen Slagter/Ministry of Education; Ruth Anderson and Bryce Cleland/Tertiary Education Commission; Chris Heath/Otago.

** Chris Heath’s Power point slides on the National Centre for Tertiary Teaching Excellence can be downloaded. Click here.

** John Jennings has contributed the following commentary on the session:

The catalyst for the agenda for QEM10 was a letter from the Minister of Education dated 24 December 2004 in which he advised of a Cabinet decision to agree to a process to be implemented during 2005 involving three elements aimed at enhancing the effectiveness of teaching and learning in tertiary education.

Culture shift towards more effective teaching and learning - the work of a Teaching Matters Forum to consult with the sector and to advise on the establishment of a National Centre for Tertiary Teaching Excellence.  Institutional academic audit deals with institutional quality processes.  It is to be hoped that the National Centre for Tertiary Teaching Excellence will provide support for quality enhancement where it can effective do so – that is, at discipline level.

This raises the question:

•         What is the appropriate level at which quality enhancement can bring about effective and positive change?

Quality assurance and risk management – a focus on quality teaching and learning in quality assurance processes.  The Minister writes:

By supporting the valuing of teaching, the pursuit of excellence and the professional development of education, quality assurance arrangements can make a positive difference.

This raises the question:

•         How can institutional quality audit contribute to effective and positive change?

Student component performance measure – successful course completion rates, course retention rates, the results of a survey of learners

The Minister writes that these:

. . . will help to secure the improvements in the quality of teaching and learning that will assist New Zealand to achieve its national social, economic and environmental goals.

I do not see any co-relation between such data and improvements.  We all know that these indicators are neither absolutes nor ‘hard’; they take no account of the very complex array of reasons why students do not succeed or do not move to higher levels of study.  I do not see how a witch-hunt of those institutions with low scores will help to secure the improvements in the quality of teaching and learning.  The Minister continues:

It will also recognise provider performance post-enrolment in keeping learners engaged in – and succeeding in – tertiary education.

Now if this does happen, and if students are better engaged in learning, then I would be much happier.

This raises two questions:

•         Course completion rates and course retention rates – how do these improve quality?

•         Is the survey of learners to be a survey of satisfaction, or a survey about engagement?

Central to the Cabinet initiatives is the engagement of students in their own learning.  Teaching involves (among other things) the Teacher, the Student, and Spaces both physical and intellectual.  The link between teaching and learning is the engagement of teacher and student with information and skills which help identify concepts and which lead to understanding and knowledge.

References:

Three publications read over the past 12 months are helpful with respect to engagement.

Mantze YORKE, Bernard LONGDEN
Retention and student success in higher education
2004, SRHE/Open University
This book is research-based (Australia, South Africa, United Kingdom).  It makes a number of points worth keeping in mind.

• The publication of indicators in any area is likely to have effects on what is being measured.
• The rationale for using retention and completion as performance indicators is weakened by government policies.
  • Less funding by government for education and more funding by the students often require students to study part time over a longer period.
  • Policies of greater access to education capture students who may not be so well prepared or informed or less likely to possess the cultural and social capital necessary to support success.
  • Policies to support greater access to life-long learning can lead students to ‘dip in and out’ of programmes
• Causes for departures are complex:
  • Institution-related (let down by the institution).
  • Student related (flawed decision making, failure to cope, life-style choices, extraneous events, student debt).

To focus on retention is mistaking the symptom for the cause; we should focus on student success through teaching, learning, assessment and institutional support services.  To bring the student success and students’ interest centre stage is to highlight the need to enhance the quality of the student experience.

The book ends by setting out 44 challenges – 24 for institutions, 12 for students, 8 for higher education systems.  Every failure cannot be laid at the doors of the institutions alone.

Hamish COATES
The scale of student engagement for higher education quality assurance
Quality in Higher Education, v.11, n1, April 2005
There is too much emphasis on information about institutions and teaching and not enough emphasis on what students are actually doing . . . the way students go about constructing their knowledge is what really influences learning . . . Recent research on the Course Experience Questionnaire has emphasized the direct educational benefits of beyond-class experiences, the value of considering a more holistic understanding of the student experience, the value that beyond-class experiences add to formal learning activities and the importance of understanding emerging dynamics of student behaviours.  All this challenges the validity of the distinction between ‘in-class’ and ‘out-of-class’ learning activities.

Ronald BARNETT, Kelly COATE
Engaging the curriculum in higher education
2005, SRHE/Open University/McGraw Hill Education
This book extends the discussion on engagement to ways we can improve the curriculum.
Their definition of engagement:

Engagement is a coming together, a merging, a fusing, pointing to mutual listening, reciprocity and dialogue and conducted in a willingness to change.  Engagement points to a situation in which one party accommodates the other and becomes somewhat transformed in the process.

Conclusion

In my view, the greater engagement of institutions, academics and students in enhancing learning is behind the Cabinet’s initiatives.  But who has control of these engagements?  Hamish Coates summarises the dilemma very well, particularly with respect to the factoring of student engagement into quality assurance.

There is much work to be done if student engagement is to be made a part of quality assurance determinations in higher education.  One challenge would involve reconciling the constructivist perspective underpinning the idea of student engagement with what is ultimately an institutional responsibility for managing ongoing improvement.  There is a tension here which stems from the fact that although students are seen as making a direct contribution to the educational process, it is one that is largely beyond institutional control.  Accordingly, institutions would need to develop approaches to manage and enhance student engagement without having ultimate control over students.

Session 2 -“Quality Assurance of Teaching and Learning – Can lessons be learned from external experience?”

Guest Speaker: Dorte Kristoffersen, Australian Universities Quality Agency

Click here to down load the Power point of Dorte’s keynote talk - this should open in a separate window.

Session 3 - Academic Audit and the Future of Institutional Quality Assurance Beyond 2006

Discussion Leader: John Jennings, NZUAAU

1. The following options were presented to the session to initiate discussion.
   Presuming rounds of audit continue, should there be:
   • a focus on one aspect for all universities ? [the status quo)
   • more attention paid to the needs of each university?
2. An annual meeting between each university and the Unit, developing a ‘quality profile’ which informs the topic of the next audit of each university.
3. Each audit focus on the needs of each university, by developing a specific terms of reference for each university audit.
4. Move from inputs to outputs, with audits of ‘bands’ of disciplines with one panel carrying out a nationwide audit.
5. Audits of very limited focus, say
   • the application  of electronic teaching and learning technologies,
   • the response of universities to the Treaty of Waitangi,
   • student engagement in the learning process,
   • the audit of academic policies, processes and procedures.
6. Undertake process audits, concentrating on:
   • processes related to the ongoing quality of universities’ academic programmes,
   • risks (‘What can go wrong?’) in activities associated with the processes,
   • controls within the processes that prevent things going wrong and detect something that has gone wrong.

Regarding processes ...

7. The Unit require a self-review, and the panel invite submissions from key stakeholders if further information desired.
8. Size and composition of panels reflect the nature of the audit being conducted for the University.
9. Panels include student members appropriately trained.

Discussion

• Reasons for audit are to assess capability, expertise and systems. Therefore, a consistent model for all universities is preferable to the ‘mixed model’, which also allows for institutions to focus on issues specific to them. Any proposed changes to the academic audit process should not impact negatively on institutional autonomy.

• Benchmarking is best facilitated by an audit that has the same terms of reference for all universities. However, the audit model must support each University’s needs.
• There needs to be a focus on the audit process itself. There is a need to explicitly engage each University between audits, so that it reaches a stage of continuous improvement.

Self-review documentation:  The cost (direct and indirect) of producing the self-review is high, and the benefits are harder to measure.  Universities want to produce documents that are specific to them, and the self-review documentation may be more helpful to the university if it focuses on how things are done rather than on what is being done.

The desire of the NZUAAU and NZVCC for audits to be focussed on creating positive improvement, as opposed to being compliance oriented, was reinforced.  If compliance oriented, the audit process may be of limited benefit to the University.

Benchmarking is best facilitated by an audit that has the same terms of reference for all universities. However, the audit model must support each University’s needs.

• There needs to be a focus on the audit process itself. There is a need to explicitly engage each University between audits, so that it reaches a stage of continuous improvement.

• Self-review documentation:  The cost (direct and indirect) of producing the self-review is high, and the benefits are harder to measure.  Universities want to produce documents that are specific to them, and the self-review documentation may be more helpful to the university if it focuses on how things are done rather than on what is being done.

• The desire of the NZUAAU and NZVCC for audits to be focussed on creating positive improvement, as opposed to being compliance oriented, was reinforced.  If compliance oriented, the audit process may be of limited benefit to the University.

• Audit Visit: Does having large numbers of stakeholders meeting the audit panel add value? Does the European experience with academic audit provide lessons for us? A number of audit approaches in Europe comprise meeting with groups, with less focus on evidence-based approaches. The general feedback from Europe is that the interaction between auditors and the institution is beneficial.

• Audit Visit: Is it necessary to authenticate the portfolio through triangulation? Are other less resource intensive methods available?

• Process Audit Pilot at the University of Canterbury: Now that a level of maturity in the audit has been achieved, the time has come for a new direction. The pilot will be watched with interest. It is important to ensure that the feedback process from the University of Canterbury to the NZUAAU is carefully structured. The structure of the feedback process should therefore be formulated upfront before the pilot commences

• Student representation on audit panels: Unanimous agreement that students be included on audit panels and that this should occur as fast as is practicable.

• Academic Audit and Student Learner Survey being developed by the Tertiary Education Commission: Will the survey results provide assurance that is not currently being met? The audit cannot be a substitute for the student learning survey and vice-versa.

• Government agency perspectives: The performance measure (retention and completion rates, student satisfaction) is designed to be all-inclusive. However, it is a strength that universities have an audit and academic approval system and the Government supports this.

• Level of engagement between audits: There is an important need to place attention on what occurs within universities between audits, to ensure that there is continuous engagement on audit issues.  Engagement between the NZUAAU and universities must be balanced against the need for objectivity during audit.

• Cycle of audits: Suggest a 5 year cycle of audits. However, there could be a process where universities provide yearly data on agreed indicators. This could be similar to the approach in the United Kingdom where, as a trade-off for external intervention, institutions publish data yearly that is publicly available as well as a 5-yearly audit.

• Audit of specific areas: This is already undertaken by NZUAAU as special contracts with appropriate government agencies. For example, checking for compliance with the code of practice for international students, and supervision of international doctoral students.

Session 4 – The quality of teaching and learning and student learning support services: student perspective

Student panel discussion

Concerning a national learner opinion survey:

• The survey should clearly identify between target groups, and the different types of learners. This could be achieved through multiple survey instruments, for example.
• The survey should investigate the nature and type of student interaction and the benefits of a diverse demographic profile for all students. For example, using a ‘buddy programme’ to pair students.
• Satisfaction based questions are generally difficult to respond to, as students do not typically have anything to compare their University experience to. Perhaps attention should be focused on students who have studied at 2 or more institutions, as they have a basis for comparison.
• Questions on importance could be more useful than satisfaction-based surveys. However, they should not be used as a justification for decreasing service provision to minority students (eg disabled students).
• Student life: A survey should cover all aspects of a student’s learning experience, in particular the social aspect. Isolation can affect students, and student groups can assist with this.
• Balance needed in the qualitative and quantitative components of surveys.

Concerning student teaching and course evaluations:

• Teaching Evaluations: There is no transparency of feedback at this stage. There are also no visible outcomes to the results of course evaluations.
• The optional nature of course evaluations means that students won’t always get the opportunity to evaluate a specific course or lecturer.This could be addressed by ensuring that there a student representatives for each course and the system works effectively.
• The timing of course evaluations is also important. Sometimes it is difficult to provide feedback until the course has been completed. On the other hand, providing an opportunity to provide feedback during the semester is also helpful. For example, providing brief feedback to lecturers on an assessment.

Concerning assessment:

• Appropriateness of assessment: There should be a variety of assessment types available to students, and a focus on what will be practical to students as graduates. For example, presentations.
• Innovation in assessment to promote expansive learning – for example, allowing a range of options (eg portfolio, presentation, multimedia) for a single assessment.
• The opportunity to participate as undergraduate students in research projects.

Concerning the difficulty in getting feedback from students:

• Utilise every mode of delivery available: mail/web/phone etc
• Allow for anonymous feedback
• Provide incentives for completing surveys
• Address reasons for student apathy: It could be that the issues contained in the survey are not relevant

Concerning e-learning:

• Learning Management Systems are useful and assist learning, in particular being able to access lecture notes etc. Also useful is the access to e-books and e-journals. These promote flexibility and help cater to different learning styles. However, the availability and depth of utilization amongst courses differs considerably
• For a number of reasons, e-learning should not be viewed as a replacement for face-to-face learning. For example, replacing lab sessions with on-line exercises is not a preferred use of e-learning.

Other issues:

• Diversity in knowledge perspectives. Opportunities to learn Maori, Pasifika and other non-western perspectives.
• Increase student-staff interaction. Encourage lecturers to talk about their background and their research.
• Promote the role of teaching-research nexus.
• Acknowledge the role of students as learners, and that they are active learners.
• Time Pressures: The shift to semesterisation from year-long courses has meant that it can be difficult to absorb and retain material. Another issue is timetabling, as it is not always family-friendly.
• Tutoring: Tutoring performs an important role in dealing with the isolation that can occur when researching at postgraduate level. Benefits include meeting staff, peer students, undergraduate students and learning new skills. It can also help with motivation. Universities should encourage more students to participate in tutoring schemes.
• Tutor training schemes: Tutor training is available, but the timing is not always appropriate. Some examples of mentoring programmes exist, but there is a lack of structure in some departments.
• Student services: Help to raise the profile of the student services available
• Environmental Practices: Universities should take the lead in focusing on sustainable practices
• Textbooks: Given the cost of textbooks, lecturers should ensure that the textbooks listed as required are going to be highly relevant to the course.
• Course Outlines: Preference for detailed course outlines, particular those with assignment deadlines, learning outcomes etc.

Session 5 – Two topical issues

Facilitator: Lorraine Stefani, University of Auckland

(a) Graduate profiles, learning outcomes and assessment

Group 1

• Graduate Profiles focus on the use of transferable skills.
• Graduate profiles are often driven by outside agencies. For example, the Association to Advance American Collegiate Schools of Business (AACSB).
• Students are not aware of the skills they are developing.
• Graduate Profiles are useful for professional programmes where there is greater rigidity in programme structure and intended outcomes, but are more problematic for liberal programmes.
• Graduate Profiles take account of the prior skills that students bring to their study. That is, we can say that we add value to graduates, but we cannot claim that we gave students all of the skills outlined in the graduate profile.
• Employers’ perception that different universities produce different skills.
• Funding mechanisms can detract from programme mapping
• Should there be a NZ National Graduate Profile?

Group 2

• Exploring the link between graduate profiles, learning outcomes and assessment. There is not a lot of evidence that graduate profiles are anything beyond statements of intent.
• Graduate Profiles are often used for marketing purposes
• Staff and students focus on assessment, which is the central feature of the university life.
• Outcomes therefore have some utility as they are taken at the course-level and are linked to assessment. However, the link between outcomes and assessment is not always widely known.
• It is easier to map learning outcomes to the graduate profile for professional programmes than it is for non-professional programmes
• It is timely to note that discussion on graduate profiles, learning outcomes and assessment, while mainstream now, would not have occurred 15 years ago. This is linked to the increasing emphasis on the development of skills as well as knowledge.
• Overall, graduate profiles have not had the impact that is often assumed.

Group 3

• CUAP’s definition of Graduate Profiles comprises the following:
• Personal attributes
• Interactive attributes
• Specific programme attributes
• Assessment activities can help evaluate whether the attributes specified in the Graduate Profile have been achieved.
• Graduate Profiles can be helpful for professional qualifications but less so for other qualifications.
• Graduate Profiles can be conceptualized as consisting of generic degree level (e.g. Bachelors) skills and specific degree skills, and – where appropriate – specialized degree requirements (e.g. Accounting). However, it may be difficult to classify liberal degrees in these terms.
• Graduate Profiles can be an important tool when in discussion with employers.
• They can be useful for professional qualifications and for curriculum design. As opposed to asking whether graduate profiles inform the curriculum or whether they are informed by the curriculum, we should instead focus on how they inform each other.
• The reality is that many staff do not see the link between graduate profiles and the subjects that they teach. Universities manage degrees and staff teach subjects. It could be that staff use profiles implicitly as they help to contextualize what is in the curriculum.
• Students value subject and course outlines that contain assessment information and learning outcomes.
• The key issue is to determine what distinguishes a university graduate from someone who is not a university graduate. The graduate profile should capture the distinctiveness of university study. i.e., the importance of research and research-informed teaching.

Session 5 - second topic:

(b) Effective use of benchmarking in the teaching and learning field

• There is evidence of various levels of benchmarking at all universities.
• Benchmarking occurs as a part of normal academic activity. For example, sabbatical, visiting staff from other universities, external moderation and other qualitative relationships. Benchmarking in this context can be closely linked to enhancement, as comparisons are made and learning takes place.
• There is need to incentivize the use of benchmarking. This could be achieved – for example - through attaching a resource implication for departments if they benchmark. However, is benchmarking an add-on activity, or should it be considered as an integral part of a university/faculty/department’s operations? If it is the latter, then is there a need for it to be incentivized?
• There needs to be an appropriate balance between quantitative and qualitative indicators. There is an inherent bias towards quantitative indicators when benchmarking. However, the most interesting findings tend to arise from qualitative investigation. For example, external moderators may be employed to look at assessment, but this may transform to discussion on other issues.
• It is important to ensure that the benchmarking is used as a means to enhance the quality of teaching and learning
• Transaction costs of monitoring – staff need to be encouraged to make comparisons and to benchmark. Professional schools typically have an advantage in this respect as accreditation promotes benchmarking. It is however problematic for the arts and humanities.
• There is a lot of data already available that could be used in benchmarking, but it is at the university level. It is subsequently largely meaningless to individuals.
• The use of dashboard indicators is a possible form of benchmarking.

Session 6 - Day 1 wrap up. Directions for QEM 1

Co-Chairs: Jenny Harper, Victoria University of Wellington; and Penny Brothers, University of Auckland

Summary themes:

• Institutional autonomy
• Who is taking responsibility and who is accountable
• Differentiation of teaching and learning within the tertiary sector
• The contrast between an emphasis on enhancement at the top, and a view from below that emphasizes compliance
• The relationship of academic audit to the national learner survey
• Emphasizing enhancement as opposed to accountability in the teaching and learning field

QEM 11 will be hosted by the Victoria University of Wellington, 28-29 September 2006.  Venue: Museum Hotel, 90 Cable Street, Wellington.

Day 2

Session 7: Parallel Workshops

Workshop A: Developing, maintaining and using teaching portfolios

Facilitator: Tom Angelo, Victoria University of Wellington

Keywords – purposive, highly selective, organized, reflective

Feedback from universities with representatives at this workshop indicated that the prevailing use of teaching portfolios was related to applications for teaching awards (both internal and national) and continuation/promotion purposes.  No university mandates the use of teaching portfolios (there is sensitivity around mandating / compliance issues).  At the individual teaching staff level, it was thought that the greatest interest in teaching portfolios resided with keen teachers, and those who participated more frequently in professional development activities.

In the UK, there is no mandated use, but ‘personal development’ planning for students is increasingly embedded in the curriculum.  In the U.S. there are wide variations – an increasingly used practice is the compiling of student achievements on CD Rom, often linked to a capstone course.

Mandating use of teaching portfolios poses the risk that they would privilege a quantity of teaching approach, rather than quality.  The PBRF use of qualitative indicators was thought to be an idea worth emulating. 

It was agreed that teaching portfolios were potentially very powerful tools for change over time, but to support this change considerable institutional tools, structured incentives and resources (e.g., to evaluate portfolios) would have to be allocated.  In addition, with the prevailing emphasis on research, many staff would take a lot of convincing to adopt portfolios.  ‘Reflection’ on teaching may be useful, but it must be against criteria which have the potential to ‘move’ teachers. While there may be advantages for many individuals, motivation of institutions to use teaching portfolios more generally was problematic.

A key discussion point was whether the focus on teaching portfolios obscured more fundamental issues, and whether the nature of the discussion needed to be adjusted.  It was suggested that more institutional emphasis be placed on compiling ‘evidence of improved student learning’, downplaying the emphasis on teaching portfolio use (which is one tool that might contribute to the larger institutional purpose).  This would be especially appropriate if greater emphasis were placed upon institutional graduate profiles (which tend to emphasize outcomes).

Perhaps the National Centre will undertake some research on teaching portfolios?

Additional resource material on teaching portfolios:

The Griffith University web site on teaching portfolios (click here) was mentioned as a good resource, and includes exemplars. Preparing a teaching portfolio (Word document) by Neil Haigh can be downloaded from this site. ( Neil Haigh and AUT have kindly made this paper available to QEM participants.) The Centre for Teaching and Learning Resources, University of Buffalo, has a portfolios site giving sample portfolios from across the disciplines (click here) .

Session 7 - second workshop:

Workshop B: Academic support for Maori and Pacific Island students

Facilitator: Patsy Paxton, AUT

The Workshop Group considered the following questions which were contributed by QEM 10 participants. 

• How do you provide good academic support when you have very few Maori or Pacific students? There are already a lot of 'extra' and 'other' expectations of the Maori and Pacific leaders (for both students and staff).

The University of Otago has a very low percentage of Maori students.

A survey is being carried out at Otago in conjunction with external consultants to look at Maori student perspectives and Treaty responsiveness.  Findings to date are that Maori students to not seek support and are reluctant to self-identify.  82% had responded that they feel culturally safe in the University of Otago environment.

The University of Canterbury is in a similar position whereby specific support is not sought by Maori students, who also often prefer not to self identify.

The other universities representatives had substantial numbers of Maori and Pasifika students and all have diverse systems in place to provide support.

• What specific support mechanisms are in place for Maori and Pacific Island students?  What works well?

A very high number of Maori and Pasifika students entering tertiary education do so as mature students, and enter under ‘Special Admission’ regulations.  As mature students, they frequently have heavy commitments to children and family, and culturally are substantially involved in whanau activities.  They frequently need support and a flexible approach to learning and assessment to enable them to successfully undertake and complete tertiary programmes.

It was agreed that numbers of Maori and Pasifika students vary across faculties, with some programmes attracting many more students than others.

One major issue that was identified across all universities is that funding to support Maori and Pasifika students is normally available under Special Supplementary Grant (as a government grant) and not embedded in as part of individual university funding.  For this reason funding can be intermittent and spasmodic, and does not satisfactorily provide the necessary platform to underpin systems which are vital to the success of these students.

AUT has a centralised support system called ‘ITMOSS’ (Integrated Model of Student Support) started in 2002, which covers all faculties (via faculty representatives), and is designed to specifically support at risk Maori and Pasifika students.  Over its three years of operation it has worked successfully in reducing the numbers of at risk students across all programmes.

The University of Auckland has high numbers of Maori and Pasifika students.  Substantial support systems are in place, including specific rooms/areas dedicated for Maori and Pasifika students to use, which has proven very successful. Targeted support of these students has made a significant difference, through peer mentoring and other systems in place.

The use of student peer mentors is common across universities and has been found to be successful.  In addition, Maori teaching staff consider it part of their role to involve themselves in the development of the whole student, and play a significant part in supporting individual students inside and outside the academic arena.  This results in a very heavy workload for such staff.  It is a common experience that Maori and Pasifika staff prefer not to say ‘no’ to these extra responsibilities, which in itself can cause difficulties.

Most institutions with significant numbers of Maori and Pasifika students offer pre-degree programmes designed to assist these students prepare for academic study. However, it has been found that since a substantial number enter as mature students under ‘Special Admission’ regulations, they consider they have been accepted at university level, and do not necessarily consider that they need such preparation. Students are frequently unrealistic in their choice of numbers of papers and take on too much.   There is a need for systems in place to identify such students and to provide intensive advice and support on an individual basis.  Such systems exist in some universities.  Overall there may well be an opportunity to increase the ability to provide such intensive advice.

• Are there any unique needs concerning “mentoring the mentors” for Maori and Pacific Island students, compared to mentoring programmes for other students?   What works well?  The uptake in some mentoring programmes is low – how can student participation be increased?

The University of Auckland has systems in place to provide some training for student mentors.  However, it appeared as though this area is not well addressed in the majority of institutions and there are opportunities for improvement.

It was agreed that in general, Maori and Pasifika students do have unique needs (as referred to above), compared with international students, and systems need to be tailored to meet these needs adequately.

• External demands and time management are acknowledged factors in the academic performance of many students, including Maori and Pacific Islands students – what works well in terms of curriculum, support etc. in addressing this issue?

Successful approaches to supporting students in time management include:

• Good timetabling (early morning classes are not appropriate for mothers with children)
• Flexible programmes in terms of length and full-time/part-time study
• The use of a variety of delivery methods, including on-line
• Support services within the university specifically directed to assist students with time management
• Mentors, at both peer and lecturer level
• Careful mentoring at admission level to ensure that students undertake appropriate levels and amounts of study commensurate with their domestic responsibilities.

• What types of quality checks can be put in place to ensure that academic support services are student-centred as opposed to ‘system-centred’? (or that we achieve the correct balance between the two approaches)?

It was generally agreed that a ‘one on one’ student-centred approach, looking at individuals, rather than a systematic approach is most helpful.  Early identification of ‘at risk’ students is essential – the earlier the better.  Therefore the ability to track and identify these students is vital.  Massey University has an approach to identify such students before the end of the first semester – at the first assignment if possible.  It is, however, difficult to do this as it requires a substantial time commitment from staff to enter all assessment results into a centralised system.

The University of Auckland is also undertaking a project to identify at risk students early on.  It is more difficult to identify such students in larger diverse faculties where there are very few core papers, and student results are much more difficult to track.

It was agreed that the development of systems to identify at risk students as early as possible to provide constructive intervention is highly desirable.

• Do we have adequate training/induction processes for academic staff – including, but not confined to, particularly international staff -- to teach students of Maori and Pacific backgrounds effectively?  How are ‘deficit/low achiever models’, and construing certain students as ‘the other,’ best dealt with?

The University of Otago has a two day induction programme for all new academic staff, which includes Treaty of Waitangi issues.  The University of Auckland also has an induction programme which includes Treaty matters. 

Although most institutions approach the question of induction of academic staff to some degree, including cultural and Treaty issues, it appears as though there are opportunities for further development in this area.

• If a university aspires to have a curriculum that is inclusive of the world views and experiences of Maori and Pacific students, how is this best ensured?

It was agreed that while in some Faculties (e.g. Education) it is easy to incorporate Maori and Pasifika perspectives, in other disciplines it is extremely difficult.  There was general agreement that all institutions, while keen to approach this successfully, were very much at sea as to how to integrate Treaty of Waitangi and other cultural perspectives into the majority of programmes and classes. 

Session 8: Parallel Workshops

Workshop A – Research supervision, doctoral retention and completion

Facilitator: David Tippin, University of Auckland

Workshop participants exchanged information about current practices related to improving completion rates and time to completion.  Annual reports and periodic student surveys are used by some institutions.  Otago requires a confirmation seminar at the end of the first year of candidacy.

It was agreed that retention and completion issues are uneven problems within each university.  Disciplinary differences and requirements (doctoral projects linked to staff research grants), and varying supervision styles (e.g., more intensive in some science-based subjects than in arts and humanities) make generalizations difficult.

PBRF-related issues: PBRF funding related to completions (including research areas and ethnicity variables) needs to be well-explained to staff, as it has implications for the institution and potentially for supervision practices.  Where PBRF awareness is high, will this impact on supervision practices and outcomes?  PBRF pressures raise questions as to where to best expend resources.  Increased intakes of international doctoral students will have an impact upon supervision practices.  Staff development remains an issue in some disciplines where traditionally a doctorate has not been required.

Improvement ideas:

• Ensuring PBRF implications are well-understood by staff.
• Encourage and monitor supportive research cultures in academic units.
• Build capacity and work with postgraduate student associations in devising strategies to assist students in completing their doctorate in a timely manner, e.g., support issues, supervision.
• Provide advice to incoming doctoral students (and even earlier at the point of initial inquiry/application) concerning realistic completion time for particular types of projects.  Three years to completion may not be feasible for some students.   

• Some factors that may influence doctoral performance are: -

• Mandatory supervision programmes.
• Financial support.
• Departmental research culture.
• Disciplinary differences – for example, some subjects attract research grants for projects. Research grants provide not only financial support, but may also reduce student isolation within their department/school, as they will be connected closely with other staff and students for the duration of the doctorate.
• Demographic characteristics – each university will have a distinctive demographic profile that can influence doctoral completions and time-to-completion. For example, age distribution within the university and between departments.

International research on completions. There is an increasing body of knowledge on doctoral completions. The issue is whether this can inform our discussion on this issue, or whether there is a distinctive NZ perspective that we need to learn. This could be achieved through cross-institutional collaboration.

Links to further information

Ministry of Education, How long do people spend in tertiary education, June 2005.  (includes data on doctoral students)

DEST study on the pedagogy of ‘good’ PhD supervision (January 2005).

U.S. Council of Graduate Schools, ‘PhD Completion Project’ (2005)

Higher Education Funding Council for England, PhD research degrees: entry and completion (January 2005)

Session 8 - second workshop:

Workshop B – E-learning, flexible and distance learning

Facilitator: John Milne, Massey University

(Click here to link to the New Zealand e-Learning quality guidelines project - opens in a separate window.)

Session 9 – Linking Policy and Practice in Teaching and Learning

Chair: Les Holborow, NZUAAU

Discussion began with the linkages between policy and practice following an institutional audit. It was found, at least by Auckland, that during the audit process it seemed that some staff were unaware of the policies that exist. They are aware of some kind of “regulatory framework” (this is how policies largely seem to be viewed) but in some instances are not aware of the specifics of individual policies.  As a more general illustration amongst the universities, graduate profiles are an example of widely available policy/guidelines, yet student and staff awareness is low – is this a question of utility, awareness, or staff not well-persuaded?

Issues and possible solutions:

• Whose role is it to motivate staff to be aware of wider institutional imperatives concerning teaching and learning, and how much effort should be put into this, particularly in an environment of post-PBRF fatigue and the privileging of research?

• Staff who are casual or part time or are tutors may be unaware of how policies apply to them – particularly policies concerning teaching and learning.  One way of addressing this (Auckland solution) is to package up the crucial points from each policy and ensure that appropriate staff members are given them via way of some kind of “Short Staff Guide to Teaching and Learning” Handbook?  This might also be useful for permanent staff.  Hardcopy might get more attention than another posting on the website.  Availability of material in multiple formats is necessary.

• Manageability. Lincoln have a system of departmental handbooks which contain, among other things, relevant policies. This seems to work well as collection of data on staff web-access to policies has shown very little traffic!

• Staff tend to consult policy when a problem/issue arises.

• Another suggestion was to ask staff to sign a statement to say they had read the policies. Noted that government agencies and private businesses usually require staff to do this (i.e. staff are required to read and know the company policies).

• Where re-interpretation or adaptation of policies at a lower level is necessary, institutional-level staff should be involved.

• Using staff feedback re student evaluations as a means of improving practices: it was suggested that it might be helpful to let staff know of the relevant policies at the right time, e.g., including the policy on student surveys with the return of survey reports to academic staff.  It could be suggested to staff: “Now that you have the student feedback, here’s what to do with it.” 

• Policy reviews should involve consultations with staff.  Staff feedback on what works in practice is needed.  A staff panel could be convened to review new policies for clarity, pressure points etc.

Comment that an increasing feature of teaching and learning is its “policy-isation” and whether this is the best way forward. What can happen is that different values are put on different activities, e.g. is research valued more than teaching now?

The Otago Vice-Chancellor has initiated a “bureaucracy-busting” project. He is seeking feedback, specific comments re policies, paperwork etc from all staff in order to identify pressure points. Change management being taken seriously.

Further research could be undertaken concerning effective means of communication and promulgation of teaching and learning policies.

National Learner Opinion Survey.  The final plenary also re-visited the proposed national student survey.  Among the issues raised in discussion were:

• The need for the survey questions to be sufficiently differentiated to take into account the unique characteristics of universities.  A debate about commonality / distinctiveness in questions needs to be undertaken.
• The intersection of the survey with internal QA processes, including internal surveys.
• The link of survey outcomes with quality.
• The educational justification of publishing the result.
• The contrast between the apparent “carrot for research” and “stick for teaching” approaches.
• The added-value of the survey for universities, who are unlikely to be affected by any funding decisions related to survey outcomes.
• The need for rock-solid methodology and pedagogy, as the survey will attract international attention.

There was a general view that the survey issue:

• could be usefully re-visited by the Vice-Chancellors, and
• is likely to be on the table at QEM11 as we may be in the throes of discussing the results!