The audit process

The New Zealand Universities Academic Audit Unit processes are fully detailed in the relevant academic audit manual.  The most recent manual is the December 2002 Academic audit manual, which was written for the Cycle 3 audits administered in 2003-2006.  A new manual is being written for Cycle 4 audits, and should be available by late October 2007. 

The Unit is regularly seeking, and receiving, feedback and comments on the quality of the academic audit process and the performance of the Unit.  In July 2007, the Board of the Unit released a statement on the 2008-2012 Cycle 4 academic audits of New Zealand universities.  That statement  summarises the changes and refinements and streamlining to be made to the process for Cycle 4 built on the experience of the university sector gained over three Cycles of academic audit.

The information below is a brief synopsis intended as an overview of the process, and to highlight important aspects of audit.

The process, highly simplified. (Click on the green buttons)

Audit overview

"In each audit, the process begins and ends with the institution's own programme of continuous improvement.  External audit begins within the institution with a self-assessment which is not just an evaluation but which includes the institution’s proposed quality improvement plans for moving forward.  The audit panel’s thinking and recommendations begin from, and move out from, the institution’s own quality improvement plans.  The institution’s implementation of its own improvement plans, as modified and enhanced by the audit panel, forms the basis of the follow-up interaction between the institution and the Unit.  The Unit and its work – both external audit and ongoing professional interaction – must add value to the institution’s own programme of quality improvement.  Audit must be owned as part of the institution’s programme of quality improvement and not experienced as an intrusion into its internal life; audit must not ignore the institution’s own initiatives or destroy the institution’s own creativity."
Extract from Academic audit manual

back

Principles for institutional quality systems

The Unit's approach to institutional audits is based on three principles –  partnership, ownership and enhancement.

• Partnership Institutional academic audit is a partnership between the Unit and the University. Audit panels include academics who provide peer review of materials offered. Audit panels also include non-academics, many of whom are involved in quality systems related to education outside the university sector; these people bring experience from outside the sector to bear on issues inside the sector. Taken as a team, the panel provides independent advice to the University about the University's own programme of self-improvement. Audit panels no longer see themselves as experts telling the University what to do. Audit panels have expertise and experience, but they see themselves as colleagues, working with the quality personnel inside the University to encourage them in their work to enhance the research, teaching, community engagement and student learning experience.
• Ownership The Unit accepts that the process of academic audit as administered by the Unit is owned by the Unit. However, the Unit is more interested in the processes of audit that lead to change, and those processes must be owned by the University. The work of the Unit, most significantly in the report and its commendations, recommendations and affirmations, is but one contribution to the whole. The work of the Unit must feed into processes of change that are owned by the University.
• Enhancement The word enhancement carries with it the notion of enhancing processes already in place in the University, rather than just imposing processes from a different context. Enhancement is about improvement and innovation, and audit teams look for ways of building on the processes already in place in a University.

The Unit is insistent that:

• academic quality is best guaranteed when responsibility for it is located as close as possible to the academic processes of research, teaching, learning and community service,
• quality assurance is a continuous, active and responsive process,
• a quality system is effective if it results in the achievement of quality delivery and quality outcomes,
• effective quality assurance in higher education requires the use of external academic and professional points of reference.
• The Unit emphasises self-assessment as a mechanism to support these principles and achieve accountability. There is an expectation that reviews of processes should be an integral part of academic planning.

back

Self-assessment

The scope of the self-assessment will depend on the scope of the audit, and the starting point for the self-assessment is the objectives that the institution has formulated.  Universities will manage their self-assessment, and the portfolio will be carefully prescribed.   The self-assessment will be carried out in the way best suited to the University’s needs and will build to the greatest extent possible on the on-going review activities of the universities.
The self-assessment will consider the extent to which there has been progress towards achieving objectives, and the effectiveness of processes that support activities related to the objectives.    Typical questions that a self-assessment addresses are as follows.

Goals and objectives

• What are the goals, objectives and planned strategies in this area?
• How were these goals, objectives and processes determined?

Progress

• What positive progress has been made in achieving the goals and objectives in this area?
• What is the relevant output/outcome data that illustrates progress?
• What key quality mechanisms and processes have been used in achieving progress?
• How effective have been the processes and activities in this area in achieving goals and objectives?
• What are the key strengths in this area?

Enhancement activities

• What are the areas in which enhancement is needed?
• What enhancement activities will be undertaken during the next planning period – say, three years?  Who will be responsible and what are the expected outputs and outcomes of those enhancement activities
• How will the University monitor the effectiveness of changes arising from the enhancement activities?

back

Self-assessment portfolio

Within the focus of the audit, the portfolio outlines the institution's quality systems, its own appreciation of the effectiveness of the systems, and its plans for quality improvement.
A portfolio for Cycle 4:

• is a concise, single document from the institution and contains judgments of the autonomous institutional community including the various sectors of the institutional communities of interest,
• contains statements of progress on the recommendations in the Cycle 3 audit report and on the university’s own enhancement initiatives identified in the Cycle 3 portfolio,
• will follow the structure and framework of open-ended questions set down in the audit manual,
• will be limited in Cycle 4 to 15,000 words with one A4 storage box equivalent of supporting reports and evidence generated by institutional processes and reviews.

back

Audit panel

The panel is selected from the Unit's Register of auditors, bearing in mind the characteristics of the institution and the focus of the audit. The final panel is agreed to in discussion with the institution.   The audit panel consults widely to test the extent to which the portfolio contains fair judgments.  In evaluating and checking the institution's self-review portfolio the audit panel will respect the objectives and values of the institution.
The role of the audit panel is to assist the institution by:

• validating conclusions and supporting and/or enhancing proposed improvement activities,
• contributing to the self-learning process by identifying aspects that require attention.

The panel evaluates the self-assessment portfolio and may request further information only where there is good reason to do so.  A clear indication will be given as to why further information is required, and further information requested will not be forwarded to the panel prior to the site visit as in past Cycles, but will be made available at the Site Visit.

back

The Site Visit

In Cycle 4, the panel will conduct a four-day Site Visit with the aim of conducting (normally) two days of interviews with a range of people from the institution and other communities of interest.  The audit panel prepares for the visit by identifying specific topics and areas for investigation, based on its reading of the portfolio and supporting documents.
The aim of the interviews is to gain clarification on issues relating to the effectiveness of systems for monitoring and enhancing academic quality.  In Cycle 4, interviews will be more carefully targeted, and to the extent possible and practicable, the main topics and/or issues to be discussed at interviews will be made clear prior to the site visit.  Wherever possible, the context and rationale underlining questions will be explained by the panel; but panels must be free to ask questions about other matters.  To the extent possible, all participants in interviews will have the chance to be heard.
The panel first reports orally to senior management at the end of the Site Visit.

back

The audit report

The audit report contains the findings of the panel based on what the panel has read and heard.  It will include commendations, recommendations and affirmations, and it will aim to assist the institution in improving its own programme of continuous quality improvement.

The formal Audit Report is first drafted by the Unit, and then refined by an iteration process involving panel members, and the institution (to ensure factual accuracy).  The draft report is sent to the University for comment on errors of fact, tone and expression, and the University is invited to make management responses to the recommendations.  The final report is authorised by the New Zealand Academic Audit Unit Board, is then sent to the University, and is released a fortnight later as a public document.

back