ADVANCING BEYOND REGULARITY: DEVELOPMENTS IN VALUE FOR MONEY METHODS AT THE NATIONAL AUDIT OFFICE 1984-1999 A thesis submitted for the degree of Doctor of Philosophy by Jeremy Stephen John Lonsdale Department of Government, Brunel University June 2000 BRUNEL UNIVERSITY PUBLICATION OF THESES Please complete in block capitals NAME: -a; - g-6A y DEPARTMENT: V G)Z N AA GN I- -tjb ELOM TITLE OF THESIS: al"N Cf tj CT 61DI 0 =vC N vftVG FDC- AAoWýýY M12-r-hbbS AFqti'E NA-'Ilw-f? t'L- A-u)) I -r DF-R Ce Iq R4 - tqq,? / PLEASE COMPLETE EITHE PART A OR PART B A. I agree that the abstract of my thesis may B. be published by the University without further reference to me. In accordance with the University's Handbook of Procedures, the Head of Library Services may allow my thesis to be copied in whole or in part without further reference to me. Such Authority shall apply only to single copies made for study purposes and shall be subject to normal conditions of acknowledgement. (See notes overleaO Signatu'ýO- Date: I request that my thesis be held under confidential cover in thelibrary for a period of _ years for the following reasons: Once the period of confidentiality has expired, I agree that the conditions set down in Part A will apply without furl reference to me. Signature: Date: Please pass this form to should sign below inýl the retention of t es cover. ý6 y6ur supervisor who ating his/her agreement to s under confidential Signature cýf Supervisor: comments: Signature (Senior Assistant Registrar) If you have completed Part A please return Part B. when complete. should be returned to this form direct to the Library with evidence the Senior Assistant Registrar (Records & of your award. Systems) in the Registry. For the attention of candidates who have completed Part A i) Attention is drawn to the fact that the copyright of a thesis rests with its author. ', A copy of ýcandidate's thesis is'supplied to the Library on condition that anyone who consults it is understood to recognise that its copyright rests with its author and that no quotation from the thesis and no information derived from it may be published Without the prior written consent of the author or the Uniyersity, as ýppropýiate. Requests for such permission should be addressed in the first instance to the Head of Library Services. 1ý1 ADVANCING BEYOND REGULARITY: DEVELOPMENTS IN VALUE FOR MONEY METHODS AT THE NATIONAL AUDIT OFFICE 1984-1999 A thesis submitted for the degree of Doctor of Philosophy by Jeremy Stephen John Lonsdale Department of Government, Brunel University June 2000 ABSTRACT This thesis examines the development of value for money (VFM) audit methods used by the National Audit Office (NAO) and considers what factors have influenced the identified changes. It also considers how developments compare with those elsewhere in Europe. The thesis is based on examination of more than 300 NAO reports, interviews with senior staff, focus groups, a thorough review of relevant literature, and comparisons with four other state audit offices. The thesis argues that VFM audit has developed into a strong form of evaluative activity, despite a number of constraints. The NAO enjoys a unique combination of the statutory power to initiate wide-ranging reviews; the right to demand documents and gain access to people; and the existence of a powerful parliamentary committee to review its work and ensure a government response to reports. Additionally, during the 1980s and 1990s, the NAO developed its VFM work, and in particular, the methods used, to become a more robust discipline. Although document review and interviews form the basis of much of the work, other methods have been added to the core repertoire, in particular, questionnaires. In addition, a broader range of methods has been used on occasions, from focus groups to systems dynamics. These developments are explained in terms of a response to broader changes in the administrative environment, in particular, new public management reforms. These have increased interest in public sector performance, enhanced the quality of information available, and given greater emphasis to service quality. Consequently, the NAO has sought to demonstrate it can 'add value' and encourage beneficial improvements to public services. Developments in methods are seen as one important way of doing this. The thesis places developments in the context of the growth of evaluative activities in the UK, whilst emphasising the particular focus of VFM work on accountability, and its links with financial audit. CONTENTS Acknowledgments 4 Index of figures 5 List of abbreviations 6 Introduction 7 VFM methods - definitions and significance Research questions and summary of methods The administrative context VFM work and other evaluative activity compared Conclusion 2 Review of related literature 18 Literature on evaluation research Literature on the development of state audit The 'audit explosion' Literature on the nature of VFM audit Audit and Evaluation Literature on the practice of VFM audit and the design of audit examinations Literature providing international comparisons Significance for the thesis Gaps in the literature 3: Methodology 43 Research design Validity and reliability 4: The development of Value for Money audit practice at the Exchequer and Audit Department and National Audit Office 54 The development of the E&AD and the NAO Development of VFM practice 'Pre-history' 'The middle ages' 'Modem times' - the passing of the National Audit Act 1983 Main developments in VFM work during the 1980s and I ý90s Comparison with other developments Other evaluatory bodies Conclusions about the development of VFM audit practice 5 Methods used in National Audit Office Value for Money studies 89 The range of evaluation methods The background to VFM methods How does the NAO portray its use of methods ? Methods used by the NAO 1984-1998 Features of each period Major trends in the use of methods for VFM work Conclusions 6 Influences on the choice of Value for Money methods 119 NAO auditors' perceptions of changes in methods The nature of the auditor's choice Auditors' approaches to choosing methods Influences on the methods chosen by VFM auditors at the NAO Factors encouraging the use of a wider range of methods Constraints on development of a wider range of VFM methods Conclusions 7 Value for Money audit methods : an international comparison 150 Comparative research - Potential and problems Nature of the states considered The development of VFM audit outside the United Kingdom Examination of methods employed by SAls in the 1980s and 1990s Analysis and discussion How does the way in which other SAls characterise their use of Methods compare with NAO ? How does the range and type of methods used by other SAls compare with the NAO ? How do the factors influencing the choice of methods compare with those in the UK ? How does the coverage of methods in the reports of SAls compare with the NAO ? Discussion Interpretation Overall conclusion 8 Value for Money methods and the nature of Value for Money work 188 What changes have taken place in the selection of VFM methods in the NAO during the 1980s and 1990s ? What are the factors that explain these changes ? How do developments in the NAO's use of methods compare with those which have taken place within the VFM work of other SAIs ? Discussion The impact of developments in methods Implications for the future of VFM Notes a 203 Appendix 1: Background to the five Supreme Audit Institutions 208 Appendix 2: Brief history of the development of performance audit in each Supreme Audit Institution 213 Bibliography 221 ACKNOWLEDGEMENTS I would like to thank my supervisor, Professor Christopher Pollitt, formerly of Brunel University and now Professor of Public Management at Erasmus University, Rotterdam, for all his help and advice. I would like to thank Dr Justin Keen, National Audit Office Fellow in Audit and Evaluation 1996-1998, for his constructive comments throughout my work. I would also like to acknowledge the considerable assistance of Professor Maurice Kogan in the final stages of preparation of this thesis. I would like to acknowledge the work of my colleagues on the Performance Audit Project team - Professor Christopher Pollitt, Dr Hilkka Summa, Robert Mul, Marit Waerness and Xavier Gierre. Our work together between 1996 and 1999 was both fruitful and enjoyable. Where I have drawn on the work of these colleagues, it is acknowledged in the text. Our book Performance or Compliance?: Performance Audit and Public Management in Five Countries is closely linked with the subject matter of this thesis. In particular, I have drawn on my work for Chapter 8 'Methods', and, to a lesser extent, on Chapter 7 'The Auditor's Craft' (written jointly with Professor Pol litt and Robert Mul) and Chapter 10 'Impacts'. Appendices I and 2 of this thesis have been published before in Performance or Compliance? I have been sponsored during my time at Brunei University by the National Audit Office. I would I ike to acknowledge the support of Martin Pfleger (Deputy Auditor General) and Caroline Mawhood (Assistant Auditor General) since 1996.1 would like to thank numerous current and former colleagues for offering their insights into VFM work, for participating in focus groups and interviews, and for commenting on draft chapters. 4 INDEX OF FIGURES Figure 1.1 VFM audit and other evaluative activities compared Figure 3.1 Influences on VFM methods Figure 4.1 Stages in the VFM audit process at the National Audit Office Figure 4.2 Proportion of NAO resources spent on VFM work 1991-92 to 1998-99 Figure 4.3 The approaches of different evaluatory bodies Figure 4.4 Changes in the characteristics of VFM work at the E&AD and NAO Figure 5.1 Potential range of characteristics of evaluation methods Figure 5.2 Methods employed in NAO reports 1986-1998 Figure 6.1 The key players influencing the VFM team's choice of methods Figure 6.2 Players encouraging the use of a wider range of methods used by VFM auditors Figure 6.3 Factors constraining the use of a wider range of VFM methods Figure 6.4 Applying methods to possible roles played by VFM auditors Figure 7.1 Characteristics of the government of five states Figure 7.2 Key aspects of VFM audit in five countries Figure 7.3 Methods used by four SAls during the 1980s and 1990s Figure 7.4 Characteristics of audit arrangements in five countries PART ONE: INTRODUCTION This thesis examines the development of value for money (VFM) audit methods used by the National Audit Office (NAO), the largest central government audit body in the United Kingdom (UK)', and considers the factors that have influenced the changes which have taken place. It also considers how these developments compare with those in a number of other European national audit offices. VFM (or performance 2) audit has been defined as 'an audit of the economy, efficiency, and effectiveness with which the audited entity uses its resources in carrying out its responsibilities' (INTOSA13 1992). It is generally associated with examination of these criteria, although some have noted that VFM work can consist of more or less than this (Keen 1998b, Summa in Pollitt et al 1999). Thus, VFM studies may consider, for example, issues of good management practice or the quality of public services. VFM work is complementary to the audit of financial statementS4 , rather than an alternative to it, with the results summarised in publicly available reports. Often, the purpose of these reports is to provide a basis for further consideration of a particular subject - perhaps through a parliamentary hearing or a special commission. In theory at least, there is an expectation that, where weaknesses are identified by auditors, action will be taken by Government to revise practices or policies. VFM is often described as a 'modem' form of audit. Increased attention to obtaining the most from public expenditure is generally taken to be a post-World War Il phenomenon (Dewar 1991), and VFM audit has been particularly prevalent, for example, in Sweden from the 1960s, and in the UK, in its current form, from the early 1980s, having been entrenched in legislation by the National Audit Act 19835. Whilst not universally practised by Supreme Audit Institutions (SAls 6), there has been a tendency towards expansion of VFM work, both geographically - for example, 1993 legislation provided for VFM audit in Ireland - and in terms of breadth of coverage, for example, in the Netherlands, where 1989 legislation extended VFM powers to cover social security expenditure. Where well established, there has been a tendency towards the creation of a VFM audit specialisation in a number of SAIs. Thus, not withstanding concerns that VFM 'has lost its way' (Bowerman 1996), there is much evidence it has consolidated its 7 position and become a permanent feature of accountability arrangements in a number of countries. VFM methods - definitions and significance Despite this apparently secure status, VFM audit is far from a static and unchanging activity. One recent observer (Power 1997: 5 1) has suggested that VFM audit practice is 4a constantly evolving construct shaped by an available repertoire of technologies and by the programmatic demands and assumptions which are made about its capabilities'. Another (Keen 1998b) has written of the 'robustness and versatility of the technical side of VFM work'. It is the evolution of this technical side of VFM and, in particular, the gathering of evidence that is the focus of this thesis. The collection and examination of evidence are fundamental parts of audit in whatever form it takes. As one authority has put it 'In no case can an auditor express an opinion or make a report without making an investigation and obtaining evidence' (Flint 1988). Thus, most audit work is directed at enabling the auditor to exercise a judgement that can be reported, and only by collecting and analysing evidence can such judgements be made. Yet, to be of value, the evidence must satisfy a number of conditions. To Flint it must be, variously, 'relevant' 'competent' and 'Persuasive', and often supported (or at least not contradicted) by other evidence. And it must have been collected and analysed using methods recognisable to others. At this stage, it is appropriate to consider what we mean by 'methods'. One general social science text describes them as 'the tools of data generation and analysis' (Sarantakos 1994 : 33), and another refers to methods being 'research procedures actually employed' (Melia 1997). One observer of VFM, Keen (1999a) has cornmented that 'Methods are, at base, what you use to uncover the truth about what is going on. VFM practitioners employ a variety of methods to arrive at judgements about specific points within studies. ' For this study, methods are taken to be the means by which data are produced and analysed to allowjudgements to be made about the activities under examination. They are regarded as stand-alone, discrete activities. The methods employed by VFM auditors cannot be considered in isolation. Those chosen will depend on a number of factors. First, the type of questions posed by auditors will influence the methods employed. Keen (I 998a) suggests that examinations 8 of effectiveness usually require significantly different approaches from studies of a more descriptive nature. Secondly, the methods employed are likely to influence the nature and extent of the impact of the work. Where, for example, methods generate original data, in many cases they are likely to be more penetrating than reliance mainly on the analysis of internal documents. Thirdly, VFM auditors do not work in a vacuum. Their background and experience are likely to influence their choice of ways to tackle a particular problem. And they are subject to outside influences. Others, such as management consultants, evaluators or academics, examine the activities of public bodies from a number of different perspectives, and their approaches have attracted the attention of VFM auditors. The methods used will, therefore, be influenced by who is doing the work, what they are looking at, and what they are trying to achieve. Finally, consideration of the methods used by VFM auditors is of more than just passing internal interest for SAls, or the limited number of outsiders who study their work. VFM reports have become a significant element of parliamentary democracy in the United Kingdom. During 1999, for example, many of the most senior civil servants in the country (as well as several senior employees of private sector firms employed by Government, such as Siemens Business Systems or Anderson Consulting) appeared before the Committee of Public Accounts of the House of Commons 7 (PAC) to answer questions on the evidence contained in NAO reports. Details of many of these reports made front page headlines in national newspapers. Yet, despite the controversial subject matter in many of the reports, most press coverage assumed, without question, that methods were appropriate and properly executed. Those the subject of the reports - often the recipients of criticism about their performance or conduct - likewise tended not to question, in the parliamentary hearings, how the NAO had reached its conclusions. That this is the case, adds to the significance of knowing more about the methods used by auditors. Research questions and summary of methods The overall aim of this study is to examine what methods have been used by audit teams at the NAO and why they have been chosen. At the outset, the research was guided by three specific questions. These are: 9 What changes have taken place in the selection of VFM methods in the NAO during the 1980s and 1990s ? What are the factors that explain these changes ? How do developments in the NAO's use of methods compare with those which have taken place within the VFM work of other SAIs ? The research questions are informed by both personal experience and previous consideration of the area. The author has been employed on VFM work for much of the 1990s, a period during which NAO staff seem to have been encouraged to look for new ways of carrying out their work. The impression gained was that teams appear to have responded by making use of a number of new methods, whilst continuing to draw heavily on more traditional means of obtaining evidence - namely interviews and document examination. The thesis will examine whether these impressions are correct. To research this subject a number of methods have been employed to form an integrated approach to answering the questions. These are: e An examination of the history of VFM work in the NAO, based on internal documents, interviews, and published material, designed to provide a historical context for a discussion of methods. This forms the basis for Part 4. An analysis of the methods used by the NAO, based on a review of over 300 VFM reports published between 1984 and 1998. This will provide the bulk of the evidence for how VFM methods have changed over time contained in Part 5. An examination of the factors affecting the choice of methods, based around focus groups of some 30 NAO VFM auditors (about 40% of the total of experienced VFM staff), as well as interviews with senior NAO staff. This will be supplemented by an examination of internal material, and is the subject of Part 6. 9 An international comparison of the methods used during the 1980s and 1990s, as well as consideration of the factors that have influenced these choices. This is based on interviews, a review of literature, and an analysis of VFM audit reports from four other national audit offices. This is the subject of Part 7. 10 The evidence generated by these methods will allow for a number of comparisons - between VFM work now and in the past (Part 4); between NAO rhetoric about methods and the actuality of what has been used (Part 5); between the principles which underpin this work and the reality of audit practice (Part 6); and between the NAO and a number of other national audit offices (Part 7). The focus of this study is mainly the NAO, the comparison with other SAls made, in particular, to enable additional insights to be made concerning the development of its work. The choice of comparator SAls is discussed in Part 3. The administrative context VFM work in the UK has not grown up in isolation. Instead, it developed during the 1980s and 1990s, a period in which, as Henkel (1991: 20) notes, governments stressed the 'values of economy, efficiency, 'value for money' and effectiveness or performance. ' Pollitt and Bouckaert (2000: 273) highlight the interest in the UK in the early 1980s in improving financial and general management in government, and increasing efficiency, noting the establishment of the Financial Management Initiative, which embraced 'the whole of central government with its philosophy of more decentralised management, more decentralised budgets, more targets and more professionalism. ' They link the establishment of the NAO and the Audit Commissiong in the early 1980s with these broader changes, alongside, amongst other things, the spread of performance indicator systems, and the introduction of general managers into the health service. Martin and Sanderson (1999: 246), too, note the efforts made to instil more business-like management into the public sector over the final two decades of the century. In their view, this 'typically consisted of a cocktail of cost reductions, devolution of management responsibilities, 'neo-Taylorist' performance management systems, new market and quasi-market mechanisms, greater 'customer focus' and new service standards. ' Collectively, such developments have been labelled 'new public management'9 (Hood 199 1, Pollitt and Bouckaert 2000). Some have characterised the 1990s as a period that saw the development of the 6 regulatory state' (Hood et al 1999). They argue that during this time, inspectors, regulators, ombudsmen and auditors all increased their activities in scale and complexity. Others have referred to the rise of 'The Evaluative State' (Neave 1988, 1998, Henkel 1991), characterising it as 'the replacement or .... the strengthening of central controls by evaluation systems which go alongside the ex ante avocation of objectives, targets and plans in place of forms of ex post self-evaluation administered by the professional leadership of the service'(CEPPP 1999: 5). Such developments have taken place over a number of years, but appear to have been given renewed emphasis under the Labour Government, elected in 1997. The Modernising Government White Paper (Cabinet Office 1999) refers, inter alia, to the introduction of 'Peer review' of Departments, the development of monitoring of performance measures, the benchmarking of progress against targets, and Public Service Agreementslo. At the same time, a number of new oversight bodies - for example, the Commission for Health Improvement for the National Health Service - have been established (Ward 2000), or are planned. Such developments have taken place against a background of broader cultural changes. Henkel (1991: 15) notes a number of trends from the late 1970s, including diminishing confidence in the validity and usefulness of professional knowledge, growing demands for professionals to be more publicly accountable, and increased efforts to locate accountability more precisely within public organisations, and locate responsibility for performance as far as possible with individuals. Henkel also suggests that the period saw 'shifts in the meaning and operation of public accountability', with it increasingly &conceived as being primarily to taxpayers and secondly to individual users of services, mirroring the shareholders and customers of private enterprise' (Henkel 1991: 16). Another important factor was the strong and intensely ideological anti-public sector ethos of the Conservative governments elected during the 1980s and early 1990s, which challenged the idea of the role of the public sector within socicty. Hughes (1998) has suggested that there were three parts to the attack: on the scale of the public sector, on its scope, and on the methods of government. Consequently, for a period of more than a decade, the public sector was depicted as too big, involved in too many aspects of society, and bureaucratic and inefficient in approach. Those who held such views were sympathetic to efforts to reduce costs and introduce private-sector approaches. Linked to this, and perhaps a consequence of the changes described above, was a growing concern during the 1990s that standards in public life in the UK were being 12 undermined. This view was definitively expressed in a report by the Committee of Public Accounts in 1994 (Committee of Public Accounts 1994b). This drew on 20 cases from previous Committee reports to highlight what they described as departures 'from the standards of public conduct which have mainly been established during the past 140 years'. The report stated clearly that, whilst it was right that public servants sought to make a success of the new ways of delivering public services, it was also important to maintain standards in conducting public business. Lapses included 'artificial' retirement settlements, inadequate financial controls, and poor stewardship of public funds. The concerns expressed in the report had a significant influence on the civil service in the second half of the 1990s and, when added to high profile ministerial resignations for a variety of misdemeanours collectively termed 'sleaze', ftielled the calls for strong accountability arrangements in government at all levels. As a result, the need for new or improved oversight arrangements was supported by those taking quite different stances towards the role of government. For those concerned about making the public sector more like the private, the fact that the state was withdrawing to more of a supervisory role meant that audit provided 'ways of redesigning the practice of government' (Power 1997: 11). Oversight could still be maintained even after the state withdrew from direct control. For those keen to ensure that public funds and standards of conduct were safeguarded despite the changes in service delivery, audit and accountability arrangements gave promise of continuing scrutiny in the new world. To all these parties, evaluation was considered an element in government control processes. 'Evaluation was seen variously as providing tools to be used in the assertion of either centralised or local managerial accountability and control, as a component in programmes of change and as making it possible for public regulation to replace public provision as a symbol of collective responsibility' (Henkel 1991: 10). The underlying theory of knowledge was positivist, and there was an assumption that 'evaluation would be surnmative, delivering authoritative judgements, based as far as possible on performance indicators or quantitative measures of input- output relationships and outcomes and set against predefined targets and standards. ' (Henkel 1991: 20). In addition, the culture of management was based on economic concepts of rationality, with a supporting management technology drawn from accountancy. Taken together, these developments meant that the concerns of VFM 13 auditors - economy, efficiency, effectiveness, waste, propriety, and value for money - were central to the administrative culture of government for much of the 1980s and 1990S. VFM work and other evaluative activity compared It remains to locate VFM more specifically amongst other evaluative activities. Figure 1.1 compares VFM audit across a range of criteria with other types of work. It suggests there is much common ground between the activities. All are ways of identifying levels of performance, reaching judgements on that performance, and making pronouncements, often in public. All aim to secure performance improvements, and all are relatively formal, structured approaches, carried out by well informed or well trained people. But Figure 1.1 also suggests some differences. First, VFM audit, as carried out by the NAO, is not part of the processes that Government applies to itself, rather it is a way in which Parliament holds Government accountable. This could have implications for the conduct of VFM work, and the way in which auditor and auditee interact. Secondly, VFM work appears to have none of the standardisation of, for example, the OFSTED school inspection process (CEPPP1999). There is no systematic programme of VFM studies, and examinations are designed to meet each subject, making VFM a varied and tailored activity. This is likely to have implications for study methods. Another way to consider these activities is to examine what key attributes they require to enable them to succeed. Four important factors would seem to be: an ability to initiate and scope the work for themselves (Henkel 1998 talks of the potential constraints on evaluators where they are commissioned); the ability to gain access to relevant information; the extent of knowledge of a range of accepted ways of collecting and analysing information; and the availability of a means of cýnveying findings and conclusions to key audiences, and having them considered and acted upon. The absence of any one element seems likely to reduce the effectiveness of the process. Thus, if there are restrictions on what can be studied or on access to key information, the study is likely to be less authoritative and valuable. Additionally, a report may be published, but if its authors have no way of being heard, it may be ignored or overlooked. And an inappropriate range of methods may produce analysis or recommendations of limited use. 14 ýJ Ec t- cu r g0U., l M 9 ce m cu ei --- ts tu E (=i u CA 0= c2. 15 = -3 E E 15 :p u cn :i 0 i-. E 0 0 j2,3 m wM U= .. 0. -V72 ce zi 0ä 2i - (U . ý: L2 ir. x0 lij (42 (JO &. ID 0mmm ci 0 0u. ci ID 00 Zx u90u iz &ý -5 A lE m tä iz oE t §=E - E - > > u ci. 0 12 fi -a lag (n "0 Z 0, u rm rj -0 kl. u Gn 1C2 x CP u u CL u0 im . , . 4. V) .-M -M .5 r_ Ili 0 ýA »9 *0 Z- c ur. 0 A tu M u0 0 -W 'ý; A .2 u oh Co rz m0 to, 00 cie M Z r - ob c2., ý > to .2 Co , 0 (4 r. 0 - a Z. 2 u 4. '. 00 to ,- E -0 ;, 3 , r- = C cj t: V) 92.. 3 r- u cu j 2 .2 - 0 ci 0 n. r3 Q> Co tu U CU tn > G Ob 2 ö c 20w rj n. uA0 .F-e ci 0 r. *5: A 2c- cu .c 1 e iz iz 0 r_ d v. 0 Co ý 2 lij 5uu2 uuEa u -d = >:, c71 u- L 0 r- U0 (n = u .2M r- 10 vi 2 le - exa uE E9) .2 > :2 E 00E> ri E > :2eE > ß. «g ', '5 u 2ý %. am>0- e e. ý= , *3 . iý, -2 iz >ý 2 -tý EU L2 *Z cn CD. E 0 u 0 -0 s 5 3 > CJ u0 .-j =0 "r m. xu LU = 0 Zu E L, 2 Ei uu- *0 m0 cj c2. 116 00ý- ,' 22 2 - Cl. e 2 l> 15 > 0 E E 0 c7, E! u 2 > . - 0 cn " r- -" CA ei (n - < tn CA u t- >, = .2 C- 0. r_ 0 e Z .2 -0 r- 0U- "CJ E im -21 cm uUu u V) 926 u GM = -k r- -ig im uUd (2-1 cu IJ cl .M M. 2 0 r- = .k E 2> LW . 52 g0 Ei .g . > 0 fi Eu"- r 04 cu Co < uu 5 q= 92 (A u "ti ;; r (A ý -m 2ý to to 40ý LL. iz u 0 eu 12 2 ýL JD g= ci -<-1 Z- 1 > LU 4z E c26 >0 91. ci = CL. - 93 gi &0 ci cu Z lw 6 .0 0 u Z 00 0 z cr, C7ý 0% as cs 17, Using such dimensions, Figure 1.1 suggests there is considerable potential strength in NAO VFM work. In particular, it appears to offer the possibility of independent, wide- ranging studies being produced for consideration by a powerful body. This body has influence and a way of requiring attention from the relevant authorities. This is due, in particular, to the combination of (a) strong powers of access to documents and people under the National Audit Act 1983; (b) a systematic process through which VFM reports are considered by the Committee of Public Accounts of the House of Commons, and government is required to respond to recommendations; and (c ) the ability to initiate examinations on major subjects under the broadly worded 1983 Act. The question then remains, to what extent the final element - the methods available - complements these other features. This is the main focus of this thesis, for which the three other elements provide much of the context. Figure 1.1 also suggests that VFM may have a combination of strengths not seen in other forms of evaluative activity. While some evaluations may be undertaken by teams with greater expertise, they often lack the processes for ensuring their work is considered, resulting in significant amounts of wasted effort (Pawson and Tilley 1997), and disappointment about lack of utilisation (Rossi and Freeman 1993). While inspectors may have similar powers of access, they tend not to have the ability to initiate such wide-ranging reviews. And while peer review may involve a collaborative and less confrontational approach than VFM work (often considered more likely to effect change), it can lack the processes to ensure that action is taken to improve performance. At the same time, however, the VFM approach is far from universally admired. Some have described it as anachronistic and adversarial, and have suggested that too few examinations lead to follow-up exercises to assess what action has been taken (Walker 1999). At other times, the NAO's VFM has been compared unfavourably with the studies of the Audit Commission (Travers 1999). These comparisons - favourable or otherwise - all help to identify the strengths and weaknesses of VFM work, be they related to the process in which it is undertaken, the nature of the topics under investigation, the skills of those carrying forward the work, and the type of methods employed. These themes are taken up throughout the thesis. 16 Conclusion Whatever the perceived strengths and weaknesses of VFM work, even its critics appear to accept that the 'never-ending search for greater effectiveness in spending public money [is] here to stay' (Kemp 1996). This study comes at a time when the activities of audit offices have never been accorded more publicity and attention, and when encouragement from legislators and governments for public bodies to ensure they achieve 'value for money' has never been greater. But it also comes at a time when the impact of audit, including whether it is a constraint on the pursuit of risk taking and innovation by civil servants, has become a matter of discussion (Public Audit Forum 199911, Cabinet Office 2000). In these circumstances, how auditors undertake their work, and what methods they use, are matters of significance. To study this subject, the thesis is structured as follows: e In Part Two, the existing relevant literature on audit, and in particular, VFM audit, is reviewed to draw out issues of significance to the thesis. In Part Three, the methodology used is explained and justified. In Part Four the historical development of VFM audit in the United Kingdom is examined to show the key features which have shaped its progress. Part Five provides the findings from the author's analysis of the methods used in more than 300 NAO examinations between 1984 and 1998. Part Six considers what has influenced the choices of VFM methods, based on extensive discussions with VFM auditors. In Part Seven, the developments in the United Kingd6m are compared with those in several other SAIs. Part Eight draws together and discusses the main messages from the earlier chapters and concludes on the implications of the developments for the NAO. 17 PART TWO: REVIEW OF RELATED LITERATURE Introduction This part examines the literature relevant to this thesis. The use of secondary sources in qualitative research has been considered elsewhere (eg. Strauss and Corbin 1990, Cresswell 1994, Maxwell 1996). There are dangers in allowing such literature to constrain or overly direct research questions. Account has thus been taken of advice that the researcher should not defer to existing literature as an authority, but rather treat it as 'a fallible source of ideas about what's going on' (Maxwell 1996: 27). The literature has been used along the lines outlined in Strauss and Corbin (1990), in particular, to stimulate theoretical sensitivity and enhance existing awareness of issues gained from personal experience. It has also been used to generate questions for interviews and focus groups. The purpose of this chapter is, therefore, to: summarise existing knowledge as background to Parts 4-7; draw out key themes of possible relevance to the thesis, to be followed up in discussions and interviews; and highlight certain gaps in the previous examination of VFM audit. To date, there has been relatively limited academic attention given to VFM audit. In contrast, the technical literature prepared by SAls is quite extensive. For the purposes of this chapter, however, this material is not considered, but instead drawn upon, where appropriate, in the remaining chapters. The academic literature is divided into the following parts: an overview of evaluative research; general work on the development of state audit; consideration of the nature of VFM audit; literature relating audit and evaluation; literature on the practice of VFM audit, the design of VFM studies and the methods of VFM work; and comparative literature. Literature on evaluative research Much has been written about the developments in evaluative research (see for example, Guba and Lincoln 1989, Henkel 1991 and 1998, Pawson and Tilley 1997). Many of the themes and debates contained within the literature have significance for VFM work. 18 Guba and Lincoln (1989) characterise developments in terms of four generations of evaluation. Initially, evaluations sought to make generalised judgements about the impact of programmes, based on randomised controlled experiments. They took a positivist position that things could be measured, and that the job of evaluation was to establish as clearly as possible what was happening using expertise and technical skills. They, assumed 'that social phenomena could be divorced from their context and that objective knowledge about them could be achieved through empirical observation and quantitatively expressed' (Henkel 1991: 20). The second generation - quantitative description - focused more on relationships between actions and performance. Proponents of this position argued that it was insufficient simply to measure, and greater attention was paid to factors affecting performance. It was 'an approach characterized by description of patterns of strengths and weaknesses with respect to certain stated objectives' (Guba and Lincoln 1989: 28). Subsequently, criticism of the scientific approach characteristic of both these approaches grew. Henkel (1998: 287) notes that 'The idea that understanding of social action required openness to the purposes and accounts of the actors and interpretation of them through dialogue challenged modes of evaluation based on observation, measurement and objectivism. '. Experimentation was criticised for failing to take account of the complexity of individual and social interaction, and of social problems. Others saw the approach as lacking external validity, and being naYve about organisational and political contexts in which policy making was taking place (Pawson and Tilley 1997). Another problem was the slow pace at which the experimental approach appeared to make progress. Increasingly, a more pragmatic and use-led model of evaluation was advocated in which 'research ought to be cpnstructed so that it is better able to be used in the actual processes of policy making' (Pawson and Tilley 1997: 12). The pragmatic approach favoured methodological skill and discipline, and saw success in the extent to which evaluations influenced policy making. It also suggested more modest evaluations. But, the approach raised concerns about the independence of evaluators and the extent to which their work was influenced by those commissioning the research. 19 Guba and Lincoln's third generation of evaluation is characterised by the making of judgements as part of the evaluation process. Under this approach, it was argued that it was not enough just to describe, but there was also an expectation that evaluators would form and offer judgements on performance, based on assessments against certain criteria agreed in advance. The evaluator 'assumed the role ofJudge, while retaining the earlier technical and descriptive functions as well. ' (Guba and Lincoln 1989: 30). , The last development was 'constructivism', or 'fourth generation' which argues that: 'initiatives and programs which go under the microscope cannot and should not be treated as 'independent variables', as 'things', as 'treatments', as 'dosages'. Rather, all social programs are constituted in complex processes of human understanding and interaction' (Pawson and Tilley 1997: 17). It draws on non-scientific research traditions in which stakeholders played a far more significant role in designing and carrying out evaluations. This put the evaluator in the position of negotiator and interpreter between the different perspectives of stakeholders (Guba and Lincoln 1989). The literature highlights a number of important issues for VFM work, of which three have been selected. First, there has been an overall shift from summative evaluation ('delivering authoritative judgements') to formative evaluation ('evaluation as a dialogue between evaluators and key actors which would generate self reflection, criticism and action for improvement among all parties' (Henkel 1998: 289)). Evaluators have sought to collaborate more with those evaluated to identify criteria and standards and to share judgements. Such an approach contrasts with the traditional line taken by auditors, who have generally sought to maintain a distance from those they are examining. Second is the issue of the position of evaluators within political systems (Henkel 1998: 290). Much evaluation is commissioned, and thus their independence is 'a negotiable, not an inalienable right'. Evaluators may operate within evaluative institutions - two examples given are inspectorates and audit offices - which have been given powers by government. In these circumstances, Henkel suggests it is easy for 'their political power to become the prime source of their influence, rather than their professional knowledge and authority. ' Evaluation may then become a coercive, rather than rational process. Both factors have a relevance to VFM work, which is designed to 20 deliver authoritative judgements, and which takes place within accountability arrangements supported by (often extensive) statutory powers and parliamentary conventions. Third, is the development of qualitative methodology in recent decades. This has provided new ways of approaching subjects which are far more open and exploratory, and which take place in more natural settings. It stresses interpretation, and allows those who are being examined to play a far greater part in the research. Such a development has raised new possibilities for all research work, including VFM audit, which had traditionally fought shy of 'softer' evidence. These issues will be picked up later in the thesis. Literature on the development of state audit Against this broad background, the remainder of this part focuses on the literature relating to audit. State audit has a long history in a number of countries (National Audit Office 1996a). Until recently, however, no major comparative examination had appeared since the 1960s, when Normanton (once employed at the Exchequer and Audit Department (E&AD) 12 , the NAO's forerunner) produced his examination of audit and accountability in the UK, USA, France, Germany, the Netherlands, Norway, Denmark and Belgium (Normanton 1966). Much has moved on since then, and there have been several significant changes in legislation affecting the SAls he examined, not least in the UK, where the 'National Audit Department' he advocated, came to pass in 1984, and where Treasury control over the work of the Comptroller and Auditor General 13 (C&AG) has been removed by Act of Parliament. Normanton's book retains more than academic interest. A number of themes have a continuing resonance. His concern that 'The [UK] urgently needs a coherent pattern of public accountability' (p422) has echoes in the debates in the late 1990s about the confused audit arrangements for non-departmental public bodies (Committee of Public Accounts 2000a and b). Equally, his belief that 'The executive needs to know more about what is achieved with the money it disburses' (p422) could be taken from late 1990s' discussions over resource accounting and performance measurement (Committee of Public Accounts 2000a and b). Even the title of one of Normanton's chapters - 'The Advance Beyond 'Regularity" - on auditors' growing interest in economy, efficiency 21 and effectiveness - is a reminder that the development of the state auditor's role ('the new mission' as he put it) has a long history. Many of Normanton's recommendations reflect concern at the limited role of the E&AD. He maintained that its concentration on regularity audit was unsatisfactory, arguing that 'control of regularity is not merely insufficient, but terribly inadequate in this epoch of heavy spending. ' He noted that other SAIs had been 'liberated from the cramping limitations of regularity', and had redirected their energies 'to make a growing contribution to the development of public administration and government as a whole -a role no other organ of the state could possibly make' (p103). Unsurprisingly, given his background, Normanton's vision of the role and potential of state audit is a positive one. He argued that, at its best, it was equal to any administrative work, and suggested that 'Its stamp and method are those of scholarly research. ' Undertaken with sensitivity, he concluded that the 'presence of such lively auditors, always in modest numbers and with no powers except the right of access to information, will prove a stimulus to rather than a drag upon the departments which they audit' (Normanton 1966: 420421). Belief in the importance of audit as an accountability mechanism is also reflected in more recent work. Lovell (1996a) has explored the role of state audit in the processes designed to maintain accountable government. He argues that the limitations of many of the other mechanisms (for example, ex ante parliamentary scrutiny of supply estimates), the high profile enjoyed by the PAC, and the rise of managerialism, have all 6significantly increased the importance of state audit' (p261). Additionally, he suggests, the lack of openness in government in the UK increases the importance of NAO reports as a source of information, as does the unsuitability for' holding governments accountable of mechanisms such as pressure groups and judicial review. White and Hollingsworth (1999), in a well-sourced review from a public law perspective, have examined the constitutional significance of public sector audit in Britain. They argue its importance has been understated, suggesting 'The system of external audit in British central government contributes to the democratic accountability of government by providing expert information to Parliament, which can be used to hold government to account. Audit also reinforces the internal system of managerial 22 accountability that operates within government. Of crucial constitutional importance is that audit links democratic and managerial accountability, allowing Parliament to a certain extent to participate in the real business of government' (p200). As they see it, 'democratic accountability is thus 'given real bite; audit allows Parliament to draw aside, at least partly, the veil of ministerial responsibility and to participate in the process by which government controls itself (pp 9-10). White and Hollingsworth see the extension of the C&AG's powers as necessary. Others have also identified areas where they consider the NAO could undertake more, or perhaps, different work. A more polemical work by Broadbent and Laughlin (1997) argues for a critical evaluation of 'new public management' reforms to assess their merit and worth. They suggest the constitutional position of the C&AG makes him well placed for this, although they note that he has tended not to undertake wide-ranging evaluations. They take issue with Roberts and Pollitt's analysis (see below) of the constraints facing the NAO in examining large controversial areas, although they accept that the legislative requirement not to question policy objectives and the political ramifications of exploring sensitive topics are significant restrictions. The 'audit explosion' Whilst several observers see scope for an expanding role for state audit as a key part of accountability arrangements, much recent literature has focused on a perceived 'audit explosion' - the rapid growth of audit and oversight in the 1980s and 1990s. Hood et al (1999) explore the scale and growth of regulation inside UK government (including that undertaken by the NAO) through empirical research. Overall, they suggest that regulation inside government has direct costs approaching il billion a year, with compliance costs imposed on those they are examining probabl; 'dwarfing' these costs (p42). Regulation has grown substantially in the public sector in terms of staff and spending over the last 20 years. They suggest that 'Iregulation seems (in many of its domains at least) to have grown in formality, complexity, intensity and specialization over the past two decades', a point which may have some relevance to later consideration of the development of VFM audit practice. They identify four main weaknesses in regulation, namely a lack of any forum for the exchange of good practice between regulators, a lack of exposure of regulators to competition, a lack of any clear 23 central responsibility for regulations, and an absence of snap inspections or random scrutinies, especially at the top and centre of government. They are particularly concerned at the apparent lack of co-ordination between the various authorities. Hood et al quote one executive agency chief executive who spoke of the 'many hobgoblins in the mist for the civil service' (p65). Hogwood, Judge and McVicar (1998) pick up this theme in their examination (from a rather one sided perspective given that they only consult with those subject to regulation) of what they call the 'pathology of accountability' or an 'accountability surfeit'. Although much of their paper deals with handling parliamentary questions and MPs' correspondence, scrutiny from the NAO is also discussed. Several Agency chief executives interviewed complained of the time- costs associated with audit investigations, which were considered to be part of the (repetitive' nature of scrutiny, overlapping with, and duplicating, a range of other reviews (p23). If Hood et al was designed to map out the terrain, Power's approach is theoretical, designed to raise questions. He starts from the position that in the late 1980s auditing 'for all the problems of image and status, was taking on new forms and being promoted in a wide variety of areas. ' His work, most notably Me Audit Society' (1997), identifies growth in the use of 'audit' in a wide variety of contexts - for example, financial audit, VFM audit, environmental audit, medical audit, quality assurance work - and argues that the explosion reflects 'a certain set of attitudes or cultural commitments to problem solving (p4)'. He goes on to suggest that it is the 'ftizziness in the idea of auditing that enables its migration and importation into a wide variety of organizational contexts', and an ambiguity about the process and its objectives which allows for considerable discretion on the part of the auditor as to what to db. He argues that VFM examinations 'are constituted by a certain vagueness which makes for considerable discretion in their implementation' (p 11). Power also links the 'explosion' of audit with wider 'new public management' developments, suggesting that 'Audit is not simply a solution to a technical problem; it also makes possible ways of redesigning the practice of government' (pl 1). Power describes audit as 'a collection of pragmatic and humble routines', which may help provide confidence about the truthfulness of statements made by auditees. And in the area of VFM work, he argues that such audits tend to prioritise 24 that which can be measured in economic terms - economy and efficiency - rather than effectiveness. I Literature on the nature of VFM audit Attention is now directed to the literature on VFM audit, in particular, what form of inquiry it is, the variety of VFM audit types, and the coverage of effectiveness. (a) The nature of the inquiry There are many definitions of VFM audit in the literature. Glynn describes it as 'a blend of both conventional auditing and management consulting' (Glynn 1985: 21). Others have characterised it as a broad, changeable mix, so that in 'dealing with efficiency, auditors may be functioning as if they were management consultants and when they deal with effectiveness, especially in terms of policy and program matters, they may be close to becoming policy analysts or indeed policy evaluators' (Gray, Jenkins and Segsworth 1993: 8). Sharkansky (1991) suggests various types of VFM audit are united by 'their concern with judging the quality of governmental activities, as opposed to the auditor's classic concern with financial records'(p2), while Power (1997) sees VFM auditing as 'an incremental repackaging and extension of existing audit capabilities ... an assembly in which new conceptions of audit have been grafted on to existing routines' (P50). Keen (1998b) argues that VFM work is retrospective, concerned with checking and verifying (rather than observing and explaining), and examines many issues separately, generally not combining data to support single, central conclusions. He further suggests that it focuses on activities associated with the three '3Es' (economy, efficiency and effectiveness), but does not generally report directly on them; uses concepts of system, control and error, though these are deeply encrusted with other material. It is also concerned with processes, rather than resource use or outcomes, and focuses on management issues, deals with systems or transactions, rather than the experiences or performance of individuals. The common assumption so far is that VFM work is audit. In contrast, Barzelay (1996, 1997) has suggested that it is not, arguing it is 'an evaluative activity of a particular sort whose name happens to include the word 'auditing" (Barzelay 1996: 19). He suggests 25 that VFM auditors, unlike financial auditors, make use of instrumental judgement, making them more like evaluators than auditors. In place of 'audit' he chose'the word 'inspection', justified on the grounds that this brings to mind the image of a professional overseer obtaining first hand impressions and information about how well a body is functioning. It is concerned with examinations of any kind of procedure, not just financial management, and with the exercise of judgement. In addition, it may or may not be based on research work, and is closely associated with accountability. In Barzeley's view these features mark VFM audit apart from financial audit and evaluation. In response, Pollitt (in Pollitt et al 1999) suggests there are strong grounds for treating these arguments cautiously. He argues that much financial audit also requires the use of judgement, and that conversely, VFM audit, like traditional auditing, can be concerned with issues of compliance. Pollitt also suggests, given the on-going developments in VFM work, it is premature to fix on a single definition of VFM audit (pl4). (b) Types of VFM audit The debate about the nature of VFM audit is complicated by the variety of types of work identified. Barzelay (1997) distinguishes between efficiency audit, programme effectiveness audit, performance management capacity audit, performance information audit, and best practice review. He suggests that some audit bodies have narrow 'product lines', whilst others have more extensive ones. He also notes that efficiency audits are the most prevalent forms of VFM audit, followed by performance management capacity audits and programme effectiveness audits. In Barzelay's view, this pattern is roughly consistent with standard theories of bureaucratic behaviour that emphasize resistance to institutional change and preferente for organizational autonomy. In his view, efficiency audits can be undertaken with little change to the ways of working and existing staff resources of an audit office. In contrast, undertaking program effectiveness studies, for example, strains the mission, role and organizational strategy of traditionally-orientated audit bodies. Bowerman (1996) also examines different approaches to VFM auditing. She suggests that there has been an evolution over time 'from a 'hands-on' approach where auditors confronted managers and public with examples of waste and inefficiency to a more 26 refined form whereby the VFM ethos has empowered managers and the auditors are consigned to their familiar role of checking management's account of VFM achievements'(pl93). Bowerman distinguishes between reviews of management systems, arrangements and procedures; VFM procedures audits; policy audits; audits of management representations of VFM; comparative performance audits; and quality audits. She does not claim this is a complete list, and argues that specific audits may encompass features of more than one of the types. Bowerman highlights drawbacks in all these forms of audit. She argues, for example, that reviews of management systems, which have caused embarrassment in the past, have now encouraged improved practices and so have largely fulfilled their purpose. Although she acknowledges that auditors may be reluctant to drop such high profile work, she suggests it may encourage auditees to see the process as adversarial. Turning to VFM procedures audits, she argues that this style of audit requires the existence of clear objectives, which are not always available. She also suggests that in the new culture of contracts, it may be outmoded to place emphasis on procedures and systems. To Bowerman, the wide variety of VFM audits may reflect the need to adjust to a variety of circumstances, but also signifies a lack of agreement as to the purpose of VFM audit. She suggests that there is no consensus as to why they are undertaken, for whom, or what methodologies should be employed. Bowerman also argues that VFM auditing is at a crossroads. With the public sector environment undergoing significant changes, the precise role of audit is far from clear. While some have suggested extended powers for auditors, she questions whether auditors should actually withdraw, their job done. Bowerman's conclusion is that 'VFM auditing may be in decline in the UK' and that auditors need to decide whether VFM audit is primarily for'management or for the public. If public accountability is its prime objective then the process could benefit from being more adversarial, rather than seeking to add value, with the consulting role done at more of an arms length than now. To Bowerman VFM audit 'has lost its way; it needs to rediscover its accountability roots' (p209). (c ) VFM audit and effectiveness One aspect of VFM work that has received attention over some years is the extent to which auditors are able to examine the criteria of effectiveness (Garrett 1986, Day and 27 Klein 1990, Gray, Jenkins and Segsworth 1993, Power 1997, Schwartz 1999). Studying the effectiveness of programme or activities is generally considered problematic for auditors, bringing them close to questioning policy, testing their skills, and raising outcome measurement problems. Several studies have tried to explain this lack of coverage of effectiveness. Anand (1988) noted that many public sector auditors had only minimal training in economic techniques, and even where junior staff were aware of the methods, their managers were given little or no training in how to manage this work. At a broader level, he suggests that other reasons for not examining effectiveness could include that making a judgement that something was ineffective might lead to a recommendation that more resources should be used, whereas auditors have traditionally been concerned with saving money. He also notes that effectiveness judgements often require the existence of clear objectives, which are often lacking. Thus, practical issues combine with broader concerns about the legitimacy of commenting on policy choices and appearing to countenance increased public spending. Pendlebury and Shreim (1990) undertook a survey of the attitudes to effectiveness auditing of auditors at the NAO, the Audit Commission and in local government. They found a majority considered effectiveness auditing should be part of their work, even though they accepted that this would make them more reliant on personal judgements, and bring them closer to considering policy decisions. The majority also identified the lack of final output measures as a major constraint to such work, along with the anticipated departmental concern that the auditor was questioning policy. Glynn, Gray and Jenkins (1992) concluded that auditors were concerned at the difficulty in satisfying normal professional standards of evidence and the increased risk associated with greater use of judgement when undertaking effectiveness audits. But, they suggest there is evidence that the NAO auditors were becoming less unwilling to undertake such more speculative studies. More recently, Schwartz (1999) has again questioned whether effectiveness auditing is really practised by most SAls. He argues that 'for a variety of reasons, most state auditors have not met the challenge of providing non-politicized, professional and objective outcome evaluations in place of agency-based evaluations which are perceived to be infected by political interests(p522). He suggests that this is largely down to the 28 political dangers facing auditors, and argues that SAIs use different strategies to cope with the effectiveness dilemma. Some avoid the subject altogether, but Schwartz suggests that the NAO has adopted a strategy of 'tinkering', using some sophisticated methodology, but not going all the way in asking relevant questions or emphasizing effectiveness deficiencies. He argues that such an approach is dangerous because such work often does not minimize political risks by using best practice in social science and evaluation work. Audit and Evaluation A further aspect of the debate about the nature of VFM audit that has received significant attention is the relationship between audit and evaluative research. Pollitt and Summa (1996) and Sloan (1994) have suggested that SAls are major players in the field of evaluation, but the relationship is far from clear. A range of positions has been taken up, with some seeing scope for a merger between the two professions and other highlighting the professional and practical differences. The important opening salvo in the debate was fired by Chelimsky (1985) who considered the relationship between auditing and evaluation in their approaches to program assessment. Chelimsky notes that auditing was originally developed as a procedure for detailed checking - for verification - and that, although the scope of auditing has widened beyond the inspection of financial statements to look at how well programs met their objectives, the concerns were largely the same. She states that audit is 'essentially normative' - relating what has been found with what is generally agreed to be a correct or proper situation - with questions asked generally for the purposes of accountability. Although evaluation also asked these questions, it also sought answers to descriptive and cause and effect questions. In addition, Chelimsky suggests that the auditor assumes that there is a correct and agreed upon way of doing things, and that criteria could be established against which to measure conditions. In contrast, the evaluator focused more on the relationship between the changes observed and the program, rather than the changes observed and those that should have been observed. Finally, auditors did not produce estimates of what might have happened in the absence of the program, whereas evaluators did. Such differences, in Chelimsky's opinion, 29 meant that even where auditors and evaluators examined the same question, their approaches and performance were likely to be different. On the subject of collecting evidence, Chelimsky suggests that auditors are concerned with evidence from authoritative sources, and place heavy weight on documentary evidence. Auditors are concerned with verifying planned program procedures. Collecting information by questionnaire or structured interview is only a &supplementary activity'. In contrast, whilst evaluators placed reliance on structured interviews and questionnaires, they pay greater concern to validity and reliability issues, and the quality of their instruments. Finally, in analyzing the data, audit reasoning is based on deduction because of the normative character of the work. Auditors seek out critical events and discrepancies. Evaluators, in contrast use inductive reasoning, based often on statistical probability and using statistical concepts. In conclusion, Chelimsky argues that, despite the different approaches, there are many similarities that suggest that the two professions can learn from each other. Evaluators should consider using cost-benefit and cost-effectiveness analyses as well as paying more attention to costs. On project design, auditors could usefully begin to use some of the techniques developed by program evaluators for designing assessments in which questions, strategies, methods and analyses mesh tightly. Whilst some might argue that Chelimsky's view of both audit and evaluation is rather narrowly focused, the debate has been taken forward by a number of others. Leeuw (1992,1996), has also focused on similarities and differences between VFM audit and evaluation. In his view the core of VFM audit is focused around management and managerial accountability, while the analysis of the relationship between public expenditure and policy instruments is also crucial. VFM audit dso strives to judge how efficient and effective management has been, which requires formally pinning down the objectives of policy instruments. Auditors are concerned with legal and procedural compliance issues, and audit questions tend to be nonnative, rather than causal or descriptive. On the issue of methods, Leeuw links audit back to its origins primarily in accountancy. He too notes the concern with evidence from authoritative documentary sources, and suggests that statistical techniques have been applied only in large-scale audits. With 30 regard to the diffusion of findings, audit seeks to suggest ways in which management can improve efficiency and effectiveness of policies, rather than advocating policy changes per se. The format in which their results are presented tend to be standardised, and the process of reporting findings, often based on discussions between auditor and auditee. In contrast, evaluators are engaged in studying the actual implementation of policy programs within society, and ask questions of a cause and effect, descriptive or prospective nature. They are not primarily interested in obtaining data from management only, instead focusing more on broader work in the field. They are also not particularly interested in assessing the economy and efficiency of expenditure on policies. In data collection, evaluators rely heavily on quantitative and qualitative methods from the social and behavioural sciences (interviews, questionnaires and observation), and use a number of statistical techniques. T'hey place great attention on the communication of their findings. Despite the differences, Leeuw considers there is much in common between audit and evaluation. There is a shared emphasis on evidence and adequate data gathering, and both are strongly linked with accountability. Both pay attention to diffusion of their findings and collect evidence with regard to the aftercare of their projects. LeeuW also considers that they complement each other. Whilst evaluators are concerned with the societal effects of public policy, auditors are concerned with the inside of the bureaucracy. In the light of this, he suggests ways in which each can learn from the other. Transfers of knowledge and experience from VFM auditing to evaluation would help to reduce the gap between them. By bringing in concepts from auditing, evaluators will be able to design their studies from a perspective moWrelevant to answering financial questions. In return, Leeuw suggests that auditors could benefit from paying more attention to social and behavioural theories, and the lessons learnt from policy impact assessments. In taking such steps, Leeuw sees the possibility of greater collaboration between auditors and evaluators. In his opinion 'it will no longer be acceptable nor efficient to have a split-up between VFM auditing and policy evaluation, or put, differently, between accountancy, public finance, budgeting and social and behavioural evaluation research'(Leeuw 1992: 65). 31 Pollitt and Summa (1997a) offer 'a more cautious interpretation of the developing relationship between SAls and evaluation. ' In their view, VFM audit shares some characteristics with evaluation, in particular, the use of certain tools and the application of criteria of economy, efficiency, effectiveness and quality. However, they consider any suggestion of a merger needs to be heavily qualified, and argue there are fundamental tensions between the roles of SAIs and those of evaluators. As a result, they consider Leeuw's vision (1996) of an audit/evaluation 'interdiscipline' unlikely. In part, they see problems because of the vagueness surrounding the identity of evaluation, and the wide variety of definitions put forward and the range of institutions deemed to be undertaking it. Comparisons with audit are thus difficult because evaluation is such a broad church. In addition, however, they consider SAls are inhibited from wholeheartedly embracing the evaluation paradigm because they are primarily working from 'a perspective of guardianship and control' which cannot be relinquished even if, as many have, SAIs take on additional roles. As a result, Pollitt and Summa consider an SAI's basic duty limits the extent to which they can 'play the part of the full-bloodied social science evaluator' or encourage risk taking. Pollitt and Summa also consider there are significant cultural inhibitions to SAIs as evaluative institutions. Some of this is due to the domination of accountants and lawyers within most SAIs. Both professions respect formal authority. In addition, auditors have a positivistic approach and a professional anxiety about hypothetical arguments, in contrast to evaluators, who are often interested in model building and a drawing on a wide range of evidence. Drawing a distinction between the aims of auditors to verify and those of evaluators to explain, Pollitt and Summa quote Chelimsky's observation (1996: 64) that the two groups 'tend to have dissimilar mind-slas'. As well as internal constraints, the authors argue that political forces also set limits on the extent to which audit institutions can indulge in broad-scope evaluations. Partly this is because such an approach would involve SAIs too closely in the criticism of policy objectives. It is also suggested that SAIs tend to steer clear of broad-scope examinations of major government programmes as too risky. Whilst they try to stand above party politics, they nevertheless must heed the sensitivities of the political system within which they operate. In conclusion, Pollitt and Summa consider that there is most likely to be 'an 32 evaluation-minded sharpening of VFM audit', in which auditors learn from evaluation and apply the lessons to improve their own work. Literature on the practice of VFM audit and the design of audit examinations The practice of VFM audit has received limited attention from outside observers. Possible reasons may include reticence on the part of auditors to explain too much about their work, and a lack of interest on the part of observers in what may appear dull routines. The first substantive examination of the NAO's approach to VFM work is Roberts and Pollitt (1994), in a case study of a report on Creating and Safeguarding Jobs in Wales (National Audit Office 1991 a). The aim was to illuminate what appeared to be some general characteristics of the NAO's VFM approach. Roberts and Pollitt conclude that, while VFM work marked an important step beyond traditional audit for regularity and/or economy, and significantly enhanced democratic accountability, it fell short of full-blooded evaluation and took place within quite prominent constraints. Some of these constraints were seen as a result of legislation, but they considered others could have been avoided without a change of statute. These factors included what they termed 'a format limitation', by which the NAO rarely specified in detail ways in which unsatisfactory aspects of performance could be improved; a focus primarily financial in scope so that management issues tended to be pursued only to the extent that they related to expenditure issues; and a dependence on Parliament and the PAC, which encouraged the auditors to remove technical or methodological detail from reports that they considered would not be of interest to MPs. The need to meet parliamentary timescales discouraged ambitious and lengthy studies, and the choice of topic and presentation of conclusions were bound to raise considerations for the NAO about the potential for partisanship amongst their MP audience. Thus, they appeared to consider that open-ended evaluations on controversial topics would encourage PAC to divide on party lines which would undermine the authority of the committee and in turn the standing of the NAO. Roberts and Pollitt had mixed views on the methodology for the study. On the one hand they considered that 'it was not a particularly sophisticated exercise' (p545), with, for example, the analysis of opportunity costs limited and the exploration of displacement 33 quite severely circumscribed. They noted criticisms raised by others about the validity of a 'bottom up' survey approach. However, they also noted that the study was significant in that it had taken the use of evaluation tools further than previous NAO work, and its analysis had been sufficient to disturb existing official approaches to the evaluation of programmes. It had also provided significant new data of use to Parliament and had resulted in a valuable hearing of the PAC. Earlier consideration of the literature on audit and evaluation drew on Pollitt and Summa (1996). This also addressed the design of studies and the choice of methods. They argue that to command authority, auditors (and others) must base their work on generally accepted tools and methods, and conclude that 'the advent of performance audit has certainly helped SAls widen the range of tools that they commonly employ .... For example, the NAO reports of the last decade or so appear to be methodologically considerably more adventurous than those of the previous period. ' To support this, they point out that auditors have made use of customer satisfaction surveys, opinion surveys, and economic models. In their view, most of these techniques (might equally have been used by evaluators working on the same topics' (pp 40-41). Despite this, Pollitt and Summa also identified three differences in the approaches used by auditors. These were that auditors did not document interviews as academic evaluators did; appeared cautious about entering into comparative analysis beyond the organisation under investigation; and produced reports that made far less reference to work on similar topics. In addition, Pollitt and Summa considered that auditors appeared to 'plan' their audits in a fairly pragmatic way, whereas evaluators 'designed' their studies. In their view, it was rare for audit examinations to involve such self- consciousness about the way different methodological compohents worked together. Instead, 'more commonly, audit teams will plan their audits in a fairly pragmatic fashion, selecting on a more-or-less ad hoc basis whatever tools and methods seem potentially useful for shedding light on the chosen program or institution'(p42). Although most observers have written from outside the world of state audit, Keen was appointed NAO Fellow in Audit and Evaluation. His remit, amongst other things, was to examine the links between VFM audit and evaluation, and outline the scope for importing more evaluation techniques into the NAO's work. One paper (Keen 1998b) 34 examines the factors that shape a VFM study, arguing that they are the product of a 'nexus' of influences. The first two are team-centred - the approaches and methods used, and the conceptual frameworks that team members bring to the studies. To Keen the current auditor's framework leads to a 'preference for retrospective studies which check on whether certain activities were undertaken, where the checking is often undertaken through careful review of documents and discussions with relevant people. ' The third factor relates to the 'beliefs and expectations of teams" - what they think, as a result of experience, is possible. Keen suggests that many team members considered the environment in the NAO not conducive to innovation, whilst others saw each study as an opportunity for new approaches. A tendency towards one or other end of the spectrum did not appear to relate to age or length of time in post. The fourth influence was the relations teams had, both within the NAO (with senior management) and with auditees. Different types of studies appeared to involve different relationships with audited bodies. Finally, the topic under investigation was important. In some cases teams felt considerable ownership of the topic and were able to develop an approach quickly and easily. In others, they struggled to find a way in. In Keen's view, the five centres influence each other. Keen also makes a number of observations about the nature of the technical side of VFM. In his view, it is robust and versatile, and can be adapted to the examination of many topics. This was due to the fact that 'VFM design is inherently flexible. It is possible to look at almost anything within any one sub-issue without affecting the overall structure of a study. ' He also argues that VFM is more complex than it at first appears, with the quality and quantity of evidence required for different sub-issues varying. Like Barzelay (1997), he suggests that the way in which VFM studies are shaped suits those producing them, and he argues that they are constructed in ways that allow agreement between auditor and auditee about the nature of a VFM study. Thus, for example, civil servants and auditors both attach importance to documents. A second article by Keen (I 999a) addresses the nature of the judgements made by VFM auditors in the course of their work. Arising out of this, he examines how VFM practitioners think about problems in the course of their fieldwork, and how they arrive at judgements. As a result of his experience working with teams, Keen concludes that, 35 at a micro-level, VFM work is 'quasi-rational, mixing intuition and analysis in different measures across the many sub-issues addressed in a study' (p517). At a macro-level, however, he suggests that overall judgements about a programme or project are rarely made. The process of forming a judgement was 'essentially atheoretical' (p52 1), with teams making inferences from substantial amounts of data. Keen also points to the lack of transparency in VFM reports in terms of explaining the nature of the thinking underpinning the bulk of individual points, or the process whereby the conclusions about each individual arguments were integrated to support overall judgements (a point also picked up in Roberts and Pollitt 1994). A particularly interesting point that Keen emphasises is the importance of the 6clearance' convention by which reports are agreed between auditor and auditee. Working with teams brought home to him the significance of getting auditees to 'agree that individual "facts' and their judgements were correct and their presentation fair' (p519). As a result he suggests that the requirement to achieve consensus dominated the design and conduct of studies, affecting how auditors viewed evidence gathering and analysis, and the significance they attached to particular key documents. Keen draws several conclusions. In the first place, in his view, different problems result in different blends of intuitive and analytical thinking, with some topics clearly lending themselves better to logical analysis than others. Secondly, VFM teams respond differently to the availability of different types of data, so that the range of options may be limited by the backgrounds of those who work on studies. Thirdly, most studies depend on an accumulation of empirical evidence, rather than argument, and on the agreement of audited bodies. Fourthly, studies tackles rarely tackle issues of causation, and tend to discuss cost and effectiveness issues separately. Findlly, Keen suggests that if VFM work is in significant part intuitive, it will be difficult to teach and learn. It also suggests that VFM work is a craft. Literature providing international comparisons To date there have been few comparative studies of audit work. In his work, Normanton (1966) concluded that there was less public accountability in the UK than in many other countries - indeed in some areas of spending he suggested that it was non-existent. In examining their functions, he identified a lowest common denominator of regularity 36 audit, and noted that, by the 1960s, some SAIs had added other discretionary work to this, and newer audit bodies had been established with these powers already provided. As noted earlier, he saw the addition of examinations of the use of resources to the repertoire of the auditor as a distinct benefit to the state. Normanton commented on the range of functions some SAIs carried out, highlighting the ways in which certain national administrations made use of the skills of the audit office. He also commented on the range of backgrounds of auditors in the SAIs he studied, noting the difference between those bodies where staff tended to be trained in accountancy (eg the General Accounting Office in the US) and those in 'the European tradition', who tended to have a legal training or to be highly trained administrators. More recently Lovell (1996b) has looked at the different approaches to state audit. His prime concern is with the relationship between the SAI and its government and parliament. Lovell identifies a range of different beliefs about the role of a SAI, commenting that each audit office is part of the continuing evolution of government, and suggesting that the varying length of time during which parliamentary democracy has been in place offers a partial explanation for different relationships between audit offices and their respective legislatures and executives. He notes that some SAIs can play roles that are supportive of government, for example, the Portuguese and German SAIs act in advisory roles for government. However, he also emphasises the contested nature of much of the development of state audit, in particular, the contentious nature of much of the development of VFM audit in many countries. Although he sees SAIs increasing in confidence, partly through the interchange of ideas between themselves, he considers that tensions may arise where such developments do not meet with the approval of national governments. a Barzelay's work on the range of types of work undertaken in several OECD countries was mentioned earlier. It suggested reasons for the prevalence of particular types of VFM audit in different countries. Barzelay (1997) argues that the political environment has a strong influence, suggesting that a VFM audit 'product line' which includes politically sensitive effectiveness audits, occurs where the audit body 'has an intense client relationship with a powerful entity for whom the policy/administration dichotomy is irrelevant. '(p255) Thus, in the US, Germany and Sweden, where there is either a 37 strong executive or strong legislature, the audit offices undertake a broad range of work, including considering program effectiveness. Although he accepts that the situation in the UK - where the NAO works primarily for the legislature in a system where the executive usually dominates, but (in his view) programme effectiveness audits are undertaken - appears to be deviant, he argues that the mandate to assess effectiveness in the UK is hedged with conditions (including the clearance convention mentioned above and discussed in detail in Parts 4 and 6) which allow the executive to influence the work. Pollitt et al (1999) compares VFM audit in five SAIs (the NAO, the French Cour des Comptes, the Dutch Algemene Rekenkamer, the Finnish Valtiontalouden tarkastusvirasto and the Swedish Riksrevisionsverket) against a background of 'new public management' changes. It notes that VFM audit in each country was differently defined and practised, rather than being a standardised package. Trends in the development of the work had also been different over time, with, for example, some SAIs giving greater emphasis than others to technical self-improvement or efforts to assess the impact of their audits. Of particular interest is the identification of four distinct roles for VFM auditors - as magistrate, public accountant, researcher and consultant. These roles are not mutually exclusive, but the authors suggest, for example, that the RRV has a tendency towards the third role, the NAO towards the second, and the Cour a close affinity to the first. These different roles, which are likely to have a significant influence on the type of work undertaken, and perhaps, the type of methods employed, are considered in more detail in Part 7. Despite these differences, the authors identify what they suggest are common 'VFM audit' values that mark the work apart from financial audit. They also identify a general increase in attention being given to the work compared with the past, and clear signs of all the SAIs making more effort to publicise their work, improve the presentation of their findings, and to expand the range of products arising from the work. They also report evidence of increased professionalism in all five SAIs, with VFM auditors becoming more expert, and in several countries making considerable use of outside consultants in their work. Yet, despite these developments, the authors suggest that VFM audit 'carries some of the unresolved problems of its youth', and is faced by many 38 strategic choices in terms of what kind of work it gravitates towards, what methods it chooses to employ, and what kind of relationships it builds with those it audits. Significance for the thesis The general evaluation literature, referred to at the start of the chapter, raises a number of issues of relevance to VFM. The shift towards formative, rather than summative, evaluation raises the issue of whether VFM could be carried out more as a dialogue between auditors and auditees, designed to generate self-reflection and improvement, or whether it is inevitably a summative process. The debate about the usefulness of evaluation also raises issues for auditors about how closely they should be tailoring their efforts to the needs of clients, and how far they can allow them to influence the way they conduct their work without compromising their independence. Both have resonance with issues about the role of audit at the end of the 20th century (Public Audit Forum 1999). The discussion about the benefits of political authority, against the potentially coercive nature of work undertaken in these circumstances (Henkel 1998: 290), links in well with the status and position of the NAO. It is particularly relevant at a time when some have argued that audit (and specifically the public and often critical PAC process) discourages risk-taking and innovation amongst civil servants (Cabinet Office 2000). The published literature on audit has examined the work from a variety of perspectives and raises a number of arguments of relevance to the thesis. A fundamental point concerns the standing and statutory power of the SAI. Such bodies are in a privileged position as regards their powers of access to documents and those who hold them. They also have a (usually) receptive audience, and the means to ensure their work is at least considered, and usually, acted upon. Almost all the literature, either implicitly or explictly, acknowledges the importance of these powers, although the arguments in their favour range from a belief in the intrinsic value of hardworking and independent auditors (Normanton), to an acceptance that the audit role is the best option currently available (Lovell). For this thesis, the most important question is what significance the formal statutory powers have for the way auditors conduct their work. Does it constrain auditors and channel them down certain routes and approaches ? Does the fact that auditors have access to departmental documents discourage innovation in methods, or 39 lead auditors to carry out audits in particular, perhaps conservative, ways ? Does the constitutional position of the audit office and the official nature of their enquiries preclude certain methods Another important issue is the current perception that there is too much regulation, oversight and audit, that there has been an 'audit explosion. ' This argument is perhaps the most substantial grounds on which the role played by state auditors can be challenged. The notion that what they do may have (or may be seen as having) negative consequences should set alarm bells ringing amongst audit offices wishing to fulfil, and be seen to fulfil, a valuable function (see Public Audit Forum 1999 for auditors' response on this issue). The question then is whether such concerns influence the choices that audit bodies make as to what type of VFM audit they carry out. Do they drive them to undertake work they consider likely to be more penetrating, provide new information, and have 'impact' ? Does fear of being labelled a burden, and having a dysfunctional effect, encourage audit offices to innovate, or has it deterred consideration of more technically complex and adventurous approaches which may, for example, require greater auditee involvement? Indeed, has the development of an 'audit society' actually led to increased innovation from auditors and encouraged greater technical complexity in regulation ? Another point is the contested and negotiated nature of much of VFM audit. Whilst the importance of the independence of the auditors, emphasised in the technical literature (INTOSAI 1992) and by White and Hollingsworth, is crucial and meaningful, at the level of particular studies, there is much that appears to constrain the SAI in terms of the ways it can look at a subject (eg Hodges 1997), its rights of access to particular types of bodies and to follow funds to final users (Lovell 1996a), the way it conducts an audit (Keen 1999a), its ability to undertake effectiveness studies (Barzelay 1997), and how it chooses to present its findings (Roberts and Pollitt 1994, Keen 1999a). This raises the question of how much freedom auditors actually have to choose their methods. The discussions of the flexibility of 'VFM audit' is also of significance for this thesis. Barzelay and Bowerman both listed a variety of different types of VFM audit, and Keen (1998b) has written of the flexibility of the VFM audit study design. This suggests there are a great many opportunities (and requirements) for auditors to choose how they 40 undertake their work, and underlines the value of a careful analysis of what factors may influence these choices. The issue then is what constrains and influences the auditors' choices. The literature suggests there may be a range of factors, including auditors' own experience, knowledge and skills, their assumptions about what is possible, expectations about the standards of the quality of evidence, professional ethics, and the political environment. This will be the subject of detailed examination in Part 6. Keen's interest (Keen 1998b) in the conceptual frameworks that auditors bring to studies, and their beliefs and expectations about what is possible and desirable raises interesting questions about what has shaped these beliefs and expectations. Are auditors' views of what is possible or not, shaped by personal experience, or are there widely accepted notions within audit offices about what is and is not acceptable ? What are the key events which have shaped the development of VFM practice ? How do those who have been in the NAO for a long time view the changes ? Auditors appear to approach new studies influenced by those that they and their colleagues have undertaken before. There is thus a strong argument for examining in more detail the way in which VFM audit has developed in the NAO. This is the rationale for Part 4 of this thesis. Finally, the literature on the work of other SAls, has raised a number of interesting issues for the choices facing the NAO. The NAO's approach to VFM work is not one that has necessarily been followed elsewhere. This suggests that an examination of the work of other SAls might shed light on aspects of the development of VFM and VFM methods in the NAO that are peculiar to the UK. On the basis of the existing literature, the nature of relationships with the executive and the legislature, the impact of the background of its auditors and the 'perceived role' of the VFNPaudit, may all have an influence on the choice of topic auditors make, their approach to these subjects, and the methods employed. Clearly a comparison with other SAls could offer valuable additional insights into the influences on the methods employed in the work of the NAO. This is undertaken in Part 7. Gaps in the literature This review of relevant literature has also helped to identify a number of gaps in the coverage of the field, several of them in fairly crucial areas of study. There is still 41 relatively limited information about what methods VFM auditors have used, and what factors have shaped the choices in their work. Much of the literature has been narrowly focused, or else quite descriptive. The literature lacks a sound base in the raw material of audit - the reports themselves. There has also been relatively little examination of the broad development of the VFM work of the NAO. Several writers have examined aspects of VFM at the NAO, but these studies have tended to take a relatively narrow perspective. A thorough examination of the broader development of VFM work at the NAO has yet to be undertaken. Part 4 is designed to provide this, drawing on the knowledge of those who have contributed to the changes, as well as documentary evidence. There have also been few attempts to go too far beyond the official publications. In almost all of the literature on VFM audit, the voices of the key participants - the auditors themselves - are barely heard. Few observers have sought their views in any systematic way, although it is acknowledged that Bowerman (ex-NAO), Power (as an advisor to the NAO), Anand (time spent as a specialist in NAO in the 1980s), Keen (two years working as NAO Research fellow) and Pollitt (responsible for reviewing NAO reports as part of the office's quality assurance arrangements between 1993 and 1995, discussed in Part 4) have all had varying degrees of involvement with the inside of the NAO. This situation has been compounded by the fact that only occasionally have auditors themselves provided insights into their activities (Dewar 1985,1991, and on a larger scale, Keemer 1984 and Sloan 1994 are exceptions), although outside the United Kingdom, Chelimsky (General Accounting Office) and Leeuw (Algemene Rekenkamer) have written from inside SAls. Former C&AGs and the current one have written articles which provide further insight into the developments (Henley 1,989, Bourn 1994), but these have been relatively few and far between. Part 6 attempts to correct this lack of a voice from within the NAO. 42 PART THREE: METHODOLOGY This part summarises the research design for the thesis, discusses the data collection methods, and considers issues of validity and reliability. The methods were chosen to produce evidence to answer the research questions outlined in Part I and, to recap, are based on the beliefs that: " to date, there has been insufficient attention paid in the research in this area to the evidence available in published audit reports; " the voice of the practitioner - the VFM auditor - has been insufficiently heard; and " insufficient attention has been given to how VFM audit practice has developed over the years, and what influence this has on the present day thinking. Research design In the absence of much existing research on the development of VFM audit methods, and given the underlying beliefs referred to above, an exploratory qualitative research approach was chosen. In keeping with the assumptions of this approach, the research design developed as the work progressed. Maxwell (1996 pl. 7-20) outlines five particular research purposes for which qualitative studies are especially suited. These are for: understanding the meaning, for participants in the study, of the events, situations and actions they are involved with; 9 understanding the particular context within which the participants act, and the influence the context has on their actions; a * identifying unanticipated phenomena and influences, and generating new grounded theories about the latter; 9 understanding the process by which events and actions take place; and * developing causal explanations. These points are relevant to the aims of this study, which is concerned with the processes by which auditors reach decisions on the choice of methods, as well as their experiences of the ways in which changes in the type of methods have come about. It 43 also considers the context in which they work and how envirom-nental factors might have impacted on the scope VFM auditors have, or feel they have, to makes choices about methods. More specifically, it is concerned with the interaction between the NAO, its clientele, and its audiences (Figure 3.1 ), and the effect this has on the selection of methods. Figure 3.1: Influences on VFM methods An inductive approach has thus been chosen that will allow examination of the complexities and processes, the variables that have yet to be identified. and the informal and unstructured linkages and processes within organisations (Marshall and Rossman 1995). This will also allow for consideration behind the detail, and permit interaction between the author and the participants to develop explanations for changes that have taken place. Grounded theorv Within this overall design, a 'grounded theory' approach has been used. Grounded theory is 'grounded' because it emerges from, and is based in, the collection and analysis of empirical data. It is defined as 'one that is inductively derived from the study of the phenomenon it represents. That is, it is discovered. developed, and provisionally verified through systematic 44 data collection and analysis of data pertaining to that phenomenon ... One does not begin with a theory, then prove it. Rather, one begins with an area of study and what is relevant to that area is allowed to emerge' (Strauss and Corbin 1990: 23). Some of the most important characteristics of grounded theory (discussed, for example, in Strauss and Corbin 1990, Sarantakos 1994) are attractive for a number of reasons. The aim of developing theory that is close to everyday behaviour - in this case audit practice - seems an appropriate one in a practical area of research, and one to which personal experience in the field can be applied. The expectation that, as long as the theory represents the reality, then 'it should also be comprehensible and make sense both to the persons who were studied and to those practising in that area' (Strauss and Corbin 1990: 23) is also attractive in that it would produce explanations that are relevant and meaningful to practitioners within the NAO. Finally, the aim of developing a theory that is based on comprehensive data and is sufficiently abstract and with variation to make it applicable to other contexts, appeals given the aim of comparing the NAO to audit offices elsewhere. Against this background, the work has involved the development of questions and hypotheses, which have been tested during the course of the research work. In particular, an overview of changes in methods used was obtained by analysing audit reports. Questions and theories about the apparent trends and developments were developed and tested in interviews and focus group discussions. In the same way, questions about the influences on VFM auditors were developed from personal experience, and also by reading internal guides. These ideas were also tested with auditors in focus groups and interviews. On the basis of responses, the thinking was refined and later interviews used to test ideas. Circulation of draft chapters to a number of experienced colleagues ('mernber checks') also helped to refine further the theories generated. Data collection methods Within this general approach, a number of discrete methods of data collection were selected, in particular, document review, focus groups, and semi-structured interviews. Together they form a coherent, integrated approach to examining this subject. All the fieldwork relating to the NAO was undertaken by the author. Where the focus was on 45 other audit offices, information gathered with colleagues during the course of the Performance Audit Project (PAP), in which the author participated between 1996 and 1999, was drawn upon, as was material gathered with three NAO colleagues in the course of the preparation of the NAO's book State Audit in the European Union (National Audit Office 1996a). Drawing on personal knowledge and experience of VFM work and methods In examining a subject in which the researcher has been involved for several years, fiexperiential data', and a high degree of 'theoretical sensitivity' is brought to the subject matter under consideration (Strauss and Corbin 1990: 41). Theoretical sensitivity is defined as having insight, an ability to give meaning to data, the capacity to understand, and a capability to separate what is pertinent, from that which is not. Much of this knowledge has been derived from participation in the activities under review, providing much knowledge that might otherwise have only have been gained through extensive background reading and basic familiarisation with the audit process. As Strauss and Corbin put it, 'this knowledge, even if implicit, is taken into the research situation and helps you to understand events and actions seen and heard, and to do so more quickly than if you did not bring this background into the research' (p42). The advantages and disadvantages of this 'insider' position are discussed further towards the end of this chapter, but suffice to say at this stage, knowledge derived from personal experience is considered a very valuable source of information in this work. Review of official literature It has been noted (Pollitt and Summa 1997b) that SAls, and in particular, the NAO, have promoted themselves more in recent years, providing rqaders with far greater information than before about their activities and, to a lesser extent, their performance. Although clearly intended to present a positive view of their work and significance, official documents such as the NAO's Annual Report and the Corporate Plan provide valuable information, including on developments in VFM audit. Consequently, all Annual Reports back to 1988 (the first year of publication in its current form) and Corporate Plans were examined for information on developments in VFM. Most information was contained in a specific section on this subject, which provided some 46 initial insights into changes in VFM. It also acted as a check on some of the assertions made by interviewees about the timing of specific developments. Review of internal guidance for VFM auditors As well as official published literature, primarily intended for external consumption, internal guidance for VFM auditors was also reviewed. These publications, especially where given strong senior management backing, can be regarded as influential, and auditors are expected, at the very least, to have consulted them during their work. As a result, they provide an invaluable insight into the 'official' rationale for all elements of the VFM process, and also include useful vignettes of cases where particular methods have been applied. In theory at least, these documents ought to have a significant influence on the understanding and perceptions that staff have of their audit work. Examination of rel2orts by the C&AG The end product of much audit work is a published report, through which the findings, analysis, conclusions and recommendations are conveyed to the various audiences of an SAI's work. Between 1984 and 1998 the C&AG presented over 500 VFM reports to Parliament. These are House of Commons papers, published and publicly available. They are usually relatively short documents (on average between 40-60 pages), which make use of graphics and photographs (see Waerness in Pollitt et al 1999 for further discussion about the characteristics of VFM reports). Whatever their merits, audit reports arc not designed primarily for research purposes. They arc written with a non-spccialist audience in mind so that they tend not to contain considerable technical detail about methods. Given this drawback, alternative sources of information about methods were considered. One option might have been the Preliminary Study Reports produced to obtain approval from the C&AG for the study proposal. These include basic methodological information in order that NAO senior management can see how the audit team intend to gather evidence. A sample of these reports was reviewed, but no central collection exists, making them hard to access. It was also established that many older ones were unlikely to be available. Another source of information on methods was the NAO's 'VFM database', containing details of methods and consultants used for studies. However, this was only established in 1996, and although staff have been encouraged to input details of past studies, it is not 47 complete. Being reliant on staff to submit details accurately also introduces the risk of error into these records, which could jeopardise the accuracy of the findings. As a result, a decision was made to rely on published reports as the best source available. Given the time constraints, it was not possible to review all 500 reports for details of the methods employed. Instead, attention was focused on three periods and all the reports published within those periods were reviewed. The periods chosen were the calendar years 1986-1988,1993-1995 and 1997-1998, in total, some 300 reports. The rationale behind the selection of time periods was that they appeared to cover one 'early' period in the recent development of VFM, a few years after the establishment of VFM audit in its 'modem' forniat; one period when personal experience suggests the practice of VFM work appeared to have become fairly settled; and a final period when there had been more concerted efforts to think about methods and study design (see Part 4). The first step was to examine the information on methods used that was contained in the published reports. The reports were examined systematically, identifying the methods used (the categories) and numbers of reports against each category identified for each year. This analysis is presented in tabular form in Part 5, in a way that allows trends in the usage of particular methods to be seen over time. Research for the second and third periods was fairly straightforward as at least a paragraph or two of explanation were usually included in the report's Introduction (see for example, National Audit Office 2000a). From the mid-1990s, many reports also included a methodological appendix or table of methods (see, for example, National Audit Office 2000a), giving much more detail. More problematic was the first Period. In some cases, little information was included on methods used, or it had to be gleaned from the main text of the report, where passing reference was made, for example, to the use bf a questionnaire. To provide some reassurance as to the accuracy and completeness of the examination of reports, two additional tools were used. The first was the VFM Database, referred to above. In addition, the NAO has produced a CD-ROM of all its reports and those of the PAC from 1969 to date. The search facility available here was used to identify references to particular methods. Focus groups A key feature of the research work was a series of five focus groups, conducted with 48 Directors, Audit Managers and Principal Auditors 14 (the three grades primarily involved in the design of VFM studies and the selection of tools). The aim was to: 0 provide insights into the attitudes, perceptions, and thought processes of auditors about the factors that influenced their choices of methods; " help to identify key dates and changes in the development of VFM methods; and " complement the data collected on methods used in VFM examinations, which provide no insight in why methods had been selected. Focus groups are considered appropriate in many circumstances, but not in others. They may be suitable for uncovering factors relating to complex behaviour or motivation, where discussions can provide insight into complicated topics, or the area of concern relates to multifaceted behaviour or motivation. They may also be valuable where the researcher would like ideas to emerge from the group, and where there is a high value placed on capturing the open-ended comments of the target audience (Krueger 1988: 44). In this project, the aim was to capture the range of opinions and experiences auditors have, and to allow participants to clarify for themselves upon what their opinions or behaviour depend (Morgan and Krueger 1992). From a purely practical point of view, using focus groups provided the opportunity to draw on the experience of a significant number of colleagues in a time period that fitted in with their working day, and to record a large amount of interactive discussion on a topic in a limited period of time (Morgan 1988: 15). The circumstances in which focus groups are not suitable (Krueger 1988: 45) did not seem to be particularly relevant in this case. The environment should not have been particularly emotionally charged; statistical projections were not needed; and there should be no major concern over the confidentiality of sensitive information. There was no obvious mismatch between the researcher's area of interest, experience and knowledge, and that of likely participants. Indeed the opposite was the case. Where research is exploratory in nature, it is often considered that it will only take a few groups with a relatively low degree of moderator-imposed structure to obtain sufficient information (Morgan 1988: 42). On this basis, five groups were planned, to be added to if findings from each were dissimilar and inconclusive. These groups provided a 49 considerable amount (90,000 words of transcript) of rich and valuable material, and it was decided not to carry out more given that the discussions became repetitive. In deciding on the size of each group, attention was paid to the need to balance practical and substantive considerations (Morgan 1988: 45), avoiding very large groups, while ensuring that research could be undertaken within a relatively short timescale. The groups were, therefore, made up of seven or eight members of staff. Most participants had between five and 15 years experience of VFM work (although some had 25 years) on a wide range of studies on different subject areas, or else valuable experience elsewhere. In view of the team nature of VFM work and the varying responsibilities of grades in different parts of the office, as well as the relatively informal nature of relations between grades in the NAO (and thus the collaborative nature of thinking on methods), grades in each of the groups were mixed. Attention was paid to the debate on the use of groups of people who know each other, including those who work together (Krueger 1988), with the possibility, for example, of familiarity inhibiting disclosure. However, given the very specialised nature of this research topic and the fact that there were only around 70 relevant members of staff to choose from, it was not considered possible in this case to undertake focus group work without using participants who knew each other. In addition, the issue of familiarity between the moderator and some of the participants arose. The moderator was known to all the participants, but in no case was the moderator working with, or for any, of the participants. Participants were provided in advance with a short summary note describing the purposes and status of the research project and the focus group. The subject was then introduced and key findings from report analysis work were outlined, before the following points for discussion were introduced. These were: how do you think the approach to the use of research tools and techniques has changed within the NAO in recent years ? " what practical considerations do you take into account when deciding whether to make use of particular tools and techniques for a VFM examination ? " what constraints do you consider there are on auditors in deciding on the use of so particular tools and techniques ? 0 what scope do you consider there is for greater use of research tools and techniques in VFM examinations ? The transcripts of the focus group data were analysed using QSR NUD. lST software to work through and code the material. Transcription proved very time consuming, although the software helped speed up the process of analysis, and provided greater rigour in the allocation of qualitative material to particular codes. A similar approach was taken with interview evidence. Interviews with senior staff To complement the evidence gained from focus groups, three sets of semi-structured interviews were undertaken. The first, and most general, interviews formed part of the work for the Performance Audit Project (Pollitt et al 1999). At the NAO, these were undertaken with the Assistant Auditor General (member of the senior management board) and three Directors responsible for VFM development, as well as two senior audit managers in the same area. A series of interviews more specifically related to VFM methods was carried out subsequently with six senior NAO staff Towards the end of the research work, further interviews were carried out to help validate the developing theories to obtain comments and reactions. In all cases, interviews were semi-structured and made use of interview schedules, but allowed interviewees to develop their own arguments. International comparisons One of the aims of the research is to compare the developments in the UK with those taking place in audit offices in other countries to gain further insights into the NAO's work. As mentioned above, this research project is closely related to the work of the PAP team. As a result, material about the activities of other audit offices gathered for the purposes of that research project was drawn upon. In particular, the use of methods for examinations had been counted by colleagues, using the same definitions developed during the researcher's analysis of NAO reports. Given time constraints and language problems, it was not feasible for me to examine a wide range of additional reports from other SAls, although a small number produced in English by the RRV were reviewed. 51 The author undertook interviews with senior officials of the audit offices in Sweden and Netherlands, and drew on interviews in Finland and France conducted by colleagues on the PAP project. In addition, information the author had gathered during interviews with senior staff of seven European audit offices during the course of the preparation of the NAO's State Audit in the European Union book (National Audit Office 1996a) informed the research. A range of literature from other SAls was examined, which provided insight into issues in these countries. Validity and reliability Maxwell (1996) defines validity as 'the correctness or credibility of a description, conclusion and explanation, interpretation, or other sort of account'(p87). In his discussion, he identifies threats to the researcher reaching accurate and credible conclusions. This section considers steps taken to minimise the risk that conclusions in this research project are incorrect. In Maxwell's view 'the main threat to valid description, in the sense of describing what you saw or heard, is the inaccuracy or incompleteness of the data' (p89). To counter this, focus groups were tape recorded and transcribed, as were interviews with NAO staff, where convenient. Interviews undertaken in the Netherlands and Sweden as part of the PAP work were not recorded, but detailed notes were written up immediately afterwards, and agreed as an accurate record with the co-interviewee. The decision was taken to review a large number of reports. Another threat was that the researcher would not make contact with sufficient participants to ensure that the understanding of the developments and the factors behind them is correct. To this end, focus groups were used to bring together as many audit staff as possible within a reasonably short time period. In addition, other NAO senior staff were interviewed separately. Because some developments took place some years ago, and many of the then senior staff had since retired, contact was made with several still maintaining an interest in audit work. Another threat was researcher bias, particularly relevant given the author's status as an employee at the NAO since 1988. This had advantages and drawbacks for the research. It made gaining access to audit staff and intemal documents and historical material relatively easy because of the proximity to these resources. Access within the NAO was 52 also not a problem because the research project was supported by two members of NAO senior management. Access to other SAIs was obtained in the author's capacity as a member of the PAP team. However, there were drawbacks. Experience as a VFM auditor may have ensured certain assumptions about what the work entails. Given the range of differing practices within the office, personal experience may only provide a limited range of insights. Another risk was that preconceptions or views would influence the responses received from those interviewed. As a result, there was minimal facilitator participation in the focus groups. In interviews, most of the questions were open, leaving participants the opportunity to give their own views. The third threat - to the theoretical validity of the account - is that insufficient attention might be paid to evidence that conflicted or was at variance to the majority of the findings. The most likely possibility of this occurring was probably in that auditors in particular parts of the NAO would carry out their work differently from others. To counter this, interviewees and participants in focus groups were chosen from across the office. The decision to examine all reports published in a particular year also ensured information was obtained from all parts of the NAO. The conclusions were thus based on a broad understanding of events. 53 PART FOUR: THE DEVELOPMENT OF VFM AUDIT PRACTICE AT THE E&AD AND THE NAO The review of the literature in Part 2 noted that, to date, there had been little examination of the broad development of VFM work at the NAO. The purpose of this part is, therefore, to examine the development of the practice of VFM work in the NAO and its predecessor body, the E&AD. This will act as an introduction to Part 5, which looks in detail at the methods used by the NAO in VFM examinations during the 1980s and 1990s. Keen (1998b) suggests that a key influence on study design are 'the beliefs and expectations of teams'. He argues that 'Experience [for some] had demonstrated to them that the environment was not conducive to innovation, a view that also appeared to be held by many directors. For others, though, each study was an opportunity to try something new, often in the form of a new research method. ' These 'beliefs and expectations', and thus their willingness to attempt new things, have been conditioned by individuals' experience of the way in which VFM work has developed - their experience, for example, of developing studies and getting them approved, clearing draft reports, working with colleagues and consultants, or working within a particular budget (see Figure 1.1). Thus any consideration of how VFM staff make their choices - in terms of subjects, approaches and methods - must be seen against a background of the way in which VFM audit practice has evolved. Focusing mainly on VFM work undertaken under current legislation accords considerable significance to the National Audit Act 1983. This raises the question of whether the Act was a watershed. Superficially, it appears to be, but in the view of one former senior NAO official it was but: cone event in many. It confirmed important features of the work of the C&AG and his staff and their relationships with Parliament. It set the stage for continuing developments in the extent and nature of the Office's work. But it was not a revolution or a breakthrough in statutory or operational terms. The NAO was built on established foundations and followed firmly in the footsteps of its predecessor, the Exchequer and Audit Department. And although that Department was itself a milestone in parliamentary audit and financial control, it was also building on 54 what went before (Dewar, unpublished: 2)' This view is supported from outside the NAO by Pliatzky (1989), who considered 'value for money audits were not a break with the past. ' Against this background, this chapter will: * chart the chronology of the development of VFM practice; provide a framework for considering the changes in the use of methods in Part 5; and help shed light on the main influences on the development of VFM discussed in interviews and focus groups (Part 6). Existing material and a range of official documents - NAO Annual Reports, PAC reports, internal guidance, and memoranda - have been drawn on to construct a coherent description of the developments. In addition, participants in the focus groups were asked to recall events they considered particularly significant, as well as discuss how they perceived developments in their time in the office. To provide internal validity, the draft chapter was circulated to a number of current and former senior staff. The development of the MAD (1867-1983) and the NAO (1984-) The public audit function in the United Kingdom dates back to the fourteenth century (National Audit Office 1996a), but was established in its present form in the mid- nineteenth century as part of a more disciplined system of central administration. In 1861 the Committee of Public Accounts was created, followed by the passage of the Exchequer and Audit Departments Act 1866. This created the position of C&AG, who was supported in this task by the E&AD, staffed with civil servants. The 1866 Act - still the basis for much of the current day audit arrangements at the end of the 20th century - required the C&AG and his staff to examine every transaction, a task that became less realistic as government activity expanded in the twentieth century. Subsequently, the Exchequer and Audit Departments Act 1921 addressed this by allowing the C&AG to rely in part on departmental systems of control, and thus examine only a sample of transactions. The following 50 years was a period of steady, if not very dramatic, development, with the MAD growing in size and extending the 55 range of bodies examined. In the late 1970s the role and function of the C&AG were considered by Parliament on several occasions (Garrett 1992). The end result of this parliamentary scrutiny, a long tussle between Parliament and the Government, and the skilful piloting of a Private Members Bill through the House of Commons just prior to the 1983 General Election, was a significant alteration in the status of the C&AG and his staff with the passing of the National Audit Act 1983. The Act followed increasing parliamentary concern about the influence that the executive - in particular, the Treasury - retained over public audit. The key changes were that: " the E&AD was transformed into the NAO; " staff ceased to be civil servants and became employees of the C&AG, who was formally designated as an Officer of the House of Commons (a position he was assumed to have held before); " statutory provision was made for the first time for reports examining economy, efficiency and effectiveness; and " the Public Accounts Commission 15 (TPAC), another committee of Members of Parliament, was established to oversee the work of the NAO and approve its budget, thereby removing it from direct Treasury control. Subsequently, the number and range of audits undertaken by the C&AG grew, and stand-alone reports to Parliament on VFM issues became an established feature of parliamentary accountability arrangements. The first major change to these arrangements came as a result of devolution for Scotland and Wales. In 1999 an Auditor General for Wales was appointed, reporting to the Welsh Assembly, and in 2000 an Auditor General for Scotland was appointed, reporting to the Scottish Parliament, and supported by Audit Scotland, a new body created by the merger of the NAO's Edinburgh office and the Accounts Commission 16 . The Comptroller and Auditor General's role in relation to Scotland and Wales diminished very significantly. Development of VFM practice This section examines how and why VFM audit practice developed within the MAD and the NAO. 'Audit practice' refers to the way in which VFM examinations have been 56 organised, planned, resourced and executed. Developments have been divided into three periods: 'pre-history' (to 1945), 'the middle ages' (between 1945 and 1983) and 'modem times (1984 to the present day), although inevitably such precise distinctions are not entirely appropriate. Looking back, a number of key themes can be seen very early on, notably, the issue of relations with auditees, the nature of VFM and what it encompasses, and the resources available for the work. 'Pre-history' Establishing an exact date when the forerunners of today's VFM work began is a difficult and probably fruitless exercise. A number of suggestions have been made. One retired NAO official wrote an article entitled 'VFM audit: the first 800 years' (Dewar 1985), noting concerns expressed by courtiers and monarchs from the twelfth century onwards about the need to ensure value from their spending. In the late eighteenth century, the Commissioners for Auditing the Public Accounts (one of the forerunners of the post of C&AG), were statutorily required to consider 'in what more expeditious and effectual and less expensive manner the services can in future be regulated and carried on for the benefit of the public' (pl 1). According to Dewar, such concerns disappeared after that until the 1880s, under an entirely different audit regime. Then, as Normanton noted, the examination of the economy and efficiency of government expenditure, rather than just regularity, became a feature of the C&AG's audit work. He was supported in this by the PAC, which stated that 'If, in the course of his audit, the Comptroller and Auditor General becomes aware of facts which appear to him to indicate an improper expenditure or the waste of public money, it is his duty to call the attention of Parliament to them' (Normanton 1966: 105). This view was not entirely supported by government departments or by the Treasury. The latter argued that, whilst departments were bound to provide information to establish the regularity of expenditure, they might refuse to explain why they had spent money in one way rather than another, although they accepted that they might have to account to Parliament for their refusal. However, further work established precedents for examinations well beyond the regularity of expenditure. In the 1890s, the PAC supported the C&AG in investigations (strictly beyond the remit of his audit) into the costs of shipbuilding, and this time there 57 was Treasury support for the work. Some years later, 'audit reports on massive waste and mismanagement in the South African War led in 1904 to a Royal Commission of Inquiry and a fundamental re-emphasis of the importance of value-for-money in public spending' (Bourn 1994). In 1911, the PAC again supported the C&AG in investigations into price discrepancies in the purchasing of horses and the absence of competitive tendering. In Normanton's view, the C&AG's victory in extracting a promise from the War Office to 'furnish explanations' in future on such matters, was the end of departmental attempts to limit the C&AG's discretionary powers in looking beyond regularity (pl08). As Keemer has commented 'the subjects on which the issue arose were modest in character but the principle was established', and later C&AGs were able to carry out more ambitious work without serious challenge as a result of these precedents (Keemer 1984: 57). 'The middle ages' The cost of World War II, post-war austerity, and the subsequent growth in public expenditure with the creation of the 'welfare state' all underlined the importance of ensuring public funds were well spent. Against this background, C&AGs developed their examinations into many aspects of government financial management. One of them - Sir Douglas Henley 17 _ commented that ' [there was a] marked acceleration in scale and coverage from the early 1950s' (Henley 1989: 258). This view is also supported by Dewar, who wrote that VFM: Gwas certainly an established element of the work done well before the Second World War (to say nothing about examinations at the time of the Boer War) and with the growth in public expenditure it developed extensively in the period between 1946 and 1983. In this 35 years or so, the Exchequer and Audit Department published some 1,000-1,500 reports on a wide range of value for money issues, to be followed up by the PAC. Major subjects pursued included: Defence procurement; contract control and pricing; project management in such areas as weapons systems, civil works, roads and hospitals; a wide range of grants and subsidies; National Health Service programmes; agricultural schemes; and assistance to industry. Indeed, value for money work became a dominant feature of the Department's audit, with 58 well developed methods and techniques' (Dewar 1991: 198). VFM practice from 1945 to the 1970s In the post-war period, the E&AD approach to VFM work was broadly as follows. Auditors, working on their own and relying on their own initiative and experience, followed up matters of interest while auditing accounts assigned to them. Individuals were 'responsible for monitoring the activities and developments in [their] allotted field, as reflected in legislation or instructions, published documents, Departmental files, management information returns and recorded transactions, and for exercising [ ... ] experience and judgement in deciding which aspects of VFM to investigate further and how to do so' (Exchequer and Audit Department 1981: 5). There tended to be little review of the plans or approaches taken, and much management time was spent checking the soundness of the end product of the work, and dealing with the 'fall out' from criticisms made of departmental practices. In the case of the larger spending departments, the approach was modified to give financial audit responsibilities to one group of staff, whilst others concentrated on contract expenditure and others stores and equipment. During the 1950s, involvement of senior staff (Chief Auditor and Director 18) in the planning and reviewing of VFM audit grew, and a number of specialist audit sections were established (Exchequer and Audit Department 1981: 9). in general, there was little discussion of proposed investigations with the audited body except to obtain relevant information or explanations of findings. This is particularly interesting given that prior to 1986 many E&AD and NAO staff were physically situated within Government departments (a practice followed today for example by the Algemene Rekenkamer, the SAI in the Netherlands). Following completion of audit work, critical findings were written up in formal minutes, supported by documentary evidence and draft audit queries. More positive findings, including good practice discovered, often went unrecorded. In terms of subject matter, up until the 1970s VFM work consisted mainly of the examination of large schemes or projects, and a wide range of individual transactions. Some subjects largely chose themselves, whilst others arose from financial audit work and 'individual intuition'. Few arose from any specific analysis of risk or evaluation of departmental systems. In addition, it was noted that 'In practice it is not always possible 59 to draw a clear distinction between financial audit for certification purposes and audit for VFM purposes because VFM enquiries may be conducted either in conjunction with certification audit or separately' (Exchequer and Audit Department 1978: 3). Many staff regarded the work as 'point hunting', designed to identify individual failures and lessons to be learned, with wider issues less important. By the end of the 1970s, some steps were taken to counter this approach, with the establishment in 1979 of a Special Studies Unit to develop new approaches to VFM work, including 'across the board' studies. However, this was overtaken by the impact of the passing of the 1983 Act. Having undertaken their work, auditors summarised the main points arising and submitted them to the department under examination (on a Reference Sheet 19 , and submitted usually at Accounting Officer2o level), along with questions requiring departmental response. The findings and the department's replies formed the basis for the C&AG's reports. These were published with the relevant sets of accounts. The C&AG generally chose to report where he was not satisfied with the responses from the auditees, or where he thought the matters examined were of sufficient interest to Parliament. Problems with pre-1983 Act VFM work There were a number of problems with VFM work during this period. Although by the mid-1970s there was greater monitoring by management of auditors' judgements on VFM work, and improvements were made in planning of the work, there was still no consistent approach to VFM coverage. Internally, it was recognised that staff shortages meant auditors were 'occupied mainly with essential certification audit and for any time remaining to be used to follow up VFM matters observed during certification tests; these tend to concern the more obvious points of economy, while deeper questions of efficiency and effectiveness were less frequently pursued' (Exchequer & Audit Department 1981: 5). Outsiders also had concerns about the E&AD's approach. Normanton's observations have already been highlighted (Normanton 1966), and twelve years later, a former civil servant questioned whether the E&AD was an effective defence against waste. His answer, part of a broad examination of 'Whitehall's overspending' (Chapman 1978) was 'to a very limited extent, and could be to a very much greater extent' (p142). In his 60 view, resources at the E&AD were too limited and too much devoted to routine book- keeping matters. There was little scope to initiate investigations, the E&AD was diffident in its dealings with departments, and there was little urgency in reporting of findings. There were a number of other restrictions on the C&AG's VFM work. Prior to 1983 he did not have specific statutory authority to carry out such examinations, and he was advised that legislation precluded him from reporting to Parliament on matters arising from his examinations other than when accounts were laid before the House. As a result, there were complaints that many of the events considered by PAC had taken place some time before they were reported (Committee of Public Accounts 1979). In addition, it was not always easy to find reports, which were published as part of government department's appropriation accounts or as memoranda to the PAC. The need to balance the office's overall audit responsibilities also had an impact. Directors had considerable discretion as to how they managed their accounts, which had led to a tendency to concentrate resources on VFM audit in view of the benefits likely to arise from the examination of the large sums involved, and in response to PAC and Parliamentary interest. Lower priority was accorded to fully documented financial audit work (Exchequer and Audit Department 1978). By the end of the 1970s, however, there was growing concern about the adequacy of this work, partly brought to a head by concern at losses of public funds by bodies such as the Crown Agents (Keemer 1984: 53). In 1978 a review of financial audit work concluded that the work was not being done to a quality that Parliament was entitled to expect. The amount of staff resources had been virtually unchanged since 1945, despite the developments in government over the period. At the same time, standards in the accounting profession had improved and there was a need to catch up (Committee of Public Accounts 1983: 7). In response to these findings, the E&AD reduced its VFM work in order to ensure that certification of accounts was being carried out to a sufficiently high standard. For a while in the late 1970s and early 1980s, VFM work was reduced from two thirds of the corporate effort, to about one third. 61 Parliamentary consideration of the VFM role These developments coincided with increased consideration of the role of C&AG within Parliament. In 1977 the House of Commons Expenditure Committee reported on the Civil Service, commenting that the system of public audit in the UK was outdated. The committee advocated changes to the relationships between the C&AG and his staff, and the Executive and Parliament, and suggested that the C&AG's audit should be extended to cover the scrutiny of the efficiency and effectiveness of the bodies responsible for the accounts that he already audited. In 1978 the Select Committee on Procedure made similar recommendations (Procedures Committee 1978), and in 1979 the PAC reported on the status and functions of the C&AG. The Labour Government accepted the need for a review of the Exchequer and Audit Department Acts, and after the General Election, the new Conservative government issued a Green Paper on 'The Role of the C&AG' (Chancellor of the Exchequer 1980). This noted that VFM audit had been a feature of the C&AGs work for many years, and was the main focus of the PAC's work. The Green Paper stated that any new statutory framework for the C&AG should provide for his examination of VFM, and 'could also in principle cover the development of the C&AG's effectiveness audit where this is a natural and sensible development of his other work. (p9)' In response, the PAC issued a Special Report on 'The Role of the Comptroller and Auditor General' in 1981, which argued for a statutory basis for VFM work, and for VFM to encompass 'effectiveness' audit as 'a logical part of the C&AG's work'. It also endorsed the wide-ranging cross- departmental studies that had started, and supported the Treasury and C&AG's view that there was no reason why the C&AG's VFM reports should necessarily be published with the accounts (Committee of Public Accounts 198 1). The increased discussion of VFM work in Parliament, and the likelihood it would be given a statutory basis, encouraged the then C&AG to establish a VFM Audit Review Committee in December 1980. A year later it reported, making more than twenty high- level conclusions and recommendations. These included that 'Parliament would be better informed about the performance of audited bodies if the C&AG provided a regular summary of the scope and findings of his VFM audit'. They suggested that a single, standardised approach to VFM audit was not feasible, and advocated a General 62 Survey for each accountable organisation through which the most appropriate matters for examination and the most suitable approach could be identified. They also advocated planning periods of longer than one year to accommodate major studies, as well as more 'across the board' investigations and effectiveness studies. They endorsed existing moves towards greater management involvement in strategic planning, methodological preliminary studies and the promotion of greater co-operation with audited bodies (Exchequer and Audit Department 19 8 1). Despite this activity inside and outside Parliament, Garrett notes that 'It was clear that the government did not wish to carry out any fundamental reform of state audit and so Parliament, increasingly concerned about the issue, took the initiative' (Garrett 1986: 423). Consequently, in January 1983 a Private Members Bill on Parliamentary Control of Expenditure (Reform) Bill was introduced with all party support. Much attention was focused on the method of appointment of the C&AG and the extent of the C&AG's rights of access. However, there was also discussion about the VFM role, and in particular, the C&AGs ability to question the merits of policy. In debate, the Government opposed many of the provisions in the bill, but did not attempt to defeat it, and it became law on the last day of Parliament in June 1983. The National Audit Act, as it was retitled, was thus a parliamentary initiative - an unusual development and in contrast to other initiatives during this period discussed at the end of the chapter - designed to modernise the ability of Parliament to scrutinise how Government spent public money. 'Modern times' - the passing of the National Audit Act 1983 The National Audit Act 1983 gave VFM work a new statutory basis and impetus. The key changes were that: qP the Act specifically allowed the C&AG to examine the economy, efficiency and effectiveness with which departments and a range of other bodies had used their resources, but prevented consideration of the merits of policy objectives; it allowed the C&AG to present reports on individual subjects to Parliament at any time the House was sitting; and it provided for VFM examinations at a wide range of government bodies, but 63 excluded the C&AG from nationalised industries, local authorities and a number of other bodies such as the BBC. The National Audit Act thus established a new form of evaluative activity - discrete VFM examinations, which were reported publicly, and which became the basis for the majority of hearings of the PAC. Within the restrictions outlined above, examinations could take place on virtually any aspect of central government expenditure. In general, each examination follows a basic process, which is shown in Figure 4.1. Figure 4.1 Stages in the VFM audit process at the National Audit Office Stage Description of activity Developing a VFM strategy and Monitoring government activity is divided between different deciding what to examine administrative areas within the NAO, broadly mirroring departments. Although topics for examination may also arise from correspondence etc, a strategic planning process is in place to generate study ideas. Planning an examination Study proposals are approved by the Comptroller and Auditor General. Feasibility work to research the main issues and to assess whether there are grounds for further work, is undertaken. Deciding on methodology and criteria Consideration is given amongst the audit team to the scope of the forjudgements study, as well as the proposed methodology. A formal Preliminary Study Report is submitted to the Comptroller and Auditor General, outlining the subject and highlighting anticipated impacts and risks. Carrying out an examination Once approval has been given to go ahead, fieldwork takes place, lasting perhaps 4-6 months. Some may be shorter. Preparing a report A draft reported is produced. Emphasis is placed on producing a document that is succinct and clearly written. Reports are aimed at a lay, rather than specialist, readership. Clearing and agreeing a report There is an expectation that the facts contained in the report presented to Parliament are agreed with the auditee. Audit teams discuss the text with representatives of the audited body to achieve a 'cleared' report. Publication Reports by the Comptroller and Auditor General are laid before Parliament and published by order of the House of Commons. Hearing of the PAC Most C&AG reports are the subject of a hearing by the Committee of Public Accounts. The senior official of the responsible Department is questioned by members of the Committee. The Committee produces its own report making recommendations for action by the Government. Government response The Government responds to the Committee's recommendations in a Treasury Minute 21 Follow up The NAO considers whether there is scope for a follow up report to examine action taken since the PAC report. Source: National Audit OtTice 1997a The arrangements link the NAO, the PAC and the organisation under examination in a process which produces two public reports (assuming that the PAC choose to hold a 64 hearing on the NAO's report and so issue their own report), a formal Government response, and the possibility of a follow-up examination by the NAO where they or the PAC were not satisfied with the response of the executive. The stages in the VFM process are shown in Figure 4.1. Main developments in VFM work during the 1980s and 1990s The place of the National Audit Act 1983 in the broader political and administrative environment of the late 1970s and early 1980s will be considered at the end of this chapter. Before that, this next section considers the themes in the development of VFM work during the 1980s and 1990s, identified mainly through systematic analysis of NAO's Annual Reports, and PAC reports on the NAO's Estimates. The chanainiz focus of VFM work Figure 1.1 indicated that, in contrast to certain inspectorates, there was no regular cyclical programme of VFM examinations, and in contrast to the validation of performance indicators or the audit of financial statements, there was no requirement for annual review of the same areas of activity. Consequently, there is a need to identify and prioritise a programme of VFM studies deemed worthy of the investment of time and resources. The previous section characterised VFM audit pre-1983 Act as influenced by its close association with financial audit, and concerned often with issues of regularity, compliance and waste. Subsequently, the ways in which C&AGs have described the nature of their activities changed markedly. Increasingly, it was characterised as examining larger, wider subjects, and doing so in a more positive and constructive manner. Although the links with financial audit were not neglected, there has also been increased emphasis on VFM work as a distinct activity. Looking back, Sir Douglas Henley noted that: 'Wider studies of departmental organisations, including their systems for financial control and the provision and use of management information, needed to be made. From the early 1980s studies looked at programmes, schemes, major activities and whole organisations, in a much broader way than formerly .... Some subjects important for government efficiency and of interest to Parliament could only be 65 effectively studied by investigating the practice of several government departments. Examples have been the organization of internal audit, the Rayner scrutiny programmes, financial reporting to Parliament, the financial management initiative, and computer security in government departments' (Henley 1989: 261). In 1987, his successor as C&AG, Sir Gordon Downey, considering the changes to VFM reports over his time in post, stated that 'the emphasis of VFM work has changed markedly in recent years. We are less concerned than we were with individual examples of waste and extravagence, although they can still be interesting and important. Now we concentrate almost exclusively on major areas of Government activity' (Committee of Public Accounts 1987). The 1983 Act was thus deemed to have provided the vehicle for a significant shift of emphasis in terms of content and subject matter. Downey's view was repeated and developed by his successor, Sir John Boum, who has emphasised the importance of focusing on the 'big issues' and moving away from looking to 'catch people out'. This change has been reinforced by the emphasis, increasingly highlighted during the 1990s, that VFM studies 'were not just cost saving exercises, but were concerned with identifying better ways of doing things' (National Audit Office 1991b). Thereafter, the C&AG emphasised the aim of undertaking studies 'where we can provide 'added value', for example, by concentrating on performance actually achieved and by making positive, constructive recommendations for improvement' (Boum 1994: 73). Such observations have been repeated on numerous occasions in the 1990s, although, as will be noted later, they have gone hand in hand with the development of a key performance indicator which recorded the value of financial savings resulting from recommendations (see Pollitt and Summa 1996 and Lonsdale in Pollitt et al 1999). The key performance target was to save the equivalent of seven times the cost of the office annually. The subject matter covered by the NAO also evolved in line with major policy developments of the day. For example, in 1991 the NAO reported that, since 1987, fifteen reports had dealt with quality of public service issues (of growing interest to the Conservative Government in the late 1980s and early 1990s), and speculated that this would grow with the development of the Next Steps initiative (National Audit Office 1991b). At the end of the decade, against a background of the Labour Government's 66 interest in 'modernising government' and improved service to the citizen, the NAO reported that it had again produced fifteen reports in the previous three years on quality of service issues (National Audit Office 1998b: 13). The 1991 Annual Report also noted the development in interest in studies related to environmental concerns and assured the PAC that whenever appropriate, these would be borne in mind during VFM investigations. Privatisation also became a regular focus of NAO attention during the late 1980s and 1990s (see Hodges and Wright 1995 and Hodges 1997), as did issues of propriety in the handling of public funds in the mid-1990s. A series of studies of contracts let under the Private Finance Initiative 22 began in 1997 (National Audit Office 1997b). The NAO also increasingly characterised its VFM studies as part of a coordinated and considered strategy. In 1985 the first analysis of the VFM programme against government expenditure was produced. Four years later, Sir Douglas Henley commented that, in the 1980s: 'A much greater degree of 'top-down' planning and supervision was required to ensure that resources were efficiently used and priorities established. Formerly, most VFM reports had emanated from the audit divisions, often though not exclusively, as an offshoot of their financial audit work. This approach fostered and rewarded individual initiative, but could not ensure that the most significant areas were covered. A strategic planning system, looking five years ahead, now operates. Senior management is involved in the choice and conduct of all major studies' (Henley 1989: 260-261). Subsequently, the NAO reiterated the importance of good planning in ensuring that the most significant and risky subjects were identified and pursued, that its work was relevant to the needs and interests of the PAC, and that the costs of studies were driven down (Bourn 1994: 73, National Audit Office 1998b). Resources The issue of what proportion of the NAO's budget could be devoted to VFM work was important from the start. In the early 1980s, the C&AG admitted resources had been spread 'very thinly' on VFM work, so that he had been able to pursue areas of waste and inefficiency, but not do anything like enough to provide the PAC with the 67 information and assurance to which it was entitled on major areas of government spending (Committee of Public Accounts 1983: 6). The PAC showed considerable interest in the subject of the NAO's resources, and in 1983 endorsed a target of a 50-50 split between VFM and financial audit by 1987. This required an effective doubling in effort on VFM work, for which the only answer was to reduce the number of staff certifying accounts and utilise all additional manpower on VFM. Initially, this proved difficult as there were problems recruiting staff of suitable quality in the mid-1980s, but by the end of the decade, resources had shifted heavily into VFM work. In 1990 the NAO reported that 53 per cent of direct costs went on VFM work (National Audit Office 1990). However, this appears to have marked a high point in terms of resources devoted to VFM work, and from then onwards the proportion declined, although a considerable sum of money was spent on VFM -a total of f9million on staffing VFM studies and consultancy support, for example, in 1997 (National Audit Office 1998b). Figure 4.2 below shows the proportion of staff and consultancy resources used for VFM work for most of the 1990s shown on a consistent basis. In contrast, the resources devoted to financial audit grew throughout the 1990s - from around 38% in 1993 to over 53% by 1998-99, in particular as preparation began for resource accounting. Figure 4.2 : Proportion of NAO resources spent on VFM 1991-92 to 1998-99 1991-92 1992-93 1993-94 1994-95 1995-96 1996-97 1997-98 1998-99 Proportion devoted to VFM (%) 43 43 41 38 31 30 27 25 Source: National Audit Office Annual Reports Despite this, the number of studies grew during the late 1980s and early 1990s, before stabilising at around 50 reports per annum from 1992-93 onwards. As resources were shifted to financial audit work, there was inevitably pressure to keep costs on individual VFM examinations down. In 1989 the C&AG reported to the PAC that efforts were being made to reduce the costs of studies, and study budgets became progressively tighter during the 1990s. In 1990 the C&AG advised the PAC that the average cost of a study was E158,000 (Committee of Public Accounts 1992: 30); by the end of 1990s, the figure was around f 18 0,000 (interviews). 68 Staffing In a number of ways, the staffing of NAO's VFM work appears not to have changed significantly since the 1983 Act, although in others it has been radically transformed. The need for qualified, experienced staff for VFM work was noted in 1970s. Then, detailed planning and the execution of major studies were seen normally as the role of Senior Auditors, subject to consultation with, and approval of, Chief Auditors and the directorate. More complex studies required Chief Auditors, working on their own or as team leaders. During the 1980s and 1990s the emphasis on using experienced staff remained. Almost all trainees started on financial audit on joining the NAO, and only had the opportunity to undertake VFM work on qualification. Teams were generally composed of an Audit Manager and a Principal/Senior Auditor, many of whom could have worked on perhaps upwards of 15-20 examinations in the past. The turnover of VFM staff has been noticeably lower than for financial audit work. In interviews, staff put this down to the intrinsic interest and high profile of the work, as well as greater difficulties in finding suitable alternative work, in comparison to financial auditors, who could, for example, join private sector accountancy firms more readily. In terms of their background, most permanent VFM staff remain graduates (in a wide variety of subjects) and qualified accountants. Despite the NAO's move from training staff through the Chartered Institute of Public Finance and Accountancy (CIPFA) to a modified ICAEW23 scheme in the early 1990s, most permanent VFM staff remain CIPFA trained accountants. At directorate level, a small number of outsiders have been brought in (see National Audit Office 1998b: 12 for a summary of the career of one director, who had previous experience with merchant banks, the HM Treasury and Glaxo) to oversee VFM studies on privatisation and PFI. In general, however, audit/accountancy has remained the background for most VFM staff despite these changes. One significant development has been that, in the later period, VFM examinations were primarily organised on a team basis, with only a few undertaken by Audit Managers on their own. In general, though, the size of the teams fell during the 1990s, with most made up, by the end of the decade, of an Audit Manager, with one or two Principal or Senior Auditors. Managers would typically be responsible for two studies at the same 69 time, perhaps at different stages, whilst more junior staff would be allocated to one at a time. Most staff would see the study through from start to finish, although promotion or planned moves for career development purposes might take a member of a team off mid-study. The growth of specialist and consultancy input If the composition of the permanent staff on VFM work has changed relatively slowly, there was a major growth in the use of outside specialist input. Prior to the 1983 Act the PAC had encouraged the E&AD to make use of secondments to and from private sector accountancy firms and government departments (see Henkel 1991 for comparison with use of outsiders for the Audit Commission's work). Several came in from departments and from private firms. In addition, a small number of outside experts were contracted in to give assistance in various fields for short periods of time. In 1987-88 the sum total of this work was two staff years, but the following year, the C&AG noted that specialist advisers were used in more and more studies, and now amounted to eight staff years' worth of input. This grew to twenty years' worth of assistance in 1989-90, the NAO's Annual Report (National Audit Office 1990) noting that advisors ranging from large accountancy firms to individual experts had helped on over 50 studies that year. Throughout the 1990s most studies used some form of consultancy input and the budget grew steadily to E5 million in 1997-98 (or about 16% of resources applied to VFM activities). Most were sector specialists or those with technical skills of relevance to specific examinations, who were contracted to undertake discrete pieces of work. Their knowledge and skills were not necessarily applicable to the on-going programme of studies, so that the approach of contracting them, rather than employing them was the most cost-effective, and ensured that those brought in had recent sector experience. In interviews, senior staff suggested consultants had had 'a profound effect' on NAO's VFM work, and that staff had got better at maximising their input over time. Reporting Perhaps the most significant and visible impact of the 1983 Act on VFM was that it allowed the C&AG to report at any time. Prior to this findings were attached to the volumes of accounts, and 'were not ... always presented to best effect, since they were 70 ff-i I often drafted in a rather flat and factual manner' (Dewar 1997: 323). Thereafter, the C&AG reported separately from the accounts, and in 1984 the first VFM reports appeared. Initial comments were unfavourable. A review of the first twelve reports criticised them for failing to explain how judgements had been reached, for their cautious tone, the lack of clarity about the seriousness of the shortcomings being revealed, and their uninformative nature (Public Money 1984). Since then there was a steady increase in output, helped in part by the free standing nature of the reports, which provided a better focus for completing each discrete examination. Sixteen reports appeared in the 1983-84 session, and 34 in the first full session (1984-85), whereas in 1997-98 the C&AG produced 50 reports. From the early 1990s a target of fifty reports was set, although this was also effectively a ceiling given the demands of other work. Allied to increased volume, the 1990s was also characterised by far greater attention being paid to the appearance of NAO reports (see Waerness in Pollitt et al 1999, but Travers 1999 for alternative view). Stand-alone reports provided the vehicle for trying new methods and enhancing presentation. Reports from the 1980s contained few diagrams and no photographs, consisting mainly of dense, uninterrupted text. From the early 1990s greater emphasis was placed on attractive and informative presentation, with graphics and photographs where appropriate. Gradually, too the introductory paragraphs of most reports began to provide a brief summary of the approach and methods, and from the mid-1990s, more and more included methodological appendices. Relations with departments One of the differences between VFM and other evaluative activities highlighted in Figure 1.1 is that VFM has its main client group, not within the executive, but instead within the legislature. The findings in VFM examinations are reported to Parliament, not to Government. When this is allied to the fact that the subject matter is chosen not by Government, but by the C&AG, an independent official (sometimes taking account of the wishes of parliamentarians), the scope for tensions in the relationship between auditor and auditee can be seen. The timing of examinations and the publication of reports, the scope of the work, the nature of the methods employed, and the criteria on which judgements are made all take on considerable significance, particularly so to an 71 executive concerned with self-presentation. Arguably, the 1983 Act provided VFM auditors with a considerable strengthening of their hand. Parliament accorded the NAO new statutory powers and reports could now appear in a more timely fashion. In these circumstances, it would have been surprising if this did not have an impact on relations between auditor and auditee. To many senior civil servants, much early VFM work did not recognise the realities under which they operated, and judgements were made against impossible standards. In their view, the points made in reports generally could not be disputed, but did not seem to merit such public airing. Despite the steady developments above, others viewed the PAC's increasing tendency to look at VFM reports rather than consider regularity and certification issue with 'some concern, not to say resentment. ' (Committee of Public Accounts 1986a: 13). A common view in Whitehall (see Pliatzky 1989) was that the NAO was always trying to push back the frontiers of its remit and to encroach on policy matters. Other aspects of the work caused friction. An essential feature of the VFM reporting procedure in the UK is that by convention (believed by one former senior official to stem from the late 1950s) C&AG's reports are 'cleared' or agreed documents, with the auditees being asked to confirm the accuracy of the facts and their presentation. Delays to the publication of VFM reports caused by adherence to the convention were identified as a problem as early as 1986. When asked about this by the PAC, the C&AG argued that the more penetrative reports became, the more departments wished to ensure they were factually correct and balanced in their presentation. He stated that some departments often became defensive. The matter came to a head in 1986, when an Accounting Officer disagreed with some of what was written in the published report during a hearing, and expressed reservations about how far he was expected to agree NAO views and conclusions. Subsequently, a joint Memorandum was prepared for the PAC by the Accounting Officer of the Department of Education and Science and the C&AG. It stated that throughout the clearance the NAO had assumed that departments understood that, in agreeing his Reports with departments, the C&AG's intention was to establish that: 1(i) all material and relevant facts had been included; 72 (ii) the facts were not in dispute; (iii) their presentation was fair; (iv) where the Report stated any NAO views or conclusions which the Department were unable to agree, this was made clear and the NAO's and Department's reasons properly represented, with all necessary balance' (Committee of Public Accounts 1986b). This is now considered the definition of the 'clearance' process, the purpose of which is to enable an agreed report to be prepared, so that the PAC can concentrate, during their hearings, on seeking reasons for problems and identifying ways of taking matters forward. Improving relationships more generally with auditees came to the fore in the early 1990s. Traditionally E&AD staff had shied away from close dealings with those they were examining. Henley, looking back on the 1980s, commented that 'By long standing tradition, E&AD's relationship with departments had been markedly at arms's length, with the overriding aim of safeguarding the C&AG's independence. This has now changed and the NAO discuss with departments likely choices for studies. There were also regular contacts between the NAO and departmental finance officers' (Henley 1989: 261). Such contacts appear to have developed during the 1990s as part of the general engagement of the NAO with the wider world (see Pollitt et al 1999). Internal guidance now makes it clear that audit teams must discuss proposed methods with departments. Departments are consulted on the choice of topics, and whilst maintaining the right to undertake work at any time, the NAO has also paid greater heed to departmental protestations that, for example, the timing of particular examinations is unsuitable. Departmental representatives are invited (and often agree) to sit on expert panels. This all emphasises that, in reality, VFM work is not possible without a significant degree of co-operation between auditor and auditee. The need for staff to be made available, visits to sites arranged, and for findings to be discussed in the course of the finalising the report, ensures that the kind of unplanned visits and random scrutiny that Hood et al (1999) favoured are rarely possible within the current VFM approach. It also ensures that going against very strongly felt views of departments, for example, about particular 73 methods, are difficult, if not impossible. For these reasons, the NAO have sought to develop relations with auditees, rather than seek confrontation. Such an approach requires care - being seen to be too cosy with government runs the risk of appearing to have been 'captured' in the eyes of Parliament. Concern at the often adversarial nature of relations between NAO and departments in the early 1990s led the C&AG to ask Sir Brian Cubbon (one time Permanent Under Secretary of State at the Home Office, who was supported by Price Waterhouse), to undertake an examination of the selection, planning and execution of VFM examinations, the way in which departments co-operated and contributed to the work, the impact of NAO VFM work in contributing to securing VFM, and the improvements that could be made (Cubbon 1992). Cubbon's report commented widely on the VFM process, noting that 'the rigorous and adversarial character of the PAC's examination casts its shadow in advance'. He made a series of recommendations, including that the NAO's selection of subjects needed to be seen to derive from an ob ective and j transparent survey of expenditure fields; there should be no assumption that an NAO examination would lead to a PAC hearing24 ; and that reports should be narrower and more tightly focused. In the light of the Cubbon report, increased efforts were made during the 1990s to ensure that departments were clearer about the NAO's work. In the mid-1990s many VFM staff attended 'client relations' courses, designed to highlight the importance of keeping departments aware of progress on studies. A process was established whereby departments were invited to comment on how they felt the study had been conducted, and their views were incorporated in the internal 'quality review' process for each report. Finding ways of assisting auditees was another response to making the essentially confrontational nature of the VFM process more 'positive'. The 1990 Annual Report, for example, detailed work carried out in addition to that for published reports, noting that 'over the past two years we have been developing new and varied types of work and outputs to support the drive for better VFM and to share our experience and knowledge' (National Audit Office 1990: 10). This included providing advice to management, computer security reviews, and work with the new Next Steps Agencies. It also included production of good practice guides arising from VFM studies, and using 74 conferences to disseminate views and best practice. Despite such efforts, tensions between the NAO and government departments came to the fore again in 1993-94. By another convention, certain departmental papers were marked 'Not for NAO Eyes'. These related to departmental responses to NAO enquiries, briefing papers for departmental appearances before PAC, papers dealing with responses to PAC Reports in Treasury Minutes, and papers about departmental relations with NAO. In 1988 the matter had been investigated by PAC, and in 1994 the Committee again became concerned after extracts of a paper marked 'Not for NAO Eyes' was published in the press. This document was a brief that warned staff how to deal with interviews with NAO staff, and explained how the NAO's fieldwork would be handled. Unease at this designation continued, and in 1997 it was ended. Specialisation in VFM work and training for VFM staff During the 1990s another important development was the creation of, effectively, a separate cadre of VFM auditors. After the implementation of the 1983 Act, VFM and financial audit work continued to be organised within common divisions, and most staff were expected to be competent in both (Committee of Public Accounts 1983: 6). This was seen as an efficient use of resources, and the two strands of work were planned separately and, to some extent, were seen as requiring separate skills. By 1991, however, there was increased specialisation in VFM, and from around this time, staff opted to concentrate on either financial or VFM audit after qualification, the aim being to encourage greater professionalism in approach. This was a recognition that the demands of both elements of the office's work were increasing, and that it was unrealistic to expect staff to be fully conversant with the growing range of financial audit requirements (the main concern), and the changing nature of VFM work. Only at the level of Assistant Auditor General - NAO's senior management - were officials responsible for both types of work. Thereafter, the needs of specialist VFM staff were increasingly recognised in the preparation of a range of specific guides (of little or no relevance to the work of financial auditors) that first appeared in 1993 and, from 1996, in a new series. Training for VFM staff also developed significantly. Increasingly outsiders such as academics were brought in to give lectures, and formal courses were augmented by a growing 75 number of 'masterclasses' and lunchtime talks by outside speakers in subjects as diverse as conflict resolution and negotiation skills (seen as relevant to handling clearance). External qualily assurance review Finally, the issue of the quality of NAO reports grew in significance during the period. The Cubbon Report drew attention to some negative aspects of the reporting process, and there were also indications that PAC were not entirely satisfied with every report it received (Committee of Public Accounts 1992). Internally, too there was a recognition that the quality of reports varied considerably. To try to make improvements, quality assurance arrangements were introduced in the late 1980s and early 1990s. Initially, quality panels were established of staff unconnected with a particular study, who provided comments on draft reports. From 1992 a reference partner system was established in some parts of the NAO. Subsequently, in 1993 the Centre for the Evaluation of Public Policy and Practice at Brunel University was appointed to undertake a 'cold' external quality assurance review of each published report. Under this self-imposed process, a team of academics produce a paper on each report, and score them, shortly after publication, against a series of criteria. The approach was designed to provide some independent view of report 6quality' for internal consumption (the scores are not published) as a means of steadily raising standards through exposure to advice, guidance and, where appropriate, criticism about the finished product and the route taken to get there. The approach has considerably more in common with some form of academic peer review or a 'colleague advisory body' (CEPPP 1999: 15) than a formal inspectorate role. In 1995 the London 25 School of Economics took over the contract, and in 1998 the criteria were extended Comparison with other developments This chapter has so far focused on the development of VFM work at the NAO yet, as noted in Part 1, the changes outlined have not occurred in isolation. Instead, they have taken place against a background of major change in the public sector in the UK, as well as in the wider field of evaluation. To shed further light on the developments at the NAO, the rest of this chapter considers briefly how other forms of evaluatory activity have changed over the same period. 76 Evaluatory activity within the Government in the UK Part I referred to the period from 1979 to 1999 as one of radical upheaval in the way public services were provided in the UK. This included, amongst other things, widespread privatisation, the use of the private sector to provide public services, the introduction of the Private Finance Initiative, customer charters setting performance targets, the creation of more than 140 Executive Agencies, quantitative methods of performance and efficiency measurement, and increased differentiation of pay. These developments run parallel with the growth of VFM during the 1980s and 1990s (Pollitt and Bouckaert 2000: 200), and according to at least one commentator (see Pollitt in Pollitt et al 1999) the relationship between the reforms and VFM is both 'intimate and complex'. The debate about the role of the C&AG that took place in 1970s was outlined at the start of this chapter. This encompassed concerns about the adequacy of the audit approach taken at that time, including the shortcomings in examination of efficiency and effectiveness, and the tendency to concentrate on 'the more obvious points of economy', procedures and regularity. Such concerns mirrored wider concerns about the adequacy of other evaluative machinery, which was increasingly found wanting in the late 1970s. Evaluatory bodies in the UK were seen as unambitious and ill-prepared for consideration of outputs and outcome (Henkel 1991: 9). As a result, although the NAO's creation, and the expansion of the VFM role, were not directly a result of government action, nevertheless, the scrutiny of the existing arrangements and their reconstitution with expanded horizons and expectations were entirely in keeping with wider trends. These included, for example, the creation of the Audit Commission and the Efficiency Unit, the introduction of managerial criteria of economy, efficiency, and effectiveness into the mandates of various central government inspectorates (Clark 1996: 25), and the Financial Management Initiative. The timing of the start of these changes is open to debate. Hennessy (1989) records the establishment of the Central Policy Review Staff and the Programme Analysis and Review system (PAR) in the early 1970s. To Pollitt, developments took place within a period of 'managerialism' running from the late 1970s to the late 1980s. During this period: 77 'The emphasis of the government's evaluatory efforts moved away from those ambitious attempts to identify effectiveness and impact which had characterized the age of enlightenment. Now efficiency and economy were the main foci. The driving manager, not the 'scientific' policy analyst, was the charismatic figure of the period. Planning went (temporarily) out of fashion. The achievement of performance targets (usually including 'savings' or 'cost improvements') became the sign of good administrative health. Many government documents gave pride of place to the pursuit of the 3 Es (economy, efficiency and effectiveness) and 'value for money' (VFM)' (Pollitt 1993). Henkel too notes that 'vigorous attempts were made [in the early 1980s] to establish economy, efficiency, value for money effectiveness and performance, as incontestable values, essential to the sustainment of political order (Henkel 1991: 12). Derlien (1990), from a comparative perspective, notes that in the UK policy evaluation was closely associated with a managerialist policy approach and an emphasis on 'VFM'. Such concerns put the focus of the NAO's work - VFM - at the centre of the political agenda, and meant its preoccupations were in tune with government concerns to control public expenditure, and improve public sector performance. Much of the NAO's work during this time reflected these concerns, of which examinations of progress of two of the most significant projects - on the Financial Management Initiative and the Rayner Scrutinies - are only the most obvious examples. Towards the end of the 1980s, and in particular, after the 1987 General Election, further important developments took place in the public sector. Pollitt characterises this as the 'Reconstructionist' phase - one marked by privatisation, fragmentation and redesign of the public sector, and the establishment of purchaser-provider regimes that required evaluation of services and performance. This required (and generated) improvements in information and clearer objective and standard setting. A key aspect of the changes in this period was the growth of interest in 'quality' and 'service to the citizen'. To Pollitt 'Quality' and 'standards' [became] the latest theme terms, and the new evaluatory arrangements [were] supposedly designed to check that service providing organizations [were] delivering flexible, cost-effective services to their citizen-users'(Pollitt 1993). 78 The development of the focus on the user as consumer provided the NAO with new scope for VFM examinations, which, as was noted earlier, it took up. NAO literature stressed strongly throughout the 1990s that, although its concern was with accountability to Parliament, this meant 'ultimately the public' (National Audit Office 1991b). Reports on quality of service, drawing on existing consumer data or based on commissioned surveys began in the late 1980s, and have continued to the current time in one form or another. The NAO has sought to capture the opinions of customers of numerous public services. At the same time, many public sector bodies looked to develop performance indicators during the late 1980s and early 1990s, and there was much activity in standards setting, inspection, quality assurance etc (Pollitt 1993). Jackson points to the NAO (along with the Audit and Accounts Commissions) as having helped to encourage these early steps (Jackson 1993: 11). Explicit assessments of agency performance against their Citizen's Charter target were produced in the mid- 1990s, and full-scale examinations of performance measurement systems, target setting and actual performance started to appear in the late 1990s (see for example, National Audit Office 1998i). Developments in this area were limited somewhat by opposition from the Government to NAO validation of performance data (Committee of Public Accounts 2000), which, as a consequence, has taken place in a somewhat piecemeal fashion. Another parallel between developments in broader evaluation and VFM work comes with the growing interest in outcomes and outputs. Pollitt refers to criticisms in the late 1980s about the neglect of effectiveness measures in favour of criteria of efficiency and economy, and notes that, from the late 1980s, public sector bodies attempted to develop 6value added' measures. As noted in Part 2, concern about the failure to address the issue of effectiveness has long been part of the discussions about VFM audit (Anand 1988, Glynn, Gray and Jenkins 1992), and Jackson called it 'the hole in the middle of the VFM doughnut' (Jackson 1993: 12). Efforts to correct this have continued during the 1990s, with NAO guidance by the end of the decade actively encouraging staff to identify ways of undertaking studies that examined the outputs and outcomes of governt'nent programmes (National Audit Office 1999a). To some (see, for example, Schwartz 1999), there is still some way to go, whatever the indications of interest. 79 Overall then the development of VIM has occurred during a period (the 1980s and 1990s) when intellectual fashion and technical changes have increased the attention that has been paid by government to issues of performance. However, developments towards the end of the period raise the possibility that the VIM approach may be becoming out of tune with new approaches to improving public services. Martin and Sanderson (1999) point to the increasing use by central government of pilot programmes as a means of modernising key public services. They consider this marks 'a new style of policy formulation and implementation which in turn calls for a different kind of evaluation support. ' Ex post analyses are regarded as less useful, and in the case of the national Best Value 26 pilot programme, for example, evaluators have been asked to act as 'change agents who are able to combine summative analysis of outputs and impacts with more formative approaches focused on developing a detailed understanding of processes. ' They add that, in these circumstances, 'evaluators therefore become more than scientists producing traditional, positivist impact assessments. They also have to engage with change (and sometimes become change agents). ' The revived interest in peer review in central government - outlined in Cabinet Office 1999 - also suggests an approach to performance improvement at variance to the style of VFM work. Again it suggests a more formative, participatory approach, in which departments learn from each other. And, even where the Government have argued that they are setting up "an OFSTED for the NHS', with all the connotations that has, the organisation itself - the Commission for Health Improvement - has emphasised that its work 'will be evidence- based, not opinion-based. Our style with be developmental, not confrontational. ' (Ward 2000). Other evaluatory bodies Finally, the development of VFM can be compared with the growth of other evaluatory bodies. During the 1980s and 1990s a number of such bodies were established (for example, OFSTED) or given enhanced remits (for example, the Audit Commission). A number of examples are summarised in Figure 4.3. In one way or another, all seek to improve performance, but as the table shows, do so in different ways, and in a variety of styles. The development of the NAO's VFM work has, therefore, taken place as others have grown up in different ways. 80 Figure 4.3 assesses these organisations against several criteria: authority, form of enquiry, client group, degree of independence, and style. All the bodies, except internal audit, have specific statutory authority for their work. In each case, this produces an element of formality to the work, and provides strong powers (for example, the right to undertake major examinations or carry out inspections, or the right to report). Whether they are OFSTED examinations, or appearances in front of the PAC, such examinations are taken seriously (see Lipsey (1999) for the effect of the latter on senior officials) because of the formal nature of the occasion and its implications for careers. Turning to relations with client groups, the NAO stands out on one extreme of a spectrum ranging from complete independence, through to total dependence on a sponsor. All the bodies in Figure 4.3 maintain an element of independence, even internal audit, despite their position supporting management. Internal audit is the most limited, although it is usual for this service to report directly to the Chief Executive of the organisation. Moving on to inspectorates, Travers (1999) suggests that 'However, theoretically independent the various 'HMIs' are, they generally look to their sponsoring department for guidance about their current priorities ... The police and fire inspectors have long operated within well-understood Home Office rules. ' The inspectorates are also funded by government grants. In the case of the Audit Commission, the willingness to be critical of both local and central government when considered appropriate, and a strong profile in the media help to assure the Commission of its own independence, even though it reports to the Secretary of State, is a non-departmental public body, and is funded, in part, by government. The NAO's position is more independent still. As noted earlier, the 1983 Act freed it from Treasury control, and the C&AG determines his own programme of work. He can report freely, and does so in published reports to the legislature. Funding comes from Parliament, not the executive. But even the NAO is not totally independent, and, as will be seen in Part 6, the need for negotiation over the timing and content of studies, ensures that there are some constraints. 81 Figure 4.3: The approaches of different evaluatory bodies. NAO Audit Inspectorates (eg Parliamentary Internal audit Commission OFSTED) Ombudsman services Authority Statute. Office of Statute. Powers Statute. Often Statute. Officer Part of internal the House of to initiate wide national bodies of the House of management. Commons. ranging reviews. with strong Commons. Ability to initiate Ability to report powers to inspect Ability to require wide ranging publicly. on basis of departments to reviews. Formal nationally provide redress parliamentary determined following mal- hearings examine criteria. administration. reports. Recommendations from hearings must be responded to by Government. Form of Audit. Audit/inspection Inspection. Investigations Audit. enquiry into complaints against public bodies. Client groups Parliament, Local and health Secretary of Members of the Management of government authorities, State, schools. public via their government departments, the Secretary of Members of departments. wider public State, wider Parliament. sector. public sector. Degree of Highly Independentof Linked to DfEE. Highly Situated within independence independent of Government. independent of Government. Government. Government. Style Public exposure Highlighting of Naming and Relatively Resource for of performance, good and bad shaming. legalistic study management, with behind performance Publication of of maladminist- who input into scenes assistance. through perfon-nance ration in order to work Often high media comparisons. tables. establish merits programme. For profile. High media High media of complaint. internal, rather profile. profile. Limited media than public, profile. consumption. No I media profile. I Sources: Travers 1999, CEPPP 1999 In terms of style, the NAO's VFM work has developed to fit in somewhere in the middle of a spectrum that runs from the approach of punishing publicly and being able to take executive action, through to behind closed doors" revision of practices. In general, the VFM approach avoids the 'naming and shaming' style, identified with OFSTED, and many reports do not name individual bodies displaying poor performance (although the PAC often expect to have the bodies identified during hearings). To date, there has been no performance tables of government departments. There is also considerably more effort made to take account of the views of those being examined - 82 in terms of criteria or methods - than the OFSTED process, although less so than processes of self-evaluation (CEPPP 1999: 16). There is also no punitive role for the NAO or PAC (in contrast the auditors working for the Audit Commission have the ability to surcharge). However, the public nature of the PAC hearing and the extensive press coverage for reports ensure there is far more transparency about the approach than, for example, the work of Internal Audit. But there are limits. The Osmotherly rules 27 (Hennessy 1989) apply to PAC hearings, ensuring some limits on what information must be provided to the Committee. Overall, this analysis suggests that the NAO has developed in a number of ways. It is a body characterised by a high degree of independence, perhaps more than any other of the evaluative bodies. It has acquired a high media profile, although perhaps not as high as the Audit Commission (which in the absence of a parliamentary committee in which to platform its work is alert to the benefits of media attention). The NAO's style has increasingly been to progress its formal role of reporting publicly on performance alongside an approach of seeking to be helpful behind the scenes. The Audit Commission Because of its status as the other main 'national' audit body, it is worth making a more detailed comparison between the NAO's work and developments in the work of the Audit Commission - auditor of local government in the UK. The Audit Commission for England and Wales was established under the Local Government Finance Act 1982 and took up its functions on I April 1983, nine months before the NAO came into being. There are a number of comparisons with the way in which the NAO came about. Like the NAO, the Audit Commission grew out of concern in the 1970s about the need for improved monitoring of public expenditure. As with the NAO, so the establishment of the Audit Commission was a contested affair, with local government, in particular, strongly opposed to the proposals. As they had with central government, the Conservative government also opposed parliamentary scrutiny of local authority expenditure (Henkel 1991: 26-27, Hollingsworth and White 1999: 142-3). Like the NAO, the Audit Commission had a clear remit to promote economy, efficiency and effectiveness, but there are a number of aspects of the Commission's approach and development that are clearly different. In the first place, from its creation, the 83 Commission regarded itself as having a proactive approach to promoting change and being responsible for helpin local authorities achieve better VFM. As Henkel put it, the Audit Commission 'made it clear that it interpreted its duties to oversee economy, efficiency and effectiveness in local government in wide terms, to include a remit to help improve local authority management and so exert a fundamental influence on it'(Henkel 1991: 28). Such an approach appears to have had an effect, and although there was considerable mistrust of the Commission for much of the 1980s (the Labour Party was committed at the 1992 election to abolishing it), by the mid-1990s, a review of the Commission was able to note that 'Stakeholders welcomed the way in which, following its 1993 strategy, Adding Value, the Commission has worked more closely in partnership with audited bodies' (Butler 1995: 9). In contrast, assisting bodies to improve VFM achieved appears to have always been a secondary objective of the NAO, and one that has developed more slowly during the 1990s. Few Government departments would accept, publicly at least, that the NAO existed to help them, primarily because the public nature of the PAC hearing, always holds out the possibility of well publicised criticism. Notwithstanding the very severe penalties that auditors working for the Audit Commission can impose on local government representatives (the Westminster Council case in the late 1990s being perhaps the most high profile example), its founding legislation also clearly required the Commission to make recommendations for improving the provision of local government services. Auditors were encouraged to disseminate good practice. In contrast, there has always been a tension in the relationship between NAO and PAC (and indeed between the NAO and the Treasury) as to where responsibility lay for making recommendations for improvements. Only in a limited number of cases have NAO studies generated formal good practice guides (although many more have included recommendations with wider applicability), and then mainly in areas where a large number of subsidiary bodies, such as colleges and universities. More recently, the approach to providing broader advice has changed, and the NAO now produces and disseminates to government bodies, a publication called Focus, which highlights key findings from reports. 84 Legislation has also encouraged the Commission to undertake or promote a comparative approach, with studies of specific themes, rather than particular bodies. This was clear under the 1982 Act, and under the Local Government Act 1992 the Commission was charged with developing sets of performance indicators to facilitate comparisons between local authorities (Bowerman 1995: 171). Some have criticised this approach, arguing that the Audit Commission has become an 'agency for homogenisation of local government', and through the use of performance indicators, has made important what can be quantified, and not quantified what can be made important (Simon Jenkins quoted in Thatcher 1999). Nevertheless, it is an approach that has been replicated in schools, hospitals and local government. In contrast, the NAO's tendency to focus on specific organisations has been noted (Summa in Pollitt et al 1999), as has a general shift away from cross-departmental examinations in the early 1990s. In part, this is likely to be a reflection of the respective audit fields. Whereas the Commission examines a field populated by some 400 or 500 bodies, all providing many of the same kind of services, the NAO more usually audits numerous stand-alone departments and agencies, undertaking unique activities. NAO staff also emphasised the constraint that the working of PAC has been on encouraging comparative work. The Committee is believed to prefer to be able to direct its questions on one Accounting Officer. More recently, however, there appears to have been some moves towards greater comparative work by the NAO, and official encouragement to the use of benchmarking (National Audit Office 1997k). Finally, recent developments appear to being pushing the Audit Commission towards more of an inspectorate role than before. According to one observer (Thatcher 1999) only some 5% of the Commission's work had, at that time, been inspection-related (including work with the Social Services Inspectorate and partnership work with OFSTED, the Housing Corporation and the Benefits Fraud Inspectorate). Until now, some 70% of its activity has involved audit through the District Audit, and some 25% on analysis, such as the national reports. It appears that this is now changing, and the creation of Best Value and housing inspectorates will push the Commission's inspectorate role to some 50% of its activity. Such a change draws an even stronger distinction between the way in which the NAO and the Commission operate. 85 Conclusions about the development of VFM audit practice This chapter concludes by surnmarising the changes in VFM work and contrasting these changes with the other, wider, movements in evaluative activity in the UK. Figure 4.4 surnmarises the developments in VFM audit practice between the late 1970s and the late 1990s. It suggests that VFM audit practice has shifted in a number of fundamental ways. Figure 4.4: Changes in the characteristics of VFM work at the MAD and NAO Latel, 970s/early 1980s Lat 1 1PPS, Same staff undertaking financial audit and VFM Staff now specialising in either financial audit or VFM work. work. Concentrated on individual examples of error, Concerned with wider issues and systems of control waste or extravagance and immediate lessons to be and arrangements for securing good VFM, with closer learned, working through PAC. and more constructive relations with audited bodies. Areas of examination and cases to be pursued Detailed strategic planning based on analysis of risks decided mainly by individuals using their to VFM, and senior management approval of studies to experience and initiative. be undertaken and issues to be pursued. Work based almost entirely on departmental Work based on a wide variety of departmental and material. external sources. Mainly concerned with fairly limited range of A far more extensive and adventurous menu of subjects and types of examination, pursued in well subjects for examination, taking more difficult issues established ways. with a wider range of approaches and methods. Work undertaken almost entirely by NAO staff and Work undertaken by combination of audit staff and limited number of seconded specialists. consultants. Little discussion with audited bodies on selection Regular contact with audited body on the selection of of examination and conduct of work. studies, issues, methodology, emerging findings and preparation of report. Examinations often undertaken by one person. Examinations undertaken by small teams. Brief factual reports submitted with accounts once Free standing reports submitted separately when ready a year, with little expression of E&AD views or throughout the year, with more NAO analysis and conclusions. conclusions. Little or no information about how work More usual to have methodological appendix in undertaken in reports. reports. Mainly drawing upon accounting and management As before, but increasingly supplemented by information, with emphasis on quantitative and qualitative evidence and 'soft' data. 'hard' data. Staff background predominantly accounting and Increasingly supported by wider range of other skills I audit. and expertise. Part I placed VFM work firmly in the range of evaluative activities in place in the UK. This chapter has examined its distinctive history, but nevertheless illustrated that the development of the NAO's VFM work has clearly mirrored some of the wider developments and interests in evaluation in the 1980s and 1990s. The growing concern with assessing (and achieving) VFM, with establishing clearer objectives, generating 86 better performance data, and with measuring 'the quality of service' and effectiveness, which were all characteristic of discussions about evaluative activities in general, have all been seen in the debates over VFM work. Concerns about 'value for money', which had been a concern of auditors almost alone in previous decades, took centre stage. Doubts about the adequacy of evaluative organisations to fulfil the roles required of them can also be seen in the misgivings amongst many about the E&AD. Although the shifts in the work of the NAO clearly mirror broader trends, in many ways the historical development of VFM is different. The pressure for the extension of the C&AG's mandate to VFM came from within Parliament, not Government, and part of a broader debate about parliamentary accountability, which also saw the creation of departmental select committees in 1979 (Drewry 1989). Thus, the focus of VFM work has always been on what has been done in the past, as much as how improvements could be made for the future, a point reinforced by the presence of PAC at the end of the process, with its concern that Accounting Officers are held accountable for the specifics of how they have used their resources in past, rather than the vaguer promises of how they will use them in the future. Any doubts that the PAC process is about accountability can be put to rest by the recent comments of the current Chairman28, who confirmed (Burkitt-Gray 1999) that the hearing was 'designed to be intimidating. ' The comparison with the development of the Audit Commission also points up the greater accountability focus in the NAO's work. Compared with the Commission's examinations, NAO's VFM has been slower to move towards a formative and collaborative approach to promoting change and helping audited bodies to effect change (a point recognised by Day and Klein 1990). It does not consult with stakeholders in the extensive way that the Commission does, although it does discuss its programme with the PAC and departments. It has been less focused on producing and disseminating good practice guidance, and has also tended to examine individual organisations, where the accountability for performance can be clearly identified, rather than take a broader comparative approaches. By the end of the 1990s the impact and implications of the VFM approach, in particular, as practised by the NAO, had become a topic of debate. Some have expressed the view that the accountability focus of the NAO's work and the presence of the PAC at the end 87 I of the VFM process act as a discouragement to innovation and risk taking on the part of civil servants. The Cabinet Secretary was quoted (Gosling 1999) - admittedly in responding to a rather loaded question - as suggesting that the NAO's role may have to change to be more like that of the Audit Commission 'The Audit Commission's way of working has been very successful, ' he said. 'I do believe that fear reinforces certain types of behaviour. The attitude to risk is important. ' The effects of devolution may also be felt in the way that audit is conducted in England. The Auditor General of Scotland wrote shortly after his appointment in early 2000 that 'We have taken the opportunity to think afresh about the process of effective democratic scrutiny and the role of public It audit'. As part of this, he suggested his approach would be more strategic than that operating in Westminster, and less concerned with 'naming and shaming' (Black 2000). But there were signs of the NAO responding. In 1999 it began a series of studies N examining good practice across government in a number of key areas - procurement and risk management being the first two - with the stated aim of highlighting good practice, rather than criticising weaknesses and shortcomings. It remains to be seen whether this is the direction for a majority of reports in the future, or whether it is an approach that will be subsumed into many reports, just as the focus on quality of service did in the early 1990s. I 88 A PART FIVE: METHODS USED IN NAO VFM STUDIES I Part 4 examined how VFM work had changed at the NAO in the 1980s and 1990s. One A area that was not addressed was the way in which audit teams collected and analysed evidence for their examinations. This part now looks in detail at this aspect of VFM Act was passed in 1983. work in order to identify trends in the development of methods since the National Audit Two statements about VFM methods - one official and one from an outside observer - act as a starting point. The first - from a document circulated to NAO staff - stated simply that 'Designing VFM examinations is a complex business and it is necessary for us to have a wide range of methods and tools at our disposal' (National Audit Office 1998k). In the second, Keen has stated that 'Methods are, at base, what you use to uncover the truth about what is going on. VFM practitioners employ a variety of methods to arrive at judgements about specific points within studies' (Keen 1999b). This chapter, therefore, aims to uncover the truth about this perceived diversity, and to assess what changes, if any, have taken place as VFM work has developed in the 1980s and 1990s. Part 6 will then look at the factors that appear to have influenced the choices. The range of evaluation methods Before examining the methods used by the NAO, it is worth considering the breadth of evaluation methods more generally upon which auditors could, potentially, draw. Methods can be seen in a number of ways. A distinction can be drawn between quantitative or qualitative methods - although some methods (eg interviews) can be used in both ways. The emphasis of the former tends to be on the testing of hypotheses and measurement. This is a scientific positivist position that argues that there is a real world out there that can be measured. From this perspective, the purpose of evaluation is to establish as definitively as possible what is happening in this real world. The emphasis of the latter is more about discovery and exploration, and tends to employ methods of data collection and analysis that are non-quantitative. Quantitative methods tend to separate the research object from its context, and assume the researcher plays a non-participant role. They seek to achieve objectivity. Qualitative methods support more naturalistic and interactive inquiry, studying real life situations. Qualitative research argues for personal involvement of the researcher in order to understand those 89 they are studying, and less emphasis on quantification. The methods employed within these different positions exhibit distinctive characteristics as illustrated in Figure 5.1. Figure 5.1: Potential range of characteristics of evaluation methods Illuminative Standardised Interactive Technocratic Inductive 4 Deductive Interpretative Instrumental Exploratory Hypothesis testing Qualitative Quantitative Within this range, there are numerous methods used by quantitative researchers, including surveys, experiments, and a range of statistical and mathematical methods. Qualitative researchers use less structured techniques of data collection. These include in-depth interviewing, verbal and text analysis, participant observation, field observation and case studies (Sarantakos 1993). Amongst more interactive methods are open-ended interviews, document analysis, life history, personal experience and self- story construction, participant observation (Denzin 1989 quoted in Melia 1997). The choice of method will be affected by the purpose for which they are to be employed. The rest of the chapter considers to what extent the work of the NAO draws on these and other methods. The background to VFM methods Any consideration of methods used in VFM work must start with an examination of the basic principles of audit to be found in the technical or theoretical literature, and in the auditing standards and guidelines that have been promulgated by the profession. It should also take account of the legislative provisions applying to audit work. Principles of audit Part I reported Flint's views that the collection and examination of evidence were a fundamental part of the audit process. He argued that 'obtaining the relevant competent evidence is the central purpose of the audit investigation purpose' (Flint 1988: 106), and stated that it is an essential premise that 'without evidence auditors have no basis on which to form a judgement and express an opinion. If there is no evidence an audit is not possible' (p3l). As a result, the subject of an audit must be 'susceptible to 90 verification by evidence. ' He adds that 'the persuasive quality of the evidence will depend on its intrinsic nature and the reliability of its source' (pl08). In addition, Flint emphasises the significance of cost effectiveness when considering the means of gathering audit evidence, postulating that an audit must produce an 'economic or social benefit'. This is an important issue for SAIs wishing to consider how they - as guardians of value for money - can demonstrate it in their own work. t Intemational audit guidelines This theoretical background provides a basis for, and is complemented by, auditing standards produced by auditors themselves. The International Organisation of Supreme Audit Institutions (INTOSAI 1992) has prepared and revised audit standards in the last 1ý twenty years. With regard to evidence collection, these standards echo the fundamental points that 'audit findings, conclusions and recommendations must be based on evidence' (paragraph 153), which should be 'competent, relevant and reasonable' (paragraph 152). They stress that the means of data collection must be carefully chosen by auditors who 'have a sound understanding of techniques and procedures such as inspection, observation, enquiry and confirmation, to collect audit evidence' (paragraph 154). Independence is also considered an essential pre-requisite for the auditor, with a key element of this that audit bodies are able to decide how they conduct their own work. This is a purist position, and it remains to be seen in Part 6 how realistic this is in the real world. Another relevant aspect of the Auditing Standards is the encouragement given to auditors to develop audit approaches and learn from other sources. Audit bodies are encouraged to co-operate with academic institutions and professional bodies, as long as this does not compromise their independence and objectivity, in order to draw on their knowledge and experience (paragraph 8 1). They 'should avail themselves of techniques and methodologies of other disciplines'(paragraph 46). There thus appears to be an expectation of innovation and receptiveness to new ideas. ýb UK leizislation The basic audit legislation in the UK, the 1866 and 1921 Exchequer and Audit Departments ActS29' provide the statutory basis for financial audit, although more than one C&AG considered the 1921 Act an adequate basis for examinations of how funds 91 had been used. It was the National Audit Act 1983, however, that provided a new statutory basis for VFM work and, as was examined in Part 4, gave a new impetus to this work. The Act provides for the auditor to have access to: 'all such documents as he may reasonably require for carrying out any examination ... and shall be entitled to require from any person holding or accountable for any such document such information and explanation as are reasonably necessary for that purpose. ' (National Audit Act 1983 section 8(l)). The 1983 Act does not prescribe any particular approach once the audit office has exercised these rights. The lack of external direction is not surprising. Although auditors may (and do) take account of the views of auditees in deciding how to conduct an audit, discretion as to how work is conducted is considered a fundamental aspect of SAI independence. As a result, in theory at least, the choice of audit methods is very much an internal, working level decision. Overview In summary, the following observations can be made about the characteristics of methods considered suitable for VFM work from theoretical and legal perspectives. There is clearly a belief that there is potentially a wide range of methods to choose from to generate suitable evidence. However, the freedom to choose is matched by certain constraints, including that methods must be: appropriate for the work being undertaken, so as to ensure that the evidence is relevant; capable of producing sound, persuasive and reliable evidence, without which auditors cannot make judgements; 0 capable of generating sufficient evidence without disproportionate cost; 0 chosen by auditors without any undue pressure from other parties; and recognised as appropriate and reliable by those who have an interest in the audit work - in particular, the auditees, but also third parties. The examination of the methods actually employed by the NAO will be considered against this theoretical, technical and legislative background. 92 How does the NAO portray its use of methods ? Before consideration is given as to what methods have been used, it is worth examining how the NAO portrays its use of methods. Official published material provides some specific insights into how the NAO characterises the methods it has used or is considering using. In particular, it indicates the NAO's views that: 9 the range of methods applied is wide and has been evolving; 9 there has been increased attention to the consideration and selection of methods; e many staff have become more outward looking in their choice of methods; 9 methods are chosen to provide robust and defensible findings and conclusions; and * methods need to be cost effective, and are chosen to fit within studies, for which the average cost has come down steadily during the 1990s. The range of methods that is used is wide and constantly evolvin In recent years, the NAO has stated on a number of occasions that the range of methods used is constantly evolving. For example, the NAO's VFM Handbook, published in 1997, noted changes had taken place in the public sector since the NAO had been set up in 1984, and argued that 'We have kept pace with these developments. ' Included amongst the signs of this was that the NAO had 'extended the range of vfm techniques and methodologies [used], and ... their complexity [and]; expanded the sources of evidence and advice we draw on, including customers and beneficiaries of public spending programmes, and other third parties and experts' (National Audit Office 1997a: ii . Later on, the same booklet states that 'To meet the vfrn challenges arising from these developments we have increased our range of methodologies, adopted innovative techniques and drawn on new sources of specialist advice. We have also increased the range and type of studies that we do. As change to make further progress is likely to be an inherent feature of public service provision, we are constantly seeking to develop new ways of carrying out vfm studies. (p6-7)'. In his introduction to the 1998 NAO Annual Report, the C&AG wrote of needing to ensure that 'we are at the forefront of applying methodologies appropriate to the day' (National Audit Office 1998b). 93 Growiniz attention to the consideration and selection of methods Such evolution has generally been seen by the NAO in terms of growing sophistication of its approach to selecting and applying particular methods. In 1998 the NAO's Director responsible for developing VFM methods argued that 'many years ago NAO teams did little planning or thinking about methodology, but ... now it was much more structured and rigorous' (National Audit Office 1998a: 2). Evidence of concerns about methodology and the selection of methods is also outlined in the NAO's VFM handbook (National Audit Office 1997a). This emphasises that 'Careful consideration should be given to the proposed approach and the research and analysis techniques which will be used (p3l)' The guidance also emphasises the importance of testing the methods and discussing them with auditees, to ensure that they are recognised as valid and appropriate. More outward lookiniz The NAO also considers its approach to VFM audit has become more outward looking and eclectic, with staff keen to look beyond the confines of the more traditional audit methods of documentary examination and interviews. The NAO has referred to its VFM work as drawing on 'on tools and techniques from many disciplines and professions: audit, social science, operational research, economics, statistics, management consultancy etc. which give us a vast range to choose from' (National Audit Office 1997g: 19). One aspect of this in recent years, has been the interest in different forms of evaluation. Some of the internal VFM guidance issued in the early 1990s drew attention to the work of the General Accounting Office in the United States 30 (although this did not spark off a rash of GAO style evaluations), while more recently, the NAO employed a Research Fellow with a health evaluation background to try to identify ways in which evaluation methods might be drawn on more fully for audit work. His thoughts on study design (Keen 1998a) were issued to NAO staff in 1998. Chosen for robustness Arising from this work, Keen noted (1998b) that the prospect of having to expose the final report to the auditees in the 'clearance' process had a profound effect on the way in which NAO audit teams operate, including on the methods they select. This aspect of 4 94 the NAO's work is reflected in the guidance staff receive. One document states simply that 'The tools and techniques you select should be robust in their design and able to withstand challenge by the department and other readers of the reports, such as academics and interest groups with specialist knowledge. ' (National Audit Office 1997g: 17). The extent to which staff consider the clearance process influences the choice of methods will be considered in Part 6. The robustness of the evidence is important not just in terms of winning the 'clearance battle'. The NAO also emphasises the significance of its role in 'producing high-quality reports which meet Parliament's expectations'. This means that staff must 'ensure that every examination is soundly based in terms of the nature, quality and strength of evidence from which we draw our findings' (National Audit Office 1997a: i). Although, as we shall see in Part 6, Parliament has generally not shown interest specifically in the methods used for VFM work, the NAO's reputation within the legislature is, in great part, based on the assumption that the teams preparing the report have used appropriate approaches in a sound and professional manner. Cost effective We saw in Part 4 how resources had shifted between VFM work and financial audit during the 1990s, and how budgets for individual studies had been significantly constrained. In these circumstances, emphasis has also been placed on ensuring that methods used are cost effective. Guidance has stated that 'Your task is to select those tools and techniques which are most suitable for your needs - those that will enable you to answer the questions in the most cost effective way .... [there should be] a clear emphasis on avoiding unnecessary and misdirected work' (National Audit Office 1997a: 19). The 1998 guidance on enhancing the impact of NAO studies, which encouraged teams to examine the use of OR methods, emphasised that such techniques were costly, but argued that careful planning, well focused work and starting with the right questions, along with expert advice, would ensure that quantified analysis was not expensive. Despite the rhetoric, however, it is not clear how auditors make judgements about whether a method will prove to be cost-effective (National Audit Office 1998c). a 95 I Methods used by the NAO 1984-1998 This analysis of NAO's published statements about the methods staff employ can now be tested by an examination of what methods have actually been used, as recorded in I the prime source of information, the published audit reports. As explained in Part 3, this involved a simple counting exercise. Notwithstanding efforts made to ensure accuracy and completeness, there are caveats to the data, which must be considered. In the first place, it was only possible to count those methods actually referred to in reports. However, as is noted later, it is clear that details are not always included in publications, particularly in reports from the 1980s. Thus, there may be an element of under-reporting. Secondly, it is assumed that auditors have recorded the methods used accurately, and no attempt has been made to check this with those involved. No other reliable source exists. Thirdly, there was considerable variation in the amount of information provided on methods used between reports. Given their role in ensuring the accountability of others for their performance, audit bodies might be expected to explain the methods employed in their work as a key part of their own accountability. And it might be assumed that, given the similarities between VFM audit and research of an academic nature, SAI reports would bear some similarity to academic research or evaluation reports that provide detailed methodologies and sources. However, these assumptions appear to be only partly true, although a significant shift appears to have taken place over time. In the 1980s few reports gave more than scanty details of methods employed. Where methods were mentioned - for example, questionnaires - few details were provided of sample sizes or questions asked etc. In contrast, a feature of reports in the mid- and late-1990s has been the increased amount of space given over to outlining methodology. All NAO reports now provide a brief summary of the approach and 4 methods used in the introductory paragraphs, and the trend has also been to include a table listing the work carried out in some detail or a detailed appendix. Summary of findings I Figure 5.2 summarises the information collected from the examination of reports. a 96 Figure 5.2: Methods employed in NAO reports 1986-1998 Method Number of reports 1986 30 1987 29 1988 27 1993 39 1994 51 1995 47 1997 38 1998 1998 40 1 4 998 0 Document examination 30 29 27 39 51 47 38 40 4 0 Interviews 30 29 27 39 51 47 38 40 40 ] Sample examinations 4 3 13 6 17 4 10 12 Systems/procedures reviews 0 1 0 0 2 1 7 8 Analysis of existing data 9 6 8 5 4 10 12 15 15 Surveys 0 4_ 2 14 20 10 12 12 Focus groups 0 0 0 0 1 3 5 4 Expert panels 0 0 0 1 0 4 3 7 Case studies 6 1 6 12 16 6 3 3 Site visits 10 10 7 12 22 13 14 13 Observation 0 1 0 2 0 2 0 3 Deadweight 0 0 0 0 0. 0 1 0 0 Displacement effect 0 0 1 0 0 0 0 0 Correlation 0 0 0 3 6 11 1 0 Sensitivity analysis 1 0 0 1 0 3 1 1 1 Decision theory 0 0 0 0 0 1 0 0 Modelling 0 0 0 1 1 0 1 1 Comparisons (including international) 2 2 3 6 5 3 3 5 Consultations with others 2 5 8 22 29 22 14 22 Bibliometric analysis 0 0 0 0 0 0 1 0 Literature review 0- 0 0 0 0 0 4 2 Benchmarking 0 0 0 0 0 0 6 0 Cognitive mapping 0 0 0 0 0 0 0 1 97 Definitions used in compiling Figure 5.2 Document examination - examination of departmental files and records. Interviews - interviews of various types with staff of audited body. Sample examination - examination of a sample of cases, chosen either judgementally or statistically, designed to allow some generalisations to be made or provide examples for the report. Systems/procedure reviews - scrutiny of departmental systems, procedures or practices designed to allow comments to be made about the robustness, adequacy or integrity of arrangements. Analysis of existing data - work undertaken by audit teams using existing departmental or other data. Surveys - surveys or questionnaires undertaken to obtain views or information. Includes those undertaken in-house or by external consultants, by telephone, face to face or by post etc. Focus groups - discussion groups organised to generate views or opinions on a specific subject(s). Expert panels - panels of specialists established by the audit team to provide input into study design or the content of the report. Case studies - use of specific examples in varying degrees of detail used to illustrate points in the report. Site visits - visits to a number of different parts of an organisation (eg regional offices) or different organisations (eg a number of colleges) for the purpose of gathering information that may be compared and contrasted in a published report. Observation - occasions where audit teams have deliberately arranged to witness particular activities in order to gain an understanding of them. Assessing deadweight - the amount of assistance given in excess of the minimum required, for example, for a company to proceed with the same investment at the same time. Displacement effect - measuring this involves assessing the extent to which, for example, assistance to one organisation has effects elsewhere (for example, in terms of job losses). Correlation- a statistical measure of relationship between variables describing the presence, direction and degree of association between them. Sensitivily analysis examines how the results of calculations or models vary as individual assumptions are changed. Decision theory - structured way of determining the best choice from a number of options and involves attaching costs and probabilities to the criteria for each option. Modellin - approach which can explain how activities cause observable results. Comparisons - visits to other organisations providing a similar service etc (in the UK or abroad) in order to gain further insights into practices. Consultations with others - third party consultations in order to generate information or opinions on the subject, on the audited body under examination etc. Bibliometric analysis - documentary data capture focusing on the publication and dissemination of subject-specific documents. Literature review - systematic review of material relevant to the subject under examination. Benchmarking - use of internal or external comparators to assess how an organisation is performing and suggest how improvements can be made. Cognitive mapgiM- means of understanding how organisations make decisions and explore factors that assist or hinder them in achieving their objectives. Features of each period 1986-88 In this period we see the overwhelming importance of document examination and interviews for collection of evidence for VFM studies. Indeed, these appear to have been the major, if not the only, methods employed in many of the reports examined. Discussions with departmental staff, examinations of samples of transactions, and the review of official files were crucial. In addition, the NAO's other main means of obtaining information were site visits (for example, 42 benefit offices were visited for a 1987 report on Incorrect Payment of Social Security Payments (National Audit Office 1987)), and the examination and analysis of existing data, perhaps from the department or bodies involved. Questionnaires and international comparisons were infrequently used. On a small number of occasions outside consultants were brought in to help. A 98 study of the Management of work at MOD research establishments in 1986 (National Audit Office 1986b), for example, used an academic to report on best practice in research management, and consultants were used to examine management information needs of the research establishments. 1993-95 In this period a number of developments also appear to have taken place. There has been a continuation in the use of more traditional methods, but Figure 5.2 also suggests a noticeable growth in the use of questionnaires and surveys, which become an important feature of NAO work in the 1990s. In 1994, for example, 20 studies published in that year made use of this method. Another feature has been the growth in consultation with third parties, with more than half of reports examined making extensive use of consultation with interested parties. Finally, a feature of the second period has been what we might call a small scale, but nevertheless noticeable, increase in experimentation and thus the use, on one or two occasions, of new methods such as focus groups, decision analysis, and modelling, which had not been seen before. i 1997-98 In this period, interviews and document examination remained the bedrock of VFM work, but there was greater attention was paid to how these methods were employed. Surveys remained another frequently used method - some studies surveyed several groups of people - as were visits to a range of sites to collect evidence. Efforts to consult widely with, and gather evidence from, third parties continued. There was greater use of focus groups and panels of experts compared with the previous period. Benchmarking was seen in a number of studies in 1997, and the period also saw the first use of a number of other methods. % Major trends in the use of methods for VFM work The remainder of this chapter discusses the main themes identified from the data in Figure 5.2. These are that overall: 0 there has been a widening of the range of methods used by NAO teams in the most recent period compared with the 1980s; 99 i0 despite these developments, certain 'traditional' audit methods, such as document examination and interviews remain the predominant methods utilised; notwithstanding this, there are signs of growing sophistication in the ways in which these more 'traditional' methods are used; 0 there has continued to be widespread use of existing secondary material, but with some new ways employed to analyse it; 0 increasingly, examinations have made use of a greater mix of methods; 0 there has been an increase in the extent of consultation with users and providers of public services, evidenced, for example, by the extensive use of surveys and the appearance of focus groups; there has been considerably more consultation with third parties; there has been growing interest in the collection and use of qualitative evidence, much of which has, in the past, been considered less suitable for VFM work; 0 in general, where they have branched out beyond more traditional audit methods, auditors appear to have drawn primarily from the social sciences. There has been relatively less drawing on economics and statistics; and, 0 although there has been growing interest in forms of benchmarking, there has been relatively limited use of international comparisons. Overall, these findings indicate that, whilst VFM work is grounded in the past of financial audit, there have been developments that suggest some changes in fashion in methods. These changes have come at a time when there has been no significant alteration to the legislation governing the work, or any fundamental change to the processes by which Parliament handles VFM reports. Equally, despite the considerable variety in the subjects covered, VFM examinations have continued to follow the well- defined gestation process summarised in Figure 4.1. Thus procedural factors or legislative change do not appear to explain the shifts. At the same time, the basic principles of audit have remained the same. This suggests that changes have not come about through fundamental changes to the way in which audit work is perceived. Each of the trends is now considered in more detail in Part 5, and Part 6 seeks to explain what 100 factors have influenced the choice of methods in the absence of any major upheaval in the basic framework of VFM audit. A widening in the ranize of methods used The data in Figure 5.2 suggest that there has been a gradual widening of the number and range of methods utilised. Acknowledging the possibility that there may be some degree of understatement in the data from the 1980s, nevertheless, the NAO has clearly made use of new methods in later years. Thus, in the mid- to late-1990s we see the use of focus groups for the first time, regular use of reference panels, some use of system dynamics modelling, the more systematic use of types of benchmarking, more explicit use of the results of literature searches, and the first application of cognitive mapping and decision theory. There has also been development in methods used previously, and recognition for example, that interviews can be conducted in a wide range of different ways. This point will be covered in more detail later in this chapter. Such findings are in keeping with official NAO views of developments discussed above, and can be seen against the background of explicit encouragement to 'innovate' and try new methods. Such a trend might also be expected to continue, given that recent guidance includes references to these new methods (see for example, National Audit Office 1999c). However, it would be wrong to exaggerate the movement. The discussion earlier also suggests that the NAO was starting from a fairly low base in terms of the range of methods used. In addition, examples of new methods given above have, to date, been seen in a relatively limited number of reports. Continued reliance on documentga and interview evidence 4 Notwithstanding the expansion in the range of methods used, it is also clear from Figure 5.2 that documentary and interview evidence have remained the two most important methods employed in VFM work throughout the 15 year period. Both were used, in some form, in all reports examined. This is in line with expectations, with statements made by auditors interviewed for this project, and is also in keeping with the work of other types of research. Documentary review is a traditional audit practice (and a basic means of undertaking research), and the significance of key documents in providing background information, evidence of how activities or programmes proceeded, and the 101 auditee perspective on problems and issues, cannot be underestimated. One NAO internal guide described file examination as 'the cornerstone of NAO and E&AD VFM work', noting that 'the examination of contemporary documents provides strong evidence of the actions, considerations and opinions of staff in respect of the issues being pursued' (National Audit Office 1993b). This is no less true of examinations of innovative projects, such as the recent PFI deals, than it is of more traditional subjects. It is also unsurprising given the nature of the culture of the audited bodies. Many, if not most, of the organisations that the NAO examines remain bureaucracies, whatever changes they have experienced in the last 10-15 years, and they continue to place importance on written communication and record keeping. In addition, developments in IT and communications have increased, rather than decreased, the amount of paper that can be prepared, retained, and retrieved when needed. As May puts it in the context of social research, 'with the increase in information available through such means, [documentary research] is becoming a more popular method of research which, alongside others, yields valuable insight into social and political life' (May 1993: 150). This is no less so for audit. The other most common means of gathering evidence are interviews. One NAO guide (National Audit Office 1993b: 16) states that 'interviewing will feature in all our studies to greater or lesser extent', a point confirmed by the analysis of reports. In part, this is natural since asking questions and seeking answers is a basic investigative practice. Much of the purpose of audit interviews for VFM work is no different from I that for any other types of researcher; put simply interviews can 'yield rich sources of data on people's experiences, opinions, aspirations and feelings' (May 1993: 91). From the point of view of an auditor new to the subject, for example, interviews help provide an overview of an activity or allow for clarification of areas of uncertainty. They are N also the opportunity to seek quick access to, and guidance about, key documents and other material. In addition, as with the NAO's access to documentary evidence, the ability to question officials when needed is a powerful tool, and one that should, in theory, help to get quickly to the important issues. And, in most cases auditees expect to have such discussions in order that they can explain their work, and assume that auditors will wish to speak to staff at several levels. 102 Developments in the use of more traditional methods Whilst documentary examination and interviews have remained key means of obtaining evidence, closer scrutiny of what these entail suggests that there have been I developments in their use. a) documentary examination Attitudes to the use of document examination appear to have developed during the 1990s. Another 1993 guide (National Audit Office 1993f) suggests, perhaps arguably, that: 'file examination does not require any special skills, and the technique results in strong evidence, largely free from examiner and examinee bias, and imposing only a light burden on the audited body. Weaknesses include the possibility of I weeding of records to give an unbalanced or incomplete view of events, the difficulty of putting documents into their proper context, and the tendency for the I examiner to believe that everything written down is true. Also, value is added to the audited body's knowledge only if further analysis is essayed' (p33). More recent guidance (National Audit Office 1999c) acknowledges that documents remain the major part of evidence used in VFM work, but advocates more methodical and sophisticated approaches to 'minimise the risks of misinterpretation and placing undue reliance on the evidence. ' For example, it draws attention to the possibility of using content analysis. The guidance also draws attention to the use of bibliometric analysis in an NAO report (National Audit Office 1997d), which was used to gauge the success of an organisation in publishing and disseminating scientific research papers. That being said, to date there is no evidence that content analysis has been used within studies, or that many of the concerns of social researchers about the interpretation of documents are considered when making use of them. The possibility of using computer packages such as QSR NUD. IST to analyse text has been suggested in NAO guidance (National Audit Office 1999c), but again there is no evidence from the reports examined that it had been used by the end of 1999. Although this would not change the nature of the outputs, it might indicate a more systematic approach to documentary examination. 103 b) Interviews May (1993: 92) states that broadly speaking there are four types of interview used in social research. These are structured, semi-structured, the group interview, as well as un-structured or focused interviews. From the review of NAO reports it would appear that each type is seen to varying degrees in the repertoire of the VFM auditor, but with the emphasis mostly on the semi-structured approach. Whilst auditors often have specific questions to ask, they are also keen to probe beyond these to gain qualitative information and encourage elaboration. Where used early on in an examination, such interviews may offer leads into new areas of inquiry. Later on they may provide texture and depth in areas already considered to be of interest. Evidence from discussions with NAO staff suggests that greater attention has been given to interview technique in the 1990s, and the office has developed its training on how to conduct interviews and minimise obstacles to their success. There are also indications that greater thought has been given by auditors to conducting interviews, coupled with a greater awareness of the variety of approaches. For example, interviews for the report on Special Compliance Ofji'ce: Prevention of Corruption (National Audit Office 1998d) used a structured approach as the basis for interviews. There are also occasions where auditors have deliberately set out to undertake interviews with a series of organisations offering a wide range of perspectives in order to build up an overall picture of a particular event. The assessment of performance was thus informed by a comprehensive view from a number of different stakeholders. At a lower level, we have also seen the first use of cognitive mapping in a study, where structured interviews with 13 key decision makers were employed to understand how decisions were made (National Audit Office 1997d). Recent NAO guidance (National Audit Office 1999c) discusses in detail the principles of interviewing and different approaches, but some (for example, Pollitt and Summa 1996) have suggested that the extent of information about interviews included in reports tends to fall short of those employed by academic evaluators. They noted, for example, that interview schedules were seldom included in published reports, and the number and types of interview, as well as the conditions, rarely published. In general, this appears to remain true, although some reports in the late 1990s provide some 104 details about the nature of the interviews. Thus, whilst Pollitt and Summa's observations appear to be valid, there are some signs of increased transparency about how interviews were conducted and what was said. And, although it is very unusual for I auditors to makeý recordings of their interviews, some experiments in doing this have been seen in 1999. How interview evidence is used in VFM reports remains an unresolved issue, and reports rarely include direct quotations from individuals (see National Audit Office 1995a as an exception). Instead, the corporate - departmental - P view, often expressed in interviews, is quoted extensively. The use of secondary data and other research Another common means of obtaining audit evidence has been through the review and analysis of secondary material. At its most basic, auditors use the work of others - perhaps the reports of internal auditors within a body, or advisers and consultants employed by them - as part of evidence gathered for a study. They may also draw on data available at the audited body or elsewhere (see Goldsworthy 1996 for an example from the NAO), perhaps examining this material from new perspectives, or using computer software to provide significantly more data than had been available within the audited body. In some cases (for example, the study of Major Equipment Storage in MOD (National Audit Office 1998n)) the NAO have undertaken a number of exercises to generate new data to fill gaps in that routinely collected by the department, or have worked with the audited body to generate new data. In another case -a 1998 report on the Deficit in the Fundfor Pensioners of the Former Water . 4uthorities - the NAO reworked existing actuarial valuations to a single actuarial basis (National Audit Office 1998m). The limited use of other research findings in audit reports is another interesting aspect of the VFM audit approach. Pollitt and Summa have noted that, in general (and in contrast to the work of other researchers), audit reports have tended not to include footnotes or bibliographies. As they put it 'A typical academic evaluation will be festooned with references to other studies and may well include some kind of formal literature review. Such papers convey a sense of the evaluators adding to some cumulative and collective body of knowledge and insight. SAI reports read more as stand-alone products' (Pollitt and Summa 1996: 41). This remains the case with many 105 0 NAO reports, although there are examples where research is acknowledged. In addition, the appearance of bibliographies in a small number of reports indicate that some audit teams are undertaking systematic literature searches for relevant published material. There may be a number of reasons for the relatively limited reference to other published material. Despite the changes outlined above, auditors are not steeped in the tradition of writing for academic audiences. As has been noted, the audience for audit reports is primarily parliamentary, and reports are prepared, in the first place, as the basis of a parliamentary hearing. The focus, therefore, tends to be short term and practical, and whatever use may be made of reports subsequently, their presentation and content are intended to reflect this timeframe. Most reports are not seen as a contribution to a debate in the way that academic work is. In addition, the fact that the report derives its status through the standing of the organisation, and the fact that it is an agreed document, also perhaps removes the need to earn credibility through demonstration of a wide and substantial evidence basis. Studies have made use of a greater mix of metbods Another trend appears to have been a tendency towards study teams making use of a larger combination of methods. In many reports from the 1980s, it would appear that evidence collection for most reports was restricted to two or three of the more traditional methods, such as document examination, site visits, examination of a sample of cases and interview evidence. In part, the concern may have related to cost. NAO guidance in 1993 warned that many methods could be applied in some studies, but: 'in practice the resource and time demands that implies cannot be met within typical NAO study constraints. And the benefits of extra complexity in design are I subject to the usual laws of diminishing returns. So we recommend that auditors identify the most cost-effective basic approach to the study, select appropriate techniques to support that methodology and view any 'triangulation' techniques as supplemental and subsidiary to the mainstream' (National Audit Office 1993f: 17). In comparison, a review of the methods employed in studies published in 1998 indicates that many made use of a combination of between four and eight (sometimes 106 10 or more) separate means of gathering and analysing information. A study on the Distribution of Lottery Funds by the English Sports Council (National Audit Office 1998f), for example, drew on ten separate discrete methods. This finding is not entirely I unexpected, and perhaps links in with NAO rhetoric that studies design has become 16 more sophisticated and complex. It is also in keeping with internal guidance on the importance of 'triangulation' and using multiple sources (National Audit Office 1997g: 20-21). The development of consultation with users and suppliers of public services One important theme discussed in Pollitt et al 1999 is the way in which SAIs as a group have engaged much more with other parties in recent years, be they auditees, the recipients of the services under examination, the audiences for their work such as parliamentarians, or the media. On the basis of this work, this appears also to be reflected in some of the developments identified in our examination of methods, in particular, the growth of the use of questionnaires by some SAIs, and the increased consultation with interested parties of all types during audits. a) questionnaires and surveys The most popular method designed to elicit the opinions of others used by SAIs has been the questionnaire or survey. The NAO has used surveys since the late 1980s. None was recorded as having been used in 1986, but four were employed in 1987. All were sent to organisations (local authorities in two cases, District Health Authorities and Social Security offices once each). In 1988 a market research firm was contracted to survey supplementary benefit recipients. Over the years, surveys have been used in a variety of ways and for different purposes. These include: e customer satisfaction surveys (eg a survey of the views of users of the National Library of Scotland); censuses to establish practices across a range of bodies (eg of health authorities and laboratories carrying out cervical cytology and colposcopy clinics); and surveys to establish customer's understanding (eg a survey of insolvency practitioners and company directors on their understanding of the law relating to company director disqualification for a study on the Insolvency Service). 107 Surveys were initially organised in-house, but they have increasingly been contracted out to market research firms, although with the questionnaire itself prepared by the study team to varying degrees. They have been conducted in various ways - by I telephone, by post, or face-to-face. An interesting aspect of the NAO's use of surveys, which is somewhat different from the position of other researchers, relates to the issue of response rates. In certain parts of the public sector - in particular, the health service - there is an expectation that all bodies will respond to NAO surveys. During a hearing on a report on NHS Supplies in 1997, the PAC expressed disatisfaction that many Trusts had failed to complete the NAO survey, and asked the NHS Executive to ensure that they all co-operated in future. Similar criticisms have occurred on other occasions - in a hearing on a Road Safety report in 1986 the committee was very critical of local authorities that had failed to complete questionnaires. As a result, some surveys of health bodies in the late 1990s received near 100 per cent response rates. Similar levels of interest have not been generated amongst other types of recipients (not subject to such direction), as the 20 per cent response rate for survey of small businessmen for the report on Contributions Agency's Customer Charter in 1997 demonstrated (National Audit Office 1997h). The effect of the greater use of surveys has been felt in a number of ways. It has increased the scale and breadth of evidence collection. In examinations of the NHS, for example, surveys of 300 Trusts generate far more data than a small sample of visits. Within resource and time constraints, no study team could hope to visit more than a small proportion of these bodies, whereas by using surveys, the NAO is able to collect more extensive data. This has increased the authority of the findings. Surveys, often of whole populations, or statistically sampled, have also allowed for greater generalisation of conclusions. By surveying the public, the NAO has also been able to give users of services a voice, introducing a more customer-oriented focus into reports. b) focus groups A development since the mid-1990s seen in Figure 5.2 has been the use of focus groups as a means of generating insights and information. Details in the table include only work for published reports, but they have also been used at preliminary study stage (see Figure 4.1). An effective example can be seen in a study of the handling of calls from 108 I% the members of the public by the police in London (National Audit Office 1995a), where focus groups explored the ways in which officers dealt with calls. One outcome of this was a list of twelve key rules for dealing with callers, which were included in the iA report as recommendations, whilst the discussions also helped identify problems in the working of the communications room, and provided quotations to illustrate the views of officers. Another example is the report on the Defence Evaluation and Research Agency (National Audit Office 1997d), which used focus groups of departmental i customers to examine how well the customer-supplier relationship was working. Participants also completed a short questionnaire. Although some focus groups have been conducted for the NAO by consultants, as an indication of its faith in the method, the NAO has since produced a guide for staff on undertaking them (National Audit Office 1997i). c) third party consultation A more substantial and longstanding trend has been the growth in the extent of consultation with third parties in the course of designing and, in particular, executing examinations. Figure 5.2 shows how significantly this has increased compared with the work in the 1980s. Many reports now include long lists of organisations that have been consulted, or with whom findings have been discussed. Provisional conclusions from health studies, for example, are routinely discussed with the Royal Colleges, the British Medical Association, and other professional and representative bodies as part of the preparation of the report. The scale of such work can be substantial; an examination of the HM Coastguard. Civil Maritime Search and Rescue (National Audit Office 19981) stated that interviews had been undertaken with 10 international organisations and fifteen UK bodies. In part, this appears to reflect a desire on the part of the NAO to ensure reports are authoritative and accurate, by allowing as some interested parties as possible to provide evidence. Given that anyone can, and many organisations do, submit evidence to the PAC prior to a hearing, there is also merit for the audit teams in seeking the views in advance, and, in addition, it increases the likelihood that errors will be identified prior to publication. 109 It d) the use of expert panels Another way in which the views and knowledge of others have been obtained has been in the use of reference or expert panels. Not seen in the 1980s, they were used on the defence area in the early 1990s to advise on study proposals and to contribute during the life of a study by advising on methodology, the interpretation of findings and the text of the draft report. Whilst Figure 5.2 indicates far from every study makes use of such panels, they have increasingly become an accepted feature of many examinations, and usually consist of outside specialists with a variety of perspectives, as well as, on occasions, representatives from the audited body. The advisory group on the study of Measures to Combat Housing Benefit Fraud, for example, included representatives from local government, the National Association of Citizens Advice Bureaux and the Benefits Agency (National Audit Office 1997j p93). A panel of specialist advisers were used for the study of the BSE crisis in 1997 (National Audit Office 1998h) to advise and give additional insights into issues underlying the topics examined, the nature and impact of the crisis and the schemes introduced. In general, the aim is again to increase the authority of the NAO's work, and to draw in as wide a range of perspectives to help enhance reports and guard against criticism for failure to appreciate particular angles. Other means of collecting qualitative evidence The appearance of questionnaires and focus groups within the methods employed by the NAO suggests that, where they have expanded beyond traditional audit methods, its teams have often looked towards the social sciences for inspiration. The guide on a 'Qualitative Techniques: Their Practical Uses in VFM Audit', issued to staff in 1993 (National Audit Office 1993b: 1), argued that widespread use was already made of qualitative information, but that 'it is possible to improve the reliability of information collected, be more penetrative in its analysis and to combine quantitative and qualitative information more effectively'. This was updated by a 1999 guide on qualitative evidence (National Audit Office 1999c) and accompanied by training courses given by outside academic staff. Figure 5.2 indicates that other means of % collecting qualitative evidence have been used, such as case studies and first hand observation. 110 % a) case studies From one perspective, most NAO examinations are case studies. Here, however, attention is focused on the use of the method within specific reports. Figure 5.2 shows that there has been occasional use of longer case studies or, more usually, shorter case examples to help highlight particular points or explore certain topics. Some NAO reports have used a small number of case studies to form the basis of a whole report (such as in a study of the management of overseas programme aid (National Audit Office 1994a) which examined aid to four countries). More usually a large number of small case examples have been used throughout many reports to illustrate particular points, provide supplementary information, or add colour and insight. Staff at the NAO considered this was important in helping non-specialists, particularly parliamentarians, understand their reports. This is less than normally would be claimed by users of the case study approach. Yin (1989), for example, states that 'A case study is an empirical inquiry that: investigates a contemporary phenomenon within its real-life context; when the boundaries between phenomenon and context are not clearly evident; and multiple sources of evidence are used' (p23). This would appear, for the most part, to go well beyond the use made of this method to date by the NAO. b) observation To many social scientists, observation is a central method of qualitative research (punch 1994, Dingwall 1997). References in a number of audit reports, however, indicate that, whilst the approach has been used to gather some evidence, on most occasions it has stopped short of what researchers might regard as ethnographic methods. Much tends to be fairly passive observation, intended mainly for familiarisation purposes (eg observing a cataract operation, or joining the Essex police on their work patrolling the M25). More participative evidence collection appears to be much rarer, although recent NAO guidance has highlighted the possibilities that, for example, workshadowing as a form of participant observation can offer (National Audit Office 1999c: 28). The work of the NAO Research Fellow (Keen 1998a) was strongly in favour of auditors observing more. Perhaps the most adventurous NAO example of participant observation arose when an auditor visited an Antarctic research station to gain first hand experience of their work (National Audit Office 1992a). Another 111 % variation was the use of the 'mystery shopper' approach for a study on Protecting and Managing England's heritage property, where staff visited sites unannounced, to assess the provision for the public (National Audit Office 1993c: 64-69). I In general, the NAO appears to have been fairly cautious about this kind of method for collecting primary evidence, considering that short of photographic records, it can be difficult to substantiate findings obtained in this way if they are disputed. Auditee aversion may also be a factor. The approach of visiting a sample of heritage sites anonymously in 1993 does not appear to have been repeated for several years (see National Audit Office 2000c for a similar approach in properties grant-aided by English Heritage, published whilst this thesis was being finalised). That being said, recent NAO guidance on qualitative methods (National Audit Office 1999c) emphasises that data generated by the 'mystery shopper' type of approach are 'usually very rich and valid and often worth the effort involved' (p30), so that it may be attempted more often in I future. Ouantitative methods a) use of economic concepts and models The evidence in Figure 5.2 suggests that the NAO has not made a great deal of use of economic concepts and models. Whilst several reports indicate an awareness of concepts such as deadweight and displacement effect, and auditors have commented on whether use has been made of such approaches by departments (see for example, National Audit Office 1986a), they have rarely been employed by auditors themselves. Roberts and Pollitt (1994) examined a notable exception in their review of a report on Creating and Safeguarding Jobs in Wales (published in 1991 and so not included in Figure 5.2). The concept of deadweight also came up in a report on Department of Trade and Industry: Supportfor Innovation (National Audit Office 1995b), where an NAO survey asked companies about the minimum level of funding they would have 16 required to go ahead with a project. The study on Creating and Safeguarding Jobs in Wales also provides the most significant example of the use of the displacement effect. Other reports have referred to it, to the extent of noting that proposal papers for new schemes did not assess possible displacement effects and quoting existing figures (National Audit Office 1986), or commented on the need for more effective project 112 appraisal techniques to minimise deadweight and displacement (National Audit Office 19971). The NAO has made occasional use of modelling, in particular, in a defence report (National Audit Office 1993d). Here, in order to look at the financial impact of departures from departmental guidelines, the NAO constructed a model of the 'average' high value fixed price contract and estimated the scale of premium payments under the Ministry's scheme. In the report on Support for Industry the NAO made use of decision theory which, by creating and then weighting a set of performance indicators, allowed schemes to be evaluated in comparison to each other, even when they were judged individually by different performance criteria. Although it did not provide an absolute measure of cost. effectiveness, it did establish the relative standings of the schemes examined. b) statistical techniques 11 Perhaps surprisingly, reports display only fairly limited use of substantial statistical analysis. Time series analysis does appear, but multiple regression analysis has been infrequently employed. Correlation analysis has been used several times in NAO reports. For example, the report on Metropolitan Police Service: Responding to Calls from the Public (National Audit Office 1995a) used it to identify how many staff a communications room ought to have for a given number of reported incidents. The report on The Industrial Injuries Scheme demonstrated a strong correlation between the number of cases dealt with by offices and the standards of their monitoring (National Audit Office 1994e), whilst the Defence ma or projects report for 1993 found no statistical correlation between defence project cost over-runs and in-service slippage (National Audit Office 1994c). In a study on Hospital Catering, the NAO compared results with patient satisfaction with meal costs, showing that high satisfaction ratings did not necessarily involve high costs (National Audit Office 1994d). The extent to which auditees have used sensitivity analysis has often been considered in examinations. A number of reports have reviewed departmental use of it and others commented on the absence of the technique in certain circumstances. The method itself has been used less frequently. In the study on the British Film Institute, the NAO tested the sensitivity of financial performance to attendance figures, whilst in Department of 113 % Trade and Industry: Supportfor Innovation, sensitivity analysis was carried out on the results (National Audit Office 1995b). c) interest in operational research A growing interest in the use of operational research (OR) techniques was referred to earlier. The interest is closely linked to a belief that use of such methods can help to identify specific financial savings. At the time of this thesis, only a few reports had made use of OR methods. An early example is an examination of Overseas Trade Services (National Audit Office 1996c), which used systems dynamics to model the interrelationships between the different parts of the export process. In 1998 the NAO established an expert panel of OR specialists to advise study teams or provide input to studies. Its existence, and the presence within the NAO directorate of a former head of the Treasury's operational research team, suggest that OR methods may become more common in future in NAO work. Drawing comparisons with other bodies and other countries a) benchmarking 0 For many researchers drawing comparisons between organisations is an obvious way of forming a view about performance. The analysis of reports suggests that this has tended to be far less so for the VFM work. Despite this, benchmarking appears to have been given a push within the NAO in the second half of the 1990s, and this is reflected to a degree in Figure 5.2. Towards the end of the 1990s it was one of the areas the C&AG was keen for staff to look at, and one experienced member of the team developing the NAO's VFM work described it as the 'success story' of the late 1990s, with the interest shown, in particular, it seems because of the potential for helping to identify opportunities to improve efficiency and make savings (National Audit Office 1997k). NAO guidance distinguishes between 'high level' comparisons between bodies; performance benchmarking, through which certain measures and indicators are used to compare performance; and process benchmarking, the systematic analysis of organisational processes. To date high-level comparisons have tended to be the most popular. Performance benchmarking using measures has, as mentioned earlier, been a feature of the work of the Audit Commission, but less so of the NAO. During the 1990s 114 greater interest has been shown in it and a number of examples stand out. A report on Entry into the United Kingdom analysed ports' performance using a weighted workload indicator, plotting passenger throughput per staff day. This provided the first hard analysis of relative performance and enabled the NAO to encourage changes in practices (National Audit Office 1995d). Another report - on National Health Service Supplies - compared prices paid for selected goods in different parts of the NHS and helped identify scope for savings (National Audit Office 1996d). Despite this, Figure 5.2 perhaps provides evidence to support the view (see Pollitt and Surnma 1996: 41) that 'SAIs appear to be slightly more cautious than evaluators about entering into comparative analysis that goes beyond the organizations immediately under investigation. ' As the authors suggest, this may well reflect difficulties with agreeing conclusions drawn on this basis, which is, in itself, a reflection of problems in establishing suitable comparisons. b) international comparisons Only a small number of reports in the 1980s and 1990s have made explicit international comparisons, although those undertaken in the 1990s appear more substantial in terms of analysis and research than the 1980s. In the former period, much of the material used is relegated to appendices, rather than utilised as part of the main analysis, and is often of a descriptive nature. Thus, there is information about how other countries undertake similar activities - for example, internal audit within central government. A report on preventive medicine (National Audit Office 1985) made reference to overseas practice in order to highlight that the (then) DHSS could do more in the same area. In the work from the 1990s there is more evidence of audit teams having ventured abroad to visit comparable organisations - for example, to Ireland and Sweden for the purposes of a study of forestry bodies, or to a number of countries for meetings with lottery operators and their auditors, for the report on Evaluating the Applications to Run the National Lottery (National Audit Office 1995e). In a 1998 report on Major Equipment Storage, the NAO commissioned consultants to investigate the methods of storage employed in the UK and benchmark them against practices abroad and in industry (National Audit Office 1998n). The NAO team then made visits to the German 115 Army and the French Air Force to examine practices. There are also cases where explicit comparisons of practice are made in reports, for example, with those of the United States, throughout the report on movements of stores and equipment to the Gulf The relatively limited use of international comparisons may be a reflection of resource constraints. One of those involved acknowledged the time and resource constraints on such work, and the need for it to begin before the main fieldwork is underway for it to be ready in time (Goldsworthy 1996: 9). There have also been problems agreeing the suitability of comparisons with auditees. Conclusions This chapter has identified the methods used in NAO reports. It has drawn out the main themes. In summary, these are that there has been a widening of the range of methods used in NAO reports since the 1980s, and improvements to the ways in which traditional audit methods have been deployed. Examinations have been better planned and increasingly made use of a greater mix of methods, although document examination and interviews remain at the heart of VFM. There has been considerably more consultation with users and providers of services, and far more effort made to obtain the views of third parties. Developments have been made particularly in the use of qualitative, rather than quantitative, methods. But there appear to be limits on the types of methods used in VFM work. The evidence suggests that auditors tend towards two of the three basic methods of social research - interviews and document examination - and make far less use of the other method - observation. While it has been suggested (Dingwall 1997) that social scientists have, in recent times, tended not to use participant observation so much either, the conclusion drawn here is that there is a more fundamental suspicion amongst auditors (and those they are examining) about this approach to evidence gathering, which ensures it has played only a limited part in VFM work, and primarily in familiarisation work, rather than in evidence to support main findings. In a process with a strong accountability focus, certain types of evidence are considered more unassailable than others. 4 In considering the range of methods employed by VFM auditors, it is worth considering how this compares with broader trends in evaluation and scientific methods. Much of what has happened in VFM appears to reflect broader developments in terms of 116 concern for methodological rigour, greater attention to taking account of the views of different stakeholders, and growing interest in qualitative methods. Improvements in the rigour with which methods are employed tie in with Keen's observation (1999b) that 'scientific methods have become more rigorous' in recent years, and that developments within the scientific tradition have led to more detailed reporting of the methods used in studies. The increasing interest in qualitative methods in VFM miffors developments in evaluation and academic research, and, having overcome their mistrust, auditors have sought to ensure they use them in as scientific a manner as possible. Greater efforts have been made to involve stakeholders in examinations, although not to the extent that 'constructivists' would. There is, however, a recognition that different parties have different views, which need to be accommodated or acknowledged in their reports. Comparison can also be made with previous assumptions about the use of methods by the NAO contained in existing research. Part 2 quoted Pollitt and Summa's statement that 'NAO reports of the last decade or so appear to be methodologically considerably more adventurous than those of the previous period' (Pollitt and Summa 1996: 40-41). This chapter has provided empirical support for this, suggesting that, in fact, things have gathered pace in the years since their observation was made. This is reflected in the slow, but steady, expansion of VFM methods, and in the broadening range of guidance that has been produced to encourage audit teams in their selection of methods. It is also reflected in a range of other initiatives such as in the increased interest in performance measurement and benchmarking, the setting up of a panel of operational research specialists to encourage interest in this field, the international methods seminar in 1998, further well sourced and researched VFM guides, and a range of quantitative courses undertaken by the LSE Methods Institute. There has also been interest in the work of RAND Europe and the consultants, McKinseys. The evidence collected here also suggests movement in some of the other issues on which Pollitt and Summa commented in the same article. Whilst few reports outline explicitly details of interviews, there are signs of greater attention to interview technique. Whilst detailed comparisons between organisations are still uncommon, the use of benchmarking of performance has grown in the late 1990s, and the rhetoric from 117 the NAO, and the evidence of a number of recent studies also suggests that this may be an area of development. We have also seen that more reports have included bibliographies and have acknowledged research work elsewhere. Finally, there are strong signs of far greater attention being paid to how studies are put together, not just in terms of methods to be used, but also the criteria to be used for judging performance. Despite these developments and others discussed in this chapter, there remains considerable evidence for Keen's view that, notwithstanding this broadening of the horizons, VFM has remained strongly rooted in traditional methods and financial/resource concerns (Keen 1999a). Ultimately, reports are prepared as the basis of a hearing by the parliamentary committee that examines the use of public funds. The use of interviews and document analysis remain crucial to VFM work, although, at the same time, increased attention appears to have been paid to developing and refining these methods. Similarly, there is support here for Power's observations on the constantly evolving nature of VFM audit practice (Power 1997). There have been many different combinations of methods used in NAO examinations, and within senior management's interpretation of the general standards for evidence, no hard and fast rules as to what a VFM should look like. The assumptions made about its capabilities have been fairly wide, and this has allowed a variety of different methods to be employed towards producing adequate evidence. Power also states that 'The development of vfm auditing has also been an incremental repackaging and extension of existing audit capabilities.. an assembly in which new conceptions have been grafted on to existing routines ... ' (p50). This chapter confirms the incremental nature of change, and the ways in which to a great extent VFM methods represent a core of well tried traditional audit methods, to which have been combined other forms of enquiry from other disciplines. Part 6 now considers how these changes appear to have come about. 118 PART SIX: INFLUENCES ON THE CHOICE OF VFM METHODS Part 5 concluded that there had been a number of important, if not revolutionary, developments in VFM methods at the NAO from the mid-1980s to the late 1990s. It also concluded that, despite these developments, VFM continued to make heavy use of documentary examination and, to a lesser extent, interview evidence. Much of the change documented in Part 5 has thus been of an incremental nature. This chapter seeks to explain how these developments came about. It does so primarily by making use of evidence from discussions held with more than 40 experienced VFM auditors about the factors they consider have influenced the choices they have made. Initially, the chapter considers whether the conclusions in Part 5 are confirmed by the perceptions of staff working on VFM studies during the period. It then examines the nature of the auditor's choice by comparing their approaches to those of others. The bulk of the chapter will examine the range of influences on the choices of methods. This provides a lead in to Part 7, which considers how developments at other audit offices compare with those at the NAO. NAO auditors' perceptions of changes in methods The evidence of change examined in Part 5 was drawn from Published NAO reports and official literature. Whilst the source material may be comprehensive, there is a possibility that such material presents only an 'official' version of developments, possibly designed to impress. Consideration was, therefore, given to what extent the trends identified from official sources accorded with the perceptions and recollections of NAO staff who had undertaken the work. In these discussions, there was universal agreement that there had been significant changes since the National Audit Act 1983. Some spoke of 'considerable' or 'continuous' change, another of 'exponential change'. One referred to 'a constant search for the magic method'. Some believed that change had accelerated in recent years, whilst others spoke of a long evolution - not necessarily incompatible views. Others commented that change had been inevitable and necessary, given both the developments in the outside world, and the new opportunities offered by the National Audit Act. More generally staff ascribed the change to one or more of a number of triggers, which are the sub ect of this chapter. j 119 The main features of change discussed in focus groups and interviews can be summarised in four broad categories. These were that: * the range of methods had broadened and VFM work had grown to encompass methods that would not have been considered in the past, although certain well tried methods remained at the heart of their work; attitudes towards the methods used in studies amongst staff and senior management had changed, and there was a greater emphasis on ensuring audit work added value. A wider range of methods were seen as allowing teams to do this, although there was a residual belief for many that the NAO's work was still very much about accountability, and even about 'point hunting' (see Part 4); the perspectives of NAO work had changed so that there was greater focus on the views of the users of public services, and also a move away from considering process to trying to assess the effectiveness of activities, and greater attempts to generalise the results; and greater attention was paid to methods at an earlier stage in the study development process, and staff were better at deciding what was an appropriate method, based on awareness of a greater selection. There was a widely held belief that VFM was better planned and executed in the late 1990s than it had been, and so was 'less of a black art'. This evidence seems to confirm the conclusions from the analysis of reports, as well as the understanding based on personal experience. The nature of the auditor's choice Attention is now given to the auditor's approach to choosing methods. The aim of this is to assess whether or not the VFM audit approach is likely to gravitate to particular a approaches, or instead, be amenable to change. The general VFM process is summarised in Figure 4.1, and has been described on a number of occasions (eg Lonsdale, Mul and Pollitt in Pollitt et al 1999, Keen 1998a and, in more detail, in NAO 1997a), as has the nature of VFM study design (Keen 1999a). It has been noted that the proposed methods are included within a study design package that also considers 120 0 resources and timing, potential impact, and potential sensitivities. In devising their plans, teams may draw on their own experience and knowledge, on ideas previously used by other teams, on suggestions from consultants or outside experts, or on advice from the VFM Guidance team. Senior management (Assistant Auditors General) 1ý discuss, refine, amend, and eventually endorse proposals. Ultimately, the C&AG has the final say, which he often exercises, perhaps suggesting amendments or offering alternatives, although in many cases he will follow the advice of staff. Figure 6.1 summarises the key players influencing the team's choice of methods. Some of these are internal to the NAO, others external. Some are direct influences - consultants appointed to assist with an examination may provide suggestions - whilst other feedback may be more indirect, for example, via the overall recommendations arising from the quality assurance process (see Part 4). Input from an element in the process may come via more than one route. Comments from the PAC, for example, may relate to a specific study, or more general concerns and observations may be fed back to I VFM staff in general, following discussions between the Chairman of the Committee and NAO senior management. Figure 6.1. The key players influencing the VFM team's choice of methods 'I Other audit ) r-O VFM Guidance teams inside ther bodies t tet earn the NAO Consultants and experts ) Audited bodies lk 4 121 Figure 6.1 also suggests that the selection process may be a negotiated one, with expectations that auditors take account of the views of others, and are not entirely free to carry out their work exactly as they want. Thus, even those to be examined may have some input into the decision making process. It is also a decision making process based partly on experience, professional judgement and trust, but one in which senior management may wish to see the piloting of certain methods, and may take some persuasion before approving a particular approach. 0 Auditors' approaches to choosing methods As was described in Part 4, most VFM auditors have a professional background in accountancy and audit. The NAO recruits graduates and trains them in these disciplines, as well as providing 'on-the-job' experience. It is clear, however, that the professional grounding does not entirely dictate the approaches to their work. A number of auditors suggested that their professional training was largely irrelevant to their VFM work, except in the sense that it had given them knowledge of government accounting practices, the principles of audit, and a tendency to see things in financial/resource terms. There is thus no instinctive resorting to set ways of carrying out their work, or any strong sense in which their professional training requires them to follow certain approaches. Instead, there were signs of a multi-dimensional approach to the selection of methods, in which audit displayed signs of being a 'craft' and a profession. The work also bears some comparison with academic research, and with the work of those contracting with consultants. Lonsdale, Mul and Pollitt in Pollitt et al 1999 (Chapter 7) considered the notion that VFM auditing was a 'craft' activity - that is, it entails the use of specialist skills to create a series of unique ('hand-crafted') products which are self-consciously intended to be made to high standards' (p105). This analogy merits further elaboration. Traditionally, craftsmen maintain considerable influence over their work. Their standards and methods are enforced by the 'guild' or professional body. Much VFM training is undertaken on the job, staff effectively working an apprenticeship with more experienced practitioners. Much of VFM work is learnt this way, with junior staff working to plans designed by more experienced colleagues, and working to audit standards similarly drawn up within the profession. 122 VFM audit work also shows many of the characteristics of a profession. It is an occupation that has developed (and is developing) a 'complex knowledge base', and in which there is 'a systematic body of theoretical, abstract, esoteric knowledge' (Pavalko 1988: 20), even though much of it is procedural and borrowed from other disciplines. Auditors trained at the NAO must pass examinations, and demonstrate professional skills and knowledge in the working environment. The training is strongly 'ideational', meaning that emphasis is on the manipulation of ideas, concepts and principles. That being said, VFM auditors lack a number of the attributes of professionals. At a work group level, there are clear restrictions on the autonomy of individual members. In putting forward their proposals for studies, they have to seek approval of the C&AG, who may (and sometimes does) reject or amend suggestions (an interesting comparison with the magistrates in the Cour des Comptes in France, who have considerable independence as to how they conduct their work). And they do not have a separate professional identity, or professional institute characteristic of many professions. Another point of comparison is with academic researchers and evaluators. Again there are similarities. There is the need to design and plan their specialised work, and draw on a body of approved ways of carrying it out. In both cases, there is an element of negotiation needed, and both need to secure entry, which may be a time consuming process. But there are differences in approaches to work. Comparing VFM auditors with academic evaluators, Pollitt and Summa (1997c) have suggested that: 'Audit teams usually 'plan' their audits in a fairly pragmatic way. Evaluators, by contrast, 'design' their projects in a more methodologically self-conscious % fashion. In the latter case, tools are selected as part of a theoretically and methodologically integrated project. In the former, tool selection may well take place on primarily functional rather than theoretical criteria. These are quite dissimilar cultures' (p99). They consider decisions on methods by academic evaluators are made within more of a defined intellectual framework. As another text put it 'Practically, methods are chosen on the basis of criteria related to or even dictated by the major elements of the methodology in which they are embedded, such as perception of reality, definition of science, perception of human beings, purpose of research, type of research units, etc' 123 (Sarantakos 1993: 33). Such an explicit and prescribed approach is uncommon amongst auditors. Pollitt and Summa's distinction has much merit, although the evidence from much of this thesis (reinforced by the inclusion of a large number of parties in Figure 6.1) is that however much audit teams may try to 'plan, this perhaps implies a degree of N control over their work out of keeping with the reality discussed in this chapter. Finally, in considering how auditors make choices, it is important to recognise that they also act as co-ordinators and commissioners of work, as well as practitioners % themselves. They are thus not constrained by the available skills of immediate colleagues, and part of their decision making process involves identifying suitable assistance (within a specified budget), capable of undertaking the work to meet prescribed quality standards and deadlines. Their position here is no different from that of anyone else sub-contracting work, but does pre-suppose some basic knowledge of the possibilities and risks of a wide range of methods. Conclusions i- In conclusion, although most auditors have had a professional accountancy training, they appear not to be tied to a particular set of concepts or approaches. Many choices are based on experience learnt on the job, and on seeing evidence of what has been successfully carried out by others. In many ways, the VFM auditor's approach to the selection of methods appears to share aspects of the characteristics of a number of other occupation types - craftsmen, professions, academics, and those commissioning the work of others. That VFM auditors work within a bureaucracy may also make it more likely that choices about methods are not made always on study-specific or technical grounds. Instead they may be made within the context of broader decisions about resources, the organisation's desire to undertake studies within particular fields, and to publish a certain number of reports within each financial year, as well as assumptions about the ability of available staff. The nature of the role of the C&AG as Parliament's auditor reporting to a committee of politicians is also likely to make the NAO sensitive to the environment in which it works. Taken together, all these factors suggest that, by not being tied to one particular approach, VFM auditors may be receptive to a wider range of ideas. At the same time it suggests there are a number of constraints on their nl, Ability to make choices. 124 Influences on the methods chosen by VFM auditors at the NAO Against this background, the key influences on the methods chosen by auditors are now examined. The chapter begins by looking at those factors that appear to have encouraged the wider use of methods, and ends by looking at factors that appear to have constrained developments. Much has been written on how and why institutions change. For example, March and Olsen (1996) emphasised the range of factors that may determine change, including competitive pressures, environmental and political conditions, and institutional origins and history. They emphasise that change can come about in many ways - for example, through major crises - but also 'through mundane processes of interpretation, reasoning, education, imitation and adaptation' (p149). Huczynski and Buchanan (1991) state that 'Organizational change may .... be triggered by a multitude of factors', some of which are external, others internal. They also refer to organisations' desire to anticipate developments and find ways of coping with them in advance. This is defined as 'proactive change'. Added to these more specific factors, are more general trends and broader shifts in social attitudes, norms and practices that penetrate all organisations, the NAO being no exception. Thus, for example, the expansion of university education in the last twenty years is likely to have contributed to the organisation's move to all- graduate recruitment. The growing informality in inter-organisational communication may also have helped to break down the more formal ways in which auditor and auditee communicate. Figure 6.2 summarises the various players that can act on an audit team and influence their choice of methods. Figure 6.2 also suggests that the receptiveness of the team will depend to a great extent on the factors summarised in the stand-alone circle such as skills, knowledge and attitudes of team members. All these factors are considered in the following paragraphs. In the diagram, the external triggers have been shown as squares, and internal triggers as ovals. 125 Figure 6.2 Players encouraging the use of a wider range of methods used by VFM auditors 'S 14 Environment PA C&AG VFM team Audit Choice of team met od Informal networks in NAO Oth! r Skills, a It udi experience, knowledge, interests, and Consultants/ attitudes of secondees members Factors encouraging the use of a wider range of methods ChanRes in the extemal envirorunent The most profound driving force behind the developments in methods appears to have been the changing environment in which the NAO has operated. This has provided both the overall impetus for change, and the stimulus for the use of specific methods. Pollitt (Pollitt et al 1999: 57) has described the possible effects on SAls of the various developments usually grouped together under the heading new public management. Of particular significance to the study of methods are that: 126 It 9 changes in the public sector might alter the size of the audit field, including bringing in new types of body, or shift the public-private boundary, and so might require new methods from different disciplines, for example, to examine private finance issues; management reform might change the pattern of organisations to be audited, leading perhaps to a large number of agencies, and so might require a different approaches to data collection (eg surveys, where visits had previously been possible); adoption of new management techniques might generate new data, and so might provide new opportunities for different types of analysis; the widespread use of new management techniques and approaches throughout the rest of the public sector might encourage SAIs to reflect upon their own practices and decide to borrow some of these techniques; the increasing emphasis on performance measurement within the public sector could encourage SAIs to give more attention to scrutinising performance data; and the general increase in interest in service quality might require new methods to be used to gauge public opinion. Material published by the NAO has drawn a clear link between the growing complexity of public administration in the UK, and the need for different and more penetrating methods to assess performance. One document stated, 'We are continually striving to improve the quality of our reports and respond to the demands of the environment in which we operate. ' One aspect of this has been to apply 'rigorous methodologies that provide robust, defensible findings and conclusions. ' (National Audit Office 1995c: 17). More recently, the director responsible for developing the NAO's work for the audit of tcross-cutting' government programmes has stated that these developments will involve Grefining our value-for-money methodology' (Whitehouse 1999). The implications of environmental changes were mentioned in the focus groups and interviews organised for this research (see Part 3). To several senior officials, such developments had brought with them changes in attitude externally towards the 127 assessment of VFM in the broadest sense, towards the role of audit generally, and towards the NAO in particular. In the first place, there was a recognition that there had been a growing interest in VFM issues outside the NAO. One experienced director suggested that by the end of the 1980s the NAO was no longer alone in commenting, on a regular basis, on VFM in government. More departments were undertaking reviews of efficiency and effectiveness, and it had become more important for NAO to devise new ways of carrying out studies that enabled it to come up with useful recommendations and underlined its unique role. As well as some encroachment into the NAO's area of work, another senior official spoke of a growing resentment within the civil service at a VFM approach perceived as aimed primarily at catching departments out. Expectations of what auditors should be contributing have changed throughout the 1990s (see for example, Public Audit Forum 1999). Part of the increased expectations has been that auditors would use their reports not simply to criticise, but also to make constructive recommendations. This implied creating new knowledge, which in turn increased the need to pay greater attention to the ways in which examinations were conducted and the methods used. Concern about the 'burden of audit' - as was noted in Part 2, a growing theme in the 1990s (Power 1997, Hood et al 1999, Public Audit Forum 1999) - intensified the demands to maximise the benefit derived from the time that auditors spent in departments. And, as this thesis was finalised, the issue was being aired again, this time in the context of whether auditors are actually a block on risk taking and innovation within the public sector. Another important aspect of the changing environment mentioned in discussions was the increased interest in measuring and reporting performance in the public sector. Pollitt and Summa (1997b: 317) have shown the importance that the NAO has attached to demonstrating its own impact. They commented that its Annual Report 'displays a consistent concern with justifying the work of the NAO and attempting to identify and measure the benefits that flow from it'. The efforts made by the NAO to quantify the impact of its work have been examined (Lonsdale in Pollitt et al 1999 chapter 10). In this sense the NAO has been heavily influenced by, and contributed to, the prevailing climate of performance measurement and reporting. The need to sustain and improve the impact that VFM work has, as measured for example in terms of financial savings, 128 been powerful driving forces to identify new methods 31 . As well as creating a different setting for audit work, the changing environment also impacted directly on the choice of methods in a number of very specific ways. In the first place, improvements in the quality and quantity of performance information, % characteristic of public management reforms, opened up new possibilities for the use of certain methods not considered possible before. A number of people suggested that, in many early NAO reports, the auditors had often been unable to say more than that departments did not have adequate data with which to manage particular activities. More recently, as noted in Part 5, the NAO had begun to look towards benchmarking, and there had also been growing interest in the validation of performance indicators (National Audit Office 1998i). Both initiatives were believed to have become more N feasible (and likely to be more valuable) with the availability of improved data. I The growing importance attached by successive Governments to the quality of public services - most notably under the 1991 'Citizen's Charter' programme, and more recently, Service First, and the Modernising Government agenda - were believed by many to have encouraged efforts by the NAO to gauge public satisfaction levels, in particular, through the use of questionnaires. As was seen in Part 5, such methods had hardly been used at all in the C&AG's reports before the late 1980s. Similarly, the fragmentation of central government has opened up the possibility, as yet not particularly exploited, of a variety of benchmarking studies, drawing comparisons between these bodies, of the type that have been a feature of the work of the Audit Commission. Overall then, changes to the external environment encouraged (and, in reality, required) a significant shift towards work that was more ambitious and wide ranging. The climate changed to one in which issues central to the work of the NAO moved 4 more centre stage and the use of resources within the public sector became more and more scrutinised. On its own, however, alterations to the environment in which the NAO operated did not guarantee a response. They might have been ignored, misunderstood or resisted. It was only because the opportunities offered, and crucially, the threats posed, by the changing enviromnent were interpreted in particular ways within the NAO, and reflected in the decisions taken, that change took place. The next 129 four sections - on the roles played by the C&AG, the VFM guidance team, the networks of staff within NAO, and individuals - examine these responses. The role of the C&AG Given the position of the C&AG as head of the NAO, it is perhaps not surprising that many considered that the holder of the post for most of the period under review - Sir John Bourn 32 - had had a significant influence on the development of VFM methods. All statutory powers of audit are vested in the C&AG, rather than the NAO as an organisation. He (and to date all have been male) employs the staff of the NAO, and has the final say on the direction taken by examinations, including the methods used. Reports to Parliament are published in his name. The C&AG's influence on VFM methods was considered to have been extensive. Several people commented that soon after taking up post, Bourn had expressed a dislike for examining 'process', and had encouraged auditors to think harder about what they were trying to do. He was also seen as a critic of 'woolly thinking', and an advocate of clear analysis. One director, responsible for overseeing VFM developments in the early 1990s, considered Bourn was responsible for encouraging the use of outside consultants - who brought with them knowledge of a wider range of methods. And he recalled that Boum had (virtually alone) been a strong supporter of several early VFM methodological developments, at a time when there was generally little interest in such matters in the NAO. More generally, he was considered to be a source of ideas from outside the audit world, for example, via his academic activities. Much of the driving force behind the NAO's response to the demands of the changing environment was thus seen to have come from the top of the office. The role of VFM szuidance team and trainin If there was widespread agreement that the C&AG had had a significant influence on the development of methods, there was more uncertainty as to how far the central VFM guidance team, their training courses and written guidance, had had an effect on changes in methods. There was agreement that having a small team (of perhaps four to six experienced staff with a track record in producing high quality studies) dedicated to thinking about and developing VFM methods was a valuable asset, and a recognition that more (and better) advice was now available about aspects of study 130 design and the use of specific methods. Opinion differed, however, over whether the team had driven change, or rather had reflected what was already taking place, consolidated guidance into a manageable form, and brought it to the attention of a wider audience. The team was seen primarily as providing official sanction to experiments that had worked, offering a distillation of existing academic/practitioner literature, and driving forward the popularisation of particular methods. Only on rare I occasions -a case cited was the first guidance issued in 1993 on qualitative techniques - were certain developments seen to have been driven from the centre. The training courses organised by VFM team were seen as a valuable way of publicising particular methods. In the 1980s, courses had popularised specific approaches by bringing in members of teams to speak about their experiences. Several people recalled lectures from social science academics in the early 1990s opening their eyes to what could be done with qualitative methods. Two major VFM 're-launches', held in 1993 and 1997, were also identified as marking new and influential steps in drawing attention to particular methods that had already been used successfully, although not introducing the methods. More recently still, the use of external training providers was believed by many to have increased the quality of courses further. The importance of informal networks If the central VFM team helped to popularise certain methods, a more informal, but in the view of many, a more potent stimulus, were the informal networks of staff. Several people talked of the importance of their 'networks' of colleagues, or referred to the NAO 'grapevine' through which they heard about what others were doing. The a informal networking appeared to have been helped by the comparatively small size of the NAO (and, even more so, of the VFM specialism within it), by the regular mixing of staff undertaking a large number of often very different studies, and by the relatively limited turnover of VFM staff, all of whom from around 1993 had ceased to combine this work with financial audit responsibilities. Many had moved through the office together over a decade or more. All these factors were believed to have created an environment in which many teams drew on each other, albeit often in an ad hoc and informal way. Interestingly, despite this, there was little tradition of exposing draft reports to open discussion, a feature, for example, of academic work. a 131 One important aspect of peer example was what were considered to be 'landmark, studies, where particular methods had been used for the first time, or were seen to have been particularly successful. To many, these had had a profound influence on what VFM teams felt they could do, one person suggesting that 'the major drivers are particular studies that demonstrate that something is possible'. There was much common ground as to which studies had been groundbreaking; several groups referred to a 1988 social security report which had made early use of a questionnaire ('which I think brought a sea change following it'). Other reports mentioned were a 1986 report on Forestry Commission (National Audit Office 1986c), which struck one (now) senior director as innovative in its use of outside consultants, and a 1992 report on The Award of the Contract for the Landing Plaffibrin for Helicopters, which had demonstrated the possibilities of short studies (National Audit Office 1993e). A study on Entry into the United Kingdom was also cited by several people as innovative 0 because of its use of focus groups (National Audit Office 1995d). Individual initiative The significance of networks and peer example leads on to the important area of individual initiative. In the absence of a set way of undertaking VFM work, and given the perceived lack of central direction referred to above, efforts made by individual members of teams in pursuing new ideas have been a prime source of change. For some it was a question of making their job more interesting, perhaps in line with Dunleavy's bureau shaping model (Dunleavy 1991), in which 'bureaucrats' value individually innovative work. In this, policy level officials (a reasonable comparator for well qualified audit staff) 'want to increase the interest of their work tasks'. Here, it seems, the 'bureau shaping' ambitions of staff may be congruent with the NAO corporate aim of developing audit methodologies. For some, efforts to try new methods were linked to the development of professional pride amongst increasingly well qualified graduate staff. There was thus personal gain to be had from taking new opportunities. For others, the search for new methods was linked to the presence or absence of personal and corporate confidence. One senior director stated that: it 132 'When you see one report with a new technique which has been made to work then everybody else begins to think whether they can make it work too and so there is a gradual growth in corporate confidence which helps and very much affects our preparedness to use techniques. ' Paradoxically, others saw the NAO's search for new methods as evidence of a lack of confidence, albeit a valuable one. Another director, who had come to the NAO from the civil service, suggested that 'there is a complex here that everyone else is doing things better and lets get them in to talk to us, which I think is good'. Others, with longer service in the NAO, suggested that staff now felt more able to explore other possibilities because there had been a shift in attitude among senior management. In particular, it was felt that the NAO was less 'autocratic'. In the past, they suggested, it had been expected that Assistant Auditors General and Directors would 'interfere' with studies, and it was seen as natural that staff carried out work in ways proven to be acceptable. Partly this was assumed to have been because many in the NAO's directorate had had little or no knowledge of non-audit methods and 'what they didn't understand and didn't trust they didn't want to know about'. In the opinion of many interviewed, it was not until the early 1990s that these people had retired and been replaced by people 'who allowed the new developments to happen'. A new generation was deemed to have assumed control during the decade and shown greater tolerance of new ideas. Overall, therefore, it seems that during the 1990s auditors felt better able to make broader choices, and felt encouraged to do so. There was also wider knowledge of what to choose from, and with the greater use of expert panels for studies or strategic planning, and the proliferation of VFM guidance, more sources from which to seek advice. There was also a growing desire amongst many to undertake more interesting and rigorous studies, and a feeling that there was an expectation that effort be put into thinking about methods. In addition, more people began to consider that internal constraints, covered later, were not too overwhelming. Role of contracted-in staff, non-accountants and consultants Moving on to another element identified in Figure 6.1, the thinking of individual NAO staff appears to have been enriched to a considerable degree by another shift in the 133 NAO during the 1990s - the introduction of more people from outside the organisation, and with backgrounds other than accounting. Part 4 referred to the increase in the number of non-accountants, secondees and consultants who began to participate in VFM work during the 1990s. Whilst still very much a small minority, many interviewed felt that this 'new blood' had been influential in bringing in new ideas about methods. Whereas in the 1980s audit staff were almost all trained or experienced accountants, thereafter, the NAO has recruited a number of people with expertise in economics, operational research, programme evaluation, performance measurement and statistics, to work on studies or be part of the VFM guidance team. Their presence was seen to have had several benefits - direct and indirect. The development of surveys on social security area in the late 1980s, for example, was directly linked by several people to the presence of a small number of researchers. Additionally, outsiders were also considered to have generated greater confidence to try new ideas. As well as more permanent additions, contracted consultants also played an important role. Part 4 showed that by the end of the 1990s the consultancy budget had reached some 10 per cent of NAO's total expenditure. The growth was a deliberate strategy of bringing in people with new skills to push back the boundaries of what NAO was able to do. There was universal agreement that the initiative had worked and that there was a strong link between the use of consultants and the growing interest in new methods. However, this had not been without its problems. Often the best consultants had previously been employed by the department, which raised issues about their independence, and made departments unwilling to accept the NAO's choice. The influence of (iuality assurance processes Another factor that grew in importance in the NAO in the 1990s was the process of quality assurance, introduced in 1993 and developed thereafter to assess the 'quality' of VFM reports. The importance of the 'cold review' process came up without prompt in all the focus groups, although views varied as to its significance. Some described it as I a useful stimulus to think about techniques because your report is effectively being marked'. Others found the process useful as a way of identifying other reports that had used particular methods successfully. Often the experience of criticism was a direct spur to make improvements to counter previously observed shortcomings. The inclusion in 134 published reports of greater detail about the methods used was also considered to have been driven by the quality assurance process, and comments and recommendations were seen by many as having helped formulate NAO guidance and create a climate where VFM auditors are expected to, at least, consider the scope for innovation. But others were less sure, and there was some antagonism to the process, particularly the lack of contact between the team and the reviewers at the LSE, who were seen as lacking an understanding of particular constraints. The influence of other audit organisations A number of groups raised the issue of the influence of the Audit Commission on the work of the NAO. This was particularly the case for staff working in the health and police areas, where the Commission and NAO share audit responsibilities (see Bowerman 1994 for a critical view of this situation). To some, the Audit Commission were seen as competition, and aspects of their work were reviewed and drawn on in considering NAO approaches (a point also picked up in interviews for Hood et al 1999). Their national studies were considered to be comparable pieces of work, so that teams had obtained ideas in particular on presentation ('more modem'), and on possible methods. The Commission was seen as particularly strong on quantitative methods, and also considered by some to be more interested in improving VFM in a very direct sense, rather than more indirectly as part of the parliamentary accountability mechanism. Less significant was deemed to be the work of other audit offices, although the NAO was considered to be far more internationally active, and thus more aware of developments than in the past. SAIs mentioned in discussion were the General Accounting Office in the United States, the RRV in Sweden, the Office of the Auditor General in Canada, the Australian (national and state) and New Zealand audit offices, and the Algemene Rekenkamer in the Netherlands. The role of the PAC The importance of the relationship between the NAO and the PAC has already been noted (see also Flegman 1980, Hollingsworth and White 1999). In summary, the Committee examines most of the reports to Parliament of the C&AG, the NAO briefs and supports the Committee prior to, during, and after its hearings. The Committee's own reports are based on the C&AG's report and evidence presented at the hearing and 135 are drafted by the NAO team. In addition, the Committee, as well as individual members, make suggestions for future examinations by the C&AG. Against this background, it is interesting that participants in only two of the five focus groups initiated discussion about the Committee's influence on the choice of methods, although once prompted, had all had views on the subject. The most common view was that the Committee had had little influence on NAO methods ('spectacularly minimal' one participant suggested), although some considered influence might be exercised behind the scenes in informal feedback between the Committee and the C&AG. A common view was that this lack of influence reflected the fact that the prime concern of committee members was with the reports' content - the findings, conclusions and recommendations. The methods used and their appropriateness were, to a large extent, taken for granted. Members were believed not to have the time or inclination to inspect them too carefully, an observation that was not meant to be a criticism. However, they were considered to like case studies - in effect, vignettes - and clear diagrammatic representations. They were also believed to be interested in user surveys. Whilst a lack of direct influence on the methods of NAO reports, there was a recognition that the Committee did have an effect in other ways. To many its presence acted as a 'generalised pressure to improve quality', but on other occasions, the Committee was seen to have had a more specific impact. An example given was in influencing attitudes to achieving high response rates on surveys. In the past, the Committee had been critical where response rates to NAO surveys have been less than 100 per cent, regarding it as very unsatisfactory that those involved did not consider a request from Parliament's audit body worth responding to. The Chairman of the Committee after 1997 (David Davis MP) was also seen as having encouraged an interest in measuring the effectiveness of service delivery. One interviewee highlighted a newspaper article (Independent 18 February 1998), in which the Chairman had encouraged the NAO to undertake greater measurement of performance. Constraints on development of a wider range of VFM methods Attention is now given to the factors auditors believe have constrained their choice of methods. Given the incremental nature of the changes identified, and the weight of history described in Part 4, it might be expected that resistance to developments might 136 have been strong, linked perhaps closely to the nature of the work and culture of the NAO. March and Olsen (1996: 149) have noted that 'institutions are usually associated with routinization and repetition, persistence and predictability, rather than with political change and flexibility, agency, creativity and discretion'. This suggests we should perhaps not be surprised to identify pressures to retain existing, well-tried methods, although this probably should be less so in an institution such as the NAO, whose raison detre is generating change. Others have examined the resistance to forces of change. Huczynski and Buchanan (1991: 532), for example, have suggested that 'change [in organisations] may be resisted because it involves both confrontation with the unknown and the loss of the familiar'. Child (1984) too has described examples of non-acceptance amongst employees when their existing working practices and close-knit groups were disturbed by change. Constraints to the developments of VFM methods have been divided between those internal and external to the NAO. These are subdivided into practical constraints - relating to issues of resources and available skills - and attitudinal. constraints - that relate more to factors affecting how auditors think and feel about their work. The different elements are summarised in Figure 6.3. Figure 6.3 Factors constraining the use of a wider range of VFM methods Practical Attitudinal internal to NAO 0 Limited resources 0 Satisfaction with existing " Restricted timescale methods " Limitations on skills and 0 Attitudes to change and risk knowledge taking 9 Belief that some methods were inappropriate External to NAO Clearance process and the 0 Departmental dislike of prospect of PAC particular methods Resources One of the most extensively discussed constraints on the use of a wider range of methods was the availability of resources ('the main consideration' according to one director). Although a number of people commented that a budget of between E180,000 and E220,000 (a range within which most studies had tended to fall in recent years) 137 ought to be enough, many reasons were given for it being a constraining factor (for example, the need to provide for the expense of an often lengthy clearance of a report, and the high cost of large scale surveys). Some considered that cost considerations had reduced the use of particular approaches, with the ability to visit a large number of NHS Trusts, and process benchmarking being two examples quoted. It had also encouraged alternative approaches to evidence collection, for example, the use of self-completion surveys (say to NHS Trusts), whereby costs were largely externalised. It had led to smaller audit teams, which had constrained the number of interviews that could be undertaken. Staff in all groups spoke pragmatically about cost considerations when deciding on the methods to use. Some talked of trade-offs between studies, with funds saved on one piece of work being used on a more resource intensive study. Others spoke of the need to justify, to themselves and their line management, spending sums of money on particular methods if evidence could be obtained in other, cheaper, ways. Yet the significance of budgetary constraints affected some more than others. For certain directors the size of the budget was the key determinant in deciding what methods to use, with, on occasions, the methods selected to fit the budget. For others, it was not much of a constraint, not least because related training or other costs could often be allocated to separate budget codes. And others felt that they could, if they put forward a sufficiently well justified proposal, get a study approved regardless of cost. Linked with this was discussion of time constraints, which many saw as a more significant issue than cost. Time considerations were seen by many to impact directly on the range of methods that could be employed, as in general, the fieldwork period for an examination was likely to last between four and six months. Time constraints meant that the NAO had tended, for example, not to conduct examinations tracking the progress of groups of individuals over time to look at outcomes. Such studies were seen as possible, but currently did not fit in with the ways in which the NAO had tended to plan its work. Gathering trend information over a long period of time was also deemed to be virtually impossible, although some questioned whether this assumption was part of a mind-set that staff had got into. They suggested that it tended not to happen because staff had not put it forward it in study proposals. 138 In general, time constraints were deemed to reflect the need for reports to fit into broader NAO strategic considerations, as well as into the parliamentary and political processes to which the NAO was a party. This meant that there was an expectation of a regular stream of timely deliveries which could be offered to the PAC. This adds weight to the suggestion that the link between the NAO and Parliament can constrain the use of methods 'because the need to satisfy political timescales may exclude the possibility of more ambitious, elaborate and lengthy studies' (Roberts and Pollitt 1994: 547). Knowledize and skills Another practical constraining factor was deemed to be the experience, knowledge and skills of staff. As has been noted, the majority of VFM staff have been trained in accounting and audit. Other skills (for example, specialised quantitative techniques, operational research, statistics) were less widespread and, although there had been a growth in relevant courses, there was deemed to have been only limited debate within the NAO about methods and methodology. As a result, some saw transferral of technical knowledge as remaining ad hoc, ('very much word of mouth and folk lore'). Sometimes even this had not worked well, and teams had Greinvented the wheel', unaware that colleagues elsewhere in the office had already mastered a particular approach. One senior director suggested that 'there isn't necessarily an awareness of what the toolkit looks like. Well, there isn't particularly a toolkit. There's a collection of tools scattered around the building which you can try and find. ' Some experienced staff added that where there had been a lack of readily available skills within the office, they had had to search externally for help. The demands of considering where to find assistance, of identifying suitable consultants, tendering the work, making judgements about the ability of consultants, and bringing them up to speed with the NAO's requirements had, on occasions, tended to deter some teams from going ahead. As a result, some felt they had relied on tried and tested methods. Satisfaction with existing methods Another important factor working against major changes in VFM work appears to have been staff perceptions of the strength of existing methods, primarily the ability to demand documents and require answers from staff working within government bodies ('our real competitive advantage over other organisations'). That auditors make 139 extensive use of, and have great faith in, interview and documentary evidence is not surprising given the statutory powers provided to them. The selection of well established methods was also seen to be guided by self imposed standards, and what audit teams and their senior management considered were appropriate ways of collecting the amount and type of evidence. In addition, there was a perhaps not unreasonable belief that particular methods - for example, document examination or interviews - were appropriate for some types of NAO study, where, for example, the bulk of the work was aimed at establishing who did what and why, and in chronicling sequences of events (for example, for much of the report on events leading to the resignation of the Vice Chancellor of the University of Portsmouth (National Audit Office 1997m), or the events surrounding the failure to publicise changes to the inheritance arrangements for the State Earnings-Related Pension Scheme (National Audit Office 2000d)). The view was also expressed that, given its constitutional position, the NAO had needed to maintain its reputation as providers of reliable reports, and so had, perhaps not surprisingly, avoided rushing to embrace new ideas without evidence that they were sound and reliable. Attitudes to change and risk taking There was also considerable discussion as to whether the culture of the NAO, and indeed audit itself, had encouraged or discouraged change and innovation in VFM method s. Several people with long experience in the NAO commented that, whilst reporting VFM issues with the accounts, it would have been highly unlikely that any data generation or more advanced analytical methods would ever have been used on a regular basis. The advent of separate reports had provided a 'broader space' within which to work, but initially, at least, the culture was not seen as conducive to change. One senior official stated that: 'The coming of VFM audit involved change. And change involved disruptions and alterations in values, attitudes, bureaucratic cultures and behaviour. It involved the co-operation and clashing of different personalities. It was the response to perceived threats and the grasping of possible opportunities, both personal and organisational. The coming of VFM audit involved the overthrow of vested interests among both the staff and clients of the audit office. And it 140 involved the disturbance of the balance of power and influence between audit offices, ministries of finance, other departments, the legislature and the judiciary. So the coming of VFM audit involved violence and conflict as much as, or perhaps more, than peaceful adjustment and development. ' For many staff the outcome of this battle had been greater encouragement from above to try new things, and fewer senior staff who were resistant to what was described as 6properly managed innovation'. This view was reinforced by one experienced evaluator who had joined the NAO in the mid- I 990s. He commented that 'I have certainly not found a reluctance to embrace most of the methods that I had been using as an evaluator outside the NAO ... I would have thought there was about as wide a range of methods being used here as anywhere in evaluation circles. ' Despite this, the lack of scope for experimentation and, in particular, the absence of a budget for research and development or experimentation were mentioned on a number of occasions as a constraint. Trialling of new methods had to be done within the scope of a specific study, yet there was seen to be only limited opportunity to diverge significantly from time or cost targets. There was also a belief amongst some that there was no room for failure. Where many acknowledged there was encouragement to be innovative, they considered this came within fairly clear boundaries, influenced in part by the importance of meeting the corporate target of 50 published reports. Another view was that the boundaries were different depending on the member of the management board, with some perceived as more adventurous than others. These findings tend to confirm Keen's view, referred to before, that 'Experience had demonstrated to [many auditors] that the environment was not conducive to innovation, a view that also appeared to be held by many directors' (Keen 1998b). Attitudes to the suitabilfty of methods In order to try to establish where the boundaries for VFM methods lay, participants in the focus groups were asked whether there were particular methods that were considered unsuitable and so were not considered. Although some specific examples were mentioned, discussion mainly concentrated on the characteristics of approaches that were deemed unsuitable. In general, methods were considered unsuitable if they: * were seen to waste departmental resources; 141 " interfered in the running of the department or distorted the outcome of departmental work; " compromised the NAO's independence, or were too closely associated with particular interests; and " questioned military, legal or medical judgements. Given the nature of the NAO's remit, methods that could be seen to waste resources were considered unsuitable and unethical. One example given was making fictitious claims to test whether it was possible defraud the child benefit system, which was deemed quite unsuitable by some, although others saw it as legitimate, not least because they audited organisations which put through 'dummies' to test, for example, their own laboratory service. Several people suggested that such methods could be seen to be interfering in the running of departmental operations, which given the nature of the NAO's work, was seen as potentially risky. Some considered 'mystery shopping' 'too sly', although again others considered it a perfectly valid approach. There were also seen to be practical and client relations difficulties to using unannounced visits (a point linked to comments in Hood et al 1999). The use of methods which might result in the questioning of a variety ofjudgments - medical, legal and military were examples given - for example, by employing experts to review decisions made - was generally deemed inappropriate. Departmental attitudes Amongst external constraints, the attitudes of departments were considered crucial by almost every one interviewed. The developments in the external environment referred to above had a profound effect on almost all of the NAO's clients during the 1980s and 1990s. Such changes appear to have impacted on the nature of the relationship between NAO and the bodies they audited, although it should be stressed that relations varied considerably between departments. In the course of a VFM examination, there are perhaps three key stages in NAO-auditee relations. First, at the outset, there are discussions about study content and methodology. Secondly, relations are maintained during the conduct of the study, when the execution of particular approaches may require discussion. Finally, there is discussion about the draft report itself - the clearance process - where the means by 142 which data were generated may be scrutinised. Departmental attitudes to methods at any of these stages were seen by many as helping to set a boundary around what teams could hope to do as part of a study. In general, departments were seen as suspicious of new methods, although less so if they had used them themselves. In general, qualitative methods were seen as more problematic than quantitative ones. Several departments were understood, for example, to dislike, and consequently to have objected to the use of focus groups. Past opposition from departments had created some caution amongst auditors in suggesting particular methods and carrying them out, and on occasions they explained that studies were re-scoped to accommodate auditee concerns, or particular methods were not used. On other occasions, however, they had gone ahead with methods which the department had disliked. More recently, NAO guidance had emphasised the need to (gain the audited body's agreement to the use of these methodologies, especially sampling and survey strategies, in order to avoid misunderstandings at a later stage about the acceptability of findings or extrapolations' (National Audit Office 1997: 3 1). There was widespread agreement that more effort was now being made by NAO than in the past, as well as a belief that many cases of opposition were deliberate attempts to prevent NAO using certain methods because of what they might identify. Several auditors were keen to emphasise that discussions with departments about methods did not present them with a de facto veto, and talked of 'dragging the department along in our trail'. Others talked of the importance of educating departments about the validity of particular methods. One said 'You have got to take them along with you haven't you. You can't force a new technique on them, you have got to explain the technique to them, because you have got to explain the results. ' Whether this amounts to 'capture' by the department would seem to depend a great deal on individuals. However, as noted in Part 4, the ability of the NAO to undertake VFM work without co-operation is limited. Whilst the choice of methods was sometimes seen as a problem, several people also considered that certain departments had shown greater interest in engaging in discussion about methods in recent years. This was seen to be linked with changes within departments and auditors felt they were dealing with 'more management style civil 143 servants', better acquainted with research techniques, used them internally, and expected those they were dealing with to use them. In some cases, auditors reported examples where the NAO team had worked quite closely with departments to develop a particular approach - systems dynamic modelling with the Ministry of Defence was one (then current) example given. Those concerned believed that such involvement would have been unlikely in the past. The prospect of PAC The departmental interest in the methods to be used by NAO was associated in the minds of many auditors with their perceived concern about the prospect of a PAC hearing at the end of the VFM process. Those regularly subject to NAO reports were seen as more 'clued up' about how to 'manage' the process. Growing professionalism, and what was seen as the greater flexibility of the public sector in recent years, with for example, the greater recruitment of specialists, the wider exposure of public bodies to outside influences, and the greater number of senior figures appointed from the private sector, were also believed to have led to different attitudes within public bodies to the criticism traditionally associated with audit. Unused to such public criticism, more auditees had responded by questioning the basis of this censure. In turn, NAO staff felt that it was even more important to have watertight methodologies. More directly, the prospect of a PAC hearing in public was considered to have an effect on the ability of NAO to use specific methods, in particular, given the Committee's perceived confrontational and fault finding approach. One focus group participant explained the difficulties in trying to organise a 'benchmarking club', designed to tease out the reasons behind varying performance in a particular field. This would have involved representatives discussing their shortcomings, and would have worked best if participants could have been guaranteed anonymity. In the context of work for a published VFM report this could not be guaranteed, as it was assumed that where cases were reported in a published report, Committee members were likely to ask who they were. Conclusions In this chapter the factors that auditors considered had influenced them when they were choosing methods were examined. It showed that auditors were not driven to particular 144 types of method by their training. The chapter then suggested that the changing environment in which the NAO had operated had been the main driver for developments in methods. This included changes in both the political environment, including the new public management reforms and the development of the 'regulatory state', and the intellectual environment, as evaluators of all kinds have sought to introduce new means of capturing the views of a variety of stakeholders. All had helped to raise expectations of the role of audit and interest in VFM in its broadest sense. The NAO's interpretation of these changes, and the translation of thought into action had come, in particular, through the lead given by the C&AG, the small VFM guidance team, and through individual teams and their informal networks throughout the office. At senior management level, there has been an understanding that external developments could not be ignored, and that the NAO had a responsibility to develop and extend its VFM approaches. This was linked to considerations of being able to demonstrate usefulness and beneficial impact. At audit team level, responses were influenced by the attitudes and skills of individuals, which had, in part, been coloured and enriched by exposure to a growing range of consultants and non-accountants. It had also been stimulated by the existence and growing importance accorded to comments arising from the external quality assurance process. But pressure had not been one way. The rather sporadic and evolutionary nature of developments, and the particular directions in which the NAO had moved, appeared to reflect the availability of particular types of staff and certain levels of resources for VFM work. Resource levels and time constraints, for example, had encouraged widespread use of surveys, but discouraged longitudinal studies. Developments have also been shaped by widely held perceptions of what were appropriate means of gathering evidence. And departmental distrust of certain methods, raised in discussions about study proposals, appeared to have drawn certain boundaries. The changes outlined in this chapter can be seen in a number of ways. Overall, much of what has been described is in line with existing theory about organisational change. The way in which the NAO's approach to VFM methods has developed can be seen in terms of the office adjusting to changes in an environment that has been complex and often dynamic. From this perspective, the addition of particular methods or the interest in new 145 approaches are aspects of the NAO seeking a new equilibrium. With major changes taking place in public administration, the NAO has sought to develop its approaches to meet these challenges and avoid being (and appearing) out of step. New methods have provided credibility as departments have become more challenging. As more demands are made on all public bodies to justify themselves, so the NAO had looked for new ways of increasing the likelihood of securing financial impact and being able to measure it. As more emphasis has been placed on the views of the citizen as consumer, so the NAO have sought to ascertain these views. Given the background of change in public administration, it would perhaps be surprising if there had not been some development. In the absence of major upheavals in audit arrangements since the 1983 Act, much of this change appears to have come about 'through mundane processes of interpretation, reasoning, education, imitation and adaptation' (March and Olsen 1996). There do not appear to have been any serious or sustained attacks on the NAO's credibility" , but steps have been taken to adapt to new demands - to 'modify the technology' (Huczynski and Buchanan 1991) - before such change was forced upon the organisation. This has been undertaken in tandem with modifications to the task (attempting to focus more on efficiency and effectiveness), modifications to the structure (the creation of new Units and the separating out of VFM and financial auditors 34) , and modifications to the people employed (recruitment of people with new skills). March and Olsen (1996: 149) also note that institutional change may be 'discontinuous, contested, and problematic'. The evidence suggests that there has been considerable discontinuity about developments in VFM methods, both in terms of the incremental, step by step introduction of new ones, and in terms of the different degrees of progress in various areas within the NAO, with some taking things forward faster than others. Change has also been contested, with departments challenging some developments, or taking time to come to accept particular methods as legitimate. And they have been problematic. Although it is not evident from published reports or official material, certain methods have been tried and initially failed to work. The changes have also raised issues about the skills available to the NAO, and about where criticising implementation becomes challenging policy. 146 I Taken together, these competing pressures appear to have produced the gradual, incremental developments outlined in Part 5. For much of the 1990s there has been a desire to enhance performance, and reduce the number of reports deemed less than satisfactory. Environmental changes and the pressures they have generated appear to have been a continuous backdrop to developments in the NAO. Other triggers have been more specific - the impetus provided by particular studies or individuals, for example. Similarly, whilst one department's dislike of a particular method may have taken some overcoming, more positive responses have helped set precedents elsewhere. In many ways this incremental development is not surprising. The NAO has continued to be responsible throughout the period for preparing reports for Parliament. It has not had the option of suspending work and considering and implementing radical change. In addition, given the generally positive way in which NAO reports have been received in the past (see Lonsdale in Pollitt et al 1999), there has been a reluctance to steer too far from proven approaches, which is entirely understandable and arguably responsible. The changes can also be understood in terms of an enlargement of the role of the NAO in the terms outlined in Pollitt et al 1999. This suggested that the NAO fitted closest to the role of public accountant, with its key tasks being to examine and report to Parliament on whether public money is spent properly and wisely. The changes and developments outlined in this thesis can be explained in terms of shifts within this grid, as indicated in the modified version in Figure 6.4, which shows the methods most commonly associated with each role. Using this 'force field' helps to demonstrate the directions of the expanding role of VFM. Developments can be seen as a series of excursions from home base (public accountant), whereby the NAO has ventured into various positions to the south (as management consultant) and south-west (researcher/scientist). This thesis makes it clear that the NAO has certainly not colonised these lands, but it does demonstrate that there is an inclination, as well as the means, to return. In time, the NAO may eventually wish to make these areas part of its own empire, raising a whole range of issues about staffing, and the role of the auditor. 147 -ir I Figure 6.4: Applying methods to possible roles played by VFM auditors f As a judge/magistrate As a public accountant Document examination Document examination Interviews Interviews Secondary material Secondary material Site visits Site visits z Surveys Benchmarking + Focus groups Performance measurement Case studies Operational research techniques Deadweight Statistical techniques Observation As a researcher/scientist As a management consultant Developments also accord with much of the recent literature on the growth in regulation discussed in Part 2 and, in particular, the growth in the scale and formality of the work. Hood et al 1999 argue that regulation in government seems to be moving in the opposite direction from that of operational management, a process they refer to as a 'mirror image'. They also suggest that there has been greater confrontation between regulator and regulatee, and growing complexity in regulation. Much of what has happened to VFM audit fits in with this analysis. The number of VFM studies produced each year grew in the 1980s, and was considerably higher throughout the 1990s than in the previous decade. Part 5 suggests that the work has grown in complexity, and VFM has grown as a distinct discipline, clearly differentiated from financial audit. The efforts made to extend the range of methods used by VFM auditors suggest moves towards greater complexity. However, other aspects of developments do not fit. Many changes outlined above suggest greater informality and attempts (often successful) to improve relations and reduce confrontation, for example, 148 by consultation on methods and the use of consultants. There has been an explicit acknowledgement of the burden of audit and the potential for audit work to discourage civil servants from taking risks in recent statements from the NAO and other audit bodies (Public Audit Forum 1999). At the same time, there seems little acknowledgement in Hood et al that the regulator - in this case the NAO - may be subject to many of the same pressures as the bodies they oversee. There is evidence that to suggest that the NAO has experienced many of the same pressures as government bodies. The NAO has considerably fewer staff in 1999 than in the 1980s, and far more work is contracted out to consultants. Performance related pay has been in place for some years, and average study costs reduced, year on year. In 1999 the first full VFM examination was contracted out. There has been a similar 'break up of the village' (non- accountants have joined the audit workforce), with 'lateral entry' at all levels of the office (newcomers have joined at director level). There is also empirical support for Power's observations on the 'audit society'. He states that 'the development of VFM auditing has also been an incremental repackaging and extension of existing audit capabilities' (Power 1997: 50), and suggests that such examinations arc 'constituted by a certain vagueness which makes for considerable discretion in their implementation' (p 11). The analysis in this thesis supports this view to a great extent. What we have seen has been an enhancement of existing evidence- gathering practices, accompanied by efforts to incorporate other methods to the repertoire. The 'vagueness' has provided the space within which auditors have been able to graft on methods from other disciplines. i 149 PART SEVEN: VIM AUDIT METHODS: AN INTERNATIONAL COMPARISON The final substantive section of this thesis compares and contrasts the developments in the NAO with those that have taken place during the 1980s and 1990s in other SAls, in particular, those in France, Finland, the Netherlands and Sweden. The overall aim is to shed light on the work of the NAO by examining comparable activities elsewhere. The chapter begins by considering the potential value of comparative research, and the problems associated with it. It then summarises the development of VFM audit outside the UK, and highlights key aspects of the work of the selected SAIs. The central section of the chapter provides an analysis of the methods used in their VFM audits during the 1980s and 1990s. The remainder compares developments at the NAO with those in the other four SAIs, and considers how this helps an understanding of the NAO's work. This chapter has a slightly different genesis from the others. In the first place, it draws heavily on the author's chapter in Pollitt et al 1999, although the argument has been developed and refined subsequently. The choice of SAIs reflects involvement in the Performance Audit Project, for which the SAls in Finland, France, The Netherlands and Sweden 35 were selected. A detailed rationale for this choice is given in Pollitt et al 1999, but in summary they are examples of the four main types of audit institution in existence within the European Union (National Audit Office 1996a: 234-235). Secondly, some of the basic research - specifically, the examination of reports published by the Algemene Rekenkamer in The Netherlands, the Riksrevisionsverket (RRV) in Sweden and the Valtiontalouden Tarkastusvirasto (VTV) in Finland, as well as the interviews at the VTV and the Cour des Comptes in France - was carried out by colleagues from the PAP team. Their work is acknowledged with gratitude. However, interpretation of the material here, as in the published chapter, is that of the author. Detailed research work by the author and three other colleagues undertaken for the NAO's book State Audit in the European Union (National Audit Office 1996a) is also drawn on. This included interviews with senior staff in all fifteen European Union SAIs and the European Court of Auditors, as well as a review of existing literature. Appendix I surnmarises the work of the SAIs discussed in this Part. Appendix 2 outlines the historical development of VFM work in these SAIs. ISO Comparative research - potential and problems Comparative research focuses on 'identifying similarities and/or differences between units at all levels, for example at a historical or cultural leve. ' (Sarantakos 1993: 7). It has also been suggested that 'insights into our own lives are thought to be enhanced by studying the ways in which different cultures and societies organize their social and political affairs and everyday lives' (May 1996: 153). Tbus, comparative research may allow us to reflect better on what is done in our own circumstances. It may also allow us to explain and understand why things have occurred in certain ways. And it does not always take a major new study to provide additional observations of value (King, Keohane and Verba 1994). Nevertheless, comparative research is not without its problems. In particular, there are issues of appropriateness (which refers to the fact that researchers cannot assume that the methods employed and the conceptualization of issues appropriate for their culture will necessarily be appropriate for another), of equivalence (which refers to the fact that meanings vary between cultures), and of differences in language (May 1996). There are inevitably difficulties for researchers in understanding the complexity of other societies and organisations with which they are not truly familiar. There are problems with generalising and explaining social relations across different societies and social contexts. Language differences, even where researchers have a proficient understanding, require a cultural understanding of words to 'allow for the equivalence of meaning', which is hard to achieve. Language problems may also encourage a reliance on official publications, which 'may then produce partial and incomplete theories' (May 1996: 162). There are strong justifications for including a comparative chapter in this thesis. The VFM work undertaken by the NAO is by no means unique, and there is value in examining whether the trends in the UK have been seen elsewhere, and whether theories developed for the NAO have a broader relevance. While audit is undertaken differently in different countries, there is an international 'community' of auditors - linked partly by international organisations such as INTOSAI and EUROSAI36 , but also through bilateral contact. As a result, there is much agreement about general concepts of auditing, and many areas of common interest and co-operation. 151 Evidence from other research (National Audit Office 1996a) also indicated that audit institutions other than the NAO had experienced significant change, for example, in terms of mandate or role. Discussions with staff at these SAls indicated that they had had to consider new ways of undertaking their work in recent years against a background of varying degrees of public sector change. This suggested that there may have been pressures on other SAIs, similar to those described earlier at the NAO. Examining how these bodies had responded in their work might throw light on the UK experience. A number of problems were experienced. Firstly, it became clear in the course of discussions within the PAP research team that there were differences in interpretation and understanding on terminology between the representatives from each country. This is far from uncommon (see Gray, Jenkins and Segsworth (1993: 4-5). The response was to attempt, at the outset, to obtain agreement on terms amongst the PAP team to try to ensure that we had equivalent understandings. There were also difficulties truly understanding what work was involved in apparently similar sounding elements of the VFM audit process. It is questionable whether it is possible to understand fully what others do without experiencing it, and (for an employee of the NAO for over 10 years) there will always be an imbalance between the author's understanding of NAO work and that carried out elsewhere. To try to minimise this problem, it was important, therefore, to seek clarification at a number of points from colleagues on the project team, and, on occasions, to draw on NAO's many international contacts. Language barriers also meant that much of the review of published material had to be restricted to that written in English, and a small amount in French. It also meant that many of the staff interviewed were those who spoke English, although the approach of interviewing in pairs (always with a colleague from the SAI being visited) meant that clarification could take place in the first language of those being interviewed. In order to try to reduce the scope for misunderstanding, prior to the PAP interview programme, the team held discussions about the set of questions to be asked, designed to ensure that they meant the same thing in different languages and would be interpreted similarly by interviewees - in effect, that our interviews would be an appropriate way of generating comparable material. Finally, the comparative researcher's perennial practical 152 restrictions of time and cost impacted on this work. The need to travel to several countries and to fit in visits alongside full time work restricted the number of people who could be interviewed. Nature of the states considered Before considering audit arrangements in the countries under examination, something should be said about the characteristics of the states themselves. Figure 7.1 summarises four key characteristics of government relevant to audit (Pollitt et al 1999). These are: e the administrative culture - whether there is adherence to a 'classic law-bound, rule-following bureaucracy', or whether the culture is more pragmatic and flexible. 9 the intensity of interest in new public management ideas.: -- the speed and vigour with which new public management reforms have been implemented, as well as their extent. * the extent of output orientation - the extent to which control systems have moved from consideration of inputs to greater emphasis on outputs or even outcomes. political system - whether a system in which executive power is concentrated in a one-party of simple majority government, with an often adversarial character (majoritarian), or shared between a number of parties, possibly with a balance between legislative chambers (consensual), or a mix (intermediate) -1 153 Figure 7.1: Characteristics of the government of rive states Administrative culture Intensity of interest in NPM Output orientation Political system Fin Legalist Low/medium High Consensual Fr Legal/ technocratic Medium (with high peaks) Medium Intermediate Neds Generalist Medium Medium Consensual Swe Generalist Low/medium (but NPM rhetoric - high) High Intermediate UK Generalist High High Majoritarian Source: Pollitt et al 1999 Figure 7.1 shows significant differences between the five states examined here. It suggests that France and Finland have the more legalist administrative cultures. It also places the UK at the more extreme end of two of the four characteristics - the intensity of interest in new public management changes and output orientation - and the only one with a 'majoritarian' political system. At the same time, there are a number of areas of common ground. Despite this diversity, Pollitt suggests four themes in the reforms that have affected all five countries during the last 20 years that have relevance to VFM audit. These are: the need to restrain expenditure; efforts to increase the productivity of public expenditure; moulding public services more closely in accordance with the convenience and preferences of their users; and strengthening 'transparency'. In his view 'the character of many of the reforms resonated vigorously with the specific, professed concerns of performance audit, in that the reforms themselves were self- consciously to do with performance' (Pollitt et al 1999: 55). It is against this background that the development of performance audit is considered. These characteristics might be expected to influence the nature of the audit work carried out in each country. A legalistic administrative culture, for example, would be expected 154 to encourage modes of evaluation that reflected concerns with legal correctness. Cultures with a greater tendency towards valuing managerialist approaches, or with sympathy for the social sciences, might influence their SAIs towards developing approaches that reflect these stances. Audit offices are, after all tied closely to broader administration, and their staff are drawn from similar sources as civil servants. Senior staff are, thus, likely to be sensitive to what is valued government. The development of VFM audit outside the United Kingdom Barzelay suggests that VFM audit is 'widespread though not universal across the OECD Member countries' (Barzelay 1996: 44), whilst the NAO has identified that 13 out of 15 of the European Union SAIs undertake some kind of VFM audit - the exceptions being the Hellenic Court of Audit in Greece and the Chambre des Comptes in Luxembourg (National Audit Office 1996a: 246). Additionally, Shand and Anand provide details of VFM audit work in SAls in Australia, Canada, New Zealand, Norway and the United States (Shand and Anand 1996). Collectively, this is by no means a comprehensive list. Given the broad and changing definition of VFM audit (see Pollitt et al 1999), it is difficult (if not impossible) to pin down when exactly the work started, and it may be more realistic to see it evolving gradually over a period of time. Shand and Anand (1996: 5 8) suggest 'in many countries performance issues as a legitimate audit concern did not really appear until the 1950s', with SAI mandates extended to encompass VFM audit from the late 1960s onwards in several cases. Pollitt has suggested that 'VFM audit as a large scale, self-consciously distinct practice dates mainly since the late 1970s' (Pollitt et al 1999: 1). More specifically, Glynn (1985: 114) refers to the then Swedish National Audit Bureau as being 'the first in Europe to adopt formally VFM auditing in 1970'. What is clear is that expansion has continued throughout the 1980s and 1990s - the Office of the C&AG in Ireland, for example, received a mandate for VFM work in 1993. More recently, several SAIs - for example, in Greece - have indicated an interest in broadening their work to encompass VFM audit. In addition, as was mentioned at the start of this thesis, where it has been established for some time it has broadened in range. In 1989, for example, the RRV's mandate was extended to cover state aided companies and state maintained foundations. And, where it has been longer established, 155 there has been a tendency towards the development of a VFM specialism, with staff working solely on VFM audit, rather than combining it with the audit of accounts (eg in the NAO, the Riksrevisionsverket and the Algemene Rekenkamer). Although there is general agreement that VFM audit involves the examination of economy, efficiency and effectiveness, Figure 7.2 highlights the variety in the ways that VFM audit is organised. The timing of the growth in VFM audit in each country is of interest. All have some kind of pre-history, but modem developments appear to stem in Sweden from the late 1960s, from the mid-1980s in The Netherlands, and from the late 1980s in Finland. The work has also grown from different roots. In Sweden, VFM audit grew out of interest in programme budgeting, and was designed initially to examine how far agencies achieved goals set by central government. Subsequently, it has broken away from this. Interestingly, from 1967 to 1987 the RRV had no financial audit responsibilities, and so VFM audit there developed quite separately from the influence of the audit of accounts. This contrasts with the NAO or VTV, where VFM work grew up as an adjunct to financial audit, and emerged from it, and the Cour, where VFM audit work has developed as part of overall examinations of public bodies, combining financial audit and the examination of performance. a 156 Gn ci cl wo to) ýwo cl -t_- .9 R e 1-. ll? ' 200-ß. 4. 0 . 0 010 c0- r- = 0 u; cn = r= 00:, 0.. = j2 u 16. AZZ r_ 2, ei ,'. ec ; >5 Z -zi e - ,Z -0 r- 2 E. (L) M- 926 öo .-W 2S ýa :E cý :. t-- - :30g- 0. Z. 0.5 0 c2. Z: Z: u rA r 13 u > im (U >% cn -= U ce (A U) c2. cm - u 926. - m - < 2 le c) 3 r- s. :e 2z 0 mi r -v r= = Z. 00 1 u = cr th fA b rA in !2 -g r- 0) c: ci r - 0 J. - , e u LA 00 _4 .-0 4> iz (L) '; cl 0ý 1 ;, 'ý r- = (L) r- A r. A . r. o- . 0 rA *0 0 s= L) 4) -0 .-e. 4. 'tj (2, ß. 5 CL, ) ä=>, 45 0, , 0-u 14 '. ' em 42 r_ ;2Eý. d -0 > 0 MRr. 0 .0 m= ýa 5. a - ýE- C, 3 01 »' - Co =xE -5 e, 0 u 92.0 "a :3 u. 0EE -3 mp r- 4) 4) r- C) -- 0s äZ J. g ý, v 41 1. - GM Ici r. (L) r UM Q<., m =. :sUub. M 0ä 9 ýa r -; -2 _ 0 :. 22M= >%. b 0. ;3 ýc er ee r2 = 0 . c3. M=0 CJ 2A> cm -"5 c) - 0muc2. bý ýn ci -0 . 52 0 Ei r= . Z; > cm 0 -5 4; 2 . E Co r= 13 -- 0 e (U c3 r. es EE r. r= 20 CD,. a fi >0 0 CA -c . ä >% '4 m c:. 78 00 2 Cl ý r. -0 r3 i tu 9 . U ue0 (Z (Z 22 4) CU 4- c) . I ' 2 cu 2 4. 90= - Li 40 t 1) = m- 00 ý cn 4-1 -Z0 cl en (L) = .=äU 'm 0 0 (n 0 A -2 -«= t! 0 ii E c2.9 . 6- A 2 -3 A= r= tý .0 -- cýa CU s. = 92. M s. = ce IM 0 .ck. (A -Z r- KA = Cb 0 -U ýo Zg& ci 1- 0 i JA 46 u Co a. E -0 43 t! E 4-1 0 ic ý)&. -. C. ) ý 1- :. ei, «S Aq ' r- U c) 1- ý, -a =- Icl s rA &ý gl. cn . Co% = Z& = CU cu 0 LJ ei A - Co :iE c) cj - ce ce 4,9 t) > 912 3 mA -ri 0= Co -e -a = x 0 r- c26 ZA tu im x =0 LLI 0 ci ui > JD = "- U 4. Jz = tu 0 u r Q GM 0- 3 ;= '. = Ei 0 tr. 00 ý - r: - >. 0 LI U00 *M uE 11 -0 ";. . - > ci A. ß. - r- Z Aý Z ÖD g2, ci fA +j CJ g- IMD2.2 ceuu 12 m , 1-- 00r. r= U -0 (A g- 2 o" ä3A ==u0 (> u c2. 0 4) --x E -e m ýa 0. a 0 t- Mu tä u :i . 0> «a 0 v. = -ý . 22 -2 öß :a0 ; ýi r ý e >--, u 0u . -2 *Z: .*= gt: &. m - c2. = e . 2 Kn ci. =29- 4- 20 0 Ici :2 - ' I g r- 0 du = JD 0 r_ IV WE ce lu 90 2 ýi = 0 -0 . .. ,Z-0ý Lw 12 - c2, 0 r: > 00 . ýa Z ci 0 ci 0E> m es r- 00 e p. uý CU = X g2 e l >, (3 U 9 cu 0 2 om 92. 0 Z0F. to 0>, < < z > > 1-1 Gn l N 22 E u; 5 tu 0 . . a. ) ci r = r_ M ;eZue. ' =- . 0 Co 0-0 tb. r- 'Z = 35' 0 t2 .- 4n . ip E -Z rA 10 = =0u m3 < cj 0m 0 4) 10 m "0 00 0 -c i; i le- = LA .. j= ci 00 . - s- CL U) c) r n . 52 .i CU g> W r. , 00 9: 6 >c 0 (L) = , &- u C: ) = Wý ce :jC=9u gl :100rZ r-4 E Mi . - CA = g: 6 2 N. r r- 0 mE r- :12 M, -CJ rA 0u cn - > 4, V) 0. 8 -12 i! 0 -bd Co ýa 'm m 't .-s. r= r C vli ýe u v; A :p (Li 42 - %Z r. 02u E e du - ý'n '* ý; 00 pý 0 (D UuX (U A , 0> C) (L) 00 g rA ý: .5 vi KZ. E- cy. , - Ici U o 00 - E-Z0c2-- r - -> 2 * 0 > Ei J2 t- C) = u J- = .. 0m fj = Q. % o t ) 0 = .-0U (Z. ul ,0 - *0 = ý: -a U (Z E=C 4- (L) t> cj 0 >% -vwl: U05. Gn m .Z e0 r_ t- c* Q % . -c2 -ý ý- (A 1 7, .-A tu g2. ü r- s. *Z 11. ) -0 ,mw . - CL.. - IU m= r 12 92 Co -2 i3 E1 'n 0 .Z -2 0 0=A0 r= -0> 25 c2. Er - r:: -. S -ý m-0 :3 r_ 0 -ti 0 r= 0 0- r= E4 c9 0 ) (a V) C 0 0 4) - M, 0 . . - t 0 A> 0% 0m E cr e0 . CL., ý5 --: 0- =. ýn u >Ee0 :3 00 ý gE >ý 0 = >Q r: aE m> J-- 0 f- 0 0 ý: (Z *- CL 0 uZ. - ID u ci E ý- ci 0 ci di U) 0 - 0W > 'u i' uZ -2 d 2 oo = 2 -. ij -ld r- u - .0 ý5 U - , 0 &A 0% ', c> m s=u (a >0 5 "0 0 0 r- m0 ID - 0- CA = ý5 = ýu (32 M Jd im. 42 0 "'0'% - 9- cm -a , 0. - r c% - -zj Im 0 21 ce. E3 u g; :% eL, 15 u-Z z3 uuu-0-0= :j .M0e m-ux0m u-mw >E0 .5", 2 ,, - . - - 11) ci c- <0 . ýq < V; l', 0 cm 0 0 0 LL. 0 42 Ei > 2 VT - = L 00 m ý: 0Am0 M im "0 0 r. 0> .-0 = 5r a, - :Eý. T r= = >, - m 0 . 4- ,0 - 0 M0 0 ý W. 0 ýe - ce =u0 x 2 < r- (Z iz F > 24 Zu -0. ý- > ce ce m , ýo Several different types of work are carried out by SAIs under the broad heading of VFM audit, ranging from the audit of the economy to programme evaluation. The Rekenkamer's work, for example, includes 'reviews of management/ administrative systems, reviews of programme effectiveness (5-10 major studies each year) and reviews of performance measurement systems' (Shand and Anand 1996: 93). From the mid-1980s it has undertaken government-wide audits, looking at the organisation of aspects of government in a number of ministries (Leeuw 1991). Generally, the VTV ficoncentrates on efficiency issues rather than programme evaluation, although there are some examples of the latter. Sometimes the [VTV] reviews the results of an agency's own internal reviews' (Shand and Anand 1996: 86). The RRV undertakes a mixture of agency specific studies and cross-government examinations, and may scrutinise the efficiency with which agencies operate, ways in which they co-ordinate their activities with other bodies, as well as their compliance with government requirements. The use made of audit reports also differs. In the NAO, as we have seen, the reports by the C&AG are examined by a specific committee, which in turn produces its own reports, to which the Government must respond. In the Netherlands, the work of the Rekenkamer is usually considered by the State Expenditure Committee, but also may be used on the floor of the lower house of the States General, or indeed by any parliamentary committee. In Finland, the VTV's work is submitted to the Parliamentary State Auditors, who include their own comments on some of these reports in their annual report to Parliament, as well as the Ministry of Finance and the audited body. The work of the RRV is directed to the agencies themselves and to the associated -1 ministry. Reports are also submitted to the Riksdag and its relevant committees, as well as the Cabinet Office. They may be included in discussions in the Riksdag, and on occasions, auditors from the RRV may be invited to appear before one of the Riksdag's Standing Committees to be questioned on published reports. Differences in the background and skills of VFM audit staff in these SAIs were addressed in Lonsdale, Mul and Pollitt in Pollitt et al 1999. This noted that, with the exception of the Cour, VFM audit was undertaken by a separate group of staff, giving 159 the discipline a distinct presence within the SAL Whereas at the NAO the trend has been to employ or train accountants, at the Rekenkamer social science and public administration backgrounds have traditionally predominated. At the RRV the majority of staff are graduates in economics and the social sciences, and often have previous work experience, whilst at the VTV many have master's degrees in accounting or business, or are postgraduates in public administration, political sciences or economics. Magistrates at the Cour are postgraduates of thetcole National dAdministration with qualifications in public law, finance and economics. Finally, the four SAls approach their work from quite distinct perspectives. Using Pollitt's four basic roles (Pollitt et al 1999), it is possible to place each SAI roughly on the grid. In the first role, the SAI is concerned primarily with compliance with the law (the Cour). In the role of public accountant, auditors are concerned particularly with assessing what public bodies have done, as against what they say they have done. They seek to establish the truth through 'exposure of actions to public scrutiny and debate' (the NAO). The researcher/scientist's concerns are with the generation of new knowledge and the production of explanations that can be defended because of their methodological rigour. The fourth role - that of management consultant - emphasises securing improvements in management, and usefulness. None of the SAIs falls easily into these two categories, but the NAO, RRV and Rekenkamer have all, at some times, shown an interest in pursuing these approaches on particular pieces of work, or combining them with other roles. Examination of methods employed by SAIs in the 1980s and 1990s Against this background the remainder of the chapter examines the'methods used by the four SAIs detailed above, and compares the results with the findings from the work of the NAO. The approach was to: e draw on the analysis of some 355 reports from Sweden, Finland and the Netherlands, undertaken by colleagues in the PAP team; * review material published by or about all four audit bodies for references to the use of methods; and 160 I e undertake interviews with staff within the SAls or draw on interviews conducted within the PAP. Figure 7.3 summarises the analysis of the review of reports published between the early 1980s and 1995. Although for the purposes of Part 51 examined a finther two years' worth of reports for the NAO, this was not possible for the four other SAIs. The NAO reports for 1997 and 1998 have, therefore, been excluded from the table. Reports were examined to a common framework; Figure 5.2 in Part 5 provides the definitions of the tenns. The caveats to the data from the four European SAls are similar to those for the NAO discussed earlier. The methods used could only be counted when they had been reported, and many of the reports contained only sparse details. Thus, again there may be an element of under-reporting. Similarly, it has been assumed that auditors have accurately recorded the methods used. The most significant additional point relates to the treatment of the Cour, which had to be different as its reports tend not to describe methods used, and many are not, in any case, published. (Instead, a record of the methods used is maintained - le dossier liasses rapport - with the evidence supporting a report. ) As a result, the examination of methods used by the Cour which follows on later in this chapter is based on interview evidence alone. Data for NAO are an amalgamation of that in Table 5.2 1ý a 161 Figure 7.3: Methods used by four SAls during the 1980s and 1990s VTVa AR RRV NAO Method 80s 90S 80s 90S 80s 90S 80s "Ods Document examination 42 49 27 48 47 52 86 137 Interviews 23 41 27 46 39 50 86 137 Review of secondary data 11 +7' 14+22 21 40 31 27 25 19 Questionnaires/surveys 4 16 1 14 15 15 6 44 Observation of activities (site visits) 24 2 5 10 12 27 47 Correspondence with auditee - - 8 1 - - - Benchmarking/ internal comparisons 12 18 - - 14 - 5 _ international comparisons 7 - 7 14 Case examples _2 5 5 6 13 34 Statistical techniques - 3 3 1 14 Consultation with third parties/expert groups 5 9 16 15 78 Modelling - 1 2 - 2 Depth interviews I - - 6 Sample examination 9 14 - 20 27 Expert interviews - 4 m - - Focus groups - - - - 4 Number of reports examined 42 I 49 1 60' 60 65 1 79 31 1 86 137 1 ' For the 1980s, the most suitable period for each SAI was chosen for examination. For VTV this was 1985-87, Algemene Rekenkamer 1983-85, the RRV 1983-85, and for the NAO 1986-88. b The first figure indicates the number of cases where results from earlier VTV audits were used as a source of information, whilst the second f igure indicates the number of reports that refer to studies by research institutions or scientific research as a source of information. 'A sample of 60 reports for each period was chosen for examination. For the first period this amounted to around a third of all VFM audits, and for the second period, around a half. 162 Valtiontalouden Tarkastusvirasto (VTV) The analysis of methods used by the VTV was limited by the fact that it has not been customary to explain the methods for data collection or analysis within reports. This was particularly true in the 1980s, although more information was available in the more recent period, along with some attempts to assess the validity of the results. Despite this, it would seem clear that, in the first period, VFM auditors mainly employed, what we might refer to as 'traditional' audit techniques such as document examination and interviews. Similar methods also predominated in the period 1993-95, when most reports stated that audits consisted of 'getting acquainted with the relevant documents' and 'having discussions' with those involved. In addition, the data suggest a slight shift in the 1990s towards greater use of non- traditional methods, although this should not be exaggerated. There appears to have been greater use of questionnaires, although all references to sending out questionnaires as a means of collecting data have been included, even where, in many cases, what was done may have been closer to corresponding with representatives of audited bodies, rather than carrying out a systematic survey. There has been a tendency to use more comparisons with other bodies, although in many reports these findings are presented in an illustrative or anecdotal way, rather than as the basis for the conclusions and judgements of the report. Finally, some comment must be made about the apparent ending of site visits. This does not seem to have been the case; rather it seems likely that the VTV ceased considering such work as a special technique worth mentioning in their reports, particularly since the State Audit Act 1947 states specifically that audits should, as far as possible, take place on the premises of audited body. Cour des Comptes Discussions with staff at the Cour des Comptes also suggested that there had been a slight widening of the range of methods used by auditors. However, the examination of documentary evidence appears to have been the bedrock of audit work here as well, with the most commonly examined documents being internal reports and instruction manuals, as well as reports from other external auditors. Interviews and questionnaires also predominated in both periods, along with the review of existing statistical and 163 accounting information. During the 1990s, other methods have been employed on 37 occasions. Meta-evaluation and meta-analysis , drawing on work undertaken by ministries, has been used, whilst case studies have been used more frequently in recent years. In the 1990s programme evaluation techniques have been developed as a way of analysing the full effect, both direct and indirect of government programmes (National Audit Office 1996a: 66). The Cour has been developing an evaluation methodology and has included text on evaluation in its Verification Manual, the magistrates audit guide. Although there has not been widespread use of economic modelling and statistical techniques, some magistrates do employ them and specific training courses have been set up. Benchmarking and other means of comparison have been used on many occasions. In a presentation to the NAO's International VFM audit methods seminar in 1998, one conseiller mailre summarised his understanding of the main types of methods used by the Cour (Mordacq 1998). His comments are broadly in line with the PAP findings. In particular, he stated that 'document examination remained at the heart of the Cour's work', and referred to interviews, questionnaires, meta-evaluation, case studies, and reviews of existing material. Algemene Rekenkamer The range of methods used at the Rekenkamer appears to have been fairly stable between the two periods, with the picture obtained from the analysis of reports confirmed in interviews with staff. The most prevalent reported methods used by VFM auditors in the 60 reports examined in the first period (1983-85) were the examination lb of documents and interviews. Analysing existing data was also a common activity, and a number of reports drew on information provided in correspondence with the auditee. Questionnaires and the direct observation of administrative processes were rarely used. in the second period (1993-95), the use of interviews and document examination again predominated, along with the examination and analysis of existing data. The most significant increase was in the use of questionnaires, whilst a number of other methods were used for the first time in the second period. These included case studies, in-depth interviews and expert interviews. The findings from our survey of reports are not 164 dissimilar to details contained in the Rekenkamer's manual of VFM audit (Algemene Rekenkamer 1996: 18). This states that 'there are various data-collection methods and techniques. In practice, the most commonly used is file examination and interviews. A written or telephone survey is also frequently used. In some cases, a panel of experts is engaged'. Riksrevisionsverket A majority of the RRV's VFM audits in both periods appear to have made use of the analysis of documents, interviews, and many undertook examinations or analysis of existing data. Figure 7.3 also shows the use of a small number of other methods. In both periods an identical number of reports made use of questionnaires, although this represented a reducing proportion. The second period saw a growth in the number of reports using explicit consultation with others, such as seminars or expert groups, as a way of gathering information. There was also an increase, from a low base, in the occurrence of benchmarking or international comparisons. In contrast, limited use appears to have been made of statistical and modelling techniques. Case studies have also been used occasionally, the term being used here to mean where auditors had visited the audited body to gather new information. In general, there appears to have been a broadening of the range of methods employed between the two periods, although the tendency has not been particularly strong, as even in the early 1980s some studies employed non-traditional audit research methods. In general, these findings accord with what the RRV has identified as the 'most common techniques' (Swedish National Audit Bureau 1993: 87-94). These were: interviews, seminars and 'hearings', experts and reference groups, direct observations, studies of documentation, case studies, questionnaires, processing statistics and making estimates. Analysis and discussion Against this background consideration is now given to five key questions. These are: a How does the way in which other SAls have characterised their use of methods compare with the NAO's view of its work ? How does the range and type of methods compare with those used by the NAO ? 165 " How do the factors influencing the choice of methods compare with the NAO ? " How does the coverage of methods in the reports of SAIs compare with that of the NAO? " Does there appear to be any relationship between the characteristics of SAIs discussed at the start of the chapter and the methods employed ? How does the way in which other SAIs characterise their use of methods compare with NAO? Part 5 considered how the NAO has characterised its use of VFM methods. Many of the same points were made in published material produced by the other SAIs, as well as in interviews. The RRV, for example, has commented that the approaches applied in VFM audit have been changing (Swedish National Audit Office 1996: 19). In particular, it has suggested that 'The link to programme budgeting has weakened and perspectives and methods from other academic disciplines such as political science and sociology have been introduced. ' According to staff at the VTV, change came quickly in their work too, following a period of administrative change in the early 1990s. They referred to the loss of their 'familiar tool kit' and the need to replace it with new ways of working. Such developments were on-going at the time interviews were undertaken, and one senior auditor considered it likely the VTV would borrow more from evaluation research methods in the future. The issue of greater sophistication in the selection and use of methods also came up. A number of senior auditors in several SAIs stated that, compared with ten years previously, considerably more thought was given to designing studies and choosing appropriate ways of obtaining information. At the Rekenkamer staý stated that attention to methods and methodology had become more systematic. At the Cour and the VTV internal committees or training courses or manuals of advice had been developed which addressed methodological issues. A number of interviewees spoke of anticipating problems in evidence collection and analysis much more than in the past, and thinking through how to reduce or eradicate them at the earliest possible stage. 166 Similarly, several of the SAIs considered their approach to VFM audit had become more outward looking. The RRV has drawn parallels between the methods employed in its work, and the evaluations and investigations carried out by research workers or government commissions, studies of the rationalisation and restructuring of government carried out by special government bodies, and the assignments carried out for public bodies by consultants. It has also drawn attention to what it sees as the fluctuating links between VFM audit and academic research. One document suggested that 'at times the links to academic research have been very strong and the audit reports have to a great extent been based on theories borrowed from different academic disciplines, while during other periods the reports have been more practically orientated, based on audit experience and common sense' (Swedish National Audit Office 1996: 20). Despite this, SAIs have drawn clear distinctions between the way they and others examine public sector activities, often emphasising what they consider the very practical nature of their role. In discussions, auditors at several SAIs made the point that they were not research institutes, and instead were concerned primarily to produce information that was 'useful' to their audiences, rather than necessarily look to innovate in methodological terms. if there is any significant difference in the reports of the four SAIs with how the NAO characterises its choice of methods, it is that there appear to be fewer references to the importance of robustness in methods for the practical purpose of defending their findings. This probably reflects the different arrangements for exposing the reports and obtaining auditee acceptance, with the NAO's 'clearance' process the most rigorous and potentially confrontational. This may well be because in the UK it " is a more significant an element in the overall audit process. Elsewhere, sound methods are justified more for their own sake, although the Cour does refer to the importance of being able to defend in its own process for exposing the content of its reports to auditees, even though there is not the same requirement to agree the form of words with clients as at the NAO. 167 How does the range and type of methods used by other SAls compare with the NAO ? In simple terms the number of different methods used was greater at the NAO in the 1980s and 1990s than it was in the work of the other SAIs. In particular, in the 1990s the NAO appears to have broadened its range of methods more than the other SAIs. After the NAO, the RRV and the Algemene Rekenkamer have the widest range of methods, with the VTV reportedly using the narrowest range. Interview evidence suggests that the work of the Cour has also tended to rely on a relatively small number of tried and tested methods. The data in Figure 7.3 do not suggest a revolution has taken place within the SAIs between the two periods examined. If anything, it suggests that SAI rhetoric is slightly in advance of practice, with the range of methods employed not yet fully reflecting the degree of interest shown in finding new ways of undertaking the work. Yet, we do see the more routine use of questionnaires (several SAIs), greater use of international comparisons (VTV), and some first attempts at modelling (RRV) and meta-evaluation (Rekenkamer and Cour des Comptes). More importantly, however, a number of the features of the NAO's work discussed in Part 5 appear to be broader trends. These are discussed below in turn. Prevalence of the use of documentary evidence and interviews First, there appears to be a similar reliance on documentary evidence and interviews. Figure 7.3 suggests overwhelmingly that documentary and interview evidence are the two most important methods employed in VFM audit across all SAIs. Just as the NAO emphasised the value of documents, so the RRV has described the study of official and unofficial documents as of 'central importance' to an audit (Swedish National Audit Office 1996: 85), and Mordacq (1998) has described document examination as at the 'heart' of the work of the Cour. The reasons given are similar to those at the NAO. They too have strong statutory rights of access and wish to exercise them. The Algemene Rekenkamer, for example, has stated that 'Although [their powers of access to documents and officials] have been debated in parliament during the post-war years, problems rarely occur in practice' (Algemene Rekenkamer 1989: 31). In extreme circumstances, the powers of other SAIs appear 168 stronger than those held by the NAO. In France, those who refuse to assist the Cour may be liable to a fine of up to 100, OOOF. Similarly, in Finland the VTV's powers under the 1947 State Audit Act allow for fines against anyone failing to provide documents on request within a set time limit, a provision which it has apparently yet to use (National Audit Office 1996a: 45). The second most common means of gathering evidence employed by SAls are interviews. Our discussions with staff at all four SAIs suggest that greater attention has been given to interview technique in recent years. The scale on which interviews are used can be extensive. A 1996 examination by the Rekenkamer on the policy of using licences by central government ministries, for example, involved detailed interviews with 200 licences holders to ask about their experiences (Boekenoogen 1998). And the work can be sophisticated. For the RRV's study of Illicit work in Sweden (Swedish National Audit Office 1998), over 100 in-depth interviews, some lasting up to four hours, were conducted with individuals active in the 'informal work sector', requiring tact, and the development of considerable trust between the interviewer and respondent. A 'snowball' method was used, with respondents asked to provide fin-ther contacts for interview. The use of secondary material Figure 7.2 shows that all SAIs make extensive use of secondary material. The VTV's work has drawn on the work of research institutes and scientific research, neither of which is very common in NAO reports (Pollitt and Summa 1996: 41). Its reports have also drawn on its own previous work, a point which may help to provide a collective view on particular subjects, which some have suggested has been lacking in NAO's work. The RRV's work on social security (Swedish National Audit Office 1995) has made considerable use of material already held within other government bodies. Associated with this has been the development of interest in meta-evaluation and meta- analysis by the Algemene Rekenkamer and Cour des Comptes. Similar work does not appear to have been carried out in the NAO. The Rekenkamer has examined the use of the large number of evaluations undertaken by government ministries and agencies in the Netherlands (Kraan and Rozendal 1997). 169 The development of consultation with users and suppliers of public services It was noted earlier that a key theme of Pollitt et al 1999 is the way in which SAIs have engaged much more with other parties in recent years, be they auditees, the recipients of the services under examination, the audiences for their work such as parliamentarians, or the media. This is also reflected in some developments in the use of methods, in particular, the growth of the use of questionnaires by some SAIs, and the increased consultation with interested parties of all sorts during audits. The most popular method aimed at eliciting the opinions of others used by SAIs has been the questionnaire or survey. All SAIs in this study have used them, although they appear to vary considerably in format and style. Examples at the Rekenkamer varied from pre-audit sets of questions, designed to elicit basic information on which the auditor can base further interview questions, to full surveys. The RRV's report on Illicit Work in Sweden (Swedish National Audit Office 1998) provides a good example of the use of a quantitative survey. A random representative sample of some 3,000 people between the ages of 18 and 74 was drawn and telephone interviews undertaken. Another development seen at the NAO, and also pronounced at the RRV, has been the growth in the extent of consultation with third parties in the course of designing and executing audits. Consultation takes a number of forms, from simply interviewing third parties, to perhaps involving them in identifying future audit projects. The RRV obtains outside views (Swedish National Audit Bureau 1993: 88-89) through seminars, where experts, researchers, and government representatives are invited to help identify new ideas, or, during a study, to discuss and test the observations and conclusions of the audit team. Such activities are not reported in VTV reports in either period, and appear less a common in the Algemene Rekenkamer's work. The predominance of social science over other methods The appearance of questionnaires within the methods employed by SAIs indicates that, where they have expanded beyond traditional audit methods, SAIs have tended to look towards the social sciences for inspiration. Another example of this trend is the use, on occasions, of longer case studies or, more usually, shorter case examples to help highlight particular points or explore certain topics. Case examples have been used to 170 inject life into their reports, a point which staff at the Rekenkamer considered was of particular importance in helping non-specialists, particularly parliamentarians, understand their reports. The RRV (Swedish National Audit Bureau 1993: 91) has also shown itself an advocate of this method, recognising its value to provide in-depth cause and effect analysis, often in combination with other methods. In contrast to the use of qualitative social science research tools, these SAls appear to have made much more limited use of economic models and concepts. At the VTV, staff indicated that economic techniques were used less now than in the past, and staff at the Rekenkamer observed that there had been a lack of economists within the office over the years. At the Cour, staff reported that there had been recent interest in cost-benefit analysis, for example, in a recent examination of family allowances. SAIs appear also to have made been fairly limited use of statistical analysis. Time series analysis does appear, but multiple regression analysis and correlation have been infrequently employed. Drawing comparisons Figure 7.2 shows that high level comparisons between audited bodies and others has been used by the VTV in the period 1993-95. Again, though these have tended to be 'high-level' comparisons between bodies. Despite this, there is evidence to support the view (see Pollitt and Summa 1996: 41) that 'SAIs appear to be slightly more cautious than evaluators about entering into comparative analysis that goes beyond the organizations immediately under investigation. ' Such problems appear to be magnified when international comparisons are considered. Although such work has been carried out - for example, reports by the RRV compared the performance of the Riksgaldskontoret (the National Debt Office) and the decentralised rail authorities, with corresponding bodies abroad (Swedish National Audit Office 1995: 15) - in both cases exact comparisons could not be drawn, and the purpose was more to widen the perspectives of the discussions. 171 How do the factors influencing the choice of methods compare with those in the UK? The factors - both internal and external - which appear to have influenced the choices made by these SAIs are now considered, as well as those that may have constrained the use of a wider range of methods. Factors working against a widening of the range of methods Perhaps the most important factor working against a widening of methods appears to be SAIs' perception of the strength of their existing tools, primarily the ability to demand documents and require answers from staff within government bodies. That SAIs make extensive use of, and have great faith in, interview and documentary evidence is not surprising given their statutory powers. Staff at the Rekenkamer, for example, suggested that the relatively limited range of methods used in recent years reflected the fact that they could obtain the answers needed from a small range of tried and tested methods. This may have had a dampening effect on the need for SAIs to look at other ways of obtaining information. In all countries the choice of methods is guided also by an SAIs own self imposed standards (influenced in part by internationally agreed guidelines), by its view of what VFM audit entails, and by judgements about what can be achieved within the timescale and budget set for a study. The choice of method will also reflect what auditors and the senior management of SAIs consider are appropriate ways of collecting the amount and type of evidence they believe is necessary. One former senior Rekenkamer official, has suggested that 'Empirical evidence from mail, telephone or face-to-face surveys is not enough foundation for auditors' judgements so the analysis and coilclusions rely heavily on documentary evidence. ' (Leeuw 1996: 52). A French official has suggested (Mordacq 1999) that the traditional system of audit, whereby the comptable (the official held responsible for public funds within his control) 'is under scrutiny for all errors and irregularities, whether material or not', has had a major and continuing influence on attitudes to financial and management audit. Thus, the nature of SAIs and audit may have encouraged a degree of conservatism that has worked against innovation. 172 As at the NAO, the backgrounds and knowledge of staff appears to have contributed to a concentration on more traditional audit methods. Many are trained in law and public administration, and have, not surprisingly, looked to these disciplines to inform their thinking. At the VTV, staff suggested that many auditors did not have a very wide knowledge of different methods that could be used, and some staff at the Rekenkamer noted the difficulties the office had had over the years in recruiting economists and bankers to complement the social science and public administration emphasis amongst their staff. It is noticeable that perhaps the most methodologically adventurous of the four SAIs - the RRV - employs more social scientists than the others. Time considerations inevitably impact on the range of methods that can be employed within the fieldwork period of a study. The fact that, in the 1980s, it was unusual for any audit examination at the VTV to involve more than 50 days' work, inevitably restricted the scope for more time consuming methods. More recently, this has grown to an average of over 200 days to carry out work leading to a final report (National Audit Office 1996a: 49), making innovation more realistic. Cost considerations were also mentioned in interviews, and these may have encouraged SAls towards more , traditional' audit methods, which have generally been seen as cheaper (Anand 1988). Factors encouraging interest in a wider range of methods Turning to the factors that might have encouraged greater interest in a wider range of methods, it seems that the changing, and increasingly complex, political and administrative environment in which SAIs have operated has had the most significant effect, as it has on the NAO. Several SAls drew a clear link between the growing complexity of public administration in their country, and the need for clearer approaches and sharper tools to assess the performance of auditees. Of particular significance for SAIs have been the improvements in the quality and quantity of performance information, characteristic of public management reform. This has also opened up new possibilities for making use of certain methods not considered possible before. In the Netherlands, for example, the expansion of programme evaluations within ministries and agencies has encouraged the Rekenkamer to developing the use of meta- evaluations. 173 Within SAls a number of factors have encouraged wider thinking about methods, including the attitude of the heads of SAIs, the example set by colleagues who use innovative methods, and the introduction of staff with new skills and experience. At the RRV, the encouragement of the Auditor General was considered to have stimulated interest in broadening VFM audit approaches, whilst at the Cour, the Premier Prisident created a committee in the late 1990s, chaired by a former President de Chambre, to consider new methods. Discussions with auditors in several SAls also revealed that another key factor in widening the range of methods has been the presence of staff within SAIs with particular skills. Staff in the Algemene Rekenkamer considered the relatively common use in their work of statistical sampling techniques in the 1990s reflected the presence for some years of a specialist statistician in the office able to provide advice. Elsewhere, changes in the background and experience of staff were regarded as important. Considering how the Cour has reacted to the new demands placed on it, Mordacq (1999) has suggested that 'we have smart young auditeurs and rapporteurs with a business administration degree who are pushing, and in the nitty- gritty of our work, more and more emphasis is actually put on system-based approaches and VFM inquiries. ' How does the coverage of methods in the reports of SAIs compare with the NAO ? Part 5 noted how the extent to which NAO reports revealed details of methodology had changed significantly over time, with more explicit information included now in most reports. Consideration has been given (Pollitt and Summa 1997b) to the amount of performance information that SAls publish about their activities. This found a wide variety of approaches. Although their work focused on a slightly different set of SAIs, the VTV, RRV, NAO and Cour des Comptes were included in this research, and a .1 similarly varied picture appears in the amount of information provided about methods in SAI reports, with the ones publishing the most information about themselves (NAO and RRV), also revealing the most in their reports about their methods. Of the four SAls, the RRV includes the most information, although reports vary considerably in the extent to which they describe the methods used. Some reports from both periods mention them only in passing, whilst others contain a more explicit 174 methodological discussion. There appears no clear connection between the nature of the project, methodological complexity, and the appearance of a methodological discussion in the report. Rather, the logic appears to have been that, when new or inventive methods are employed they are covered in some detail in the published report. More recently, the description of methods has been done on a more regular and relatively standardised basis in the RRV's reports, and also, on occasions, in some considerable detail (eg Swedish National Audit Office 1998). Explanations of the methods used in studies do not appear to be given the same prominence in the other SAIs. At the Cour des Comptes details of techniques used are not included in reports, and, at the time interviews were undertaken in 1997, the Algemene Rekenkamer's Board had not encouraged the inclusion of methodological details in reports, although in some cases there are brief references to the work undertaken. At the VTV, methodology has been explicitly addressed in only a few reports, although in the 1990s some have discussed methods and sources of data, even in some cases the validity of the results. Overall then, the evidence suggests that SAIs vary significantly in the extent to which they explain their methodology, although, as in the NAO, there has been a tendency in recent years to provide more information. Discussion This chapter concludes by surnmarising the work of the four SAIs, and comparing it with the findings from the NAO. The range of methods used by the SAIs has remained relatively stable in the 1980s and 1990s, although some new ones - for example, in- depth interviews, meta evaluation - have been used, and refinements have been made to existing ones, for example, by more sophisticated questionnaires. Pollitt's reference to 1ý the period being one of 'occasional experiments' (Pollitt and Summa 1996) appears appropriate, although staff in the four SAIs were at pains to highlight that they had given greater attention in recent years to how they planned their work and applied methods. Such qualitative improvements are hard to measure, but there is no doubt that considerable thought has gone into developing VFM audit, and many reports, such as on the black economy (Swedish National Audit Office 1998), have tackled complex subjects. 175 Of the four SAls examined, the most adventurous - marginally so in terms of the number of methods, but more significantly perhaps, in terms of their content and execution - appears to have been the RRV. A number of notable studies have combined a range of interesting and innovative methods. The RRV has also published a substantial analysis of its own role (Swedish National Audit Bureau 1993, summarised later as Swedish National Audit Office 1995), covering the context for its work, the historical background, and the organisation and execution of VFM audit. This demonstrated well the office's impressive pedigree in Sweden. Elsewhere, the interest in developing a wider repertoire of VFM audit methods appears to have been taken forward more slowly. At the Rekenkamer, senior staff expressed an interest in extending into using econometric techniques, cost-allocation methods and cost-effectiveness analysis, suggesting that they too are considering ways in which their work can develop, but, at the time of the interviews, this had not yet fed into published reports. Perhaps less methodological innovation is evident in the work of the VTV and the Cour, but even here there is evidence of a growing interest in extending the boundaries of VFM audit. Thus, Pollitt's conclusions, drawing on this author's work, appear about right that: 'Perhaps one could say that, since the mid-1980s, the Swedish and UK SAIs have each given some prominence to increasing their methodological self- consciousness and sophistication, whereas the Dutch, Finnish and French SAIs have, in different ways and degrees, been somewhat more reticent. That is not to imply that the Algemene Rekenkamer, VTV, and Cour are in some sense lagging behind in a race. (Indeed, each of these has attempted some ambitious studies,.... ). However, they seem to have made less of a public or professional issue out of 'tools and techniques'. There is less evidence, in their cases, of descriptions of their methods in their reports, of technical handbooks or mandatory training courses or any of the other paraphernalia of self-conscious technical self- improvement' (Pollitt et al 1999: 198-199). 176 Comparisons with the NAO It is clear from the evidence in this chapter that many characteristics of the work of these SAIs are similar to those of the NAO, for example, how they have characterised their use of methods, the predominance of document examination and interviews, and the use of secondary material and data. It is also noticeable that all four SAIs make use of questionnaires for a range of purposes. Indeed, the uniform picture regarding the use of documentary evidence, interviews, secondary data, and to a lesser extent, surveys, suggests that these are the basic methods of VFM audit. (A view further reinforced by a comparative study of the Swedish and Danish SAIs (Kjar 1996)). It is also noticeable that the four SAIs appear to have made no greater use of statistical techniques, economic concepts and models than the NAO. Many of the influences working for and against the expansion of methods also seem similar. The traditional conservatism of the audit approach and SAls as institutions, as well as a belief that existing approaches to VFM audit can continue to provide the answers, have not encouraged innovation. Time and cost constraints were also cited as reasons for relatively limited changes. And at the Algemene Rekenkamer and the RRV, as at the NAO, there were concern that the balance of staffing for VFM audit examinations required refinement, and that there was a need to bring in people with new skills and able to apply different methods and approaches. However, despite much common ground, there is evidence to suggest that the NAO has pushed things further during the 1990s in several ways. In particular, it has: * used a broader range of methods over the period, and extended in some areas that other SAls have not. It has also not particularly avoided any methods used elsewhere, with perhaps the exception of meta-evaluation; demonstrated greater interest in methodological developments, although a similar level of interest has been shown at the RRV; developed further the consultation with users and producers, making it a more central part of its work, with perhaps the RRV again the closest in approach; 177 taken ftuther the quality assurance process, which has systematically exposed published reports to outside scrutiny in a way that others have not; pushed the use of outside consultants as part of study teams fin-ther than other SAIs; and om tended to provide more information in its reports about the methods used than other SAls (whatever misgivings some critics in the UK may have on this point). Interpretation It now remains to try to explain these conclusions, firstly, by returning to the characteristics of the administrative/political culture summariscd in Figure 7.1, and secondly by examining the key characteristics of audit arrangements in the countries under examination in Figure 7.4. This is done, acknowledging the limitations on any analysis based on only covered five countries. On the basis of the analysis in Figure 7.1 three observations are made. First, it appears that those countries with traditionally the more legalistic administrative cultures - Finland and France - are associated with relatively less innovation in VFM work. In France, for example, the audit approach conducted by the Cour des Comptes has focused very significantly on public bodies' compliance with the law, and for this reliance on more traditional audit methods has been deemed satisfactory. As one observer put it, in this culture 'The actions of both the individual public servant and the individual citizen will be viewed within [a] context of correctness and legal control, (Pollitt et al 1999: 32). Although there has growing interest in the social sciences in Finland, the administrative classes have traditionally had a legal training (Pollitt and Bouckaert 2000: 221). Such an approach is thus likely to influence the way in which performance is judged. Secondly, both the SAls identified as having overseen the greatest innovation in VFM - the NAO and the RRV - are situated in countries with high output orientation. (This is also true of Finland, although here the relatively small size of the SAI and the shorter tradition in VFM as a discrete activity may have reduced the opportunities to develop new approaches to date). In Sweden 'performance management, based on a shift from 178 an orientation to input and procedural controls to a system based on the achievement of measured results, became a central philosophy of the public management reforms from the late 1980s onwards. (Pollitt and Bouckaert 2000: 262). In the UK, successive governments have encouraged the use of performance indicators and league tables. A climate in which there is an orientation towards measuring outputs would seem to fit well with a strong interest in developing VFM work, even if, as yet, many SAIs have not pushed the audit of effectiveness particularly far. Thirdly, Part 6 suggested that changes in the administrative environment had had a significant influence on the changes at the NAO, and this is reflected in Figure 7.1. The link is less clear for other SAls, although Pollitt (Pollitt et al 1999: 197) suggests that while Sweden has seen relatively low intensity implementation of NPM changes, the rhetoric of management reform has been 'quite radical and insistent', and sufficient for Hood to categorise Sweden amongst the major reformers (Hood 1998). Ultimately, Pollitt concludes that 'it might be said that the rank order of SAIs in terms of special tools and methods for performance audit roughly corresponds with the rank order of the corresponding governments in terms of the intensity of its public management reforms. ' (Pollitt et al 1999: 199). Finally, there appears to be no simple relationship between the nature of the political system and the extent of developments in VFM methods. Sweden and the UK have very different systems, and the categorisations of the Finland and France are also different. The overall nature of the political system may not, in any case, reflect the atmosphere in which audit is considered. In the UK, for example, although the political system tends to be adversarial, there is a cross-party approach to considering the reports of the NAO in the PAC, and reports are produced by the Committee on basis of consensus. In effect, party-political differences tend to be put aside as politicians combine to take on the administrative elite, and legislature seeks answers from the executive. High-level administrative factors may offer some limited insights, but more may be gained by looking specifically at characteristics of audit arrangements in each of the countries. The categories are: the position of the SAI within the constitution (whether it is linked to the executive or legislature may influence the type of work it does and hence 179 what methods it uses); the strength of the processes for handling the reports of the SAI (based on the suggestion that those SAls with very formalised parliamentary procedures for considering audit reports need to do less to make their work appealing to their prime audiences than those where there is greater uncertainty about how reports will be handled); whether there is a tradition of VFM as a separate activity (the assumption being that separating it from financial audit ought to increase the scope for methodological development); the type of staff employed (the background of staff should influence the methods they employ); the role of the SAI (using the classification in Pollitt et al 1999); and the range of product types (using the classification in Barzelay 1997). Figure 7.4 summarises the evidence. a 180 wi CL) .c W Z cc im fi ei ; Lo . s2 0 2: rn rq fol le (4 2 4. U- CD -5 .==, Ei E -ý0 = CD -5 0=9 Ici s. u2 . 2> r4 ý :i rA Ici O(D < ýi 0-2U, -ci t: (D *5: 2, r r- 1 4. ) 0 2 ý; t: V'& ý 10 O(D s l , Z en Z tu -0 ,0 eL & CL +0 a (: S-GC% :, e wý - 2 r. M Ei 0 b. = = %d uu ' lýI 0E l', ý0 Z 0 t' M (Z =0 I = 0U ci- 0 0 ri u M 12 C t) n Z: .5 CA u 12 0 -, 2 9 . 0 g cu Ln Ci. u1 in - 00 re Z '2 rA >0 ý- U 4 cn (n En Lg g-. e 10 e0 ;9 15 2 - ;2 ýA U) ci u c2.2 9w2. .2 bo 5u Z. m r_ 0 0 92- x 4) 42 gyo M. 42 c4.2 x 4) c4 U r_ 2 "ö u 0 ýc 2 iz Z u2 :D I--, CI ý. w m %ýo ON C: 0 . 19 z 2 0 W rA r- 0 u X 14 -23- C/) 4) 0 The different constitutional positions of the SAls ensure a range of relationships between them and the executive and legislature. This might be expected to result in different types of work, requiring different audit methods, and more or less confrontational relationships between auditor and auditee. A position closer to the executive might, for example, lead to the SAI undertaking work for the government, and could potentially ensure better relations between the SAI and those it is examining. In the case of one of the two SAIs closest to the executive - the RRV - specially commissioned (and funded) work is undertaken for the Government, but there is no evidence that this uses significantly different methods to examinations chosen by the RRV itself. In addition, although the audit system in Sweden may be less confrontational in theory than that in the UK, there is no evidence that auditees are any more welcoming of the RRV's work and criticisms because it is also a part of the government. Indeed it seems that often agencies treat RRV with considerable suspicion, seeing it as an 'instrument of the Ministry of Finance' (Ahlback 1999). Thus, the constitutional positions of the two SAIs identified as having the more innovative track r6cords - the NAO and RRV - do not appear to be determining the extent to which they have pushed their interest in VFM audit methods. The RRV's methods do not appear noticeably different because it is a government agency. In her comparative examination of the RRV and the Danish SAI Rigsrevisionen, Kjar (1996) suggests that the latter's work had been more limited in scope and focus, and that one reason for this was that the very formalised parliamentary procedures for dealing with their work. This had meant there was little need to develop procedural and methodological expertise, or focus on politically or economically interesting questions. Regardless of the efforts made, reporis would be dealt with in a! well defined way. In contrast, Kjar suggests that the RRV had greater need to make its work attractive and penetrating because there was not the same certain route for dealing with its reports. If this is to hold true for other SAIs, then the NAO's work - which is routinely examined by the PAC in a system which is arguably as 'watertight' as any in terms of ensuring recommendations are considered and acted on - would be limited in scope, presentation and methodology. As we have seen, this is not the case, the evidence suggesting that, despite its monopoly supplier position, the NAO has developed its work for the PAC 182 considerably in the 1990s. Conversely, the VTV - which has no direct official contact with the Finnish Parliament (National Audit Office 1996a) - might be considered to have the greatest need to push its work furthest in order to gain attention, but again this does not accord with the evidence. This kind of institutional explanation does not, therefore, seem the answer. Organisational and staffing factors may play a part. One possible contributory factor is the existence of VFM work as a separate entity. In Part 4 it was noted that until the early 1990s, many staff at the NAO undertook both financial audit and VFM work. After that, the growing tendency towards specialisation was formalised, the argument being that both disciplines were now too complex to expect staff to become proficient in both. Of the five SAIs, VFM work was undertaken as a separate discipline in all but the Cour, where fewer developments were seen. In contrast, at the RRV, performance auditors have been a distinct grouping for many years, and at the Rekenkamer there had been separation - albeit less complete - for some years. Perhaps the best that can be said is that it is a contributory factor. Linked to this is the composition of SAI staffing, and there appears to be more evidence of a relationship with the degree of innovation demonstrated by the SAIs. At the RRV, where we saw the most innovative developments, many staff have social science and research backgrounds, and so are likely to be familiar with a range of research methods. At the NAO, there is not the predominance of these skills and experiences, but by bringing in specialists more and more, using specialist panels, and employing consultants on almost every study, they have broadened considerably the breadth of knowledge and experience available. a The number of staff may also have an influence on developments in methods. Of the five SAls considered here, the NAO is the largest - with 750 staff, of whom around 180 work on VFM studies (National Audit Office 1996a: 240). Because of this the NAO has had scope to establish a separate team - consisting of experienced and high quality staff - whose job is VFM development in the broadest sense. This has included monitoring developments, developing training courses, Raising with outside bodies, producing guidance for staff, and charting the course for the future of the NAO's VFM work. Their 183 presence has given the pursuit of new methods a push, as was considered in Part 6. At the RRV more than 100 staff work on VFM, nearly twice the number at the VTV. The different roles played by SAls identified in Pollitt et al 1999 were referred to earlier. In that analysis it was suggested that SAls could play roles as public accountants, magistrates, researcher/scientists or management consultants. They tended towards one role or other, but at times took on different ones. The role that the SAI takes on appears to have a significant bearing on the range of methods which it has employed. In the case of the Cour, for example, the strong association with the magisterial role, with its concerns for legal compliance, may help to explain the relatively limited range of methods used, and its primary use of documentary examination and interviews. In contrast, the RRVs use of a wider range of methods and its interest in qualitative methods in particular, reflects its tendency towards the researcher/scientist role. In the NAO's case, Part 6 recorded that there had been greater efforts to explain why changes have occurred (researcher/scientist), and towards advising how improvements could be made (management consultant). This had required the NAO to draw on a range of new methods. The NAO's role and perspective has altered over the last 10-15 years, and this helps to explain the very noticeable (and still on-going) search for new ways of undertaking its work. There is also an important link between the developments in VFM audit methods and the extent to which managerialist thinking has penetrated the SAL Overall, all the SAls examined have become aware of the need to develop their work, not just in terms of methods, but also in terms of making their work more accessible and valuable to end users (see Pollitt and Summa 1997b). In doing this they have been responding to the changing environment in which they operate, which have varied in each of our countries. An integral aspect of these reforms has been the development of greater interest in producing performance information. Pollitt and Summa examined in detail the way in which various SAls reported on their own performance. They concluded that the 'degree to which each SAI has adopted a managerialist discourse mirrors the extent to which managerialism has penetrated the state apparatus generally in the country concerned. (p332)' Of our five countries, the UK has seen the most enthusiastic embrace of public management reform, and Sweden was also associated with considerable reforming zeal. 184 The NAO and RRV appear to have taken on board the culture of performance indicators, performance improvement and efficiency gains in their own work. The NAO's Annual Report, more than those of any other of our SAIs, 'displays a consistent concern with justifying the work of the NAO and attempting to identify and measure the benefits that flow from it. ' (Pollitt and Summa 1997b: 317). This includes explicit data on savings achieved from its recommendations, and the success in having recommendations accepted and acted upon. The implication of this is that the NAO has become increasingly concerned with ensuring that its examinations generate identifiable financial savings, and this has generated a culture in which methodological development is actively sought out. Similarly, the recent RRV Annual Reports have included far more performance information than previously, including quantified financial impacts from examinations. Linked to this are cultural issues around institutional self-confidence. Part 6 reported the views of a former civil servant-turned-NAO director, who interpreted the constant search for new methods by the NAO as a sign of a lack of corporate self-confidence. In a fast changing world, where there has been some suggestion that others (private sector accountants) could do the work, or others do it better (eg the Audit Commission), the greater interest in methodological change could be interpreted as a sign of uncertainty. Conversely, the less vigorous search for new methods, for example, seen at the Cour could be taken as a sign of an organisation more confident of its position (as one of the three grands corps, one of the most prestigious institutions in French administration) and its approaches, which have proved successful in the past, and which are not under challenge. Finally, a further possible explanation could link to the volume of audit output. This is based on the assumption that the greater the number of reports published, the more scope available to experiment with new methods. Figure 7.4 suggest there may be some link between a high volume of reports (column (g)) and a wider range of methods, with both the NAO and the RRV publishing in the region of 40-50 reports, more than the others. Conversely, the VTV published the fewest during the 1990s. Less evident is any relationship between the number of distinct VFM product types and the extent of methodological development. It might be expected that the broader the range of products, 185 the greater the range of methods that might be required. Barzelay (1997) credits the RRV with the largest range of product types, suggesting a possible relationship, but the NAO is considered to produce only two main types of report. Without wanting to be accused of special pleading, this figure may reflect the position in the mid-1990s, and may be out of date. Nevertheless, on the basis of the current evidence, the link is, at best, unclear. Overall conclusion overall, the international comparisons suggests the following factors have pushed the NAO's work slightly faster than in other SAIs. In particular: * the NAO appears to have had greater scope to experiment for the simple reason that it prepares more reports each year than the other SAls. This has given it more opportunities to try new methods; the NAO appears to have experienced stronger pressures than the other four to justify their own performance. This is, in part, a reflection of the fact that new public management thinking in the UK has been pushed furthest, creating an environment in which the NAO has been expected (or has interpreted the need) to justify its own performance. A key part of this has been thinking about methods. Linked with this is the fact that transparency in performance and data collection has been pushed furthest in the UK, so much so that the Treasury decided in early 2000 that they had established too many performance indicators; this is associated with the fact that the NAO is the only one of the five SAIs that attempts to quantify systematically the financial impact of its work. This is clearly the driving force behind many of the developments, including recent interest in benchmarking and operational research; 40 the NAO has more systematically exposed its work to outside scrutiny, which has become an integral part of its review and development process. No other SAI has systematically sought comments on, amongst other things, the appropriateness of the methods used on each study. The views of outsiders have helped to develop a climate in which methodological improvement is considered important; 186 e whilst there has been discussion in all countries about the right skills for VFM audit work, the NAO has most exposed its auditors to working with consultants and other experts, who have drawn attention to new ways of undertaking VFM audit; and although there has been an interest in developing VFM audit in several of the SAIs, there appears to have been greater dissatisfaction with existing methods at the NAO than elsewhere, and perhaps more of a sense of permanent change prevalent amongst those guiding the development of VFM work. 187 PART EIGHT: CONCLUSIONS - VFM METHODS AND THE NATURE OF VALUE FOR MONEY WORK This final part draws together material from the previous chapters as the basis for conclusions about the development of VFM methods and the nature of VFM work. The basic questions which framed the research were: " What changes have taken place in the selection of VFM methods in the NAO during the 1980s and 1990s ? " What are the factors that explain these changes ? " How do developments in the NAO's use of methods compare with those which have taken place within the VFM work of other SAls ? Several pieces of original work were undertaken to provide a multi-dimensional picture of developments. Part 4, which surnmarised the history of VFM work in the NAO, primarily from the 1950s onwards, provided the context against which changes in the use of methods can be understood. Part 5, based around a systematic examination of published audit reports, charted the development of VFM methods in the 1980s and 1990s, and offered a summary of the timing and extent of the changes. Part 6 complemented this by examining the factors which auditors believe influenced their choice of methods. In order to obtain additional insights into the factors which have fashioned the experience of the NAO, in Part 7, comparisons were drawn with developments at other SAls. What changes have taken place in the selection of VFM methods in the NAO during the 1980s and 1990s ?a Part 5 concluded that there had been a widening of the range of methods used in NAO reports since the 1980s, and improvements to the ways in which more 'traditional' audit methods were deployed. Many studies have continued to draw on secondary material, but new means of analysis have also been used. Examinations have been more rigorously planned, and have made use of a greater mix of methods. There has been considerably more consultation with users and providers of services, and with interested third parties. 188 Auditors have extended their work in certain directions, in particular, in the use of qualitative methods, looking more towards the softer social sciences than, for example, statistics. Despite these changes, VFM has remained strongly rooted in traditional methods and financial/resource concerns. The use of interviews and document analysis remain crucial to VFM work, although increased attention appears to have been paid to developing and refining these methods. These on-going shifts in the type of methods used by VFM auditors reflect a conscious attempt to extend the range of methods, particularly, in the second half of the 1990s. This interest is also demonstrated, for example, in the proliferation of internal technical guidance produced to encourage audit teams in their selection of methods, which increasingly, has encouraged the use of different approaches. What are the factors that explain these changes ? Part 6 examined the factors that auditors considered had influenced them when they were choosing methods. It argued that, although many VFM auditors were trained as accountants, they tended not to rely on this training when considering how to approach their examinations. As a result, many were open to other influences, making VFM work potentially quite responsive to outside influences. It suggested that the changing environment - political, administrative and intellectual - in which the NAO had operated had been the main driver encouraging the use of new methods. There had been growing expectations of the role of audit, and more emphasis within Government on achieving 'VFM' from public services. The NAO's interpretation of these changes, and the translation of thought into action had come, in particular, through the lead given by the C&AG, through the efforts of a small technical guidance team, and through individuals and their informal networks throughout the office. At senior management level, there had been a recognition that external developments could not be ignored, and that the NAO had a responsibility to develop and extend its VFM approaches to meet new demands and demonstrate that it could 'add value', at a time when all public bodies have been under pressure to do the same. At team level, the extent of change has been influenced by the attitudes and skills of individuals, which have, in part, been coloured and enriched by exposure to a growing range of external consultants and non-accountants. Another stimulant had been the external quality assurance process. 189 However, pressure has not been only one way, and the rather sporadic and evolutionary nature of developments, and the particular directions in which the NAO had moved, reflected the availability of particular types of staff and certain levels of resources for VFM work. Time constraints, for example, had encouraged widespread use of questionnaires, but discouraged longitudinal studies. Developments have also been shaped by widely held perceptions of what are appropriate means of gathering evidence, which are, in turn, influenced by accepted audit standards, drawn from a financial audit. Departmental distrust of certain methods, raised in discussions about study proposals, appeared also to have drawn certain boundaries. How do developments in the NAO's use of methods compare with those which have taken place within the VFM work of other SAIs ? Part 7 examined the developments in the methods used by four other European SAIs. It concluded that the range of methods used had remained relatively stable between the 1980s and 1990s, although some new methods, and improvements to existing methods had been introduced. Staff in those audit offices examined highlighted the greater attention given in recent years to how they planned their work and applied methods. As in the NAO, they tended to characterize themselves as pragmatic and concerned with using cost-effective ways of generating sufficient information for their needs. There are other parallels, with document examination and interviews, along with the analysis of secondary material, clear features of their work. It is also noticeable that that they have made much use of questionnaires, and that, like the NAO, all four have made only limited use of statistical techniques and economic models and concepts. Of the four SAls, the Swedish Riskrevisionsverket appears to have been the most methodologically adventurous. Despite the considerable common ground between the SAIs, the NAO appears to have pushed its methodological development ftu-thest. It appeared to have used a slightly wider range of methods, demonstrated greater interest in such developments (along with the RRV), and developed further its consultation with users and producers. It had taken further its quality assurance process, and made greater use of outside specialists and consultants to help enlarge the range of approaches used. It also tended to provide more 190 information about methodology in its reports. By way of explanation, Part 7 argued that the NAO had perhaps had greatest scope to experiment simply because it produced a larger number of reports than the other SAls. It also suggested that it may have experienced greater pressure to justify its own performance am had the others, which was a reflection of the greater prevalence of new public management thinking within the UK. Developing the methodological content of its reports (alongside, for example, enhancing their appearance) has been seen as a response to such pressures. Discussion Parts I and 4 located VFM audit within the array of evaluative activities that have developed in the last twenty years in the UK. They suggested that VFM was potentially a powerful tool, given the statutory powers to initiate studies, the strong powers of access to documents and people, and the robust mechanisms through which the results of examinations are considered. They also suggested that the final factor that would influence the strength of this work was the methods used. This section draws some overall conclusions about developments of methods for VFM, and considers the impact and implications of changes identified. Conclusions about developments in VFM methods at the NAO Developments have taken place in VFM methods in a number of ways. First, it seems clear that although changes in the range of methods used by VFM teams at the NAO during the 1980s and 1990s have been incremental, rather than revolutionary, there has been a distinct broadening of approach to VFM work. The 'toolbox' has acquired new tools, and older ones have been refashioned, or the instructions for use, revised. There is evidence of some quickening of the pace of change towards the end of the 1990s, although the overall picture is of a core of methods and approaches, supplemented from time to time with new means of obtaining and analysing evidence. In some cases, these have found favour and have been incorporated into the mainstream repertoire. Others have been tried on occasions but have failed to satisfy the needs of auditors. Nevertheless, as a basic conclusion, VFM work is a broader based activity at the end of the 1990s than it was in the 1980s. 191 Secondly, this broadening of the range of methods used in their work has increased the amount of original data generated by auditors. Rather than relying primarily on reviewing documents and reporting auditee views (as had often been the case in the past), auditors have increasingly been able to offer views independent of those they are examining. Collecting original data, undertaking their own site observations, or seeking the views of service users and third parties, all help to add to the sum of knowledge on a specific subject, and to generate new insights of greater value to the NAO's audiences. A recent NAO study on hospital-acquired infection (National Audit Office 2000b), for example, produced the first comprehensive database on hospital practices in this area. As well as forming the basis for the published report, the database was passed on to the NFIS Executive for further use and analysis. Something of this nature would not (and could not) have happened in the 1980s. Developments in methods have freed auditors far more from reliance on material generated by those whom they are seeking to assess. The widening of auditors' repertoire has, therefore, helped to enhance the independence and objectivity of VFM work. And, in cases such as the one above, it has also increased the value of NAO work. Thirdly, auditors have refined and extended existing ('traditional') audit methods, and increasingly combined them in a more planned manner. Whereas once auditors might have been satisfied with one or two approaches to generate material for a study, more audit reports are now based on multiple sources of evidence, increasing the likelihood of auditors getting behind the important issues, and producing valid conclusions and valuable recommendations. Thus, while some would argue that there is still some way to go, there is strong evidence that VFM has become a more robust and better planned activity, designed to explain more, rather than simply criticise. a A fourth point is that auditors have increasingly sought to gauge and reflect the views of service users, including the general public, by integrating methods such as customer surveys and focus groups into their repertoire. Increasingly, they have generated evidence on the impact that public service performance has on citizens, be they patients, benefit claimants, or users of utilities. Recommendations arising from such examinations feed directly into improving the quality of services provided. VFM work has moved beyond traditional audit concerns of cost, and examined on a regular basis, the quality of public 192 services. In doing so, it has sought to offer more of a customer focus, which is strongly in line with wider developments of the past decade or more in the UK. Linked to this is the fifth point, that auditors have consulted more widely with interest groups and outside experts as a routine part of their work. There is more evidence that audit teams have drawn on, or are aware of, published literature, ensuring that they have tapped into a wealth of external knowledge. This has enhanced the credibility of their work amongst a broader range of specialists, and, because many such people participate as reference partners, has ensured that their recommendations and conclusions have been scrutinised more widely before they are reported to Parliament. In turn, this has increased the auditor's own accountability, as they have had to justify their approaches, conclusions and recommendations in greater details than ever before. VFM teams have engaged more with specialists and practitioners working in the same field, and have drawn more on their knowledge. Whilst audit and, for example, academic research remain some way apart, considerable bridge-building has taken place. A sixth point is that the aspirations of many of those undertaking VFM work, and certainly those responsible for overseeing its development, have grown. At all levels within the NAO there are now greater expectations of the work, and this is, in part, related to growing awareness of what methods are potentially available. It is also reflected in more ambitious internal guidance, and specific encouragement to try to identify subjects and methods which are most likely to lead to high impact reports, preferably with identifiable financial savings. It can also be seen in the use of outside specialists to investigate and help develop VFM work, and the high quality of experts brought in to contribute as consultants or reference partners. The ambitions of many involved with NAO's VFM work are, therefore, considerably greater than they were in the 1980s. Finally, the evidence in this thesis suggests that the developments in methods have been influenced by the position and status of the NAO, a point highlighted in the review of existing literature for Part 2. In particular, it would appear that the statutory process within which it operates (providing published reports for Parliament), the specific conventions that have grown up around the work (eg the clearance of reports), the need to 193 obtain a large measure of co-operation from those being examined, and the expectations of key audiences, in particular, members of the PAC, about the robustness of the approaches used, all contribute to constraining the scope for experimentation. At the same time, however, the official nature of the NAO has made it highly conscious as an organisation of the expectations that are placed on it to lead by example, and demonstrate its own value for money. Awareness of the concerns within government about an 'audit society', and a desire not to be branded as an unhelpful influence, has sharpened the need to develop the methodological side of its work. The nature of VFM work Attention is now paid to the wider issue of the nature of VFM. VFM work can be considered in a number of ways. On the basis of this work, it is primarily a summative activity, seeking to make authoritative judgements based on the collection of evidence. To a small extent, there have been some signs of more formative designs, with some examples of bringing together participants to discuss the subject matter, and to develop ideas for improvements and change. But the focus on assessment against targets, and the collection of quantitative data using objective and disciplined procedures, emphasises that auditors are seeking to make judgements by gathering together sufficient authoritative evidence from 'out there'. The perspective has been largely positivist, trusting in scientific methods, rather than taking a more interpretative approach, designed to understand why people have acted as they have. And it is primarily about control and accountability, although again, there have been some shifts towards a more developmental approach, with greater emphasis given to identifying good practice, rather than simply reporting weakness and failure. Finally, in Part 5 'he saw that VFM had grown increasingly to draw on qualitative methods, rather than simply relying on quantitative approaches drawn from accounting. VFM has thus clearly not been immune to some of the trends in evaluation noted in Part 2, even if it has not embraced them fully. For all that, however, much has not changed significantly. First, despite these developments, at the start of the 2l't century, VFM remains an evaluative activity with an overriding focus on resource use (such as economy, efficiency and effectiveness, and the 194 quality of publicly funded services), and those factors that influence resource use (such as the adequacy of management systems, and the existence and effectiveness of processes for ensuring propriety). In addition, the perspective remains very much directed at the past - at what use has been made of resources - even though there is now far more attention paid to recommendations for the future, and in highlighting good practices. Secondly, whilst VFM work has grown apart from financial audit in several, very practical, ways, it still remains influenced by it. During the 1990s the two elements of NAO work have increasingly acquired separate identifies. They have been organised in separate areas and, with the exception of one or two specific posts, it is only at the most senior levels of the office where staff oversee both types of work. In addition, the disciplines look towards different intellectual and professional sources for ideas and influences. Yet, despite this, financial audit continues to influence VFM work. This is perhaps inevitable given that VFM work is undertaken within an organisation that audits accounts, as just one element of overall responsibilities. The need to fulfil statutory requirements to audit accounts helps set restrictions on the proportion of the overall NAO budget that can be devoted to VFM work, and hence average study costs. At a high level, this has meant a limit to the number of studies that can be published each year. And within certain specific fields, the need to prepare detailed reports on particular (qualified) accounts has meant that certain organisations have experienced little VFM work. For example, there have been no VFM examinations of the Child Support Agency, but a report on its qualified accounts appears annually which covers the major issues around its performance. Thirdly, despite a broadening of the influences, VFM work remains constrained by assumptions about what is acceptable evidence, derived very much from a financial audit tradition. Whilst NAO teams demonstrate a willingness to experiment at times, they tend to approach new methods with circumspection. Auditors have primarily drawn on only two of the three central areas of research techniques (Punch 1994) - interviewing and documentary examination - leaving observation largely unexplored as a basis for evidence gathering. As Keen notes, there has been a tendency for audit to gravitate towards methods that are acceptable to be both auditor and auditee, and this can be 195 explained principally in terms of the demands of the often confrontational nature of the report clearance process. This suggests that there are some methods and approaches to VFM that are not acceptable as means of collecting and analysing evidence. Given the nature of the work - prepared for Parliament and the basis on which senior civil servants are held accountable publicly for their performance - there seem to be limits to the scope for experimentation. Fourthly, despite the refinements in the way in which VFM is undertaken, some have commented on the 'essentially atheoretical' nature of VFM, noting that 'different findings were not fitted into an explicit theoretical framework, and questions of causation were not considered' (Keen 1999a). Some of the practitioners interviewed argued strongly that their work was 'practical' (considered the opposite of theoretical) and not the same as academic research. Others felt criticism of VFM for lacking a clear theoretical framework was unfair, and considered that many more recent studies had theoretical or conceptual frameworks, even if they were not necessarily on display. The debate remains unresolved. Fifthly, whatever the rhetoric about the choice of methods being key to the independence of the auditor, the evidence suggest that VFM is, to a significant extent, a negotiated process, in which audit teams must take account of a number of views and influences, both internal and external, in deciding how they will approach a particular subject. There may even be an element of compromise in the choice made, with the views of other parties, including those they will examine, and resource constraints (particularly, time) playing a big part in what is selected. It is clear, therefore, that VFM auditors cannot always do exactly what they want, notwithstanding the statements made in the professional guidance (INTOSAI 1992). Finally, whilst there have been moves to take on a more management consultancy or research approach (the shifts within the forcefield in Figure 6.4), the heavy reliance on documents and official material, emphasises that VFM remains strongly linked to accountability and guardianship. The presence of PAC at the end of the VFM process ensures that accountability will always be foremost in the minds of participants in the process. The concerns of this committee about standards in the conduct of public 196 business expressed in 1994 (referred to in Part 1), has continued to influence how they view organisational performance, in particular, where problems have arisen. This suggests there may always remain tensions between the NAO's primary role of public accountant (providing information to Parliament), and the more management consultancy approach (offering advice and guidance to departments) encouraged by some inside and outside the office, that highlights good practice. The impact of developments in methods Against this background, it remains to assess briefly what impact these developments have had, and what are the implications for the future of VFM work. Consideration has been given (Lonsdale in Pollitt et al 1999) to the issue of the 'impact' of VFM work. This acknowledged the difficulties of disentangling the effect of specific audit work from other changes. It also suggested that the impact of a particular report was contingent on a range of factors, including the kind of questions that the auditors asked, and the type of methods used. The impact was dependent, too, on factors such as the response of those audited, or the timing of the report (eg whether it appeared when the subject was topical or legislative change possible). Attention was also drawn to the potentially wide range of impacts that could stem from a VFM report. These were likened to a 'ripple effect' on a pond, with the intensity of the impact strongest at the point of contact - the audited body where the examination took place (where changes might be made to specific weaknesses identified) - but becoming weaker as the waves spread out towards other bodies undertaking similar work (eg schools or health authorities other than those visited), or having similar features, but outside the same sector (eg other types of organisations with similar personnel functions), and finally, towards public bodies as a whole (eg in influencing what is acceptable behaviour for public servants). The efforts made by SAIs to identify ways of measuring and reporting their impact have been examined (Pollitt and Summa 1997b, Lonsdale in Pollitt et al 1999). In recent years, the NAO has reported regular increases in the number of recommendations and amount of financial savings arising from its work (see National Audit Office 1999b). It has also been able to report regularly that more than 90 per cent of recommendations from the 197 PAC's reports have been accepted by Government. Leaving aside reservations about the appropriateness of such measures (which are not subject to external audit review, but are agreed with departments, who have little to gain by exaggerating the impact of their auditors), as well as the possibility that in many cases the recommendations may to some degree reinforce developments that would have taken place anyway, and acknowledging that the identification of such impacts may well have been more systematically pursued in recent years, the data seem to suggest increased impact in the late 1990s, than towards the earlier part of the decade. The question then is to what extent can changes in methods be held responsible for increased impact. Disentangling the effect of changes in methods from all the other developments described in Part 4 is difficult, but on the basis of this research, it seems reasonable to conclude that the changes in methods outlined in Part 5 have contributed significantly to an overall enhancement of the impact of VFM work. In particular, it seems that developments in methods have brought new perspectives to reports (eg from users of public services), given greater rigour to VFM work (eg improvements to questionnaire design), and offered new ways of examining performance (eg benchmarking). The NAO clearly considers there is a direct link between particular methods and achieving and identifying financial savings more easily, highlighting, for example, operational research techniques as a key way forward in this respect (National Audit Office 1998c: 2). As a result, greater emphasis has been place on considering potential impact at the start of the work. implications for the future of VFM This examination concludes by considering the implications of this work for the future development of VFM work at the NAO. Some observers (eg Bowerman 1996, Pollitt in Pollitt et al 1999) have suggested that VFM audit is at a crossroads, and that SAIs must make choices about how they wish to conduct their work in future. Official sources too have suggested that audit should adapt in certain ways if it is to provide value (Cabinet Office 1999,2000). In February 2000, the Government announced a review of central government audit arrangements for the 21" century, which may have implications for the way in which VFM is undertaken, even if it does not examine VFM work specifically. At 198 the same time, the newly appointed Auditor General for Scotland has suggested that VFM work will be undertaken differently for the Scottish Parliament (in particular, it will take a 'more strategic approach', with greater 'emphasis on supporting improvement) (Black 2000), a fact that may have a knock-on effect on activities at Westminster. And unfavourable comparisons have been drawn by some commentators about the overall approach of the NAO compared to the Audit Commission (see Travers 1999). All these developments suggest that VFM work may not stand still (or be allowed to). More generally, the Labour Government has shown itself to be interested in measuring and justifying public sector performance, and can be expected to push forward new ways of evaluation. Such developments in other aspects of the evaluation of government performance suggest that comparisons will continue to be drawn between VFM work and other modes of evaluation. The interest in peer review in Whitehall, for example, (Cabinet Office 1999), and the formative approach to evaluation of Best Value plans (Martin and Sanderson 1999) suggest a government more inclined to co-operative means of Performance appraisal, at variance to the highly public, and often confrontational, VFM approach. Such developments may add to the pressure on the NAO. Whatever the choices, there are a number of lessons from this examination of the developments in VFM methods. e First, it is clear that many of the changes and enhancements of methods have been associated with the requirement to examine major government policy initiatives or new approaches to public administration. Major changes in the way in which public services are provided are certain to continue in the future (see Cabinet Office 1999), and it is likely (if not ineAtable) that VFM work and the methods used to execute it, will need to continue to develop in tandem with these changes. Secondly, the apparent willingness of the NAO to consider new methods bodes well for its ability to respond successfully in this fast changing administrative environment. Having adapted VFM work to the demands of examining quality of services, privatisations and PFI contracts, it seems more than likely that the NAO will be able to adapt it to other initiatives. As this work was being 199 completed, the NAO announced that it would begin a series of multi- departmental studies, aimed at highlighting best practice across Government. Some have suggested that the world is changing and the need for VFM auditing is in dispute (Bowerman 1996). But there is plenty of evidence that those undertaking VFM work understand the need to adapt to these changes. Thirdly, the developments in the work of other evaluative bodies discussed above, which work more closely with the executive than the NAO, suggest that it may be unwise for the NAO to steer a course too far away from their more formative and participative approaches to performance improvement. Such approaches may need to form a larger part of the repertoire of the NAO. This implies a need to develop ways of overcoming suspicion of the VFM process amongst government departments, and of pulling those subject to examinations more into the process of evaluation, whilst safeguarding the auditor's independence. Fourthly, the development of more sophisticated ways of approaching their tasks, combined with its powerful rights of access to information, the ability to initiate examinations, and the existence of formal procedures through which its voice can be heard, suggest that VFM audit could continue to be a key means through which the executive is held to account by the legislature in the UK in the twenty-first century. However, this unique combination - official status and statutory powers of access, technical skills, and a process for being heard - could make it a target for the executive. Arguments over access to new types of 6arms length' body and private contractors providing. public services, have continued throughout the 1990s, as the forms of government have changed, and more business is done by outside bodies (Committee of Public Accounts 2000a and 2000b). It would be ironic if VFM work grew in strength and rigour, just as the extent to which it could be applied was diminished by an inability to examine much of the new mainstream of government business for lack of powers of access. 200 Fifthly, the NAO's increasing interest in ensuring that its work is seen to be of value suggests that it will continue to move towards using methods that allow it to identify financial savings and specific improvements in the delivery of public services. Such a trend is already under way, with the growing interest in benchmarking, and efforts to encourage staff to look closely at the scope for using operational research techniques in examinations. This has been justified specifically on the grounds that it should help to apply logic, common sense and rigour to problems in ways that should help identify potential financial savings. The pressures to demonstrate 'added value' will grow as there is greater understanding of 'compliance costs', which to date, few oversight bodies or those subject to their work, have sought to calculate (Hood et al 1999). A sixth point is that the growing range of methods employed and the need to continue to develop the VFM audit discipline raises interesting issues about the types of qualification that staff working in this area hold, the types of training they receive, and their competencies. Interviews at several SAIs indicated that senior staff had questioned whether the balance of skills and competencies that staff had were right for more difficult or innovative studies. Changes have already been seen, where they have followed a deliberate policy of recruiting staff with previous experience, and where a range of specialists from disciplines such as OR, statistics, evaluation and have been added to the audit and accounting skills of the majority of staff. The question of whether it is reasonable to ask so much of individuals has been raised elsewhere (Pollitt et al 1999). This is linked with the question of whether it is sensible for an organisation to try to master so many skills. SAls may need to decide whether to focus on using a relatively small number of methods, or whether they wish to attempt to gather together staff from a very wide range of disciplines. Finally, the use of more evaluative methods raises interesting questions about the aims and objectives of VFM audit, in particular, the extent to which it is about holding bodies to account, or about encouraging and assisting change. In 201 any final assessment, the question remains how far, as currently constituted, staffed and positioned, SAls are likely to be able to effect change and improvements within government. They clearly can act as a catalyst for change by highlighting poor performance and providing information that is a starting point for consideration of change. But many observers see them constrained by their status and constitutional positions, by their lack of executive power, and by expectations of how they should conduct themselves. In the end, the question remains whether and how its role in publicly exposing failure can ever be reconciled with a more collaborative approach to performance improvement. This in turn raises issues about the relationship between the NAO and the Committee of Public Accounts that is at the heart of dilemma for the NAO about its role and objectives. a 202 NOTES Central government audit arrangements in the United Kingdom changed in 2000. In February 2000 an Auditor General was appointed for Scotland, working to the Scottish Parliament. There is also an Auditor General for Wales, reporting to the Welsh Assembly. Currently, this post is held on a two-year contract by Sir John Bourn, who remained as Comptroller and Auditor General, reporting to the Westminster Parliament. 2. 'Performance audit' is the common term for this kind of work throughout much of the world. However, the NAO refers to 'Value for Money' or VFM examinations and for the purposes of this thesis this term is used. 3. INTOSAI is described as the professional organisation of supreme audit institutions in countries that belong to the United Nations or its specialised agencies. INTOSAI supports its members by providing opportunities to share information and experiences about auditing and evaluation. It was founded in 1953 and now includes over 170 SAIs. INTOSAI has developed auditing standards to provide a framework for the establishment of procedures and practices to be followed in the conduct of audits throughout the world. 4. The main purpose of financial audit in the UK is to form an independent opinion on the financial statements produced by an organisation. During the audit, the auditor will perform a variety of procedures, depending on the judgement of the audit staff and the nature of the account, to enable sufficient evidence to be gathered on which to base an audit opinion. 5. The main pieces of legislation relating to the National Audit Office are the Exchequer and Audit Departments Acts of 1866 and 1921, and the National Audit Act 1983. The first two have provided the basis for financial audit. At the time of writing the Government Resources and Accounts Bill was being considered in Parliament. This provides for the replacement of the old style departmental appropriation accounts (prepared on a cash basis) with resource accounts (on more commercial style accruals basis). 6. Supreme (or state) Audit Institutions (SAls) is the collective name for national audit bodies throughout the world. 7. The Committee of Public Accounts of the House of Commons was established in 1861. It is one of the senior committees of the House and meets twice a week when Parliament is sitting. By convention it is chaired by an Opposition MP. It consists of 15 members, with the balance of the parties on the Committee reflecting the balance in the House of Commons. Hearings of the Committee usually focus on a single report by the Comptroller and Auditor General, where the senior official(s) from the relevant Department are questioned on the evidence presented in the report. 8. The Audit Commission for Local Authorities and the National Health Service was established under the Local Government Act 1982. This brought local authority auditing in England and Wales under the control of a single body. In 1990 the Audit Commission's role was extended to include the National Health Service authorities, trusts and other bodies. Under the Local Government Act 1992, the Commission was given additional responsibilities in respect of the production of annual comparative indicators of local authority performance. Further legislation was passed in 1998 and 1999 regarding the Commission's performance reporting work. The Commission has four main functions. These are: to appoint auditors to all local government and NHS bodies in England and Wales; to set standards for those auditors through the Code of Audit Practice; to carry out national studies designed to promote economy, efficiency and effectiveness in the provision of local authority and NHS services; and to define comparative indicators of local authority performance that are published annually (www. audit-commission. gov. uk). 203 9. 'New public management' is the broad term for a wide range of reforms undertaken by governments in Western Europe, North America and Australasia during the 1980s and 1990s. It encompasses privatisation, marketisation, the introduction of quality systems and performance measurement regimes, as well as the restructuring of public administration, and the contracting out of services to the private sector. 10. The Comprehensive Spending Review, published in July 1998, announced that each Government Department would agree a Public Service Agreement. The Agreement would include each Department's objectives and measurable efficiency and effectiveness targets. Progress will be monitored by a continuous process of scrutiny and audit, overseen by a Cabinet committee. 11. The Public Audit Forum was established in 1998 by the four national audit agencies - the National Audit Office, the Northern Ireland Audit Office, the Audit Commission, and the Accounts Commission for Scotland. 'It was created to provide a focus for developmental thinking in relation to public audit. The main role of the Forum is consultative and advisory. Its remit includes providing a strategic focus on issues cutting across the work of the national audit offices, building on co-operation between them, establishing definitions of what constitutes public audit, and advising on the applications of standards and the practices of the auditors of bodies delivering public services' (www. Rublic-audit- forum. 90VAik). It was established after concern had been raised during the mid-1990s that audit bodies were not sufficiently co-ordinated. 12. The Exchequer and Audit Department was created by the Exchequer and Audit Departments Act 1866. It was headed by the Comptroller and Auditor General. The office was transformed into the National Audit Office in 1984. 13. The post of Comptroller and Auditor General was created by the Exchequer and Audit Departments Act 1866. It was an amalgamation of the functions of the Comptroller of the Exchequer, who had authorised the issue of public monies to departments since 1834, with those of the Commissioners of Audit, who had traditionally presented the government accounts to Treasury. The Comptroller and Auditor General was given the task of examining every appropriation account and reporting the results to Parliament. He was supported in the task by the Exchequer and Audit Department. The Comptroller and Auditor General is appointed by the monarch on an address presented by the House of Commons (National Audit Office 1996a) 14. The following are the main grades in the National Audit Office. New entrants start as Assistant Auditors. On entry they professional accountancy training. On passing examinations they become Auditors. On passing their final examinations and being considered to have sufficient experience, they are appointed as Senigr Auditors. Subsequent promotions are by promotion board. Usually, Principal Auditors lead the fieldwork on VFM studies. Audit Managers (equivalent to Grade 7 in the Civil Service) manage the day to day running of studies. Directors have overall responsibility for relations with individual departments and their related public bodies. The NAO has six Assistant Auditor Generals (equivalent to Grade 3s in the Civil Service). The Deputy Comptroller and Auditor General is the senior official within the NAO, the C&AG being an outside appointment. 15. The Public Accounts Commission JPAQ was established under the National Audit Act 1983. The Commission is a committee of MPs that considers the NAO's plans and budget proposals. The Commission reviews the budget and makes a recommendation to the House of Commons to accept it. 204 16. The Accounts Commission for Scotland was established in 1975. It examines the use made by Scotland's 32 local authorities, fire and police boards of L9 billion of public money. on I April 2000 Audit Scotland was established, comprising staff employed by the Accounts Commission and the National Audit Office in Edinburgh. All of Audit Scotland's work on local authorities will be carried out on behalf of the Accounts Commission. 17. The Comptroller and Auditor Generals holding office during the post-war period were: Sir Gilbert Upcott (1931-1946), Sir Frank Tribe (1946-1958), Sir Edmund Compton (1958- 1966), Sir Bruce Fraser (1966-1971), Sir David Pitblado (1971-1976), Sir Douglas Henley (1976-1981), Sir Gordon Downey (1981-1987) and Sir John Bourn (1987-). 18. The post of Chief Auditor has since been replaced by Audit Manager, and is the equivalent of Grade 7 in the Civil Service. 19. A description of the purpose of a Reference Sheet is given in Committee of Public Accounts 1979 p xiv. This states 'Major matters arising from E&AD's audit are taken up by Reference Sheet summarising the relevant facts and posing a number of questions relating to them. They are normally addressed to the Accounting Off icer of the Government department concerned, usually the Permanent Secretary ... Reference Sheets and departments' replies form the basis of the C&AG's reports on the accounts .... Matters taken up by Reference Sheet but not subsequently reported on are usually further pursued in correspondence with departments. ' 20. The senior official in each central government department and other body is designated the Accounting Officer. The responsibilities of the Accounting Officer are laid out in a memorandum from HM Treasury. Accounting Officers are accountable to Parliament for the use of the funds provided to their department. Accounting Officers appear before the PAC to be questioned on the reports of the Comptroller and Auditor General. 21. A Treasury Minute is the formal Government response to the recommendations in the report from the Committee of Public Accounts. 22. The Private Finance Initiative is an example of public/private partnerships. It typically involves public sector clients specifying services which they wish to purchase and, through competition, selecting private sector suppliers to provide them. PFI deals usually place heavier responsibilities on the private sector than conventional public procurements. Associated with this is a heavier share of the risks of the project. PFI has been used, amongst other things, in hospital building, IT change projects and prison building (Committee of Public Accounts 23rd Report 1998-99 'Getting Better Value for Money from the Private Finance Initiative'). 23. The Chartered Institute of Public Finance and Accountancy (CIPFA) describes itself as 'the leading accountancy body for public services, whether in the public or private sectors. Many existing members occupy position of influence in Local Authorities, NHS Trusts, Central Government, Housing, Education, Water, Electricity and Gas companies. The Institute of Chartered Accountancy of England and Wales (ICAEW) is the institute for some 116,000 chartered accountants. 24. Not all NAO reports are considered in a hearing of the Committee of Public Accounts. The NAO recommend which reports the Committee might wish to take, although ultimately decisions are made by the Chairman of the Committee. Some reports may be deemed unsuitable if, for example, considerable progress has been made towards implementing any recommendations made in the report (in which case there is little scope for Committee to have any impact), or if the report is considered particularly technical. 205 25. The criteria were: Administrative and Managerial Context; Structure and Executive Summary; Graphics and Statistics; Scope; Methods; Conclusions and Recommendations; and Enhancing the NAO's reputation. 26. The 'Best Value' performance framework took effect from I April 2000. It requires all local authorities to publish annual best value performance plans and review all of their services every five years. In these plans, authorities must show how they are continuously improving their services. 27. The 'Osmotherly rules' govem the appearance of officials before select committees and the extent to which they are required to provide information. 28. The recent Chairmen of the Committee of Public Accounts have been Sir Robert Sheldon (1983-1997) and David Davis (1997-) 29. At the time this thesis was being completed a piece of legislation - the Government Resources and Accounts Bill - was progressing through Parliament. This provides the basis for the change from cash to accruals accounting in central government. The Bill amends and replaces much of the 1866 and 1921 Acts. 30. The General Accounting Office is the SAI of the United States. It describes itself as the investigative arm of Congress. 'The GAO's mission is to help the Congress oversee federal programs and operations to assure accountability to the American people. GAO's evaluators, auditors, lawyers, economists, public policy analysts, information technology specialists, and other multi-disciplinary professionals seek to enhance the economy, efficiency and effectiveness, and the credibility of the federal government both in fact and in the eyes of the American people. GAO accomplishes its mission through a variety of activities including financial audit, program reviews, investigations, legal support and pol icy/program analyses' (www. gao. gov/main). 31. The main elements of the impact measurement process at the NAO are: the number of recommendations made in reports; the proportion of recommendations in PAC reports accepted by government in the Treasury Minute; and the amount of savings arising from NAO recommendations (see Lonsdale in Pollitt et al 1999). 32. Sir John Bourn, the Comptroller and Auditor, General was appointed Comptroller and Auditor General in 1988. He was previously Deputy Under Secretary of State at the Ministry of Defence. Prior to that he had worked at the Treasury, the Northern Ireland Off ice and the Civil Service College. 33. There has been some criticism of the National Audit Office's VFM work. Bowerman 1994 criticises the lack of co-ordination between the NAO and the Audit Commission. Some (see Travers 1999) have suggested that the NAO's work is not as m6dem or well presented as that of the Audit Commission. Other critics such as Sir Peter Kemp (see Accountancy Age 16 October 1997) have suggested that the NAO's work and the PAC hearings discourage innovation and makes civil servants cautious and risk averse. 34. The NAO is divided into six Units. Unit A includes Corporate Policy, Personnel, and Finance; Unit B audits the Home Office, Department of the Environment, Transport and the Regions, the Ministry of Agriculture, Fisheries and Food, and various central departments; Unit C audits the Departments of Health, Social Security, and Education and Employment, as well as the Inland Revenue and Customs and Excise. Unit D is responsible for financial audit guidance and International work; Unit E audits the Ministry of Defence, and the Foreign Office, as well as private finance initiative deals; and Unit F undertakes audit work 206 on privatisation, regulation, and Modernising Government themes, as well as providing VFM guidance. 35. The audit arrangements in Sweden and Finland are unusual in that the SAI is situated within the executive and there is also a parliamentary audit body. In Sweden Riksdagens Revisorer (Parliamentary Auditors) is composed of members of the Riksdag, and is supported by an office of some 25 officials (see National Audit Office 1996a). They produce high level reports. 36. EUROSAI - the European Organisation of Supreme Audit Institutions - currently groups together 42 SAls on the European continent. It aims to promote professional co-operation and the exchange of information and documentation, study public audit, and ensure the unification of terminology in the field of public audit. It is one of the regional groups of INTOSAL 37. Meta-evaluation involves the examination of the quality and usefulness of evaluations. Meta-analysis draws on existing evaluations to summarise and synthesise the results of a great many studies. 207 APPENDIX 1: SUPREME AUDIT INSTITUTIONS -A SUMMARY OF THEIR STRUCTURES AND HISTORY' Valtiontaloudentarkastusvirasto (VTV) -Finland The VTV was established under the 1947 State Audit Act, although its first forerunner was established in 1824. It is 'an independent and neutral government agency responsible for ensuring the legality and soundness of the government's financial management and compliance with the national budget. ' The VTV examines the revenue and expenditure of the Council of State, government bodies (including ministries, state offices and agencies), state enterprises, and recipients of state subsidies such as local communities, public utilities, and joint stock companies in which the state holds a majority share. The VTV comes under the jurisdiction of the Ministry of Finance, but under the Finnish Constitution its work prograrnme is independent. It has access at all times to all documents it may require, and under the 1947 Act can fine anyone failing to comply with requests. The VTV is headed by an Auditor General, appointed by the President of the Republic on a proposal of the Council of State. The appointment is for life, but with a retirement age of 65. The Auditor General and three Audit Councillors form the Audit Council, which considers the office's financial plan and draft budget, audit planning instructions and the guidance issued to state bodies. The office is divided into four operational units, two responsible for performance audit, one for financial audit, and the other for internal services. The VTV has around 120 staff, the vast majority of whom work in Helsinki. All audit staff have university degrees, with postgraduate qualifications in accountancy and business common. a There is no direct official contact between the VTV and the Finnish Parliament; instead relations are conducted through the Parliamentary State Auditors, who are responsible for overseeing the finance of the state and supervising compliance with the budget on behalf of Parliament. The VTV is required by statute to provide the Parliamentary State ' This Appendix was published in 'Performance or Compliance ?: Performance Audit and Public Management in Five Countries' by Pollitt C, Girre X, Lonsdale J, Mul R, Summa H, Waerness M. It was prepared by the author. 208 Auditors with its annual plan of operations and submits all reports to them for information. Cour des Comptes - France Under the French Constitution, the Cours des Comptes is charged with assisting Parliament and Government in supervising the implementation of the state budget. It wa's established in its present form in 1807, although its predecessors date back to the fourteenth century. The Cour's ftinctions have been greatly increased in the course of the last fifty years, and its organisational structure, duties, and powers are now set out in an Act of June 1967. The audit of social security, public enterprises, and nationalised industries was added in 1976, when audit bodies previously responsible were merged with the Cour. The audit of local corporations, agencies, and companies has, since 1982, been devolved to regional chambres des comptes, established to carry out this role. The main task of the Cour, according to the Constitution, is to assist Parliament and Government in checking the proper execution of the State budget and the annual law on Social Security. However, it is independent of both institutions, and ranks, together with the Council of State and the Cour de cassation as one of the grands corps de Vita, the senior administrative bodies in France. The Cour audits the accounts of the State, the state autonomous agencies (itablissements publics nationaux) and of state-owned companies (enterprises publiques nationales). It may, at its discretion, audit the accounts of entities, public or private, which receive financial assistance from the state, or its agencies and companies. It also audits the accounts of the independent units that run the health insurance and pensions system (organismes de sicuritj sociale). The Cour is headed by the premier prisident, appointed by decree from the President of the Republic. It is divided into seven chambres, each headed by a President. Each chambre adopts an annual work programme on an entirely independent basis, and the Cour's overall work programme is drawn up from these proposals by the premier prisident after discussion with the other Presidents. In addition, the parquet gineral, headed by the procureur giniral, provides advice on legal matters and acts as intermediary between the Government, the Cour, and the judicial authorities. The Cour consists of some 500 people, 250 of whom are magistrates appointed for life by decree of 209 the President of the Republic. Auditors are post-graduates from the tcole Nationale d'Administration, often with qualifications in public law, finance and economics. Many spend a significant part of their career in other jobs and then return to the Cour later in life. The Cour's audit work involves establishing that the accounts present a true and fair view of all transactions, as well as that these transactions have been performed according to financial regulations and in the most effective way. The latter work, known as contr6le de Ja gestion (or performance audit), has steadily developed in recent years. Algemene Rekenkamer - The Netherlands The Algemene Rekenkamer is an independent body, which supports the work of the States General. It has a history dating back to 1814, but state audit in the Netherlands can be traced back to the fourteenth century. Its role was most recently formulated in the 1976 Budget and Accounting Act, and it is 'responsible for auditing state revenue and expenditure'. The Rekenkamer has no judicial powers and is not a court. Its budget is set following discussions with the Ministry of the Interior and the Ministry of Finance. The Rekenkamer's audit field is extensive, covering some 50,000 separate bodies. it can also examine public limited companies and private companies where share'capital is partly or wholly owned by the state, and corporate entities which have been given state loans or guarantees. The Rekenkamer reports to the States General. Any parliamentary committee can discuss its reports, but they are usually considered by the State Expenditure Committee, which has the authority to call ministers to account and seek further evidence from the Rckenkamer. The Rekenkamer undertakes both regularity and performance audits. Its regularity audit is systems based and relies heavily on the work of ministerial internal audit divisions. The Rekenkamer examines the divisions' audit programmes and the conclusions they have reached, and the Rekenkamer's staff undertake their own work around this. The Rekenkamer has a Board of three Members, including the President, each of whom is appointed (for life or until state retirement age of 70) by the Government from a selection made by the lower house of the State general. The composition of the Board is politically 210 balanced. The Rekenkarner has six audit divisions, divided into audit sections which are based in the government ministries. It has some 320 staff, of whom 200 are directly involved in audit. The majority of auditors are graduates in statistics, economics, and law. Riksrevisionsverket (RRV) - Sweden The RRV was established in 1961 when the National Accounts Board and the Government Audit Bureau were merged, and was reorganised most recently in 1992. It is the central administrative body for government accounting and auditing and is subordinate to the Ministry of Finance. It is headed by an Auditor General, appointed by the Government for a six-year term which can be renewed. The RRV has discretion as to its work programme, but undertakes many government-commissioned examinations each year. The RRV's audit mandate covers state agencies including their regional and local offices, public utilities, national subsidy transfers to households, businesses, and government bodies, and certain state owned companies. The RRV does not audit the Parliament and associated bodies, the Cabinet Office, or the ministries, which are examined by Parliament's own audit agency, the Riksdagens Revisorer. The RRV has access to all information it requires, and under Sweden's public access to information legislation, must allow access to its completed audit files. Until 1998 the RRV was divided into three departments - for Performance Audit, Financial Audit, and Financial Management and Accounting - each headed by an Assistant Auditor General. In July 1998 the Financial Management departments was removed from the RRV and established as a separate agency - the National Financial Management Authority. Until this change, the RRV had a staff M around 500, mostly based in Stockholm. All audit staff are graduates, predominantly in social sciences and economics. National Audit Office (NAO) - United Kingdom The NAO was established in 1983, taking over the responsibilities of the former Exchequer and Audit Department, created in 1866. It is headed by the Comptroller and Auditor General (C&AG), who is appointed by the monarch on an address by the House 211 of Commons, the motion for which is made by the Prime Minister. The C&AG is independent of government, and has full discretion as to his work programme. He reports to Parliament and is an officer of the House of Commons. Resources for the NAO are determined directly by Parliament. The audit function of the C&AG is primarily in two parts - the certification of some 650 accounts, and the examination of the value for money of specific programmes, projects and activities. He also has inspection rights to some 4,000 other bodies that receive public funds. The C&AG provides his audit opinion in a certificate published with each set of accounts. His reports are examined by a committee of the House of Common, the Public Accounts Committee. In addition to his audit role, the C&AG undertakes a smaller Comptroller function, which makes him responsible for giving authority for Treasury requests for funds. The NAO is divided into six units, five of which undertake both financial audit and value for money work, and the sixth which provides central services. Each is headed by an Assistant Auditor General. The NAO has a staff of 750, but in addition makes extensive use of contracted-in consultants and agency staff. Audit staff, who are mostly university graduates trained by the NAO as accountants, specialise in either financial audit or value for money work. 212 APPENDIX 2- THE DEVELOPMENT OF PERFORMANCE AUDIT' Finland The origins of state audit in Finland can be traced back to medieval times, although the first predecessor of the Valtiontalouden tarkatusvirasto (VTV), the General Revision Court, was established in 1824. This body's mandate covered the auditing of the accounts of all public bodies, including military operations. In the early twentieth century, the Senate's economic committee commented on the very limited nature of state audit, which paid attention only to formal aspects of the accounts, rather than whether audited authorities had acted as 'appropriately' and 'carefully' as could have been expected. The committee recommended that the Court be given an extended mandate, but there were no changes before Finland became an independent state. The 1919 Finnish Constitution provided for the examination of the appropriateness of financial management, as well as the legality of expenditure. Subsequently, the 1923 Audit Decree stated that the tasks of the State Revision Office (established under the Constitution) included investigating 'whether any wastefulness or unnecessary expenditure or expenses exist' (State Audit Office 1996: 2). In the 1940s, the requirements of wartime gave an additional impetus to these considerations. A separate unit was established to audit the accounts of state bodies, with the audits focused particularly on the 'appropriateness' of the use of public funds. After the war, the Act on the Auditing of Public Funds 1947 made audit of the appropriateness of public expenditure explicit in legislation, allowing the newly established audit office, the VTV to undertake what can, in a broad sense, be considered as performance audit. The focus of attention during this period was very much on the appropriateness of expenditure, rather than the performance of public bodies. Some years later, at the beginning of the 1970s, discussions began within the VTV, focusing on the idea of auditing activities of public bodies. In 1971 a government committee on state audit proposed that appropriateness audit be extended and redirected to look at well defined topics or projects. This work would focus on the planning and programming systems for the audited activities, their economy and efficiency, co-ordination with other activities, 2 This Appendix was published in 'Performance or Compliance ?: Pcrfonnance Audit and Public Management in Five Countries' by Pollitt C, Girre X, Lonsdale J, Mul R. Summa H. Wacmcss M. It was prepared by the author. 213 and the possibilities for development. In 1978 another committee, investigating the structure and development of central government, spoke out against 'mere numerical and legalistic' audit, advocating that the VTV examine the efficiency and effectiveness of administrative activities. By the mid-1970s, this new approach to audit had already been adopted by the VTV, at least in its explanatory rhetoric. During the 1970s and 1980s its work - focusing on how well activities were planned, how well they linked with broader policies, and on their efficiency, economy and profitability - was referred to as 'activity audit' and considered a more advanced form of audit, in contrast to more 'limited' and 'legalistic' financial audit. To begin with, however, it was not treated as a separate discipline, but in 1981 the distinction between 'activity audit' and financial audit was made for the first time in VTV documents, and activity audit was allocated resources separately. In the early 1980s a further government committee examined principles for the development of state audit. It took as its starting point that activity audit needed to be developed further, to be clearer methodologically and to receive increased resources. The committee's report offered conceptual refinements to the definition of 'appropriateness audits', 'efficiency audits', 'effectiveness audits' and the criteria to be used in them, and finally proposed that a greater proportion of the VTV's resources should be devoted to activity audit. Before 1987 the VTV's audits of ministries and agencies combined financial and value for money work, and all audit reports included value for money observations. The VTV was organised into units according to the branches of administration, and in planning their work no distinction was made between performance oriented and financial audits. After the reorganisation in 1988, performance audit in its current, separate format was established, and increased effort was put into developing its own profile, and methods and ways of working. During the 1990s there were a number of developments in the VTV's performance audit work. Examinations tended to focus on broader, cross sectoral topics rather than single organisations or sub-functions. There was a greater tendency towards openness in the VTV's relations with auditees and the general public and, after 1993, greater emphasis on the appearance of published reports. During the 1990s the VTV has produced around 214 twenty reports a year, and in 1995 it published, for the first time, a collection of summaries from all eighteen performance audit projects completed during the year, a practice now repeated annually. In 1996 the VTV underwent further organisational changes which split the Performance Audit unit into two. France Providing a description of the development of performance audit in the Cour des Comptes is made difficult by the fact that it is not treated as a separate discipline in the way that it is in the other SAIs examined in this book. The most important provision relating to the Cour des Comptes is the 22 June 1967 law on its role and organisation. Articles L111-3 and L111-4 provide for the Cour to examine aspects of economy, efficiency, and effectiveness. The former refers to the Cour considering 'the good use of public money', and the latter states that the Cour 'audits the accounts and the management of the public firms. ' This involves inquiring into the organisation of the audited body, the number, level and pay of its staff, its procurement policy, its equipment policy, its costs compared with its achievements, and the usefulness of its activities to the state or to the general public. Since 1991 the Cour has published a series of rapports publics particuliers. These address particular subjects, in many cases assessing the adequacy of management and making recommendations. Up to around 200 pages long, their size allows for significant discussion of the activities under examination. From around the same time, there has been increased interest within the Cour in evaluating public policies, and from 1992, the premier prisident has chaired the INTOSAI working group on evaluation. By the mid- 1990s, roughly two-thirds of the Cour's resources were devoted to the assessment of financial management and performance in the broadest sense (National Audit Office 1996a: 65). Since 1996 the Cour has, under a new article in the Constitution, produced 'le rapport sur la sicuriti sociale' following its examination of the execution of the annual law on the ways and means of social security. From 1997 the annual public report has included a section on the impact of previous reports. 215 The Netherlands The Algemene Rekenkamer was established in 1814, although the origins of state audit date from several centuries earlier (National Audit Office 1996a: 142). Under the first Budget and Accounting Act, which became law in 1927, the Rekenkamer had the task of auditing the efficiency and/or effectiveness of public policy and expenditure (Leeuw and Engwirda 1993/4: 16). After the Second World War, attempts were made to reduce public expenditure in The Netherlands by improving the performance of government activities. The Algemene Rekenkarner began to conduct the forerunners of performance audits, in particular, with emphasis on aspects of efficiency and the organisation of the public service, issues which were of most interest to the States General (Algemene Rekenkamer 1989: 25). In the mid- I 950s discussion arose in the States-General about the emphasis which should be placed by the Algemene Rekenkamer on performance audit, and the office was encouraged to carry out more audits of efficiency and effectiveness, examining the reasons behind government decisions and whether objectives had been achieved. It was also suggested that the audits should draw attention to undesirable side-effects, conflicts with other goals, or the lack of available information at the time of decision making (Algemene Rekenkamer 1989: 26-29). It was during the 1950s that performance audit assumed the status of an independent aspect of audit work within the Rekenkamer, and from the middle of the decade staff began to examine and report on policy goals. Despite this, changes were still under consideration. In 1956 the Minister of Finance appointed a committee to undertake a general review of the Budget and Accounting Act, with the aim oi providing a statutory basis for developments that had been going on since the 1920s, including the role of performance audit. However, a new Act was not passed for twenty years, when powers to conduct performance audits were incorporated into the 1976 Government Accounts Act. This stipulated that the Rekenkamer should pay attention to the 'efficiency and effectiveness of government management and of the organisation and performance of the public service. ' (Leeuw and Engwirda 1993/4: 17). In addition, the Rekenkamer was to present information in its Annual Report which might enable the Parliament to assess 216 government policy, and could be asked by Parliament to conduct specific performance audits (Algemene Rekenkarner 1989: 27-28). Despite the new legislation, in the early 1980s only a small proportion of the Algemene Rekenkamer's work could be classified as performance audit, and it was only in the following years that 'a conscious effort [was made] in favour of the selection, planning and implementation of performance audit on an increasingly systematic basis (Algemene Rekenkamer 1991: 35). As the number of examinations grew during the decade, the work was also developed in other ways. In particular, in 1985 the Rekenkamer reported on its first government-wide audit, looking at the organisation of accounting divisions across government ministries. These broad audits grew in number during the mid to late 1980s, looking, for example, at the control of the budgetary process, and at the efficiency and effectiveness of the management of 722 central government subsidies (Leeuw 1991: 26). In 1989 the remit of the Rekenkamer was extended to include inspection of welfare benefits and state aided bodies, permitting performance audit in these areas. Article 59 of the Governmental Accounts Act also enabled the Rekenkamer to examine both the regularity and the performance aspects of corporate entities performing a function regulated by statute, or where the State owned more than 5 per cent of issued capital, or which received a State grant, loan or subsidy. Under a ftirther amendment to the Budget and Accounting Act, made in 1992, the Rekenkamer's role was specified in more detail as 'auditing the effectiveness and efficiency of management, which includes all possible aspects of the internal management at departments of the public service; auditing the effectiveness and efficiency of the organization; and auditing the effectiveness and efficiency of the policy implemented' (Leeuw and Engwirda 1993/4: 17). Greater consideration was given subsequently to the focus of the Rekeilamer's work, and in 1995 the Board announced an office goal of producing studies examining the effectiveness of government activities (personal interviews). The Rekenkamer's reports developed significantly during the late 1980s and 1990s. Traditionally, it produced a single annual report, and occasionally separate 'interim' reports. In the late 1980s the number of interim reports grew, and between 1990 and 1995 compendiums of perforniance audit reports were published each June and December. In 217 1996 the Rekenkarner broke with its tradition of issuing one large annual report containing details of all its audit activities, and now publishes an annual regularity report relating to all departments, and about 30 separate reports, one for each audit (Witteveen 1996: 28). Sweden An important forerunner of performance audit in Sweden was the work of the State Sub ect Auditors, whose role developed during the Second World War as a response to the need to control the activities of government agencies and ensure that they were sound, appropriate and economical. The Riksrevisionsverket (RRV) was established in 1961, on the merger of the National Accounts Board and the State Subject Auditors. Its development in the 1960s was closely related to the development of the Programme Budgeting System (which grew up between 1965 and 1975), as well as the modernisation of the budget process and long term planning within state agencies (Swedish National Audit Bureau 1993: 36). The growing interest in performance audit was one of the main arguments for establishing the RRV. Performance auditing began to develop after the RRV was restructured in 1967, with the aim of giving it 'a strategic policy role as a force for renewal in government administration' (Swedish National Audit Bureau 1993: 36). Financial audit was taken out of the RRV's remit and decentmlised to the agencies, so that the main activities of the RRV were performance audit and supporting and developing financial management. The RRV's role was designed as a part of the overall state financial administration system, comprising planning, budgeting, accounting, and auditing. The RRV's task was to examine how agencies reached the goals set by central government, with recommendations intended mainly to make agencies aware of the possibilities of more efficient resource utilization, made possible by better planning and follow-up. The aim was for the RRV to undertake regular audits (every five years was the original intention, but this was never implemented) of agency financial administration, objectives, plans and outcomes, identify weaknesses and make recommendations. These recommendations were usually submitted to the agency itself (Swedish National Audit Bureau 1993: 36- 37). 218 In the 1970s it became clear from public debate that there was a need for broader and deeper analysis in performance audit, to look at the conditions, role, and functions of agencies in the wider context. Many examinations had, in practice, already gone beyond internal questions and perspectives and other audit approaches and methods were developed. By the end of the 1970s agency audits covered a much wider context, and this led to more extensive analyses and the need for new competences. The link to programme budgeting weakened, and perspectives from other disciplines, such as the political sciences and sociology, were introduced. Increasingly, audit reports were submitted to the Government, in line with the view that problems could only be dealt with at central government level. During the 1980s performance audit developed further as the pressure for change in the public sector grew. In 1983 the State Agency Management Commission was set up to look into the problems of the public sector, which was widely considered too complex and ineffective. Parts of the Commission's examination covered the role of RRVs performance audit work, and supported the RRV's new type of broad 'systems orientated' audits which assisted government decision making. There was a growing expectation that more effort be put into analysing possible alternatives approaches and their consequences, and making more explicit recommendations. The overall expectation was that performance auditors would play a more active part in the reform of the state administration. In 1987 financial audit, which had been outside its remit for twenty years, was returned to the RRV, and for a while grew in strength with the development of the results based budgeting and full accruals accounting. In 1989 the RRV's performance audit mandate was extended to include the auditing of state-owned companies and state maintained foundations. At the same time, a single Performance Audit Division was founded (to replace the three existing ones) to help consolidate and co-ordinate audit activities. Performance auditors tried to take a more active role in the work of bringing about change in government administration and examining those problems considered urgent. Two years later the Swedish name of the performance audit activity was changed from (directly translated) 'management audit' to (directly translated) "effectiveness audit'. 219 Since 1993, performance auditors have also participated in the annual auditing of agencies' performance reports. In 1994 the Government gave the RRV the task of identifying the scale of abuse of social benefits and allowances, and subsequently, it has reported on early retirement pensions, unemployment insurance funds/unemployment cash allowances, housing benefit, sickness benefit and parental allowances (Sandberg and Ivarsson 1996: 29). a 220 BIBLIOGRAPHY Anand P (1988) 'Monitoring and auditing VFM in the UK: The scope for quantitative analysis' in Financial Management andAccountability 4 (4) Winter pp253-270 Ahlback S (1999) Att kontrollera staten. Den stalliga revisionens roll I den parlamentariska demokratin (Monitoring Government: the Role of the State Audit in a Parliamentary Democracy), University of Uppsala, Uppsala Algemene Rekenkamer (1989) Algemene Rekenkamer. - past, present andfuture The Hague: ' Algemene Rekenkamer Algemene Rekenkamer (199 1) Vigilent and resolute: the challenges facing the Netherlands Court ofA udit The Hague: Algemene Rekenkamer Algemene Rekenkamer (1996) Manual of performance audit The Hague http: //www. rekenkamer. nl Barzelay M (1996) 'VFM auditing and the new public management: changing roles and strategies of central audit institutions' in OECD Performance audit and the modernisation of government Paris: PUMA/OECD pp 15-56 Barzelay M (1997) 'Central audit institutions and performance auditing: a comparative analysis of organizational strategies in the OECD' in Governance: An International Journal of Policy and Administration Vol 10, No3 July pp 235-260 Black R (2000)'An end to the blame culture' in Public Finance May 5-11 Boekenoogen M (1998) 'The granting of licences by central government' Paper presented to International Value for Money methods seminar (National Audit Office 29-30 September) Bourn J (1994) 'National Audit Office: Force for Change' in Flynn N (ed) Change in the Civil Service London: Public Finance Foundation pp 69-74 Bowerman M (1994) 'The NAO and the Audit Commission: co-operation in areas where their VFM responsibilities interface' in Financial Accountability andManagement, February pp47-63 Bowerman M (1995) 'Auditing performance indicators: the role of the Audit Commission in the Citizen's Charter Initiative' in Financial Accountability andManagement 11(2), May ppl 71-183 Bowerman M (1996) 'The rise and fall of VFM auditing' in Lapsley I and Mitchell F (eds) Accounting and Performance Measurement: Issues in the Private and Public Sectors London: Paul Chapman Publishing pp 193-212 Broadbent J and Laughlin R (1997) 'Evaluating the 'new public management' reforms in the UK: a contitutional possibility ?' in Public Administration Vol 75 Autumn Burkitt-Gray A (1999) 'The scourge of the perm secs' in Government Computing September 221 Butter P (1995) Review of the Audit Commission (conducted by PJ Butler assisted by Bain & Company) London Centre for the Evaluation of Public Policy and Practice (1999) The OFSTED system of school inspection: an independent evaluation Uxbridge: CEPPP Chapman L (1978) Your Disobedient Servant. ý The Continuing Story of Whitehall's Overspending Harmondsworth: Penguin Cabinet Off ice (1999) Modernising government London: HMSO Cabinet Office (2000) Wiring it Up - Whitehall's Management of Cross-Cutting Policies and Services London: Performance and Innovation Unit Chancellor of the Exchequer (1980) The Role of the Comptroller and Auditor General London: HMSO Chelimsky E (1985) 'Comparing and contrasting auditing and evaluation: some notes on their relationship' in Evaluation Review Vol 9 No 4, August pp 483-503 Chelimsky E (1996) 'Auditing and evaluation: whither the relationship T in Wisler C (ed) Evaluation and Auditing: Prospects for Convergence, New Directions in Evaluation, Number 7 1, Fall, Jossey-Bass Child J (1984) Organization: A Guide to Problems and Practice (2"d edition) London: Harper & Row Clark D (1996) 'Open government in Britain: discourse and practice' in Public Money and Management January-March Committee of Public Accounts (1979) The work of the Committee of Public Accounts and the status andjunctions ofthe Comptroller andAuditor General London: HMSO Committee of Public Accounts (1981) The Role of the Comptroller and Auditor General London: HMSO Committee of Public Accounts (1983) Supply Estimates 1984-85 for Exchequer and Audit Department and National Audit Office London: HMSO a Committee of Public Accounts (1984) National Audit Offices Estimates for 1985-86 London: HMSO Committee of Public Accounts (1985) NationalAudit Office Estimates 1986-87 London: HMSO Committee of Public Accounts (1986a) NationalAudit Office Estimates 1987-88 and Corporate Plan 1987-88 to 1991-92 London: HMSO Committee of Public Accounts (1986b) Redundancy Compensation Payments to University Staff London: HMSO 222 Committee of Public Accounts (1987) NationalAudit Office Estimates 1988-89 and Corporate plan 1988-89 to 1992-93 London: HMSO Committee of Public Accounts (1988) NationalAudit Office Estimates 1989-90 and Corporate Plan 1989-90 to 1993-94 London: HMSO Committee of Public Accounts (1990) National Audit Office Estimates 1990-91 and Corporate Plan 1990-91 to 1994-95 London: HMSO Committee of Public Accounts (1991) NationalAudit Office Corporate Plan 1991-92 to 1995- 96 and Estimatesfor 1991-92 London: HMSO Committee of Public Accounts (1992) NationalAudit Office Estimates 1992-93 London: HMSO Committee of Public Accounts (1993) National Audit Office Estimate 1993-94 London: HMSO Committee of Public Accounts (1994a) NationalAudit Office Estimate 1994-95 London: HMSO Committee of Public Accounts (I 994b) The Proper Conduct ofPublic Business London: HMSO Committee of Public Accounts (1995) National Audit Office Estimate 1995-96 London: HMSO Committee of Public Accounts (1996) NationalAudit Office Estimate 1996-97 London: HMSO Committee of Public Accounts (1997) NationalAudit Office Estimate 1997-98 London: HMSO Committee of Public Accounts (2000a) Government Resources and Accounts Bill London: HMSO (4 1h Report 1999-2000) Committee of Public Accounts (2000b) Government Resources and Accounts Bill London: HMSO (91h Report 1999-2000) Creswell J (1994) Research Design: Qualitative and Quantitative Approaches Thousand Oaks: Sage Creswell J (1998) Qualitative Inquiry and Research Design - Choosing Among Five Traditions Thousand Oaks: Sage Cubbon B (1992) Examination ofthe Impact ofthe NAO and PACprocess on valueformoney in Government Departments: Report by Price Waterhouse in association with Sir Brian Cubbon GCB (unpublished report) 3 Day P and Klein R (1990) Inspecting the Inspectorates York: Joseph Rowntree Memorial Trust Derlien H-U (1990) 'Genesis and structure of evaluation efforts in comparative perspective' in Rist R (ed) Program Evaluation and the Management of Government New Brunswick NJ: Transaction Publishers Dewar D (1985) 'Value for money audit: the first 800 years' in Public Finance and Accountancy, August 30 pp 10- 12 223 Dewar D (1991) 'Current issues in auditing: the audit of central government' in Friedberg A et al (eds) State Audit and Accountability: A book of readings Jerusalem: State of Israel State Comptroller's Office pp 196-208 Dewar D (unpublished) draft chapter of an untitled study of the history of the NAO entitled 'A Sense of History' Dewar D (1997) 'The audit of central government' in Sherer M and Turley S (eds) Current Issues in Auditing London: Paul Chapman Publishing Dingwall R (1997) 'Accounts, interviews and observations' in Miller G and Dingwall R (eds) Context and Method in Qualitative Research London: Sage Drewry G (1989) The New Select Committees 2 nd edition Oxford: Clarendon Press Dunleavy P (199 1) Democracy, Bureaucracy and Public Choice, Harvester Wheatsheaf, Hemel Hempstead pp 174-209 Exchequer and Audit Department (1978) Management Review of the Exchequer and Audit Department: Report of the Steering Committee unpublished report London Exchequer and Audit Department (198 1) Review of Valuefor Money Audit London: Exchequer and Audit Department (unpublished) Flint D (1988) Philosophy and Principles ofAuditing. An Introduction, Basingstoke, Macmillan Education Flegmann, V (1980) Called to Account: The Public Accounts Committee of the House of Commons 1965-66 - 1977-78 Farnborough: Gower Garrett J (1986) 'Developing state audit in Britain' in Public Administration Vol 64 Winter pp421-433 Garrett J (1992) Westminster - Does Parliament Work ?, London: Victor Gol lancz Glynn J (1985) VFMAuditing in the Public Sector, London, Prentice Hall International Glynn J, Gray A and Jenkins B (1992) 'Auditing the three Es: the challenge of effectiveness' in Public Policy and Administration Volume 7 No 3 Winter a Goldsworthy D (1996) 'The Highways Agency's bridge programme: lessons learned from a VFM study'. Paper preparedfor the UK Evaluation Society annual conference, London 19-20 September. Gosling P (1999)'Go on-line urges mandarin' in Public Finance 17 December -6 January Gray A, Jenkins B, Segsworth B (1993) Budgeting, auditing and evaluation: function and integration in seven government New Brunswick, NJ: Transaction Books Guba E and Lincoln Y (1989) Fourth generation evaluation Thousand Oaks CA, Sage Henkel M (199 1) Government, Evaluation and Change London: Jessica Kingsley Publishers 224 Henkel M (1998) 'Evaluation in higher education: conceptual and epistemological foundations' in European Jo urnal of Education, Vo 13 3 No 3 Henley D (1989) 'Value for money audit' in Henley D et al (eds) Public Sector Accountability and Financial Control Chapman and Hall. Hennessy P (1989) "itehall 2 nd edition London: Fontana Press Hodges R and Wright M (1995) ' Audit and accountability in the privatisation process: The role of the NAO' in Financial Accountability and Management 11 (2) May pp 153-170 Hodges R (1997) 'Competititon and efficiency after privatization: the role of the NAO' in Public Money and Management January-March pp3 542 Hogwood B, Judge D and McVicar, M (1998) 'Too much of a good thing ? The pathology of accountability' Paper presented at the Political Studies Association Annual Conference, University of Keele, 7-9 Apri I Hood C (199 1) 'A public management for all seasonsT Public Administration Vol 69, No I pp 3-19 Hood C (1998) 'Exploring variations in public management reforms of the 1980s' in Bekke H, Perry J, Toonem T (eds) Civil service systems in comparative perspective Bloomington and Indianapolis: Indiana University Press Hood C, James 0, Jones G, Scott C and Travers T (1998) 'Regulation inside government: where new public management meets the audit explosion' in Public Money and Management April- June Hood C, Scott C, James 0, Jones G and Travers T (1999) Regulation inside Government: Waste- Watchers, Quality Police andSleaze-Busters Oxford: Oxford University Press Hughes 0 (1998) Public Management and Administration: An Introduction 2nd Edition Basingstoke: Macmillan Press Ltd INTOSAI (1992) Auditing Standards, International Organisation of Supreme Audit Institutions Huczynski A and Buchanan D (199 1) Organizational Behaviour - an Introductory Text Hemel Hempstead: Prentice Hall Jackson P (1993) 'Public service performance evaluation: a strategic perspective' in Public Money and Management October-December Keemer P (1984) State Audit in Western Europe: A Comparative Study unpublished M Phil thesis, University of Bath Keen J (1998a) From Checking to Observing: Reflections on Study Design London: National Audit Office 225 Keen J (1998b) 'The audit nexus: shaping VFM audit. ' Paper presented at the European Institute for Advanced Studies in Management conference 'Accounting for the New Public Management', Venice, September Keen J (1998c) 'The main theoretical approaches to evaluation' Paper in 'Securing Better Value for Money' (internal report to the National Audit Office). Keen J (I 999a) 'On the nature of audit judgements: the case of value for money studies' in Public Administration Vol 77 No 3 pp509-525 Keen J (I 999b) Constructivism and Postmodernism: Implications for VFM work (Unpublished paper for NAO, February) Kemp P (1996) 'Call Whitehall to account' in The Times 9 May King G, Keohane R and Verba S (1994) Designing Social Inquiry - Scientific Inference in Qualitative Research Princeton: Princeton University Press Kjar A (1996) Rigsrevision i Danmark og Sverige (State Audit in Denmark and Sweden. A report from a project called 'Public sector - conditions and flatures') Institute for Political Science, University of Copenhagen. Kraan A and Rozendal P (1997) 'Using evaluations in performance auditing : The case of the Netherlands Court of Audit' Paper presented at the 1997 EES Conference Stockholm Krueger R (1988) Focus Groups: A Practical Guidefor Applied Research 2"d Edition Newbury Park, CA: Sage Publications Leeuw F (1991) 'Characteristics of government-wide performance audits' in Leeuw F (ed) Auditing at the Netherlands Court ofAudit: case studies The Hague: Algcmenc Rekenkamer Leeuw F (1992) 'Performance auditing and policy evaluation: discussing similarities and dissimilarities' in The Canadian Journal ofProgram Evaluation Vol 7 No I pp53-68 Leeuw F (1996) 'Auditing and evaluation: bridging a gap, worlds to meet? ' in Wisler C (ed) Evaluation and Auditing. Prospects for Convergence, New Directions for Evaluation Number 71, Fall, Jossey-Bass Publishers a Leeuw F and Engwirda M (1993/4) 'Effectiveness auditing in the Netherlands: history, charter and case studies' in EVROSAI 1993/94 pp 16-20 Lipsey D (2000) The Secret Treasury London: Viking Lovell A (1996a) 'Notions of accountability and state audit: a UK perspective' in Financial Accountability and Management 12(4) November Lovell A (1996b) 'Evolutionary developments in state audit: a European Union perspective' in Research in Government Non-proji't Accounting, Volume 9, pp79-99 226 March J and Olsen J (1996) 'Institutional perspectives on political institutions' in Governance 9 (3) pp248-264 Marshall C and Rossman G (1995) Designing qualitative research 2"d edition Thousand Oaks: Sage Martin S and Sanderson 1 (1999) 'Evaluating public policy experiments - measuring outcomes, monitoring processes or managing pilotsT in Evaluation, Vol 5(3) pp 245-258 Maxwell J (1996) Qualitative Research Design -An Interactive Approach Thousand Oaks: Sage May T (1993) Social Research- Issues, Methods and Process Buckingham: Open University Press Melia K (1997) 'Producing 'plausible stories': interviewing student nurses' in Miller G and Dingwall R (eds) Context and method in Qualitative Research, London, Sage Miller G and Dingwall R (eds) (1997) Context and Method in Qualitative Research London: Sage Mordacq P (1998) 'How to gain assurance as to the proper use of public money'. Paper presented at the International VFM methods seminar, 29-3 0 September 1998, National Audit Office Mordacq P (1999) 'La cour des comptes and les chambres regionales des comptes: safeguarding texpayers' money in the French system of government' in Justice and Money (ed by Joelle Godard and David Guild) Hume Papers on Public Policy Volume 7 No I Spring 1999 Morgan D and Krueger R (1992) 'When to Use Focus Groups and Why' in Morgan D (ed) . ful Focus Groups -Advancing the State of1he Art Newbury Park: Sage Success Morgan D (1988) Focus Groups as Qualitative Research London: Sage Mul R (1996) Between Government Basking and Consulting: Strategies of six Supreme Audit Institutions. Unpublished paper for Netherlands School of Government National Audit Office (1985) National Health Service: Preventive Medicine London: HMSO National Audit Office (I 986a) Department of Trade and industry: Assistance to Indust? y under section 8 ofthe Industrial Development Act 1982 London: HM SO National Audit Office (1986b) Ministry of Defence: Management of Work at Research Establishments London: HMSO National Audit Office (I 986c) Review of Forestry Commission Objectives and Achievements London: HMSO National Audit Office (1987) Department of Health and Social Security and Department of Employment: Incorrect Payments ofSocial Security Benefits London: HM SO National Audit Office (1988) Annual Report London : National Audit Office 227 National Audit Office (I 989)Annual Report London: National Audit Office National Audit Office (1990) Annual Report London: National Audit Office National Audit Office (1991 a) Creating and Safeguarding Jobs in Wales London: HMSO National Aud it Office (199 1 b) Annual Report London: National Audit Office National Audit Office (1992a) The British Antarctic Survey: Management of Major Capital Projects and Scientific Programmes London: HM SO National Audit Office (I 992b), 4nnual Report London: National Audit Office National Audit Office (I 993a) Annual Report London: National Audit Office National Audit Office (I 993b) Qualitative Techniques: Their Practical Uses in VFM audit (intemal NAO guide) National Audit Office (1993c) Department of National Heritage: Protecting and Managing England's heritage property London: HMSO National Audit Office (1993d), 4ccountingfor inflation in defence procurement London: HMSO National Audit Office (I 993e) Ministry of Defence: TheAward of the Contractfor the Landing Plaýfbrmfor Helicopters London: HMSO National Audit Office (1993f) StudyDesign (intemal NAO guide) National Audit Office (I 994a) Overseas DevelopmentAdministration: Management of overseas programme aid London: HMSO National Audit Office (I 994b). 4nnual Report London: National Audit Office National Audit Office (1994c) Ministty of Defence: Major Projects Report 1993 London: HMSO National Audit Office (1994d) National Health Service: Hospital Catering in England London: HMSO National Audit Office (I 994e) The Industrial Injuries Scheme London: HMSO National Audit Office (I 995a) Metropolitan Police Service: Responding to Callsftom the Public London: HMSO a National Audit Office (1995b) The Department of Trade and Industry: Supportfor Innovation London: HMSO National Audit Office (I 995c) Annual Report London: National Audit Office National Audit Office (I 995d) Entry into the United Kingdom London: HMSO National Audit Office (I 995e) Evaluating the Applications to Run the National Lottery London: HMSO National Audit Office (1996a) State Audit in the European Union London: National Audit Off ice 228 National Audit Office (I 996b) Annual Report London: National Audit Office National Audit Office (1996c) Overseas Trade Services: Assistance to Exporters London: HMSO National Audit Office (I 996d) National Health service Supplies in England London: HMSO National Audit Office (I 997a) Valuefor money Handbook London: National Audit Office National Audit Office (I 997b) Annual Report London: National Audit Office National Audit Office (1997c) HMPrison Service: Prison Catering London: HMSO National Audit Office (1997d) Ministry of Defence: The Defence Evaluation and Research Agency. ý Review ofperformance London: HMSO National Audit Office (1997e) Wind Up of Leeds and Bristol Development Corporations London: HMSO National Audit Office (1997f) Scottish Office Department of Health: Cataract Surgery in Scotland London: HMSO National Audit Office (I 997g) Designing vfm studies. A Guide London: National Audit Office National Audit Office (1997h) 7he Contributions Agency's Customer Charters London: RMSO National Audit Office (19971) Focus Groups: How to apply the technique to vfin work London: National Audit Office National Audit Office (1997j) Department of Social Security. ý Measures to Combat Housing Bene/11 London: HMSO National Audit Office (1997k) Benchmarking. How benchmarking can help in vfin examinations London: National Audit Office National Audit Office (19971) The Scottish Office Development Department: Sales of Scottish New Towns'Commercial and Industrial Properties London: HMSO National Audit Office (I 997m) University ofPortsmouth London: HMSO National Audit Office (I 998a) International VFM methods seminar: September 29-30 London: National Audit Office a National Audit Office (I 998b) Annual Report London: National Audit Office National Audit Office (I 998c) Striving to achieve more: improving the financial impact of our work (internal paper) National Audit Office (1999d) Special Compliance Office: Prevention of Corruption London: HMSO National Audit Office (I 998e) Countering Anti-competitive behaviour in the telecommunications industry London: HMSO 229 National Audit Office (1998f) The Distribution of Lottery Funds by the English Sports Council London: RMSO National Audit Office (I 998g) Department of the Environment, Transport and the Regions: HM Coasiguard. Civil Maritime Search and Research London: HMSO National Audit Office (1998h) Ministry ofAgriculture Fisheries and Food and the Intervention Board., BSE. ý the cost of a crisis London: HM SO National Audit Office (I 998i) Benefits Agency: Performance Measurement London: HMSO National Audit Office (I 998j) Performance Measurement London: National Audit Office National Audit Office (1998k) 'The vfm toolkit. Are you using the right method for the right jobT wallchart produced by National Audit Office VFM Guidance team. Internal document. National Audit Office (19981) HM Coasiguard. Civil Maritime Search and Research London: HMSO National Audit Office (1998m) Deficit in the Fund for Pensioners of the Former Water Authorities London: HMSO National Audit Off ice (I 998n) Ministry ofDefence: Major Equipment Storage London: HMSO National Audit Office (1999a) What are taxpayers gettingfor their money? A value for money guide to examining outputs and outcomes London: National Audit Office National Audit Office (I 999b) Annual Report London: National Audit Office National Audit Office (I 999c) Getting beneath the surface -A handbookfor using qualitative evidencefor valuefor money examinations London: National Audit Office National Audit Office (2000a) Inpatient Admissions and Bed Management in NHS Acute hospitals London: HMSO National Audit Off ice (2000b) The Management of Hospital Acquired Infection in Acute NHS hospitals London: RMSO National Audit Office (2000c) Access to properties grant-aided by English Heritage London: HMSO a National Audit Office (2000d) State Earnings-Related Pension Scheme: 1hefailure to publicise reducedpension rightsfor widows and widowers London: HMSO, Neave G (1988) 'On the cultivation of quality, efficiency and enterprise: an overview of recent trends in higher education in western Europe, 1986-1988' in European Journal ofEducation Vol 23, Nos I and 2, pp7- 22 Neave G (1998) 'The evaluative state reconsidered' in European Journal of Education Vol 33, No 3 230 Normanton E (1966) The Accountability and Audit of Governments Manchester: Manchester University Press OECD (1996) VFMauditing and the Modernisation ofGovernment OECD, Paris Pavalko R (1988) Sociology of Occupations and Professions Itasca, Illinois: FE Peacock Publishers Pawson R and Tilley N (1997) Realistic Evaluation London: Sage Pendlebury M and Shreim 0 (1990) 'UK auditors' attitudes to effectiveness auditing' in Financial Accountability and Management 6(3) Autumn pp 177-189 Pliatzky L (1989) The Treaswy under Mrs Thatcher Oxford: Basil Blackwell Pollitt C (1993) 'Occasional excursions: a brief history of policy evaluation in the UK' in Parliamentary Affairs 46,3 pp3 53 -3 62 Pollitt C and Summa H (1996) 'VFM audit and evaluation: similar tools, different relationships? ' in Wisler C (ed) Evaluating and Auditing. - Prospects for Convergence, New Directions for Evaluation, Number 71, Fall, American Evaluation Association Jossey-Bass Publishers pp29-50 Pollitt C and Summa H (1997a) 'Evaluation and the work of Supreme Audit Institutions' in M- C Kessler et al (eds) Evaluation des politiques publiques, Paris: I'Harmattan. Pollitt C and Summa H (I 997b) 'Reflexive watchdogs ? How supreme audit institutions account for themselves' in Public Administration Summer 1997 Vol 75 Number 2 Pol I itt C and Summa H (I 997c) 'VFM auditing: travellers' tales' in Chelimsky E and Shadish W (eds) Evaluationfor the 21' century. a handbook; Thousand Oaks, California, Sage pp86-108 Pollitt C, Girre X, Lonsdale J, -Mul R, Summa H, Waemess M (1999) Performance or Compliance? Performance audit and Public Management in Five Countries Oxford: Oxford University Press Pollitt C and Bouckaert G (2000) Public Management Reform Oxford: Oxford University Press Power, M (1997) The Audit Society. Rituals of Verification Oxford: Oxford University Press Procedures Committee (1978) Select Committee on Procedure T" Report, Session 1977-78 London: HMSO Public Audit Forum (1999) Implications for Audit of the Modernising Government Agenda London: Public Audit Forum Public Money (1984) 'First twelve reports' in Public Money September pp 67-70 Punch M (1994) 'Politics and ethics in qualitative research' in Denzin N and Lincoln Y (eds) Handbook of Qualitative Research, Sage pp 83-96 231 Riksrevisionsverket (1996) Housing Benefit -a performance audit on an instrument of social and housingpolicy Stockholm: Riksrevisionsverket Roberts S and Pollitt C (1994) 'Audit or evaluation ?a National Audit Office VFM study' in Public Administration Vol 72 Winter pp527-549 Rossi P and Freeman H (1993) Evaluation: A systematic approach Thousand Oaks: Sage Rueschemeyer D (1986) Power and the Division ofLabour Cambridge: Polity Press Sandberg B and Ivarsson S (1996) 'Fraud and abuse of the welfare system in Sweden' in EUROSAI no 3 pp 29-32 Sarantakos, S (1993) Social Research, Basingstoke, Macmillan Press Limited Schwartz R (1999) 'Coping with the effectiveness dilemma: strategies adopted by state auditors' in International Review ofAdministrative Sciences vol 65 pp 5 11-526 Sharkansky 1 (1991) 'The development of state audit' in Friedberg A et al (eds) State Audit and Accountability -A Book ofReadings Jerusalem: State of Israel, State Comptroller's Office Sloan N (1994) Performance audits and Other Evaluative Studies Compared Unpublished M Phil thesis, Brunel University State Audit Office (1996) The State Audit Office of Finland: general introduction Helsinki: State Audit Office Stewart D and Shamdasani P (1990) Focus Groups Theory and Practice Thousand Oaks: Sage Stewart J and Walsh K (1992) 'Change in the management of public services' in Public Administration Vol 70 Winter pp499-517 Strauss A and Corbin J (1990) Basics of Qualitative Research - Grounded Theory Procedures and Techniques Thousand Oaks: Sage Strauss A and Corbin J (1994) 'Grounded theory methodology: an overview' in Denzin N and Lincoln Y (eds) Handbook of Qualitative Research, Thousand Oaks Sage Swedish National Audit Bureau (1993) Performance Auditing at the Swedish National Audit Bureau Stockholm: The Swedish National Audit Bureau 0 Swedish National Audit Office (1995) Performance Auditing at the Swedish National Audit Office Stockholm: Swedish National Audit Office Swedish National Audit Office (1996) Handbook in Performance auditing. Theory and Practice Stockholm: Swedish National Audit Office Swedish National Audit Office (1998) Illicit work in Sweden -a report on a welfare state dilemma Stockholm: Swedish National Audit Off ice Thatcher M (1999) 'Riding audit's big bang' in Public Finance 5-11 March, London Travers T (1998) 'The day of the watchdog' in Public Finance 16 October, London 232 Travers T (1999) 'Should the inspector be a bruiserT in Public Finance November 19-25 London Walker D (1999) 'Call in the Auditor' in The Guardian 18 October Ward S (2000) 'No monkey business' in Public Finance 14-20 April, London White F and Hollingsworth K (1999) Audit, Accountability and Government, Oxford University Press: Oxford Whitehouse M (1999) 'Modemising Audit Too' in Overview August/September pp 13-14 Wisler C (ed) (1996) Evaluation andAuditing. Prospectsfor Convergence, New Directions For Evaluation Number 71, Fall, Jossey-Bass Publishers, American Evaluation Association Witteveen T (1996) 'Relations between the Netherlands Court of Audit and Parliament: a Dutch perspective' in EUROSAI no 3 pp26-28 Yin RK (1989) Case Study Research: Design and Methods, Newbury Park CA, Sage a 233