RIAM:Overview: The Assertion Linked Systems Based Audit (ALSBA)
- 1 Introduction
- 2 What Are Assertions?
- 3 For What are Assertions Used?
- 4 How do we Establish Assertions?
- 5 What is the Assertion Linked Systems Based Approach?
- 6 What is Threat Testing?
- 7 How Do We Document Systems?
- 8 What Are Some of the Types of Reviews Conducted Within the ALSBA?
- 9 How Do We Report?
- 10 BackLinks
The keys to the method are structure and focus. With RIAM, Internal Audit applies a Systems and Assertion Based approach to answer targetted 'questions' about an audit area. Questions focus the review, while assertions define the criterion for the answer.
Many systems based approaches merely measure the compliance of an organisation's staff with a particular system. The ALSBA method is a substantial enhancement to this commonly used approach. The RIAM auditor analyses compliance of the system process with a strategic and/or tactical purpose, compliance of practice with procedure and the awareness and readiness presented by potential (risks and opportunities) in the system itself.
The avoidance of checklists makes the auditor adaptable by permanently adopting a 'learning' posture. More importantly, the process is universal in that the same logic structure can be applied from the strategic level through to transactional compliance level, and from 'hard' financial processes to 'soft' subjective process.
Very large organisations present some particular challenges for the systems based audit, including coordination of teams across multiple jurisdictions, locations and organisation units. Here we present an overview to the technical aspects of the RIAM SBA, in Conduct of the Very Large Audit we explore the method in detail in both the large audit and small audit context.
What Are Assertions?
The figure on the preceding diagramme summarises the Assertion Linked Systems Based Audit analytic structure. The process starts with the five areas of Internal Audit's "Scope of work" within which Assertions are defined. Support for the selected Assertions is classified into management's 10 Control Classes (areas for management action). The systems built by management to support the Assertions within the Control Classes will have identifiable "Control Attributes" identical to those used in our Control Implementation Service, and are classifiable according to the "Type" - preventive, detective or corrective.
The concept of Assertions is the core of a RIAM Systems Based Audit. Assertions are truths that we wish to express about a system. They formulated as statements of "fact" about a system. Examples of typical Compliance Assertions for financial aspects of a Grants Scheme might be:
a. Grant expenditure is bona fide (ie that acquittals are for actual grants and for services appropriate to grant activity);
b. Grant data reported/processed is:
- Attributed to the proper period,
- Accurately calculated,
- Correctly and appropriately accumulated,
- Accurately recorded,
- Correctly disclosed,
- Properly authorised with respect to transactions (ie grantee approved costs and the Commission is satisfied that the amount is for an appropriate expense),
- Providing benefits to which grantees are eligible,
c. The relevant management directions and legislation are observed:
- Payments are in accordance with legislation, and
- Approvals for grants are in accordance with the legislation (ie properly vetted by the Grant Committee and approval is given by the Board); and
d. The assets of the organisation are efficiently, effectively and otherwise appropriately protected and applied (ie having an appropriate process of grant approval that assures projects are of an appropriate standard, and that Commission resources are used efficiently).
For What are Assertions Used?
When we say a given system is operating satisfactorily we mean that our review has tested the truth of a set of assertions and we have found that they have been sustained. Thus testing the assertions is the purpose of the audit.
Assertions are the focus and underlay the structure of the RIAM analytic method. All review activities, findings, discussions and recommendations must be able to be tied back to the review's assertions.
The result is that both the auditor and the auditee have a precise understanding of the level of comfort a given review offers.
Assertions have another huge advantage for the auditor: They allow us to frame focus questions about a system in "yes" or "no" form, which are answered by proving or disproving the assertions. For example, the question "Is system XYZ operating effectively?", is, by its nature, subjective. My meaning of the word 'effective' may be radically different from your understanding of theat same word. If we say, "effective in this context means accurate and timely" then we both know that neither of us meant "authorised and consistent", or "fair and equitable".
Thus by combining a focus question, with the assertions that define a "yes" answer, we as auditors, can give management and the governance committee what they want: certainty. We do not need to hedge for the unknown - because we have stated clearly our context specific meaning.
Thus we say that assertions are the definitions of the audit project focus question.
For a detailed discussion of assertions and example assertion sets in various kinds of systems see:
How do we Establish Assertions?
A reviews assertions are agreed with the auditee management before a review commences. In many cases, such as financial balances audits and quality audits, we are able to recommend appropriate assertions. In other reviews, particularly those specifically requested by management, the managers will have a clear idea of particular "Questions" they wish answered by the review.
The establishment of "Questions" is the first step in selecting audit assertions.
During the entrance interview phase of the audit management identifies a number of questions about the target system they wish to have answered. The auditor then proposes a series of Assertions, the sustaining of which will constitute an affirmative answer, and the suppressing of which will constitute a negative answer. These assertions are agreed with management.
What is the Assertion Linked Systems Based Approach?
The objectives of the reviews are summarised as:
- Document the procedures in operation within the section so far as they relate to the target activities;
- Collect sufficient data and analyse that data to support assertions that address management's critical success factors represented by questions they request audit to answer;
- Identify risk and efficiency exposures to the organisation and the critical success factors of management;
- Recommend relevant and practicable changes in the systems and procedures to management where these exposures are present; and
- Form an opinion as to the overall reliability of the systems in place and as modified.
Meeting The Objectives
The structure of the approach, diagrammed above, that meets the audit objectives has four phases. Here we summarise those phases. A more detailed discussion of these phases mapped into the context of both small team and large multi-location, team audits is explored in:
PHASE 1: FAMILIARISATION, SCOPE AND PLANNING
This topic is explored in more detail in:
PHASE 2: DOCUMENTATION AND SYSTEMS ANALYSIS
PHASE 3: TESTING AND RESULTS ANALYSIS
PHASE 4: REPORTING AND FOLLOW UP
Establishing the framework
The key principles of the framework include:
- Interviews to scope and focus the review and involvement of Management and Staff throughout the process;
- Ensuring agreement as to the purpose, focus, scope, boundary, approach and findings of the review;
- Assertions as criteria for evaluation.
- Application of Risk Analysis, not just at the Planning stage, but also the Threat Analysis stage when assessing Systems Design, and the Reporting Stage when finalising recommendations. The Audit Risk is the risk that the audit will provide a wrong opinion. This is a function of:
- The Inherent Risk in the organisation
- the risk that an error is likely to occur;
- The Control Risk
- the risk that the control system will not prevent, detect or correct the error; and
- The Detection Risk
- the risk that our procedures will not identify the existence of a material error.
- The Inherent Risk in the organisation
The ALSBA uses Assertion focussed Risk and Threat analytic procedures to minimise this risk.
- Risk and Threat analysis aims to minimise the cost of reviews by keeping procedures tuned to the real exposures, and when combined with assertions, raises the certainty that our systems opinion is correct.
- Use of a variety of report and presentation styles to best communicate information; and
- The Internal Auditor MUST become part of the management & systems improvement process, not a disinterested, occasional observer.
- Analysis of control systems performance in meeting objectives.
- Clear discussion and specific recommendations to provide improvements.
What is Threat Testing?
Threat testing is an approach to assertion testing used as an alternative to a Desired Control Model. RIAM supports both concepts.
The key benefits of threat testing are:
- Controls analysis is kept current to the ACTUAL systems in place rather than an out-of-date control model;
- The audit process recognises and supports improvement and change in systems - essential for environments where Total Quality Management is operating;
- By evaluating the sources of possible problems, the process RESULTS in the development of Desired Control Models;
- Management is involved in the assessment of risks of systems failure;
This is a brief outline of the Threat Testing process :
- Each assertion is examined in turn. For each assertion a list of causes for failure of an assertion is prepared based on experience, statistical sampling, management advice, consultant advice, and checklists, etc. These causes are called threats. To each threat a probability of occurrence may be assigned if desired (perhaps based on historic samples).
- Each threat is then applied to the control system model (developed during the systems documentation phase) to investigate the probability of the system preventing the threat (ie. mitigating the risk). This probability is expressed as a probability of system failure.
- The risk of the threat occurring multiplied by the risk of system failure (Control Risk) is probability of the assertion not being sustained in operation.
The sum of all such threat related probabilities is the total risk of assertion failure in the system.
How Do We Document Systems?
RIAM working papers are designed to form a "tree" or pyramid with the apex being the opinion of the systems in operation, and the base being the detailed "views" or models of the organisation's systems and the testing results verifying aspects of the system's operations.
|1||Final Audit Report and Other Relevant Files|
|2||Supervisor, Manager & Partner Reviews and Follow Up|
|3||Engagement Letters, Contract and Contacts|
|4||Action Plan, Client Follow Up and Correspondence|
|5||Matters for Manager & Partner Attention|
|6||Matters for Review Next Audit|
|7||Planning Documents and Audit Program|
|8||Work & Time recording Schedule|
|9||Background and Organisation Details|
|10||Organisation Objectives, Operating & Financial Policies, and Performance Measures|
|11||Strength & Weakness Schedule|
|12||Control System Documentation and Conclusion|
(Control Questionnaires, flowcharts, checklists and narratives)
|13||Records of Interview|
|14||Legislation and Management Directives - Compliance|
(Including Important Contracts and Agreements)
|15||Analysis and Tests of Transactions, Processes and Account Balances|
|16||Other Background Data and Notes|
The Index for The Standard RIAM Audit File
The foregoing index shows that the files are self contained units including not only plans and tests, but also:
- date records of client contacts;
- relevant legislation and directions;
- full internal and external cross references;
- systems documentation; and
- organisation background and structures.
Section 12 of the file contains the detailed analysis of the systems under review:
|2||Objectives (Purpose) of the Control System||12.|
|3||Framework of Analysis (Assertions to be supported)||12.|
|5||Overview of the Control System (Principal Flows)||12.|
|6||Control System Flowcharts/Documentation||12.|
|7||Files & Records in the System||12.|
|8||Cycles in the System||12.|
|9||Transactions and Value||12.|
|10||Documents in the System||12.|
|11||Segregation of Duties||12.|
Index for Section 12 of the Standard Audit File - Control System Documentation
The continuation of the "tree" structured analysis is evident in the above index. Each subsection contains further structured working papers, the details of which can be found in the volume "Standard Forms & Papers" of this series.
Methods of Analysis
The working papers require that systems design is analysed BEFORE any testing is performed. While prewritten test programs can be used, the full benefit of the method is received when the systems analysis is performed using the various systems models:
- Segregation of Duties Chart
- Client Provider Analysis
- Key Quantities (transaction values and volumes)
- Cyclic Events
- Annotated Data Flows, Narrations and/or Document Flows
- Key Controls structured by their "data flow focus":
And evaluated within the assertion/control attribute structure outlined earlier.
The Working Paper's documentation of the system culminates in the Assertion Matrix and Control Strength & Weaknesses Chart. The Systems Analysis of section 12 of the file is summarised in these charts.
Within the assertion structured systems model are many subsystems. These are documented throughout the documentation "tree". Each subsystem should be documented in the way that best suits our analytic needs. For transaction flows this might be some type of annotated data flow, for a delegations analysis it might be an organisation chart, and for a risk analysis it might be a Fitzgerald Matrix, etc.
There are a number of techniques available to the auditor for use in documenting systems of internal control, such as:
- Process and Document Flows
- Annotated Data Flows
- Organisation Charts
- Segregation of Duties Chart
- Assertion Matrix
- Lancaster Modelling
- Algorithm Pseudo-programming
Irrespective of which method is chosen, documentation should include:
- the origin of every document and record in the system
- all processing that takes place on the document
- the disposition of every document and record in the system
- a description of internal controls operating within the system
What Are Some of the Types of Reviews Conducted Within the ALSBA?
Management Assurance services utilising the ALSBA cover the full range of Internal Audit work including:
How Do We Report?
Ultimately the product produced and of greatest significance to management is the report. Our reporting is standardised to ensure consistency of structure, coverage, presentation, language and quality.
The significant features of our reports include:
- Standardised structure;
- Systems documentation and flow charts;
- Every finding is presented with: "Observation", "Risks and Implications", "Recommendations", and "Management Comment" sub sections;
- Clear, specific and relevant recommendations, not vague references to the need to "review" an area or "correct a problem";
- Clearly argued risks and implications of each finding. An observation is analysed by:
- The assertions affected,
- Risks and exposures from the observation,
- Arguments in favour of the breach and audit's comment on that argument;Inclusion of and focus on Action Plans; and
- Linking of findings to a clearly stated premise for the finding's importance: The Assertions affected.
Although the Report structure is one of the aspects of RIAM specifically tailored to the client, most adopt a close variation of one standard structure. RIAM includes five distinct report structures to assist clients identify their reporting needs.
The report is presented under the following headings/sections:
- Executive Summary
Provides a summary of the purpose, objectives, assertions, approach, scope, the overall opinion, key findings and issues arising.
- Objectives and Approach
Addresses the "How" and "Why" of the review, and defines the assertions on which the conclusions and findings are based.
- Scope and Boundary
Clearly defines the matters covered by the review, and most importantly the matters excluded from the review.
- Brief Description of the System Reviewed
Covers the Purpose of the Section/Systems, The People and Organisation Structure, the Principal Activities of the Section/Systems, Documents and Records (both manual and computer) and the Reports Produced from and to the Section/Systems.
- Checklist of Findings, Recommendations and Action Plans
Presents in Landscape form a summary of the findings and recommendations in section 6 under the headings: "Findings" and "Recommendations". Tables include boxes for Action Plans to be referenced or detailed. This section assists in monitoring and following up responses to audit recommendations by the Audit Committee.
- Detailed Findings and Recommendations
The findings and recommendations have a standard structure:
- The observed facts, relevant legislation, directions and industry relevant information.
- Implications and Risks
- Assertions suppressed or supported.
- Principal risks and exposures.
- Arguments in favour of, or reasons for, the breach and audit's comment.
- Summation of audit's conclusion as to risk or exposure.
- Numbered, clear, specific and relevant recommendations for action.
- Where alternatives are identified either by audit or the client they are presented and evaluated.
- Management Comment
- Management's response to the issues raised and action taken. After discussion and exit interviews the vast body of your recommendations should be accepted by management. If not, you have not done your job correctly!