System Under Development Audit of the Legal Case Management System - Planning - Final - August 2019

Internal Audit Division

Cat. No. J79-8/2019E-PDF

ISBN: 978-0-660-32259-9

As recommended by the Departmental Audit Committee, subject to approval by the Director of Public Prosecutions, on August 2, 2019.

Approved by the Director of Public Prosecutions on August 15, 2019.

Cette publication est également disponible en français.

This publication is available in HTML formats on the Internet at https://www.ppsc-sppc.gc.ca/eng/

© Her Majesty the Queen in Right of Canada, 2019.

Table of Contents

• 1.0 Executive Summary
  • 1.1 Objectives and Scope
  • 1.2 Audit Conclusion
  • 1.3 Summary of Recommendations
  • 1.4 Statement of Assurance
• 2.0 Introduction
  • 2.1 Background
  • 2.2 Objectives and Scope
  • 2.3 Methodology
• 3.0 Findings
  • 3.1 Governance
  • 3.2 Project Management
  • 3.3 Budget
  • 3.4 Internal Controls
  • 3.5 Testing Plan
  • 3.6 Implementation Plan
• 4.0 Conclusion
• 5.0 Overall Management Response
• 6.0 Management Action Plans
• Appendix A – Audit Criteria
• Appendix B – List of Abbreviations

1.0 Executive Summary

1.1 Objectives and Scope

The objectives of this audit were to provide management with an independent assessment of the progress, quality, and attainment of the project objectives at defined milestones within the project and an evaluation of the internal controls of proposed business processes during the various phases in the development cycle where enhancements can be easily implemented and processes adapted.

Given the significance and size of the project, an audit report will be provided at the end of each of the project milestones and a final report will be provided upon completion. Due to the agile Legal Case Management System (LCMS) development approach, activities such as internal controls identification and implementation, testing, and implementation planning are at an early stage or have not started. These will be further assessed during the next audit.

The planning and examination phases of the audit were conducted between August 2018 and February 2019.

1.2 Audit Conclusion

The project governance could be improved by assigning and acting upon responsibilities and accountabilities; implementing and/or complying with change, issues and risk management procedures; and timely updating and disseminating of information to stakeholders. Internal controls, testing, and implementation will be further assessed during the next audit.

1.3 Summary of Recommendations

This report contains recommendations to the project Executive Co-sponsors, who should:

• strengthen the project governance by establishing the Terms of Reference for the Project Steering Committee; define, assign and communicate roles, responsibilities and accountabilities of the project’s stakeholders and ensure that they are exercised accordingly.
• ensure that the project management plan and risks and issues log are updated, maintained, and communicated to key stakeholders.
• ensure a communication plan is prepared and used.
• review the financial information for accuracy and completeness and regularly update and monitor the budget.

1.4 Statement of Assurance

In my professional judgement as the Public Prosecution Service of Canada’s (PPSC) Chief Audit Executive, sufficient and appropriate audit procedures were conducted and evidence gathered to support the accuracy of the conclusions reached and contained in this report. The audit’s findings and conclusion are based on a comparison of the conditions, as they existed at the time of the audit, against pre-established and approved audit criteria that were agreed upon by PPSC’s management. The findings and conclusion are applicable only to the entity examined. The conduct of the audit was in accordance with the Treasury Board of Canada Policy on Internal Audit.

I would like to thank the LCMS project team, management, and stakeholders for their cooperation and assistance during the conduct of this audit.

Cathy Rodrigue
Chief Audit Executive

2.0 Introduction

2.1 Background

The Public Prosecution Service of Canada (PPSC) has, since its creation in 2006, relied on the Corporate Service Provider’s (CSP) iCase, a legal case management and timekeeping system, which will be decommissioned after the ongoing implementation of a new system. This prompted the PPSC to look for an iCase replacement. The PPSC considered four options and decided to customize the CSP’s solution to meet its specific requirements.

The Internal Audit Division (IAD) originally launched the System under Development Audit – Legal Case Management System (LCMS) on January 11, 2016 and provided advice to the LCMS project and reported to the Departmental Audit Committee once on the project’s progress. The audit was placed on hold in February 2017 while awaiting project and spending authorities from the Treasury Board of Canada (TB), which were obtained in May 2018. Because of the delays, we launched a new internal audit to assess the project as it moves forward.

The project cost over the 2015-2016 to 2020-2021 is estimated at $5.58 million plus an ongoing $0.9 million.

Although modified, the IAD conducted this System under Development Audit - LCMS in accordance with the PPSC’s 2018-2019 Risk-based Audit Plan, which was approved by the Director of Public Prosecutions on June 15, 2018. Due to the lapse in time, we relaunched the audit with a modified approach.

2.2 Objectives and Scope

The objectives of this audit were to provide management with an independent assessment of the progress, quality, and attainment of the project objectives at defined milestones within the project and an evaluation of the internal controls of proposed business processes during the various phases in the development cycle where enhancements can be easily implemented and processes adapted.

Given the significance and size of the project, an audit update report will be provided at the end of each of the project milestones and a final report will be provided upon completion. Due to the agile LCMS development approach^{Footnote 1} activities such as internal controls identification and implementation, testing and implementation planning are at an early stage or have not started. These will be further assessed during the next audit.

The conduct of the planning and examination phases was between August 2018 and February 2019.

2.3 Methodology

The audit complied with generally accepted auditing practices and was conducted in accordance with the TB Policy on Internal Audit.

The audit methodology included, but was not limited to:

• interviews with:
  • the Project Sponsor;
  • the Project Manager;
  • the Information Technology Manager;
  • Project team members;
  • the Steering Committee members;
• a review and analysis of the project documents; and
• an analysis of financial system data.

3.0 Findings

3.1 Governance

Adequate governance ensures that the project was defined and approved by senior management. It also ensures that procedures are defined to keep management informed of the progress so that they can respond to issues as they arise.

We expected the project to have a defined scope and an established governance with clear responsibilities and accountabilities, including approving milestones and deliverables.

The Project Charter (Charter), approved by the Executive Council, and the Project Management Plan (Plan) were developed in March 2016. The project governance and scope were defined in both documents, including roles and responsibilities of the Steering Committee (Committee), Executive Sponsor^{Footnote 2}, and Project Team. The documents also identified the Project’s phases, milestones, and deliverables.

The Committee was established in March 2016 by the PPSC Executive Council to oversee the project. However, there were no Committee Terms of Reference defining key areas, such as membership, responsibilities and accountabilities, frequency of meetings. We also found that the responsibilities and accountabilities of the Committee, including those of the Chair and the Executive Sponsor, are inconsistent between the Charter and the Plan. During interviews, we found the Committee members had a lack of clarity regarding their responsibilities and accountabilities, as described in the Plan and Charter, and a lack of information regarding the project. Since its creation, the Committee has met only six times, limiting their ability to fully undertake their role as a governance body. In addition, the Project Manager and not the Executive Sponsor initiated the meetings.

Defining roles and responsibilities can bring clarity to a project and ensure that individuals with the right skill and/or authority undertake tasks and approvals. Responsibility for managing scope changes was defined and procedures were in place to obtain approval of scope changes. The division of responsibilities was appropriate, although responsibilities for the authorization to operate^{Footnote 3}, change management, and certification and accreditation were not assigned.

Weaknesses in project governance may cause the project to not meet the needs of the PPSC or to deliver on time and on budget.

Recommendations:

1. The project Executive Co-sponsors should strengthen the project governance by establishing the Terms of Reference for the Project Steering Committee.
2. The project Executive Co-sponsors should define and assign roles, responsibilities, and accountabilities of stakeholders, including those for the Executive Co-sponsors and Project Steering Committee. Consideration should be given to sharing this information with the Project Steering Committee and all project members.
3. The Project Steering Committee should ensure that roles and responsibilities are exercised accordingly and decisions are documented, where required.

3.2 Project Management

Project management controls ensure adequate oversight of the project, appropriate involvement by stakeholders, iterative evaluation of risks, monitoring of issues, and escalation of issues, where required.

We expected a detailed and up-to-date Plan, an integrated and fully assigned project team with a contingency plan to replace team members, a risk and issue management system, a communication plan, change, issues and quality assurance procedures.

The Project Management Institute states that a project plan can be used to guide both project execution and project control. The Plan should be expected to change over time as more information becomes available^{Footnote 4}. We found the Plan was last updated in September 2017 and did not reflect changes in both budget and schedule. The Plan was not detailed enough to include processes, sub-processes, activities, and related resource assignment, which enables scope, budget, and schedule control.

The project has no contingency plan to replace resources on an interim or permanent basis in instances where the project experiences staff departure or prolonged absences. However, the current project team members are all assigned project work and are fully utilized. All relevant business units, with the exception of IM and security, are involved in the project. The latter would ensure that information management and security requirements are considered during and after the definition of business requirements and rules in accordance with TBS policies and directives. An IM manager was recently hired at the PPSC and an IT security resource will soon be hired for the project.

The project has no defined quality assurance procedures to ensure that project deliverables are properly created, documented, and approved. This assurance function provides the oversight that management could use to ensure the project was proceeding as expected. We were advised that a quality assurance professional will soon be hired.

The project has a risk and issue management system but stakeholders were not involved in determining acceptable risk tolerance for the PPSC. In addition, the project risk and issue log, a useful tool for monitoring risks/issues, was not maintained.

There was a change management procedure but it was not followed for the one scope change that occurred. Removing the integration of the Integrated Financial and Material System (IFMS) with LCMS from the project scope was informally approved by the Executive Sponsor with no involvement of the Committee, as required by the Charter and the Plan. Changes made without the proper oversight or authority can weaken a project.

The project had a 2016 draft communication plan. A well-defined and documented communication plan will establish why, what, when, how, and to whom to communicate information regarding the project. The Project Management Institute states that “identifying the informational needs of the stakeholders and determining suitable meaning of meeting those needs is an important factor for project success.”^{Footnote 5}

The lack of project management controls may cause the project to fail in meeting the PPSC’s needs on time and on budget.

Recommendations:

4. The project Executive Co-sponsors should ensure that the Project Management Plan is updated and maintained to reflect changes. When changes are made, the Plan should be shared with key stakeholders.
5. The project Executive Co-sponsors should ensure that the risk and issues logs are maintained and monitored and information is shared with stakeholders.
6. The project Executive Co-sponsors should ensure a communication plan is prepared, that at a minimum, addresses the current phase of the project.

3.3 Budget

A sound project budget ensures the accuracy and completeness of information that is necessary to manage the project. We expected a project-detailed costing information that is accurate, complete, and current.

We found that the project had its own cost centre reflecting only items associated with the project. We also noted that the budgetary estimate was inaccurate, such as salaries for 2019-2020, and that it only included the first two months of 2020-2021 while the project is expected to rollout in June 2020. In addition, there was no provision in the budget to pay for professional services after March 2020, although professional services would still be required.

There was $595,384 unused in 2018-2019 and was reallocated as part of the annual exercise. We were advised that this money, if required, will be requested. However, there are no guarantees that the funds will be available therefore leaving the project vulnerable.

Inaccurate, incomplete and outdated financial information may cause to go over budget or not be able to finish the project.

Recommendation:

7. The project Executive Co-sponsors should review the financial information for accuracy and completeness and regularly update and monitor the budget.

3.4 Internal Controls

Efficient internal controls for application data and processes ensures that controls are designed in the planning phase when it is efficient to modify, enhance or aggregate them.

The audit expected controls requirement to be developed, risk assessed, and prioritized before their implementation. The audit found that internal control requirements are being developed although they are not documented. In addition, a control risk assessment was not prepared by the project team to identify and prioritize internal control requirements. The project approach is to examine the CSP controls, prototype them and present them to subject matter experts to determine if they are required and set priorities.

The audit team will further assess this area during the next audit of LCMS.

3.5 Testing Plan

An adequate testing plan provides for testing at the various stages of development and includes a definition of the types of tests to be performed, the timeframe for testing, and documentation requirements.

We expected that the project have a timely comprehensive testing plan which provides for adequate time to address all testing requirements before rolling out the system.

The audit found that the project had not fully established testing requirements and that a testing coordinator will be hired in summer 2019 to assist with test planning, testing, and fixing errors.

This may leave the project with limited testing time before the expected June 2020 rollout. The audit team will further assess this area during the next audit of LCMS.

3.6 Implementation Plan

An implementation plan ensures a minimum disruption during the initial implementation of the new system and thorough vetting of the processes prior to “throwing the switch” to the new system.

We expected the project to have a data conversion plan and migration strategy, a pilot test plan, a back-out plan, a readiness assessment plan, a training plan, and a transition plan.

We found that the project had a draft data migration strategy (from iCase to LCMS), including data conversion (from iCase format to LCMS format), which was neither finalized nor approved. As well, the project had not developed a pilot test plan (to verify the system under real time operating conditions), a training plan, a back-out plan (to restore the system to its original state if the implementation failed) or a readiness assessment plan (assess integration and readiness of all LCMS components, including with external dependencies). However, we were advised by the Project Manager that he will be hiring a test coordinator to develop, among other things, a pilot test plan.

The interface between IFMS and LCMS will not be fully automated during the first LCMS release, meaning that there must be an interim process in place before full automation of the interface. The audit found that that there was no detailed description of the process for the transition period.

The audit team will further assess these areas during the next audit of LCMS.

4.0 Conclusion

The project governance could be improved by assigning and acting upon responsibilities and accountabilities, implementing and/or complying with change, issues and risk management procedures, and timely updating and disseminating of information to stakeholders. Internal controls, testing, and implementation will be further assessed during the next audit.

5.0 Overall Management Response

The audit report has appropriately pointed out areas for improvement in the management of the PPSC LCMS project.

The management responses provided indicate that most of the recommendations are being addressed; however, there are reasons why the organization will choose to not fully address a specific area and to accept some risk. Taking into consideration the size and mandate of the PPSC, the project team is necessarily limited in size. While larger organizations might be able to devote more time and resources to a larger project team, the PPSC has chosen to limit the number of resources available to the project. The PPSC has a limited pool of information technology and project management expertise to draw from. The project team is comprised of 3- 4 full-time and 3-4 part-time individuals, with supplementary expertise provided by professional services contracts. This smaller project team size allows the PPSC to manage and control the overall costs of the project. In addition, the legal knowledge provided by the subject matter experts, required to successfully implement an LCMS, draws on resources who are already dedicated to fulfilling the mandate of the organization.

The PPSC does not have the resources to devote the time and money to a project that a larger organization might be able to bring to bear. There are some areas where actions are possible and others where management has chosen not to take action and assume the associated risks.

6.0 Management Action Plans

Appendix A – Audit Criteria

Audit Criteria - Planning

1. Management provides adequate governance over the project to ensure that the project is adequately defined and approved by senior management. Procedures are defined to keep management informed of the progress. Communications and escalation procedures are in place to allow management to respond to issues as they arise.
2. The project management controls ensure adequate oversight of the project (e.g. financial, meeting deadlines), appropriate involvement by the stakeholders, iterative evaluation of risks, monitoring of issues, and escalation of issues where required.
3. The budget process is accurate, complete, and provides the information necessary to manage the project.
4. Internal controls are designed in the planning phase, when it is most efficient to modify, enhance or aggregate controls.
5. The project plan provides for adequate testing at the various stages of development, including definition of the types of tests to be performed, the timeframe for testing and documentation requirements.
6. The implementation plan is developed to ensure minimum disruption during the initial implementation of the new system and thorough vetting of the processes prior to final “throwing the switch” to the new system.

Appendix B – List of Abbreviations

CSP

Corporate Service Provider

IAD

Internal Audit Division

IFMS

Integrated Financial and Material System

IM

Information Management

IT

Information Technology

LCMS

Legal case Management System

PPSC

Public Prosecution Service of Canada

TB

Treasury Board

ToR

Terms of Reference