Just-in-Time Audit of PTIF & CWWF - Governance Phase

July 2017

Table of Contents

Executive Summary

The overall objective for this audit is to assure Infrastructure Canada (INFC) senior management of the effectiveness of core controls which will strengthen implementation of the Public Transit Infrastructure Fund (PTIF) and Clean Water and Wastewater Fund (CWWF). 

The just-in-time audit is being undertaken in three phases: Phase I (completed) assessed the agreement and project approval processes; Phase II (current) assesses governance, and Phase III (future) will assess payments and reporting.

The Audit and Evaluation Branch (AEB) concluded that the governance and risk management processes and tools in place were generally effective.

  • Oversight Committees are established and meeting as intended. There are tools and templates in place for reporting. This, together with formal and informal communications, encourages consistency across the regions, although the capacity of smaller communities to provide information remains a risk.
  • Internal governance provides an effective forum to operate and implement the PTIF and CWWF consistently across the regions and make program-level decisions based on issues and questions raised at committee meetings.
  • Documentation is in place to record decisions made.

Three recommendations were made to further improve the governance of the programs, reduce risk and realize efficiencies:

  1. The ADM, POB should ensure that federal Co-Chairs of Oversight Committees make sure that future progress and outcomes reporting is timely, complete and sufficiently detailed.
  2. For future programs, performance indicators should be established, with input from the provinces and territories, to ensure that data is available and that reporting expectations are aligned throughout the program lifecycle.
  3. Key performance issues with Program Information Management System (PIMS) should be prioritized by Information Management/Information Technology (IM/IT) and Program Operations Branch (POB) and addressed as soon as possible. Improved communications between the business community and the PIMS SME team would ensure that ongoing investments in the system reflect the specific needs of all programs. As part of ongoing system improvements, lessons learned from programs, including PTIF and CWWF, on the usability and effectiveness of the system should be discussed within the business community and provided as requirements to the PIMS SME team.

Background

In Budget 2016, the Government of Canada announced an investment of $120 billion in infrastructure over 10 years, including $60 billion in new funding for public transit, green infrastructure, and social infrastructure, to better meet the needs of Canadians and better position Canada's economy for the future.

The Government of Canada proposed to provide $11.9 billion from this plan of which $3.4 billion would be be used to upgrade and improve public transit systems and $2.0 billion would be be used for investments in water and wastewater projects.

Public Transit Infrastructure Fund (PTIF)

Public transit infrastructure investments aim to strengthen communities and grow the economy. This $3.4 billion program is supporting projects that result in increased capacity, enhanced service or improved environmental outcomes. These projects are contributing to rehabilitating public transit systems, enhancing asset management, modernizing and optimizing transit systems, and supporting planning for future expansion.

Clean Water and Wastewater Fund (CWWF)

This $2 billion program is providing critical project funding for water and wastewater infrastructure to ensure communities have access to safe, efficient, and environmentally responsible water and wastewater systems. Projects are supporting the rehabilitation of water treatment and distribution infrastructure and wastewater and storm water treatment, collection and conveyance infrastructure, as well as initiatives that improve asset management, system optimization, and planning for future upgrades to water and wastewater systems.

Management Control Framework

The Committee of Sponsoring Organizations (COSO) of the Treadway Commission's Internal Control – Integrated Framework broadly defines internal control as "a process, effected by an entity's board of directors, management and other personnel, designed to provide reasonable assurance regarding the achievement of objectives in the following categories: efficiency and effectiveness of operations; reliability of financial reporting; and compliance with applicable laws and regulations."

At INFC, departmental officials developed a management control framework for the administration of PTIF/CWWF. This framework was approved by the Departmental Management Committee.

The control framework defines the major program risks, the controls in place to mitigate them, the expected outputs generated by the application of the controls, and accountability for each control. This information is available to all program personnel and is used to administer the program.

Audit Objectives and Scope

Audit Objective

The overall audit objective is to assure INFC senior management of the effectiveness of core controls which will strengthen the implementation of the PTIF/CWWF programs. 

This audit is being undertaken in three phases over the 3-year duration of the programs: Phase I (completed) assessed the agreement and project approval processes; Phase II (current) assesses governance, and Phase III (future) will assess payments and reporting.

Governance has been defined to refer to structures and processes that are designed to ensure accountability, transparency, responsiveness, rule of law, stability, equity and inclusiveness, empowerment, and broad-based participation. Governance also represents the norms, values and rules of the game through which public affairs are managed in a manner that is transparent, participatory, inclusive and responsive. Governance therefore can be subtle and may not be easily observable. In a broad sense, governance is about the culture and institutional environment in which citizens and stakeholders interact among themselves and participate in public affairs. It is more than the organs of the governmentFootnote 1.

For the governance phase of the audit, the objective is to assure INFC senior management that the governance structure in place for PTIF/CWWF Projects with respect to the agreements and risk management framework is adequate and effective.

Audit Scope

The scope of this audit assessed governance during the period of June 1, 2016 to March 31, 2017.

Audit criteria were sourced from the Terms and Conditions of the Treasury Board Submission and the Management Control Framework, and potential risks described next.

Risk Assessment

Potential Governance Risks

The table below presents the potential areas of risk identified during preliminary interviews and through a review of program-related documents. These are potential risks that may occur given the nature of the processes involved and the inherent operational conditions of the business. These potential risks are addressed in the audit program.

Potential Risk Events

Risk Level

Risk 1: There is a risk that oversight bodies have not been established or do not have a mandate that is clearly defined and implemented.

Medium

Risk 2: There is a risk that oversight bodies are not receiving sufficient information to make informed decisions.

Medium

Risk 3: There is a risk that the program will not be delivered within the defined timeframe or that funding will not be spent as intended if there is insufficient monitoring of the progress of projects.

Medium

Risk 4: There is a risk that unclear roles and responsibilities and a lack of communication with stakeholders could lead to non-alignment with program terms and conditions.

Medium

Risk 5: There is a risk that employees have insufficient guidelines, tools, procedures and training.

Medium

Governance Phase: Lines of Enquiry

This audit assessed the governance of the program to ensure that:

  • Committees are established in a timely manner and in accordance with program Terms and Conditions and Management Control Framework
  • Internal governance committees and oversight committees are effective in terms of risk management and issue resolution related to PTIF/CWWF
  • Formal and informal discussion and decision making are documented

Governance Phase: Approach

The audit included various tests, as considered necessary, to provide reasonable assurance on the management of the programs. These tests included, but were not limited to: interviews, observations, file walkthroughs, review of supporting documentation, attendance at key meetings (e.g., Oversight Committee, PTIF/CWWF (SQUAD) Committee, etc.), and analytical reviews.

Documentation was reviewed for a sample of the Oversight Committees. One jurisdiction was selected from each of the regions (British ColumbiaFootnote 2, Ontario1, Quebec1 and New Brunswick).

The scope also included INFC's PTIF-CWWF (SQUAD) Committee and oversight provided by the Program Implementation Committee (PIC) related to these programs. Excerpts from the Terms of Reference for these committees are found in Annex A.

The audit was conducted in accordance with the Treasury Board Policy on Internal Audit and the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing.

Obervations and Findings

Line of Enquiry 1: Committees Established

P/T Oversight Committees are established and meeting as intended. Internal Committees have been established and are monitoring the implementation of the PTIF and CWWF Programs, providing effective internal oversight.

Observations – Oversight Committees

  • Roles and responsibilities of program management, including Oversight Committees, are clearly defined in the Management Control Framework and were well-understood by interviewees
  • P/T Oversight Committees have been established in all sampled provinces, with terms of reference clearly defining each committee's mandate, roles and responsibilities and committee membership
    • Four of the seven P/T Oversight Committees reviewed were formed within the required 60 days of signing the bilateral agreement. The other three were formed after the 60-day deadline had passed:
      • British Columbia PTIF (70 days)
      • Quebec CWWF (84 days) and Quebec PTIF (109 days)
  • The current composition of all sample P/T Oversight Committees is consistent with the membership requirements defined in the committees’ terms of reference
  • All committees sampled have held their first meeting and have scheduled upcoming meetings according to the schedule defined in their terms of reference
  • INFC Directors also have regular, informal conversations with their P/T Co-Chairs and counterparts between scheduled Oversight Committee meetings to ensure that the programs are being implemented effectively.

Observations – PTIF-CWWF (SQUAD) Committee and Program Implementation committee

  • The PTIF-CWWF (SQUAD) Committee has been established and is providing a forum to effectively and consistently implement the PTIF and CWWF across the regions. Interviewees indicated that the SQUAD is one of the more effective working-level committees in the department.
  • The SQUAD is making program-level decisions based on issues and questions raised at committee meetings. The SQUAD regularly discusses program expenditures and financial issues, tools and guidelines, progress reports, and reporting deadlines.
  • Although they are not the sole focus of PIC, PTIF and CWWF are regular agenda items at PIC meetings. The PIC receives regular updates from the Chair of the SQUAD, discusses the SQUAD's decisions and any other issues raised, and when necessary, advises on how the how SQUAD should proceed.

Observations – Matrix Governance Structure

  • The matrix program governance structure supports regular, open and transparent communications between all staff levels. This is evident as Directors co-chair the P/T Oversight Committees and are members of the PIC, while Analysts have multiple opportunities to connect by attending the PTIF-CWWF (SQUAD) Committee and the P/T Oversight Committee meetings.
  • Strong channels of communication and working relationships have formed, both internally and externally. Interviewees indicated that there are ongoing and frequent communications to discuss questions, concerns and risks regarding the projects and the programs.
    • Program Analysts are in regular contact with each other and their P/T counterparts.
    • Program Analysts also have frequent discussions and opportunities to connect with their Directors to ensure the Directors are aware of any issues that come up.
  • An additional feature of governance that supported the implementation of the PTIF/CWWF was the bi-weekly calls that the Deputy Ministers of Infrastructure Canada held with their counterparts in each province and territory. These calls have served as an opportunity for the DM to raise key issues with their counterparts to ensure that actions were being taken to submit projects or progress reports, or to discuss any outstanding issues. These calls served as a "safety valve" in the governance structure to allow any issues to be discussed.

Line of Enquiry 2: Risk Management and Issue Resolution

Internal governance provides an effective forum to consistently operate and implement the PTIF and CWWF across the regions and make program-level decisions based on issues and questions raised at committee meetings.

Observations - Tools

  • There are tools in place to monitor and report on the implementation of PTIF and CWWF:
    • The Master Risk Tool is used by Analysts to assess and monitor individual project risks. Risks are re-assessed semi-annually within the tool using the progress reports submitted by the P/Ts.
    • PIMS is a database that acts as a central repository. It is mainly used by Analysts to enter and track program information including bi-lateral agreements, amendments that have been made, project lists, progress reports, claims, and payments made by provinces.

Additional tools such as the Open Data Portal, the decision tracker and the Infrastructure Funding Report (IFR) effectively support the monitoring and reporting of the PTIF-CWWF Programs.

Observations – Risk identification

  • High risk projects are formally reviewed semi-annually at P/T Oversight Committee meetings.
    • We observed that while INFC prepares a summary of the highest risk projects for discussion at the P/T Oversight Committee meetings, other P/Ts, such as BC, do their own summary. For the Atlantic region, we noted that there was an additional P/T Oversight Committee meeting held two months after the first Oversight Committee meeting to discuss the projects with highest risk.
    • The frequency and forums to discuss medium- and high-risk projects was established at the PTIF-CWWF (SQUAD) Committee. Oversight Committees are expected to meet semi-annually, at a minimum, to discuss medium- and high-risk projects. High-risk projects are expected to be discussed more frequently by the Oversight Committee through formal emails and teleconference meetings. It will be critical to monitor high-risk projects as they progress towards delivery.
  • When required, program risks and issues are escalated and discussed at the PTIF-CWWF (SQUAD) Committee and PIC meetings. Examples include recent discussions regarding the procurement of buses and the treatment of cost overruns on projects.

Observations – Reporting

  • The sampled provinces are using the provided templates for progress reports and outcomes reports and are therefore meeting basic requirements. A centralized reporting tracker is used to monitor the submission of progress and outcome reports.
  • Many reports were not submitted on time and are past due. The following statistics were reflected in the tracker (as of March 24, 2017):
 

Progress Reports

Outcomes Reports

Submitted on Time

4

3

Submitted Late

8

4

Past Due (Not Submitted)

9

13

Not Yet Due

4

5

TOTAL

25

25

Findings

  1. Timeliness of Reports - We noted that many of the progress and outcomes reports were received late, past the six-month requirement, or have not been received. As per POB, this could be the result of the project lists and progress reports having the same or overlapping due dates. In this case, the P/Ts have prioritized the allocation of funds over reporting requirements. P/Ts also indicated that their internal approval processes created delays in reporting. Although the competing timelines have weakened reporting to date, it is expected that upcoming reports will be more timely now that most or all projects have been identified. The timeliness of reporting will be critical to fully monitor program success and manage potential risks and issues.
  2. Reporting Detail (Progress Reports) – Based on a review of progress reports submitted by the sampled provinces, the level of detail supplied across the regions for completing the template is not consistent. Reports that have more information should be considered a best practice because they are easy to  understand and are a good example of what effective monitoring and reporting should look like.  
  3. Outcomes Reporting – Performance measurement strategies have been developed for both programs, defining the outputs, outcomes and associated key performance indicators to measure success. The performance indicators were established quickly, as bi-lateral agreements were being negotiated and signed. POB indicated that it is likely that P/Ts will not be able to provide measurements for some of the smaller municipalities that do not have the capacity to report on the performance indicators. There is a risk that incomplete performance information and inconsistencies between P/Ts will impede the ability to report on national performance.
  4. PIMS - Interviews indicated that the PIMS tool is perceived as cumbersome and concerns were raised about the duplication of effort and data integrity. Some of these issues result from business decisions that were made after the IM/IT consultation with the PTIF-CWWF (SQUAD) Committee, which are now viewed as impractical. For example:
    • PIMS is a system where Analysts record project information that is used for reporting and to manage funding. In order to support project or program data analysis, it is necessary to extract data from PIMS using IBM's Cognos Business Intelligence suite.
    • Some reports are available on a real-time basis and regular reports can be emailed automatically to Analysts if required. However, in order to produce a custom report, a report request must be sent to IM/IT which may require some time to be developed.
    • PIMS is not perceived as an efficient tool due to too many mandatory fields and business rules that are part of the design.
    • Currently, PIMS does not support an upload functionality, and Analysts are unable to copy and paste data from the spreadsheets received from the P/Ts.  Therefore, all information submitted by a P/T must be manually entered in PIMS.

Line of Enquiry 3: Documentation of Decision Making

P/T Oversight Committees, PTIF-CWWF (SQUAD) Committee and PIC all maintain clear records of decision for all committee meetings.

Observations

  • Regular informal discussions are occurring, and are documented.
  • The PTIF-CWWF (SQUAD) Committee maintains a decision tracker document that tracks all actions and decisions made during committee meetings. Additionally, the tracker shows if updates have been made to a decision and whether or not the decision has been approved by senior management (e.g., PIC). This allows the committee to track decisions throughout the life of the programs.
  • The decision tracker is used by the Chair of the SQUAD to update PIC on any issues and decisions that need to be made for the program.
  • Through our testing, we noted that decisions recorded in committee records of decision are included in the decision tracker, and that decisions in the decision tracker were documented as a record of decision.

Recommendations

Based on the key findings indicated in the report, we recommend:

  1. The ADM, POB should ensure that federal Co-Chairs of Oversight Committees make sure that future progress and outcomes reporting is timely, complete and sufficiently detailed.
  2. For future programs, performance indicators should be established with input from the provinces and territories, to ensure that data is available and reporting expectations are aligned throughout the lifecycle of the programs.
  3. Key performance issues with PIMS should be prioritized by IM/IT and POB and addressed as soon as possible. Improved communications between the business community and the PIMS SME team would ensure that ongoing investments in the system reflect the specific needs of all programs. As part of ongoing system improvements, lessons learned from programs, including PTIF and CWWF, on the usability and effectiveness of the system should be discussed within the business community and provided as requirements to the PIMS SME team.

Recommendations and Management Plan

Management Response and Action Plan

#

Recommendation

Management Response and Action Plan

OPI and Due Date

1

The ADM, POB should ensure that federal Co-Chairs of Oversight Committees make sure that future progress and outcomes reporting is timely, complete and sufficiently detailed.

Agreed. Management accepts this recommendation.

INFC has been working collaboratively with PTs to clarify the importance of providing timely, accurate and complete reporting. The Federal Co-Chairs of INFC will endeavor to work with his/her PT counterparts to ensure progress and outcomes reporting is timely, complete and sufficiently detailed. The Federal Co-Chairs will undertake the following actions:

  • They will ensure performance measurement and progress reporting is a standing item on Oversight Committee meeting agenda in order to address issues on an ongoing basis.
  • They will thoroughly review ongoing progress and outcomes reporting and work through the established Oversight Committee to address any deficiencies in reporting.

ADM POB targeted Completion date:

To be initiated immediately and completion for all Oversight Committee meetings by December 31, 2017

2

For future programs, performance indicators should be established, with input from the provinces and territories, to ensure that data is available and reporting expectations are aligned throughout the lifecycle of the programs.

Agreed, Management accepts this recommendation.

In preparing Phase II of the Investing in Canada plan INFC has consulted extensively with PTs. The Assistant Deputy Minister of the Program Operations Branch established a FPT committee specifically to seek input from PTs on performance indicators and data availability. In finalizing Phase II programs, INFC will ensure program outcomes are aligned with PT expected outcomes for infrastructure investments.

  • POB PI DG will ensure the development of indicators for outcomes that can be measured with available P/T data and can serve to report on programs' performance on an ongoing basis over the duration of the programs.
  • POB Regional DGs will ensure that all final signed agreements include applicable indicators as well as requirements for recipients to report on progress.

POB PI DG targeted Completion date:

December 31, 2017

POB Regional DGs targeted Completion date:

March 31, 2018

3
  1. Key performance issues with PIMS should be prioritized by IM/IT and POB and addressed as soon as possible.
  2. Improved communications between the business community and the PIMS SME team would ensure that ongoing investments in the system reflect the specific needs of all programs.
  3. As part of ongoing system improvements, lessons learned from programs, including PTIF and CWWF, on the usability and effectiveness of the system should be discussed within the business community and provided as requirements to the PIMS SME team.

Agreed. Management accepts these recommendations.

INFC will ensure that ongoing improvements to PIMS are based on continuous discussion with all system users. It must be understood that PIMS is a Financial and project/program management system. PIMS data is stored in a central database (IDW) where reporting is done using IBM Cognos Business Analytics

  1. IMIT and POB will work together to define the key performance issues with PIMS. Going forward, POB analysts will report all PIMS performance issues using contact functions available directly in the system and issues will be prioritized and resolved.
  2. and c.
    1. POB Regions, PI and IM/IT will work together to improve communication between system users and PIMS Subject Matter Expert Working group (SMEs). Furthermore, SME recommendations on system development related to program delivery will be presented to POB Program Implementation Committee (PIC) which will provide guidance and prioritize PIMS changes.
    2. The current governance around development of PIMS will be reviewed and recommendations will be made to HMC, to ensure that ongoing investments reflect the specific needs of all programs, the ongoing usability and effectiveness of the system. (This review will be conducted in the context of new programming for phase II and associated systems.)

POB PI DG and DG IM/IT targeted Completion date:

October 31, 2017

POB PI DG, and DG IM/IT targeted Completion date:

October 31, 2017

POB PI DG and DG IM/IT targeted Completion date:

March 31, 2018

Annexes

Annex A – PTIF/CWWF (SQUAD) and PIC - Terms of Reference

PTIF/CWWF (SQUAD) Committee

The PTIF/CWWF (SQUAD) Committee is established as a forum for providing advice and making decisions in a consistent manner. The Committee is mandated to meet at minimum once a month, and establishes priorities and tracks progress of activities through the PTIF/CWWF (SQUAD) Committee Work plan. The Committee is chaired by the Director, Program Operations for Ontario, and membership of the Committee consists of representatives from each POB Region, Aboriginal Consultation Environment Services, Horizontal Integration/Performance Measurement, Policy, Communications, and Corporate Services. The committee oversees the development, implementation, and maintenance of program policies, processes, tools and frameworks, while ensuring that stakeholder views are considered and/or reflected in the management of program implementation issues to generate consistency across regions. Recommendations to the PIC and advice on program priorities are provided as required. The forum is in place to effectively share best practices and lessons learned across programs and jurisdictions.

Program Implementation Committee (PIC)

The PIC provides program Directors General and Directors with a venue to horizontally discuss regional and program-specific issues and ensure that consistent program management practices are implemented across all programs. The committee is mandated to meet on a bi-weekly basis to identify issues requiring discussion and decision by the Program Committees as well as approve and endorse Program Committee priorities, issue briefs, policies and work plans, in line with the Decision Making Framework. Decisions and issues discussed at the PTIF CWWF Committee meetings are reported at PIC meetings. The Directors in Program Operations Branch who co-chair the P/T Oversight Committees are members of PIC.

Annex B – Interviews

Interviews were conducted with the following stakeholders:

Interviewee

Position Title

Chad Westmacott

Director, Program Operations, Ontario

Laura Di Paolo

Director General, Program Operations and Program Integration

Bogdan Makuc

Director, Governance and Reporting

Maxine Bilodeau

Director, Program Operations, West

Johanne Lafleur

Director, Program Operations, Atlantic

Maria Montilva

Program Analyst, Atlantic

Sheila Armstrong

Senior Policy Analyst, Ontario

Heidi Craswell

Policy Analyst, West

Elizabeth Doherty

Co-Chair, Ontario-Canada CWWF Oversight Committee

Tara Faganello

Co-Chair, BC-Canada CWWF Oversight Committee

Annex C – Documents Reviewed

  • PTIF-CWWF Management Control Framework and Terms and Conditions
  • For the sampled P/T Oversight Committees:
    • Terms of reference
    • Exchange letters
    • Agendas and records of decision
    • Project Lists
    • Progress and Outcomes Reports
  • PTIF-CWWF (SQUAD) Committee terms of reference, records of decision and decision tracker
  • PIC terms of reference and records of decision
  • Program tools and guidelines, including Analyst Guide, Progress and Outcomes Reports templates, Audit Plan Template, PIMS Guideline, Master Risk Assessment.

Annex D – Audit Criteria

Line of Enquiry

Audit Criteria

Line of Enquiry 1:

Committees are established in a timely manner and in accordance with program Terms and Conditions and Management Control Framework

1.1. A clear and effective governance structure is established and documented.

1.2. Provincial/Territorial-Canada Oversight Committees have been established and are monitoring for the effective implementation of the PTIF and CWWF Programs.

1.3. The PTIF/CWWF (SQUAD) Committee has been established and is monitoring for the effective implementation of the PTIF and CWWF Programs.

1.4. The Program Implementation Committee (PIC) continues to be an effective internal oversight body.

Line of Enquiry 2:

Effectiveness of the internal governance committees and oversight committees in terms in terms of risk management and issue resolution related to PTIF/CWWF

2.1 Senior management and oversight bodies receive sufficient and timely financial and non-financial information to effectively manage risks and monitor issue resolution throughout the delivery of the PTIF and CWWF Programs.

Line of Enquiry 3:

Documentation of formal and informal discussion and decision making

3.1. Senior management and oversight bodies maintain sufficient documentation to support decisions made through the delivery of the PTIF and CWWF Programs.

Footnotes

Footnote 1

http://www.unesco.org/new/en/education/themes/strengthening-education-systems/quality-framework/technical-notes/concept-of-governance/

Return to Footnote 1

Footnote 2

BC, Ontario and Quebec had separate Oversight Committees for each fund.

Return to Footnote 2

Date modified: