Combined Internal Audit and Evaluation of the Investing in Canada Infrastructure Program (ICIP)

Final report – April 19, 2024

Copyright

© His Majesty the King in Right of Canada, as represented by the Minister of the Office of Infrastructure of Canada, 2024.

Cat. No. T94-73/2024E-PDF
ISBN 978-0-660-73200-8

Table of contents

• Executive summary
• Program overview
• Objectives and scopes
• Conclusions and key findings
• Recommendations
• Annex A: Audit – Detailed findings and conclusions
• Annex B: Evaluation – Detailed findings and conclusions
• Annex C: Management action plan
• Annex D: Methodology and limitations
• Annex E: List of acronyms

Executive summary

Program background

The Investing in Canada Infrastructure Program (ICIP) is a large federal funding program that has undergone significant evolution since its launch in 2018.

• In 2019, ICIP went through a series of changes to streamline and introduce new program authorities.
• In mid-2020, as part of the Government of Canada's response to the COVID-19 pandemic, ICIP was amended with the addition of a new stream. Provinces and territories were able to transfer up to 10% of their original allocation to the new COVID-19 Resilience Infrastructure Stream (CVRIS) to fund eligible projects that originally had to be completed by the end of 2021 (2022 in the territories and in remote communities).

The deadline for provinces to send projects for approval through ICIP was accelerated by two years (originally March 31, 2025) to March 31, 2023, as announced through Budget 2022. Territories have until March 31, 2025, to fully commit funding to projects. INFC is currently developing the next generation of programming.

A combined audit and formative evaluation

The objective of the audit component was to provide reasonable assurance that the ICIP has established effective oversight and ongoing monitoring processes. The internal audit component included all four original streams (i.e., excluding the COVID-19 Resilience Infrastructure Stream^{Footnote 1}) for the period between April 1, 2019, and June 30, 2022.

The objective of the formative evaluation component was to support future program design by reviewing program design and delivery options, focusing on promising practices and lessons learned. It also reviewed and the extent to which the program gathered data to assess progress towards results and Gender-Based Analysis Plus (GBA Plus). The evaluation component looked at the four original streams (i.e., excluding the COVID-19 Resilience Infrastructure Stream) for the period between April 1, 2017, and March 31, 2022.

Overall key findings

The evaluation found that:

• ICIP contributed to the overall core infrastructure needs of Canadians and remains relevant in addressing Government of Canada's priorities and objectives.
• ICIP's design resulted in targeted communities and diverse populations receiving program funding.
• ICIP made progress towards measuring performance achieved at the immediate outcome level, with the opportunity to enhance the current Performance Measurement Strategy (PMS) to report on long-term outcomes.

The internal audit found that:

• While oversight and monitoring processes are in place and operating as intended, there is an opportunity to further improve their efficiency; and
• Lessons learned from ICIP should inform and contribute to departmental program harmonization efforts. In addition, the system-based controls introduced in ICIP should be leveraged as part of future system enhancements.

Conclusions

• The evaluation concluded that, to report on progress towards long-term outcomes and fully assess the impact of ICIP on targeted communities and diverse populations, INFC will need to undertake the planned refinement of ICIP's Performance Measurement Strategy.
• The Internal Audit concluded that, with reasonable assurance, INFC has established effective oversight and ongoing monitoring processes for managing ICIP. Some opportunities for improvement were identified related to improving the efficiency of some processes, notably risk management.

Recommendations

As the ICIP intake is now closed in provinces, the Audit and Evaluation Branch recommends the following:

1. The oversight processes for ICIP should be updated to improve efficiency as:
1. The program's focus will increasingly shift to project implementation issues, there is an opportunity to review and recalibrate the membership of Oversight Committees to ensure the members are best positioned to discuss the matters expected to be brought to the committee going forward; and
2. Monitoring processes continue to be revised to ensure reporting requirements are aligned to the associated risk level.
2. As the ICIP project intake is now closed in provinces, the performance measurement strategy should be updated to refine the objectives and requirements for intermediate and long-term results to ensure the availability of data and the feasibility of reporting progress toward expected relevant outcomes.

Leveraging ICIP for future programming

ICIP is the single largest program in INFC's history, funding a variety of different asset categories, with a wide range of project sizes. This makes ICIP an ideal program to take lessons learned in how the program was implemented to inform future program design, as well as department-wide harmonization efforts.

• Some notable program successes over previous programs include the availability of performance data and embedding of some system-based controls in processes due to the increased use of automated data-entry and linking between systems.
• There are also lessons learned about program design, GBA Plus and performance measurement strategies that can be leveraged for future programming, including:
  • Ensuring that future Performance Measurement Strategies should be designed to include the required information and performance data to support INFC in reporting on results, GBA Plus, national targets and any upcoming government-wide horizontal considerations such as sustainable development goals, quality of life framework, etc.
  • Developing targets and baselines for all expected outcomes as well as data sources and methodologies during program design.

Investing in Canada Infrastructure Program

Program overview

The Government of Canada (GoC) committed over $180 billion in public transit, green, social, trade and transportation, and rural and northern infrastructure through the Investing in Canada Plan.

As part of the Investing in Canada Plan, Budget 2017 provided details of $33 billion in funding initiatives under the Investing in Canada Infrastructure Program (ICIP) to be delivered by Infrastructure Canada (INFC).

ICIP provides long-term predictable funding and support for provincial, territorial, municipal and Indigenous priorities while remaining focused on the GoC's objectives of creating long term growth, improving the resilience of communities and transitioning to a clean growth economy, improving social inclusion, and improving economic outcomes of Canadians.

This allocation-based program is being delivered through Integrated Bilateral Agreements (IBAs) with provinces and territories (P/Ts). Negotiations with each P/T began in 2017 and culminated with the signing of IBAs with each jurisdiction throughout 2018.

ICIP funding streams

ICIP is divided into the following five funding streams:

• Public Transit to support new transit networks, service extensions and active transportation infrastructure that will transform the way that Canadians live, move and work.
• Green Infrastructure to support greenhouse gas emission (GHG) reductions; enable greater adaptation and resilience to the impacts of climate change and climate-related disaster mitigation and ensure that more communities can provide clean air and safe drinking water for their citizens. This stream includes three sub-streams: Climate Change Mitigation; Adaptation, Resilience and Disaster Mitigation; and Environmental Quality.
• Community, Culture and Recreation Infrastructure to support community, culture and recreation infrastructure and build stronger communities and improve social inclusion.
• Rural and Northern Communities Infrastructure to support projects that improve the quality of life in rural and northern communities by responding to rural and northern specific needs. The stream includes the $400 million Arctic Energy Fund to address energy security in the territories.
• COVID-19 Resilience Infrastructure, supported by a transfer of up to 10% of the original allocations under the above streams, supporting pandemic response and economic recovery efforts, as well as provide a national top-up to support ventilation improvements in public buildings.

Program evolution

ICIP is a large and complex program that has undergone significant evolution since it was first launched in 2018.

In mid-2020, as part of the GoC response to the COVID-19 pandemic, ICIP was amended with a new stream. P/Ts were able to transfer up to 10% of their original allocation to the COVID-19 Resilience Infrastructure Stream (CVRIS), to fund eligible projects that originally required to be completed by the end of 2021 (2022 in the territories and in remote communities).

During the later part of 2020, focus shifted to the negotiations and signing of amendments to each IBA to implement CVRIS changes.

• 2019 ICIP undergo a series of changes to streamline and introduce new program authorities based on provincial and territorial requests.
• In August 2020 the new COVID-19 Resilience Infrastructure Stream (CVRIS) was announced to respond to the immediate pressures and concerns as a result of the pandemic.
• Budget 2021 provided $120 million in new funding, specifically for ventilation projects, which would be delivered through CVRIS.
• In mid-2021, the required completion date for CVRIS projects was extended to March 31, 2023 (2024 in the territories and remote communities). Building on the $150 million to improve ventilation in public and community buildings announced in the 2020 Fall Economic Statement, the government provided as well $70 million over three years, starting in 2022-23, to support ventilation projects through CVRIS.
• Budget 2022 announced that the GoC would accelerate the deadline for provinces to submit projects for approval through ICIP to March 31, 2023 (from March 31, 2025). Territories may still submit projects until the March 31, 2025, deadline.

Application process

Each P/T established its own process to manage the intake process from ultimate recipients.

Once applications were received, each P/T jurisdiction did its own prioritization exercise. This allowed each P/T to design an intake process that met its needs, based on its own priorities and capacity.

Following the P/T-led intake, assessment and recommendation process, potential projects were submitted for INFC's consideration through the Infrastructure Recipient Information System (IRIS).

IRIS is a web-based application to manage the federal intake, capturing the information required by INFC to assess eligibility and meet federal requirements under the ICIP, such as:

• project tombstone data
• narrative description of the project
• summary of project financials
• outcomes and indicators applicable to the project
• information to support federal obligations, such as Indigenous consultation, or environmental assessment requirements.

Once submitted, the department reviews a project's eligibility, finances, risks, environmental and Indigenous considerations, and expected outcomes.

Projects must demonstrate alignment to at least one immediate-level result to be eligible. Projects must also provide relevant performance data to inform progress towards expected results.

ICIP status

As of December 31, 2023: 88.5% of the program's allocation has been committed^{Footnote 2} to over 6,000 approved projects.

Provinces have submitted projects toward their remaining allocations.

Figure 1: ICIP Status, As of December 31, 2023

Outcomes-based approach

An outcome-based program is a performance-oriented approach to fund programs or projects where funding is provided based on the achievement of predetermined outcomes or results, rather than the general outputs achieved by funding. To better understand whether programs deliver their intended results, the funding is aligned to meaningful and measurable outcomes or targets.

Whereas most INFC programs are funded based on the eligible asset categories, ICIP was INFC's first outcomes-based program designed to determine eligibility based on a project's ability to meet specific outcomes, as outlined in the Integrated Bilateral Agreement signed with each province and territory.

As part of ICIP's design for the outcomes-based approach, P/Ts were required to identify specific performance indicators for each immediate outcome at project submission. These indicators would be monitored through progress reporting and at the end of the projects. P/Ts were required to submit project-specific data that demonstrates a contribution toward immediate outcomes only.

Underlying the outcomes-based approach is a series of indicators meant to provide a clear mechanism to identify and report on progress toward key infrastructure results. P/Ts reported on indicators associated with the immediate outcomes. The longer-term outcomes will be collected through various data sources such as Statistics Canada and INFC's external data team to measure the intermediate and ultimate results.

Program outcomes

Immediate Outcomes: results that are directly attributable to the outputs of an organization, policy, program, or initiative. In terms of time frame and level, these are short-term outcomes, and usually relate to increased capacity, increased quality or access to the infrastructure asset.

Intermediate Outcomes: results that are expected to logically occur once one or more immediate outcomes have been achieved. In terms of time frame and level, these are medium-term outcomes, which are usually achieved by the end of a project/program and usually when there is a change of behaviour or practice level among beneficiaries.

Ultimate Outcomes: are the highest-level results that can be reasonably attributed to a policy, program, or initiative in a causal manner, and are the consequence of one or more intermediate outcomes. An ultimate outcome usually represents the raison d'être of a policy, program, or initiative, and takes the form of a sustainable change of state among beneficiaries.

Infrastructure Canada

The Communities and Infrastructure Programs (CIP) Branch within INFC is responsible for the administration of ICIP. Within the branch, ICIP is managed by two primary groups:

• The Regional Programs Operations and Engagement Directorate is the primary point of contact between INFC and the P/Ts, including engaging stakeholders, conducting project reviews and approvals, and providing project-level oversight and monitoring to ensure compliance with IBAs signed with each P/T. This Directorate is divided regionally, with members responsible for specific provinces and territories.
• The Program Integration Directorate supports regional operations, through the development of monitoring tools, frameworks, and guidance, providing training and advice, and ensuring program integrity. This directorate also includes the Environmental Assessment & Indigenous Consultation Division, responsible for the review and approval of environmental assessment requirements and Indigenous consultation obligations for all projects across funding programs.

In addition, groups across the department provide support in various areas, such as communications, finance, reporting and performance management.

Agreement management

The administration of ICIP is governed by the terms and conditions established in the IBA signed with each P/T. The IBAs have been amended multiple times since the program's inception to account for changes, such as the creation of the COVID-19 Resilience Infrastructure Stream.

As the IBA reinforces how ICIP operates, any changes to the terms and conditions require an amendment that is ratified before those changes can go into effect.

Oversight of the agreement is provided through Oversight Committees, which are co-chaired by a representative from Canada and the relevant P/T. Formal meetings occur, at least once a year, to monitor compliance with the terms and conditions of the IBA; act as a forum to resolve issues or address concerns; and review and recommend (as necessary) amendments to the agreement.

Objectives and scopes

Internal audit objective and scope

Objective

The objective of the internal audit component of this engagement was to provide reasonable assurance that ICIP had established effective oversight and ongoing monitoring processes.

Scope

The scope of the internal audit component of this engagement included all four original streams (excluding the COVID-19 Resilience Infrastructure Stream) for the period between April 1, 2019, and June 30, 2022.

Statement of conformance

The internal audit portion of this combined engagement conforms to the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing, as supported by the last results of the Quality Assurance and Improvement Program.

A detailed methodology and approach can be found in Annex D.

Audit criteria

1. ICIP is supported by effective governance and oversight processes.
2. Risk Management processes are in place and effective to manage program and project-level risks that might prevent the achievement of ICIP's expected outcomes.
3. ICIP's internal controls, including monitoring and reporting processes, have been established and are operating as intended to effectively support the delivery of the program.

Evaluation objectives and scope

Objective

The Evaluation engagement's objectives were to:

1. Support future program design by looking at a) program design and delivery options with particular attention to promising practices and lessons learned, and b) how a program can collect data to support progress towards results and GBA Plus.
2. Meet the legislative requirements of the Financial AdministrationAct (FAA) that requires an assessment of relevance, and effectiveness of all INFC grant contribution programs once every five years.

Scope

The ICIP evaluation covered the timeframe of April 1, 2017, to March 31, 2022, and focused on the four original streams of public transit, green infrastructure, community, culture and recreation and rural and northern communities. Excluded from this evaluation was the newest stream of ICIP, the COVID-19 Resilience Infrastructure Stream, as it will be evaluated separately within the next five years as outlined in INFC's IAEP 2021-22 to 2024-25.

Methodology

The evaluation used multiple lines of evidence to gather information in support of the findings. This included a review of program documents, literature review, program data, external data, survey with ultimate recipients and key informant interviews.

Evaluation questions

The evaluation questions were developed based on a discussion with program staff to support the department to inform future program design and delivery and meet legislative requirements.

1. Was ICIP responsive to changing needs/priorities?
2. What progress has been made towards ICIP's immediate outcomes as outlined in the program's logic model and performance measurement strategy?
3. Is ICIP on track to have the data to report on progress toward its intermediate and ultimate outcomes?
4. Are the national targets outlined in the Integrated Bilateral Agreement with each province and territory realistic/achievable?
5. Has the outcomes-based approach to ICIP implementation been effective?

Conclusions and key findings

Internal audit conclusion and summary of key findings

The audit concluded with reasonable assurance ICIP has established effective oversight and ongoing monitoring processes, with an opportunity to improve their efficiency.

Key Findings	Why is it important?
While oversight and monitoring processes are in place and operating as intended, there is an opportunity to further improve their efficiency.	Within ICIP, some monitoring processes are the same regardless of project size, complexity, or risk. For example, progress reporting requirements established in the IBAs are the same (i.e., semi-annually) for most ultimate recipients. Given the high volume of approved projects for funding, this does not lead to the most efficient use of resources, for both recipients and INFC, as project monitoring is a highly manual process. Monitoring efforts should therefore be aligned with each project's associated risk to maximize available resources. Given the scale of ICIP, both in terms of funding and number of projects, efficiently deploying resources is critical to ensure oversight and monitoring processes are completed in a timely manner.
Lessons learned from ICIP should inform and contribute to departmental program harmonization efforts. In addition, the system-based controls introduced in ICIP should be leveraged as part of future system enhancements.	As INFC matures as an organization, and the volume of active programs and projects increases, the need to standardize tools and processes is increasingly important to facilitate movement of employees between programs and ensure consistency. IT-based system controls are an essential part of a well-functioning system of internal controls, and these should be leveraged in any future system enhancements or replacement and minimize the use of paper-based controls. These efforts will allow INFC to better monitor the oversight and focus the efforts of limited resources on the most important program areas.

Evaluation conclusion and summary of key findings

The evaluation concluded that ICIP design and delivery were responsive to P/Ts' core infrastructure needs and aligned with Government of Canada priorities. Continued improvements are needed to ICIP's performance measurement strategy, and to future programs.

Key Findings	Why is it important?
ICIP contributed to the overall core infrastructure needs of Canadians and remains relevant in addressing the Government of Canada's priorities and objectives	ICIP's multiple streams and outcomes-based design aligned projects to federal government priority areas while also addressing some P/T priorities. ICIP funding helped the Government of Canada meeting its objectives to address areas of core infrastructure that were identified as needing investments.
ICIP's design resulted in targeted communities and diverse populations receiving program funding	As per the Treasury Board's Directive on Results (2016), evaluations are required to consider government-wide policy commitments, including GBA PlusFootnote 3. ICIP's Rural and Northern Communities and Community, Culture and Recreation Infrastructure streams were designed to target funding to communities and diverse populations, including Indigenous peoples, vulnerable populations, and official language minorities. However, at the time the program was launched, it was not a design requirement for ultimate recipients to gather and report on GBA Plus data. Without this information, the evaluation was unable to fully assess the direct benefits of ICIP funding to these diverse communities and populations. Given GBA Plus data is not being collected at the ultimate recipient level, INFC's Evaluation Team analyzed ICIP project location data with Canada 2021 Census Data. The analysis confirmed that communities and diverse populations, such as rural and northern communities' Indigenous populations received project funding. Looking forward, new programs should consider collecting GBA Plus data to fully assess the impact of program funding on targeted communities and diverse populations.
INFC designed a Performance Measurement Strategy (PMS) at the time ICIP was launched. As is normal practice, the PMS remains evergreen and data sources continue to be updated to reflect emerging data sets. The evaluation found that ICIP made progress towards measuring performance achieved at the immediate outcomes level, with the opportunity to enhance the current PMS to report on long term outcomes.	ICIP is INFC's first outcome-based program. As per ICIP's PMS, as part of each project approval submission, P/Ts were required to provide performance data on the project's anticipated outputs and immediate outcomes. Final project reports are provided by P/Ts, to confirm whether a project has achieved its intended immediate outcomes. The evaluation found that the initial data is available to measure progress towards immediate outcomes. ICIP's PMS also included intermediate and ultimate outcomes that are leveraging existing data from Statistics Canada and other federal departments. The evaluation found that some of the long-term outcomes do not have identified data sources. The program also identified national targets against which the program could measure progress towards outcomes if projects submitted were aligned. The evaluation found that not all the national targets remained relevant or had data, and not all targets had set baselines. Establishing baselines shows that a reference point is determined and will be used to measure progress to the expected results which are linked with the targets. Having clear and achievable targets is important to measure the extent to which a program is meeting its outcomes, allowing for course correction. At the time of the evaluation, the program management showed a planned update of the PMS. This is an opportunity to address key challenges, such as the relevance of performance indicators, baselines and targets, and feasibility of data collection.

Recommendations and considerations

Recommendations

1. We recommend that the oversight processes for ICIP be updated to improve efficiency.
1. The program's focus will increasingly shift to project implementation issues, there is an opportunity to review and recalibrate the membership of Oversight Committees to ensure the members are best positioned to discuss the matters expected to be brought to the committee going forward.
2. Monitoring processes continue to be revised to ensure reporting requirements are aligned to the associated risk(s).
2. As the ICIP project intake is now closed in provinces, the performance measurement strategy should be updated to refine the objectives and requirements for intermediate and long-term results to ensure the availability of data and the feasibility of reporting progress toward expected relevant outcomes.

Considerations for future programming

Going forward, INFC should ensure that lessons learned from ICIP are applied to future programs, including:

• Design and delivery: Ensuring that future programs have the required information and performance data to support INFC on reporting on results, GBA Plus, national targets and any upcoming government-wide horizontal considerations such as sustainable development goals, quality of life framework, etc.
• Performance measurement: Developing targets and baselines for all expected outcomes as well as data sources and methodologies during program design.
• Process harmonization and system controls: Ensuring that lessons learned are leveraged and inform the ongoing initiative of program management harmonization across INFC's suite of programs. Specifically, expanding upon existing IT-based system controls under ICIP will promote efficiency, reduce potential errors compared to paper-based controls, and provide a framework to inform future system enhancements (or replacement).

Annex A: Audit – Detailed findings and conclusions

Key Finding #1: Opportunity to improve the efficiency

Audit Criteria
ICIP is supported by effective governance and oversight processes.	ICIP's internal controls, including monitoring and reporting processes, have been established and operating as intended to effectively support the delivery of the program.
What is in place and working well?
Oversight committees have been established, are meeting regularly, and discussing required topics. Records of Discussion are kept, and there is evidence that action items are followed up. Both recipients and INFC staff have lauded the effective communication, and the willingness to raise and deal with issues at the working level as concerns arise.	There is an established process to review and approve project progress reports. Streamlining of the progress report approval process allowed approvals to be delegated to regional managers or regional program analysts from regional directors, allowing INFC management to focus on projects that needed more attention.
What is missing and why it's important?
The discussions taking place at the committee often focus on detailed project issues. These involve a level of knowledge that is more granular than the current co-chairs would be able to speak to, requiring both sides be accompanied by teams that can speak in detail about these topics.	Within ICIP, many processes are the same regardless of project size, complexity, or risk. While progress reporting is done through a system, it is still a highly manual process. Given the volume of projects, this is not an efficient use of resources for both recipients and INFC.
What should be in place?
As project intake is now closed in provinces, the topics of discussion at oversight committees will increasingly be focused on individual project implementation issues, and eventually project and program closure. The membership of committees should reflect the level of knowledge and authority expected to discharge the committee's mandate.	While some level of oversight of all projects is necessary, limited resources should focus efforts on projects that are areas of greatest risk (e.g., those with implementation challenges, and/or those with significant funding)

Governance and oversight

Oversight Committees (OC) have a formal role in monitoring the implementation of the IBA in each jurisdiction, and function as a formal forum for Federal and P/T officials to discuss and resolve issues related to ICIP.

Each OC has a term of reference that sets out committee membership and procedures. Each OC is co-chaired by two senior officials - one federal and one P/T representative - and is required to meet at least annually. Each OC is supported by working-level discussions that occur throughout the year.

The federal co-chair is currently at the ADM level, and in most cases the P/T co-chair is also an ADM (or equivalent).

During these meetings, the federal co-chairs are usually accompanied by the Regional Director General, Regional Director/Manager and/or regional teams to provide additional support on specific topics that may arise during a meeting.

The audit found that OC meetings were occurring as needed, that members or their delegates discussed the matters as required under the IBA and committee terms of reference, and there was some evidence that action items stemming from a committee meeting were later actioned.

Program lifecycle

Senior-level official participation made sense during the early life of ICIP, as discussions may have required decision makers that agree on a mutual understanding of foundational matters, and how each party was interpreting the IBA.

Once the foundational ground rules were established and agreed upon, meetings increasingly turned to more operational and project specific issues, requiring detailed or granular project level information, better suited to working-level members.

Now that project intake is closed in provinces, future meetings are likely to increasingly focus on project specific issues.

Both INFC and P/T representatives commented on the strong (informal) communication that happens regularly, and that there is a clear path to raise issues between formal OC meetings.

Potential for improving efficiency

Both INFC and P/T officials indicated that as OC meetings increasingly focus on project-level discussions, a significant amount of time and effort is required to prepare briefing materials for the co-chairs on the specific details of projects that may be discussed.

In most cases, working level staff are in frequent discussions (in some cases weekly) to address specific project issues.

Recommendation #1. a)

We recommend that the membership of Oversight Committees be recalibrated based on the program evolution allowing the appropriate level to discuss the matters to be brought to the committee going forward.

Internal controls – progress reporting

Progress Reports are a key internal control measure to ensure that funds are used for their intended purposes, and to confirm that the project aligns with program outcomes. They are required semi-annually for most ultimate recipients, and annually for ultimate recipients that are municipalities, regional governments, or Indigenous communities with a population below 5,000.

Progress reports must contain specific information, such as:

• cashflow updates
• confirmation that the project is on track to achieve expected results
• any risks and mitigation strategies, if required; and
• forecasted and actual construction start.

Under ICIP, progress reporting requirements are a standardised approach (except for the small communities' exception noted above). All projects are required to report on the same metrics, in the same level of detail and frequency, regardless of the project risks and materiality.

This approach significantly affects the workload of regional program analysts, who must spend considerable time on the monitoring of a high volume of low-risk/low=materiality projects.

The graph below shows the program breakdown of approved ICIP projects (as of April 1, 2023) by federal contribution.

Most approved projects have a federal contribution less than $3M, and in total represent 9% of the program's funding allocation. This is contrasted against projects with a federal contribution of more than $3M, which represents more than 90% of the program's funding allocation.

Approved ICIP projects by federal contribution show that there is a high proportion (83%) of projects accounting for only a small proportion of total program funding (as of April 1, 2023, excluding the COVID-19 Infrastructure stream)

Figure 2: Approved ICIP projects, As of April 1, 2023

With over 5,000 active ICIP projects to monitor, INFC has made efforts to streamline the progress reporting process, including:

• System enhancements to simplify some of the data entry and processing between systems
• Simplifying the review process for projects that meet certain conditions (e.g., low materiality)
• A tiered approval process, where program analysts and managers can approve progress reports, in certain circumstances.

Enhanced oversight

Certain projects - notably very large transit projects - have enhanced project monitoring that goes beyond the semi-annual reporting requirements of most projects. In addition to an increased reporting frequency, some of those projects also have dedicated oversight committees, or sub-committees, to focus on specific projects, or a sub-set of projects that would meet at various frequencies.

Potential for improving efficiency

Project monitoring is a highly manual process. Given the large volume of projects, efforts should be aligned with each project's associated risk.

Some streamlining of the oversight process has already happened, and it is recognized that ICIP's terms and conditions and IBAs place constraints on the ability to tailor oversight and monitoring processes. That said, wherever possible, processes should focus efforts on projects that are larger, more complex, and/or have high risk rating.

Recommendation #1. b):

We recommend that project monitoring processes be reviewed to ensure reporting requirements are aligned with the associated risk level.

Key Finding #2: Harmonization and system-based controls should be leveraged

Audit Criteria
Risk Management processes are in place and effective to manage program and project-level risks that might prevent the achievement of ICIP's expected outcomes.	ICIP's internal controls, including monitoring and reporting processes, have been established and operating as intended to effectively support the delivery of the program.
What is in place and working well?
Project risk assessment guidance is provided to staff through training and guidance documents that are readily available to program delivery staff on the intranet. There is a standard risk assessment template, which allows for consistent documentation.	ICIP introduced some system-based controls to flag projects that may be subject to payment conditions before payment can be issued to the recipient. The general claims process follows a similar process to other INFC programs, which are mature, and well documented. The audit found no significant issues with claims. Recipient audit plans had been established and were compliant with IBA requirements.
What is missing and why it's important?
INFC's current processes involve a significant amount of manual review and reconciliation, which increases the potential of human error. The program information management system does not keep a complete, reportable, audit trail of control overrides and justification.	Many monitoring processes, even when completed within the system, are highly manual processes that are subject to human error. Existing tools, such as recipient auditing, could be better used to focus the efforts of limited resources.
What should be in place?
As INFC matures as an organisation, and the volume of active programs and projects increases, system-based controls are an essential part of a well-functioning system of internal control.	The capability and functionality of payment controls, regarding their tracking and overriding, should be refined (and expanded to other INFC programs), to reduce potential errors, better track control effectiveness, and provide a more comprehensive audit trail. Processes and controls should be right sized to align to the risks that are being mitigated, as this will increase efficiency. There are ongoing efforts within the department to harmonize various processes across programs, which should continue and be automated to the extent possible.

Risk management

Risk management is an important process to ensure that program and project goals are met.

Program-level risks

A comprehensive Program Risk Profile for ICIP was developed in 2019. That document identified program risks and associated mitigation strategies to ensure successful program implementation. At the time of the audit being completed, the Program Risk Profile was being updated to reflect the program's lifecycle and was completed in September 2023.

In compliance with Treasury Board's Policy on Transfer Payments, a comprehensive Management Control Framework (MCF) for ICIP was developed. The original ICIP MCF was approved in 2019, with the expectation that it would be revised throughout the life of ICIP. The ICIP MCF was last updated in July 2021 and, at the time of the audit, was being updated.

Project-level risks

Under the four original ICIP steams, all projects undergo a comprehensive risk assessment as part of INFC's due diligence. Guidance documents exist that detail how to conduct a project risk (re)assessment, what factors should be considered, as well as prompts or examples that help promote consistency of application.

This audit noted improvement in the consistency in the risk assessment process for ICIP projects, compared to what was found in the audit of previous programs.

The audit confirmed that the project risk (re)assessments were completed in a manner consistent with the guidance.

Considerations for improvement

A mature risk management process would see risk mitigation measures (such as the detail and frequency of the monitoring regime) adjusted as program or project risks evolve.

While INFC has matured its risk management processes, they are still highly manual. There is also a lack of consistency between INFC programs, which adds complexity.

Integration between programs, systems and eventual process automation are large endeavours that require significant resources to design and implement.

To the extent possible, INFC should consider integrating some automation to the risk management process as it explores existing system enhancements or replacement.

Key program internal controls

Project changes

INFC has developed a comprehensive ICIP Project Change Policy, including a scope change assessment table and communication templates that guide program analysts when assessing project changes. These guidance materials facilitate consistent treatment between projects and across jurisdictions.

The audit found that the Project Change Policy was followed in all cases of major project changes in the sample of projects selected for further analysis.

System controls

Historically, INFC programs have been heavily reliant on manual control processes to ensure compliance with various control points, such as checklists that are reviewed and signed off at various stages.

Under ICIP, some system-based controls were implemented to flag conditions that may impede the ability to process a project claim.

These flags include:

• pending project change;
• outstanding progress reporting; or
• outstanding environmental assessment or indigenous consultation conditions.

Once these flags are triggered, the system will not allow a payment to be processed until the flag is cleared.

The flags were able to be overridden, but only through intentional action by regional analysts, and some conditions (such as those related to environmental assessment or Indigenous consultation conditions) could only be removed by users with a specific role and ability.

While control overrides are logged, and in some cases notes to file were kept to explain the reason, the system does not maintain a reportable audit trail of control overrides and justification.

Claims process

Program guides and procedures related to ICIP claims processing were developed, along with accompanying tools and templates. This guidance defines and supports the key players involved in claims processing.

The audit found all claims were processed following the established procedures, including documentation to support the required approvals, in accordance with the Financial Administration Act.

Potential for improvement

INFC continues to fine tune its internal controls, and ICIP saw the notable introduction of system- based controls to prevent project payments while certain conditions existed.

To the extent possible, INFC should explore expanding such controls to other programs, and consider expanding the capability/functionality of those controls as part of system enhancement or replacement.

Program harmonization

As INFC's mandate, and active program suite continues to expand, there is a growing need to standardise processes to improve quality and efficiency. Standardized processes also help system integration and automation, as business rules do not need to be customized for each program.

Recipient auditing

The Policy on Transfer Payments requires that departments implement cost- effective oversight and internal control systems. Oversight includes recipient monitoring, which includes recipient audits.

A recipient audit is an independent assessment to provide assurance on a recipient's compliance with a funding agreement. The scope of a recipient audit may address any or all financial and non-financial aspects of the funding agreement. These audits can be used to ensure that individual projects, and the program, are managed effectively.

Under ICIP, there are mandatory minimum requirements that recipient audits be conducted in each jurisdiction over the life of the program. The audit found that those minimum auditing requirements were being implemented, or planned, in each jurisdiction. However, the audit also noted that those recipient audits were largely being planned and conducted in isolation from other programs. Attempts have been made to group ICIP recipient audits with other programs to take advantage of efficiencies, however, given the nature and scale of ICIP, it is often not operationally possible.

The benefits of recipient auditing can be amplified when those audits are coordinated and implemented at the enterprise level. When considering the enterprise as a whole, lessons learned in a specific audit can be used by others. For example, an audit examining the project management and claims processes of a recipient may be useful to a different program interacting with the same recipient.

Harmonization

A concrete example of an attempt to harmonize program delivery processes at INFC is recipient auditing.

Recipient auditing plans were generally put on hold during the early phases of the pandemic, and INFC took advantage in the fall of 2020 to establish a Recipient Auditing Working Group to support the development of a department-wide framework. That work resulted in the approval of the INFC Recipient Auditing Framework in the fall of 2021.

Since the initial framework was developed, an increasing number of programs have been managed outside of the Communities and Infrastructure Programs Branch. At the time of the audit, responsibility for implementing the recipient auditing framework was being transitioned to a team within the Corporate Services Branch to ensure an enterprise approach.

Program advice, tools, harmonization and support

Within the Communities and Infrastructure Programs Branch, a team has been created with a focus on standardising and harmonizing processes across programs.

This team plays a crucial role in program governance to foster greater consistency and alignment across the branch and department, and increased effectiveness and capacity in program design and delivery through the following priorities:

• Harmonized and standardized program tools, requirements and processes
• Development and delivery of training & advice related to program requirements.
• Guidance to support analysts in the Treasury Board process for projects.
• Surge capacity to support delivery of priority files across the branch.
• Program analytics in support of effective program design and delivery.

To reduce program management complexity, administrative burden and improve oversight, INFC is taking a deliberate and systematic approach to standardize its program management practices while retaining the flexibility needed to meet the specific outcomes of each program.

Potential for improvement

As one of the largest programs at INFC, in both value and volume of projects, the lessons learned in the implementation of ICIP can be leveraged in improving the processes of existing programs, as well as informing the design of future programs.

INFC should continue to prioritise harmonization efforts, across regions, processes, and programs.

Internal audit conclusion

The audit concluded, with reasonable assurance, that ICIP has established effective oversight and ongoing monitoring processes.

The following opportunities for improvement were identified to improve the efficiency of existing ICIP oversight and monitoring processes:

• Review the membership of Oversight Committees to ensure the identified co-chairs and members are best positioned to discuss the matters expected to be brought to the committee going forward; and
• Review project monitoring processes to ensure reporting requirements are aligned to the associated risk(s) level (s).

In addition, the following opportunities for improvement were identified as considerations:

• Ensuring that lessons learned are leveraged and inform the ongoing initiative of program management harmonization across INFC's suite of programs. Specifically, this work will promote efficiency, reduce potential errors, and provide a framework to inform system enhancements (or replacement) to implement common system-based controls.

Annex B: Evaluation – Detailed findings and conclusions

Key Finding #1: ICIP met the overall core infrastructure needs of Canadians and remains relevant in addressing the Government of Canada's priorities.

ICIP contributed to the overall core infrastructure needs of Canadians as funding was disbursed targeting asset classes requiring investments.

ICIP's multiple streams and outcomes-based design components aligned projects to federal government priority areas while also addressing some provincial and territorial priorities.

Description of the core infrastructure needs in Canada

Figure 3: Percentage of core Infrastructure assets in Poor/Very Poor/Fair condition

Source: Canadian Infrastructure Report Card 2019

Very Poor: The asset is unfit for sustained service. It is near or beyond its expected service life and shows widespread signs of advanced deterioration. Some assets may be unusable.

Poor: There is an increasing potential for its condition to affect the service it provides. The asset is approaching the end of its service life, the condition is below the standard and a large portion of the system exhibits significant deterioration.

Fair: The asset requires attention. The asset shows signs of deterioration, and some elements exhibit deficiencies.

According to the Canadian Infrastructure Report Card 2019, about 27% of Canada's core public infrastructure i.e., roads and bridges; culture, recreation and sports facilities; potable water; wastewater; stormwater; public transit; and solid wastewater are in very poor, poor or fair conditions and needs investments as seen in Figure 3.

In addition, the Study on Addressing the Infrastructure Needs of Northern and Aboriginal Communities (2014) highlighted that people living in northern Canada, including the Indigenous communities face a significant infrastructure deficit exacerbated by a number of environmental factors, including remoteness, a brief construction season, and higher costs of building and maintaining infrastructure.

The critical infrastructure needs in northern communities includes local and regional roads, air and water transportation infrastructures; energy infrastructure, telecommunication, water, sewage, and solid waste management.

ICIP met the core infrastructure needs of Canadians

Figure 4: Total number of ICIP projects and approved funding by asset categories, April1, 2018 to March 31, 2022^{Footnote 4}

Source: Infrastructure Funding Report

The evaluation found that ICIP funding was aligned between areas of core infrastructure that were identified as very poor, poor or fair and needing investment in the 2019 Infrastructure Report Card (figure 4).

Furthermore, public transit received the most funding (approx. $13 billion) under ICIP given public transit was identified has a federal priority and usually the cost/project ratio for public transit is higher (more than $13B for 315 projects) compared to other assets, like recreation facilities (less than $1B for 608 projects).

The analysis of project funded showed that other asset categories that received significant funding (around $1 billion) included green energy, drinking water, wastewater and recreation, all areas where infrastructure was identified as needing investments.

ICIP funding was allocated to federal government priority areas

Table 1: Federal government objectives align with ICIP funding

Stream	Sub-stream	Objectives
Public Transit		Improved capacity of public transit infrastructure; Improved quality or safety of existing or future transit systems; and Improved access to a public transit system.
Green Infrastructure	Climate Change Mitigation:	Better capacity to manage more renewable energy; Improved access to clean energy transportation; More energy efficient buildings; and Improved production of clean energy.
	Adaptation, Resilience and Disaster Mitigation	Increased structural or natural capacity to adapt to climate change impacts, natural disasters, or extreme weather events.
	Environmental Quality	Upgraded wastewater treatment or collection infrastructure; Upgraded drinking water treatment and distribution infrastructure; and Better capacity to reduce or address soil or air pollutants.
Community, Culture and Recreation Infrastructure		Improved cultural infrastructure, like museums and Indigenous heritage centres; Supported upgrades to recreational facilities, like arenas and both indoor and outdoor recreational spaces; and Improved community infrastructure, like community centres and libraries.
Rural and Northern Communities Infrastructure	Rural and Northern Communities	Improved food security; Improved road, air or marine infrastructure; Improved broadband connectivity; Increased access to more efficient or reliable energy sources; and Improved education or health facilities (specific to the Truth and Reconciliation Commission's Calls to Action).
	Arctic Energy Fund	Supported energy security in communities in the North, including Indigenous communities, by investing in upgrades to existing fossil fuel-based energy systems, as well as supplementing or replacing these systems with renewable energy options—improving energy reliability and efficiency as well as reducing pollution.

Source: Infrastructure Canada - Investing in Canada Infrastructure Program

The review of program data, key informant interviews with provinces and territories and survey results with ultimate recipients illustrated that ICIP funding was allocated to projects in federal government priority areas.

Furthermore, the analysis of all provincial and territorial infrastructure plans (except for Nunavut where a plan was not available) illustrated that their priorities were aligned with ICIP streams.

Interviews with provinces and territories highlighted that extended eligibility criteria and amendments to bilateral agreements allowed P/Ts to better align their priorities.

However, more than 50% of the provinces and territories noted that their priorities didn't always align with the desired federal outcomes for eligible projects under the program.

ICIP allowed the Government of Canada to advance its objectives

Figure 5: Total number of ICIP projects and approved funding by asset categories, April1, 2018 to March 31, 2022^{Footnote 5}

Source: Infrastructure Funding Report - ICIP

The analysis of the number of approved projects and federal program contribution under ICIP's four different streams showed that projects were funded in areas that advance the Government of Canada's objectives.

Within the evaluation period (April 2018 – March 2022), the data showed that Green Infrastructure Stream (GIS) had the highest number of approved projects (1,175) and the second highest approved funding ($5.4 billion).

The Public Transit Infrastructure Stream (PTIS) has the highest funding allocation ($11.4 billion), with 437 approved projects (figure 5).

Key Finding #2: ICIP's design resulted in targeted communities and diverse populations receiving program funding

The Treasury Board's Directive on Results (2016) requires evaluations to consider government-wide policy commitments, including GBA Plus. The evaluation examined program results through an inclusiveness lens to identify potential areas to consider in the development of future infrastructure programming.

ICIP's Rural and Northern Communities Infrastructure Stream and Community, Culture and Recreation Infrastructure Stream were designed to target funding to rural and Northern communities and diverse populations including Indigenous peoples, vulnerable populations, and official language minorities. For instance, based on a comparison of ICIP project locations with Canada's 2021 Census, the evaluation found that the program funding to rural communities exceeded the share in the Canadian population while were only 1% less than their Canadian population share for Indigenous and Northern Communities.

The evaluation noted consultations with diverse population groups were not a requirement during the program design. Going forward, future programming should include the ability to integrate GBA Plus considerations to move beyond knowing which group received funding to understanding the impact of infrastructure programming on diverse groups.

Location data shows some positive outcomes for targeted communities and diverse populations

Table 2: Comparing ICIP investments by diverse groups to overall Canada share

Group	ICIP projects share (%)	Overall Canada share (%)
Indigenous Population (Indigenous Identity)Footnote 6	4%	5%
Remote Communities (Index of Remoteness)	2.2%	7.3%
Northern Communities (CSDs over 60 parallel latitude)	1.2%	2.2%
Rural Communities (1,000 or less in population)	35%	18%
Vulnerable (poverty rate based on market basket measure)	5.9%	8.1%
Population by first official language spokenFootnote 7	ICIP projects share (%)	Overall Canada share (%)
English-only population in Quebec	12.8%	13%
French-only population across Canada except Quebec	12.7%	13.5%

Sources: Data are from Census 2021 and program data on projects between 2018 and March 31, 2022, for all streams except CVRIS.

The program location data analysis shows some positive outcomes for groups, where ICIP project shares exceed or are very close to overall Canadian shares. This suggests that ICIP funding has reached a high percentage of these populations. For example, rural and northern communities benefitted greatly from investments under the Rural and Northern Communities Infrastructure Stream and Indigenous communities benefited greatly from investments under the Community, Culture and Recreation Infrastructure stream.

Key Finding #3: ICIP made progress towards measuring performance achieved at the immediate outcome level, with the opportunity to enhance the current Performance Measurement Strategy (PMS) to be better positioned to report on longer term outcomes.

INFC developed the ICIP PMS at the time of program launch. As is normal practice, the ICIP PMS is evergreen and continually is continually updated to address gaps and identify data sources, targets, and baselines, especially for intermediate and long-term outcomes.

As part of this evaluation, the PMS was reviewed to ensure that ICIP had set realistic/achievable performance indicators, baselines and targets, and that the program would have the data to be able to report on progress towards its immediate, intermediate, and ultimate outcomes. The evaluation was able to confirm that ICIP has early data from the applications to report on progress towards immediate outcomes.

The evaluation confirmed that some of the longer-term outcomes, both intermediate and ultimate, have data. It also revealed the need to update the PMS as some of the long-term outcomes do not have identified data sources.

INFC integrated national targets that are aligned with federal objectives into ICIP's indicators, The evaluation found that not all national targets remained relevant or had data. In addition, most of the outcomes do not have baselines or targets.

Having clear baselines^{Footnote 8} and achievable targets are important to measure the extent to which a program is meeting its outcomes, allowing for course correction, if required, as the program is being implemented.

It is recommended that INFC proceed with the planned refinement of the PMS to address the remaining gaps, such as the relevance of the indicators, baselines and targets, and the feasibility of data collection.

Current State on ICIP's performance data

The National Targets^{Footnote 9}

• Model share: Increase by 20% for public transit and active transportation
• Public transit: Maintain at least 96% as the % of people in a municipality with a transit system that live in the service area of their transit system
• GHG emissions: Contribute to reduction of GHG emissions of 10 megatonnes (Mt)
• Drinking water advisories: Reduce by 40% the number of long-term drinking water advisories in non-reserve communities
• Wastewater: Increase the number of wastewater systems achieving compliance with federal effluent regulations: 95% to 100% for high risk; 85% to 100% for medium risk
• Accessibility: 100% of federal-funded public facing infrastructure will meet the highest published applicable accessibility standard in a respective jurisdiction
• Broadband: Increase by 3% points to (93%) the number of rural households that have access to the highest broadband speed range available in their jurisdictions based on 2015 CRTC data
• Energy efficiency: Increase efficiency of electricity generation by 4% (kilowatt hour per litre/m3 of fuel used) in communities that are dependent on fossil-fuel for electricity generation

There are 83 indicators for immediate outcomes outlined in the PMS. P/Ts were to select which indicator(s) were relevant to their projects and submit performance data as part of their initial application.

• The evaluation found that initial data exists from project applications.Once all final projects reports are submitted by P/Ts^{Footnote 10}, it will be possible to tell the complete performance story at the immediate outcomes level.

ICIP has 12 indicators for intermediate outcomes and 15 indicators linked to ultimate outcomes against which INFC needs to gather data and set baselines and targets.

• At the time of the evaluation, the PMF leveraged existing data from Statistics Canada and other federal departments. The evaluation found that not all long-term outcomes had data against which to measure progress and very few had baselines and targets.

The program also set 8 national targets against which to measure progress towards outcomes.

• The evaluation looked at a sample of indicators, including those for the 8 national targets, to see how many had targets and baselines. Although it is common at this stage of PMS implementation, it found that not all outcomes have baselines or targets. Moreover, not all national targets remained relevant or had data.

Evaluation conclusion

The evaluation concluded that ICIP's program design and delivery were responsive to P/Ts' core infrastructure needs and aligned with Government of Canada priorities. Continued improvements are needed to ICIP's performance measurement framework, and to future programs.

ICIP was INFC's first outcomes-based program and set a foundation for future programming.

By incorporating lessons learned, leveraging ICIP to set improved, relevant indicators, and targets for future program design will allow for more informed decision- making and project selection for funding.

Conducting the evaluation at the end of the program intake provided an opportunity to apply lessons learned from the evaluation to inform future program designs:

• The need for a balanced approach to program design e.g., broad enough project eligibility criteria to accommodate the vast infrastructure needs across P/Ts in Canada, but narrow enough to meet and report on desired targets;
• During program design, establish performance targets and baselines along with clearly outlined data sources and methodologies to be able to report on progress towards all program outcomes;
• Conduct consultations with diverse groups including rural and Indigenous communities in program design including the performance measurement strategy; and
• Incorporate the collection of GBA+ data to target diverse groups for program funding to facilitate telling the performance story around the benefits of ICIP funding for diverse populations.

Annex C: Management action plan

Recommendation	Management Action Plan	Key Deliverables	OPI and due date
We recommend that the oversight processes for ICIP be updated with to improve efficiency. Given the ICIP intake is closed in the provinces, and the focus will increasingly shift to project implementation issues, there is an opportunity to review and recalibrate the membership of Oversight Committees to ensure the members are best positioned to discuss the matters expected to be brought to the committee going forward; and Monitoring processes continue to be revised to ensure reporting requirements are aligned to the associated risk(s).	The memberships for the ICIP OCs have been reviewed and a recommendation to delegate the federal co-chair role to the Director General (DG) Regional Program Operations & Engagement (RPOE) was approved on September 5, 2023. This decision is being implemented and communicated to provincial and territorial counterparts. Several measures to help streamline and facilitate monitoring and reporting processes under ICIP have been implemented. These include: Tools - Improvements to the progress report analysis tool and the development of a new tool with built-in logic checks. These checks make it easier for analysts to quickly find problems which must be reported and sent back to provinces and territories for correction; as well as identify items that may need further consideration or monitoring (e.g., forecasted construction dates on or close to deadlines). Reporting content - Streamlining the progress report content by removing data points collected through other means and by simplifying the outcomes and indicators section of non- final reports and keeping only relevant fields such as project finances, schedule and risk and mitigation strategies. Streamlining cashflow review - In keeping with the streamlined project approval process for projects under $10M in total eligible costs, cashflows for projects under $10M in total eligible costs that are submitted as part of the progress report are not reviewed by regional teams or entered into PIMS and are kept for reference only. Projects under $10M in total eligible costs represent less than 20% of total program contributions and have very little impact on the overall program cashflow.	Delegation of the membership to DG-level, as appropriate. In Spring 2023, Program Integration (PI) implemented a streamlined approach to progress reporting under ICIP, whereby May 2023 progress reports were deferred to November 2023, essentially moving projects with semi-annual reporting requirements and total eligible costs of $3M or less to annual reporting. This approach is in line with the risk-based approach for program delivery and reporting described in the program's Ts&Cs and has provided some respite in terms of progress report volume while ensuring that INFC continues to obtain updated information on most of its contribution funding and that regional teams remain effective and efficient. ICIP program managers adopt the principle of continuous improvement and are always looking for further areas of efficiency, streamlining and risk management. This work will continue for ICIP's duration.	PI and RPOE; September 30, 2023; Completed PI; Spring 2023 and ongoing; Completed
As the ICIP project intake is now closed in provinces, the performance measurement strategy should be updated to refine the objectives and requirements for intermediate and long-term results to ensure the availability of data and the feasibility of reporting progress toward expected relevant outcomes.	Agreed. The Communities and Infrastructure Programs Branch continues to maintain the evergreen ICIP performance measurement strategy and will proceed with the planned update by incorporating changes that refine the data and processes to support reporting towards ICIP's intermediate and long-term results.	Updated ICIP Performance Measurement Strategy	CIP September 2024

Annex D: Methodology and limitations

Internal audit methodology

In accordance with INFC's 2021-22 to 2025-26 approved Integrated Audit and Evaluation Plan (IAEP), the Internal Audit Directorate undertook the audit component of the Combined Internal Audit and Evaluation engagement of the Investing in Canada Infrastructure Program (ICIP).

Risk assessment

A risk-based approach was used to establish the objective, scope, and approach for this internal audit engagement.

Considering these risks, detailed audit criteria and sub-criteria were developed to guide the audit field work and form the basis for the overall engagement conclusion. The audit criteria were also derived from the Committee of Sponsoring Organizations of the Treadway Commission (COSO) Enterprise Risk Management Framework. COSO is an international organization dedicated to improving management performance by developing and sharing best practices in internal control, risk management, governance, and fraud deterrence.

Document review, interviews and walkthrough

The internal audit engagement included various tests, as considered necessary, to provide reasonable assurance on the overall engagement conclusion. These tests included, but were not limited to, interviews, walkthroughs, review of supporting documentation, and other audit procedures.

All project files were reviewed as part of the testing procedures. The field work was substantially completed on April 28, 2023.

Internal audit findings were communicated to the Office of Primary Information (OPI) to validate facts and to confirm the clarity, accuracy, and completeness of the information reported.

Population and sampling strategy

The population data was extracted on August 24, 2022, from a report containing a complete list of ICIP funded projects that were approved between April 1, 2019, and June 30, 2022.

That report included a total of 2,910 approved projects from the four original streams of ICIP.

A judgmental sample of 52 projects was selected.

The primary considerations were:

• projects with a risk rating of medium-high or higher
• projects where claims had been paid.

Secondary considerations were:

• funding stream
• region
• program contribution
• funding disbursed
• project-level risk.

Scope limitation

The project management system identifies projects that are currently subject to a condition that needs to be flagged (such as: pending project change, outstanding progress reporting, outstanding environmental or indigenous consultation conditions, etc.). The system does not provide reporting on projects that previously had flagged conditions.

Of particular interest to this engagement was the testing of controls related to projects with a pending project change.

We relied on our judgmental sample of 56 projects, where we conducted a manual search and identified 22 that had some previous project change.

Based on the selection method, the results of the project change process cannot be extrapolated.

Evaluation methodology

Lines of evidence

The evaluation used multiple lines of evidence to gather information in support of the findings. The analytical methods used for this evaluation were tailored to the nature of the data available. The evaluation design and level of effort were calibrated with available INFC resources.

Document review

The document review provided context for program changes between 2018 and 2022 and assessed the alignment between the program's theory of change/logic model as outlined in the PMS and the available data and examined different models of program design. Notable documents included:

• Treasury Board (TB) Submission,
• Terms and Conditions,
• Performance Measurement Strategy,
• Bilateral Agreements with provinces and territories, and
• external reports such as the 2021 and 2022 OAG reports.

To assess INFC, signatories and ultimate recipients' roles and responsibilities, as well as the progress towards expected outcomes. The document review also examined program documents from other INFC programs and evaluation reports and program documents from other programs under the Investing in Canada Plan to collect information on alternative program design and delivery models, and performance measurement strategies to support future program development.

Literature review

The literature review looked at academic literature on performance design, program design and delivery and data collection to assess the alignment between ICIP's theory of change, outcomes, and performance indicators. It also identified lessons learned and promising practices related to different program design and delivery approaches, namely the outcomes- based approach, and to data collection, particularly concerning programming for diverse groups (GBA Plus).

Interviews

58 key informant interviews were conducted to gather lived experiences and perspectives on ICIP's ability to be adaptable and flexible in addressing changing needs and priorities, the ongoing relevancy of existing national targets, and the effectiveness of the outcomes-based approach to ICIP's implementation. This included factors that facilitated or hindered, unexpected outcomes, and perspectives on alternate design and delivery models. Interviewees included INFC officials (27), representatives from other government departments, agencies, and crown corporations (9), and provincial and territorial representatives (22).

Program & external data review

The program and external data review examined INFC's internal program data available from IRIS and PIMS to assess progress towards outcomes and targets. External data from Statistics Canada, Canadian Urban Transit Association and Natural Resources Canada examined to assess if the department is on track to have the data needed to report on intermediate and ultimate outcomes as well as targets. In addition, both types of data were also used to assess the need to adjust the current eight targets or establish new ones.

Survey

A survey was administered to gather perspectives from ultimate recipients (e.g., local governments, private for-profit and not-for-profit organizations, transportation authorities and Indigenous entities) on the following elements:

• extent to which ICIP was flexible and adaptive in addressing changing needs/priorities;
• progress made towards providing programming for diverse groups (e.g. Indigenous, Official Language Minority Communities; people living in rural and remote areas; and vulnerable populations); and
• the effectiveness of program implementation, including factors that facilitated or hindered the approach, as well as
• unexpected outcomes.

From the list of 1,517 ultimate recipients, the survey was completed by 439 ultimate recipients, resulting in a response rate of 29% and an error rate of ± 3.9% (based on a 95% confidence interval). Representation by stream was consistent between the sample and completed surveys. Representation by province was consistent except for Newfoundland, where ultimate recipients were under-represented and Saskatchewan, where ultimate recipients were over-represented.

Evaluation limitations and mitigation strategies

• The document review had limited information on stakeholder perspectives. Interviews and surveys were conducted to mitigate the information gap and collect perspectives from provinces, territories and INFC staff (interviews) and ultimate recipients (surveys).
• The literature review evidence is limited by a lack of access to a comprehensive academic literature database. Upon completion of the document and literature reviews, we conducted a gap analysis to identify any areas lacking in sufficient information and conducted interviews to help gather information to fill these gaps, such as on different approaches to program design and delivery and on the outcomes-based approach more specifically.
• When conducting interviews, there is a risk that interviewees' responses will be subjective and contain biases. To ensure interviewee perspectives are captured as intended, information collected through interviews was validated by interviewees and balanced with data from the other lines of evidence.
• In terms of data, as infrastructure projects take time to build and impacts are observed a few years after construction ends, the evaluation assessed whether ICIP is on track to have the data to report on progress towards its intermediate and ultimate outcomes rather than to indicate that there are no data for outcomes that are too early to achieve. Data emerging from the Canadian Core Public Infrastructure Survey were largely unavailable as the data release timing was not coordinated with the evaluation and/or was delayed, thus these data could not be included in the evaluation.
• The evaluation team worked with the Contractor to design the survey accounting for accessibility, technological capacities, and cultural sensitivities to maximize survey responses. Generally, the ultimate recipient (UR) that completed the survey represent all the URs. However, URs from Newfoundland were somewhat under-represented and URs in Saskatchewan somewhat over-represented in the final tally of completed surveys. Representation by stream was fairly consistent between the original sample and those who completed the survey.
• The survey results do not include URs from Québec or the territories as neither chose to participate in the survey. Given the population in these regions, it is likely that impact on key groups (e.g., people living in northern or remote communities, people living in official language minority communities) may be understated in this report as a result. Additional questions were asked in interviews to representatives of the Territories and Québec to reduce the information gap.
• URs with more than one ICIP funded project (some of which were in different streams) were asked to consider their overall experiences in their survey responses. There is some potential for error when assessing results by stream if recipients had different experiences between their different projects. However, given that these recipients made up approximately 10% of the sample, their impact was likely minimal.

Annex E: List of acronyms

ADM

Assistant Deputy Minister

BN

Briefing Notes

CANCEA

Canadian Centre for Economic Analysis

CIP

Communities and Infrastructure Programs

CUTA

Canadian Urban Transit Association

DAC

Departmental Audit Committee

FTE

Full-Time Employee

GBA Plus

Gender-Based Analysis Plus

IAD

Internal Audit Directorate

INFC

Infrastructure Canada

IT

Information Technology

MCF

Management Control Framework

OPI

Office of Primary Interest

OC

Oversight Committee

PI

Program Integration

RA

Recipient Audit

RASCI

Responsible, Accountable, Supporting, Consulted and Informed

RPOE

Regional Program Operations & Engagement

TB

Treasury Board

Ts&Cs

Terms and Conditions

UR

Ultimate recipient

VfM

Value for Money