Table of Contents

• Executive Summary
• Background
• Audit Objective and Scope
• Audit Approach
• Audit Findings
• 4.1 Expression of Interest
• 4.2 Merit Assessment
• 4.3 Changes Following Project Approval
• 4.4 Projects Outside the Competitive Intake Process
• 4.5 Program Management Control Framework
• Conclusion and Recommendations
• Management Response and Action Plan

Executive Summary

The overall objective of this audit is to provide reasonable assurance on the effectiveness of the processes, controls and governance structure in place during the project intake and assessment phase of the Disaster Mitigation and Adaptation Fund (DMAF) program lifecycle to ensure that projects are assessed in accordance with the terms and conditions of the program and aligned with the expected results.

The audit covered the period between May 1, 2018 and September 30, 2019.

The Audit and Evaluation Branch (AEB) concluded the DMAF program has put in place processes, controls and a governance structure that ensured projects were assessed in accordance with the terms and conditions of the program and aligned with the expected results. The audit found project assessments were conducted in a fair and consistent manner, and projects submitted to the Minister for consideration were in accordance with the program terms and conditions.

While the audit did not find any deficiencies, it did note some areas for improvement in order to further the effectiveness and efficiency of the program:

1. It is recommended that, to manage the expected high demand of the next intake, the Assistant Deputy Minister responsible for DMAF assess how the eligibility criteria, merit criteria, and any potential filter(s) will ensure the best use of departmental resources available while continuing to align with program objectives. The application of additional filter(s) should be included in the program application documentation to ensure potential applicants are aware.
2. It is recommended that the Assistant Deputy Minister responsible for DMAF ADM, CRED finalize the update to the Fund’s management control framework, as well as related material, prior to the next intake.

Background

Budget 2017 announced $2 billion to establish the Disaster Mitigation and Adaptation Fund (DMAF) as part of the suite of programs under the Investing in Canada Infrastructure Program.

The DMAF is a new national, competitive, merit-based contribution program intended to support infrastructure projects designed to mitigate current and future climate-related risks and disasters triggered by natural hazards, such as floods, wildland fires, droughts, and seismic events. The DMAF aims to support large-scale disaster mitigation and adaptation projects with total eligible costs of at least $20 million.

The competitive, merit-based aspect of the program means that all projects are assessed against eligibility and merit criteria. Infrastructure Canada (INFC) makes recommendations for project approvals to the Minister of Infrastructure and Communities. The Minister has the authority to approve projects within an intake process, as well as outside of the competitive process for urgent and emergent situations.

The DMAF’s expected immediate outcome is to contribute to the construction or rehabilitation/modification of public infrastructure assets that will increase structural and natural infrastructure capacity to adapt to climate change impacts, disasters triggered by natural hazards, and extreme weather. Recipients, and in some cases Ultimate Recipients, will be accountable for achieving this immediate outcome.

The expected intermediate outcome of the DMAF (1 to 3 years post project completion), is that infrastructure investments contribute to increasing the ability of communities to adapt to and withstand climate change impacts, disasters triggered by natural hazards and extreme weather events.

The expected long-term outcome of the DMAF is to increase economic, environmental and social resilience, and that infrastructure is managed in a more sustainable way.

Eligible recipients:

The DMAF is managed through Contribution Agreements signed with successful applicants. Subject to specific conditions, eligible recipients include:

• A province, territory, or a municipal or regional government;
• A public sector body;
• A public or not-for-profit institution when working directly with a municipality;
• A private sector body, including for-profit organisations and not-for-profit organisations;
• A band council within the meaning of section 2 of The Indian Act;
• A First Nation, Inuit or Métis government or authority; and
• An Indigenous not-for-profit organisation.

Delegation of authority:

The Minister of Infrastructure and Communities has delegated authority to approve projects where the federal contribution is up to and including $50 million. Projects with a federal share of more than $50 million, or subject to other conditions (such as a project within the Minister’s riding) require Treasury Board approval.

Intake and project approval:

There are two possible streams for projects to be considered for funding: a competitive intake process, and urgent and emergent situations.

Under the competitive intake process, applicants apply directly to INFC for funding and projects compete to be selected based on their ranking relative to a number of merit criteria and other factors (e.g., number of projects per proponent). INFC staff conduct the eligibility and merit assessments which require a consistency verification amongst the DMAF analysts. To further support the assessment process and to ensure alignment with horizontal Federal priorities, INFC has sought considerations from advisory committees, namely an Other Government Department (OGD) Committee and an External Review Committee (ERC) to review departmental assessments prior to ministerial decision on project approvals.

The INFC assessment, in combination with the OGD committee and ERC review results, are taken into consideration by the Deputy Minister, who recommends projects to the Minister of Infrastructure and Communities. The Minister (or Treasury Board) approves projects under the DMAF in accordance with the delegation of authority.

Projects facing an urgent and emergent situation can be considered outside the competitive process, in consultation with the Minister of Public Safety and Emergency Preparedness Canada. Projects under this stream are assessed against the same eligibility and merit criteria as projects submitted through an intake process. The Deputy Minister then recommends projects to the Minister of Infrastructure and Communities. The Minister (or Treasury Board) also approves these projects, in accordance with the delegation of authority.

Program organisational structure:

Unlike most other INFC programs where the analysts administer multiple programs concurrently, the DMAF program is managed by a dedicated team of 12 employees, led by a Director. The program was initially part of the Program Operations Branch; however, in January 2020 there was a re-organisation of departmental responsibilities where the DMAF program was moved to the new Communities and Rural Economic Development (CRED) Branch.

Audit Objective and Scope

The overall objective of this audit was to provide reasonable assurance on the effectiveness of the processes, controls and governance structure that were in place during the project intake and assessment phase of the program lifecycle to ensure that projects were assessed in accordance with the terms and conditions of the DMAF program and aligned with the expected results.

The audit assessed the management of the DMAF’s project assessment process from May 1, 2018 to September 30, 2019, which captures the first competitive intake of the DMAF program, as well 21 projects assessed outside of the competitive process.

Audit Approach

This audit used a 'just-in-time' approach. This audit assessed the first competitive intake and project assessment phase of the DMAF program. Future audits will likely focus on program implementation and governance (monitoring, payments, and reporting), as well as any future intake processes. The timing and scope of upcoming audits will be determined during annual updates to the Audit and Evaluation Branch’s Integrated Audit and Evaluation Plan.

The audit included various tests, as considered necessary, to provide reasonable assurance on the sound management of the program. These tests included, but were not limited to, interviews, observations, walkthroughs, review of supporting documentation, attendance at key meetings, and analytical reviews. Further, the “just in time” auditing approach was leveraged to enable the audit to report on the subject matter within a shorter timeframe than under the traditional retrospective approach.

Our sampling included coverage of both approved and non approved projects in three distinct streams: those projects submitted at Expression of Interest (EOI), projects invited to submit a full application as part of the competitive intake, and projects assessed outside the competitive process. The total number of transactions is shown below:

DMAF Phase	# of Projects	Federal Share of Funding Requested ($ millions)
Submitted at Expression of Interest (EOI)	168	5,500
Eligible to submit a full application	134	4,400
Invited to submit a full application	99	3,000
Requested to submit at a future intake	35	1,400
Projects assessed outside the competitive process	21	602

Audit findings were communicated to the auditee to validate facts and to confirm the clarity, accuracy, and completeness of the information reported.

The audit engagement was conducted in accordance with the Treasury Board Policy on Internal Audit and the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing.

Audit Findings

4.1 Expression of Interest

There was an expectation that the Expression of Interest (EOI) assessments were conducted in a fair and consistent manner, in accordance with the program terms and conditions.

Lower Capacity Applicants

Project submissions under the DMAF were subject to a two-step assessment process. The first step was the EOI stage where the application outlined the proposed project and provided information for INFC to assess whether the project met the program eligibility criteria. Successful applicants were then invited to submit a full application, where an assessment of merit was completed.

During the first intake process, significant outreach activities were undertaken by DMAF staff, including providing applicants with guides that detailed program parameters and what was expected in the application forms, hosting interactive webinars led for potential applicants, promoting an email address for applicants to submit any questions, and offering an opportunity to request one-on-one calls with the DMAF staff for project-specific questions. The audit found many applicants took advantage of these tools.

The audit found that the assessment of projects during the EOI phase was done in fair and consistent manner.  However, there were some notable differences in the proportion of applications deemed ineligible depending on the type of applicant.

While applicants of all types had projects deemed ineligible, it was noted that certain types of recipients, such as small municipalities, not-for-profit organisations, and Indigenous communities, had a higher proportion of EOI applications deemed ineligible. Part of this difference could be attributed to applicants with more resources or prior experience with departmental programs (which is much more likely in the case of higher capacity applicants) being more conscious of what projects would be eligible and being more selective in which projects were put forward. 

Type of applicant	Applied	Ineligible	% ineligible
For-profit organisation	3	2	67%
Indigenous Community	19	7	37%
Not-for-profit	14	5	36%
Territory	7	2	29%
Municipality	89	15	17%
Province	36	3	8%
Total	168	34

To be clear, the audit did not examine the specific circumstances of each project/applicant deemed ineligible. 

The use of multiple outreach tools is positive, and indicates the DMAF program recognized potential limitations of some recipients and made efforts to address them. Management is encouraged to continue and consider expanding their outreach activities with potential applicants during the next competitive intake process, in particular for lower capacity applicants or those with less familiarity with federal contribution programs. As DMAF is a merit-based program, it is important that INFC continue its efforts to ensure the program remains accessible to all applicants.

Deferred projects

Prior to the launch of the EOI phase, INFC consulted with provinicial and territorial governments multiple times to get a sense of the volume and nature of requests that may be received under the DMAF. The level of demand for the program surpassed INFC’s expectations following those consultations.

Following the EOI assessment, 134 projects were deemed eligible, with a total federal share requested in excess of $4 billion. Given the onerous nature of the full application process for both the applicant to prepare, and for INFC staff to assess, a decision was made to add another filter which prioritized projects with earlier construction start dates. This decision resulted in 35 eligible applicants being deferred to a future intake for funding consideration.

Despite best planning efforts, programs do not always roll out as anticipated and it is not unreasonable for program management to make adjustments as a program evolves to deal with unforeseen circumstances. The audit found the decision to defer projects was well supported and presented a reasonable approach given the magnitude of program oversubscription. 

However, given the competitive and merit based nature of the DMAF, the use of additional filters during the first intake, while reasonable given the above noted circumstances, potentially led to some meritorious projects being deferred.

Why is this important?

Some applicants may have made a different decision to expend their resources in submitting an application had they known additional filters may be applied which could reduce their chance of immediate intake.

As it is anticipated any future intakes will also be oversubscribed, developing options on how additional filters may be used will increase the likelihood the next intake will also be managed in an effective and efficient manner for both applicants and the department.

4.2 Merit Assessment

There was an expectation that projects were assessed in a fair and consistent manner, in accordance with the program terms and conditions.

Applicants to the full process were provided with an electronic form to complete and submit, and asked also to submit any other documentation required to assess the project. The DMAF team used this information to assess the merit of projects and conduct a risk assessment for each project, using standardized tools. Following the assessments, each project was peer reviewed to ensure that assessments were conducted in a fair and consistent manner. 

Following the peer review confirmation, a standardized project fiche was created for each project that summarized the project, costs, intended outcomes, and the assessment results. The project fiche was shared with the OGD Committee and the ERC, who provided written feedback and considerations for each project. The project fiche was then updated with the relevant OGD and ERC comments and shared with the Deputy Minister, and ultimately with the Minister.

During the audit, the merit assessment process was repeated by the audit team for a sample of 12 projects. The audit confirmed that the merit assessment scores and the risk assessment results prepared by the DMAF team were fair and consistent with the established procedures. However, the audit noted that none of the merit criteria had minimum requirements and as such, projects cannot be screened out, which may result in the suboptimal allocation of funds. However, as previously mentioned, the level of demand for the program was higher than anticipated following consultations with provincial and territorial governments prior to the launch of the EOI phase. Given that the level of subscription to the program was higher than anticipated, a decision was made to not set a minimum for the merit criteria that could have been detrimental to the program.

Why is this important?

How the project application and selection process is designed is a delicate balance between openness, efficiency, and ensuring projects align with desired outcomes. While it is recognized that the use of merit criteria without minimum score requirements was a policy decision, the audit wanted to highlight this as an area for potential improvement for future intakes. The use of minimum requirements on more objective criteria is a best practice, supporting the selection of projects with stronger scores which can improve the efficiency of project assessments, and the maximization of outcomes.

4.3 Changes Following Project Approval

Project approval is based on an application prepared at a point in time. There is a risk that, following initial approval, the project could evolve and substantially change as the project is further refined.

For example, under the DMAF, projects are required to have a minimum of $20 million in eligible expenditures to meet the eligibility requirements. As most projects were submitted based on preliminary designs, the related cost estimates are subject to a relatively high level of fluctuation (this variation could be as high as +/- 20 to 30%).

The audit found that projects that were at or near the $20 million minimum, as part of the project assessment sufficiently highlighted the risk that total eligible costs may ultimately be below the $20 million minimum. There are existing measures in place to minimize the risk to INFC, such as the negotiation of, and various provisions within, the project’s contribution agreement. However, there is no clear guidance or procedures on how to deal with a project if estimated eligible expenditures fell below $20 million, or if the project had a significant change in scope.

Why is this important?

Project changes will invariably occur. The nature, scope, and size of those changes may have an impact on project eligibility. It is anticipated that any project facing this issue would require an analysis of the specific facts and circumstances to determine potential next steps.

This is an important area to monitor as projects move from approval and into implementation.

4.4 Projects Outside the Competitive Intake Process

There was an expectation that the delivery of the DMAF program would be done in a fair and consistent manner, in accordance with the program terms and conditions.

Under the DMAF, projects can be considered outside of the competitive intake process in cases where the Minister of Infrastructure and Communities, in consultation with the Minister of Public Safety and Emergency Preparedness Canada, identified a specific area of concern due to urgent and emergent situations. Projects meant to address such urgent or emergent situations and submitted outside of the competitive intake process are still required to meet the eligibility requirements and be assessed against the same merit criteria as projects submitted through the competitive process.

The audit found, in all cases, that the Minister of Infrastructure and Communities and the Minister of Public Safety and Emergency Preparedness Canada exchanged letters concurring that the projects submitted outside the competitive intake process were considered urgent and emergent situations.  

Projects outside the competitive intake process followed a similar assessment process to the one described as part of the merit assessment. It should be noted that those projects were not peer reviewed, nor were they reviewed by the OGD committee or the ERC. While the input gathered from these consultations may have provided some useful considerations, this likely would not have had an impact on project approvals, as the ERC and OGD committees do not make project recommendations. The audit also repeated the merit assessment process for a sample of six projects, and confirmed that the merit assessment scores awarded by the DMAF team were compiled in a fair and consistent manner, and in accordance with the program terms and conditions.

4.5 Program Management Control Framework

There was an expectation that there is a management control framework (MCF) in place, and that it considered the identification, assessment and mitigation of program risks.

In developing the methodology and merit criteria used in the DMAF program, consultants with specific expertise in climate adaptation were brought in to provide advice and guidance. In addition, other government departments were consulted, particularly Public Safety, whose merit-based National Disaster Mitigation Program was used as a baseline for the DMAF.

The audit found that the MCF should adequately mitigate the identified program risks and establish a control framework that would contribute to the achievement of program objectives. Notably, the governance structures described within the MCF took advantage of various existing committees and procedures to avoid the duplication of effort.

The MCF, by design, provides a high level overview of the practices and procedures. There is an opportunity for the MCF to better reference where to find more detailed information such as analyst guides, operational tools, or public documents like application guides.

It should be noted that the MCF was not approved prior to the first competitive intake process. However, the audit found the practices and procedures outlined within the MCF, including the identification and mitigation of risks, were followed despite the lack of formal approval.

Following the EOI and assessment phases of the competitive intake process, the DMAF team performed lessons learned exercises that identified processes that worked, as well as some opportunities for improvement for consideration prior to the next intake. This exercise considered tools, engagement with potential applicants, governance and information management, as well as some program design elements. Doing a lessons learned is considered a sound management practice and the findings can be used as a tool to review the MCF. The January 2020 departmental reorganisation also requires that the MCF be updated so that responsibilities and accountabilities reflect the new reporting structure.

Why is this important?

The management control framework is a core document that articulates the practices and controls deemed necessary to ensure the program achieves its objectives. Given the lessons learned following the first intake and the previously mentioned departmental reorganisation, there are sections of the existing MCF and related program material that need revision.

Conclusion and Recommendations

Conclusion                                                                                 

The DMAF program has put in place processes, controls and a governance structure that ensured projects were assessed in accordance with the terms and conditions of the program and aligned with the expected results. The audit found project assessments were conducted in a fair and consistent manner, and projects submitted to the Minister for consideration were in accordance with the program’s terms and conditions.

Recommendations

The audit did note some areas for improvement based on the key observations in the report to further the effectiveness and efficiency of the program:

1. It is recommended that, to manage the expected high demand, the Assistant Deputy Minister responsible for DMAF assess how the eligibility criteria, merit criteria, and any potential filter(s), will ensure the best use of resources available while continuing to align with program objectives. The application of additional filter(s) should be included in the program application documentation to ensure potential applicants are aware.
2. It is recommended that the Assistant Deputy Minister responsible for DMAF finalize the update to the DMAF management control framework, as well as related material, prior to the next intake.

Statement of Conformance

In my professional judgement as Chief Audit and Evaluation Executive, the audit conforms to the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing and the Government of Canada’s Policy on Internal Audit, as supported by the results of the Quality Assurance and Improvement Program.

 

____________________________

Isabelle Trépanier
Chief Audit and Evaluation Executive
Infrastructure Canada

Management Response and Action Plan

#	Recommendation	Management Response and Action Plan	OPI and Due Date
1	It is recommended that to manage the expected high demand, the ADM responsible for DMAF assess how the eligibility criteria, merit criteria, and any potential filter(s), will ensure the best use of resources available while continuing to align with program objectives. The application of additional filter(s) should be included in the program application documentation to ensure potential applicants are aware (Risk Category: Program design and delivery)	The Assistant Deputy Minister responsible for DMAF will ensure the criteria and any potential filters are in place to manage the expected high demand and to ensure the best use of resources. This information will be communicated to potential applicants through outreach activities, the website and the DMAF Applicant’s Guide.	ADM, CRED October 31, 2020
2	It is recommended that the ADM responsible for DMAF finalize the update to the DMAF management control framework, as well as related material, prior to the next intake.  (Risk Category: Business process risk)	The Assistant Deputy Minister responsible for DMAF will ensure the Management Control Framework (MCF) is updated before the next intake to reflect any changes to the program design.	ADM, CRED August 31, 2020