Evaluation of the New Building Canada Fund
July 2018
Audit and Evaluation Branch

PDF Version (Size: 591.44 KB)

Help on accessing alternative formats, such as PDF, PPT and ZIP files, can be obtained in the alternate format help section.

Executive Summary
Program Background
Evaluation Context
Major Findings

Relevance
Effectiveness (Design and Delivery)
Effectiveness (progress toward the achievement of outcomes)
Efficiency

Conclusions and Recommendations

Conclusion
Recommendations
Management Response and Action Plan

Annex A: Methodology

1.0 Executive Summary

Program Overview

Infrastructure Canada's New Building Canada Fund (NBCF) is one component of the Government of Canada's $53-billion New Building Canada Plan (NBCP). Since 2014, this $14-billion Fund has been supporting projects of national, regional and local significance that promote economic growth, job creation and productivity.

There are three sub-components under the NBCF:

The $4-billion National Infrastructure Component (NIC) provides funding for projects of national significance. It focuses on projects that have broad public benefits and that contribute to long-term economic growth and prosperity.
The $10-billion Provincial-Territorial Infrastructure Component (PTIC) supports infrastructure projects of national, regional and local significance. It contributes to economic growth, a clean environment and stronger communities. The PTIC is divided into:

$9 billion for National and Regional projects (PTIC–NRP); and
$1 billion for projects located in communities with fewer than 100,000 residents through the Small Communities Fund (PTIC–SCF).

Evaluation Objective and Scope

This evaluation covered a three-year period from the Fund's inception in 2014-2015 to 2016-2017. While the focus of this evaluation was on the design and implementation aspects, it also examined how effectively the NBCF delivered on its mandate.

Key Findings

Relevance

This evaluation found that the program is relevant, since NBCF addressed PTMs' evolving infrastructure needs by funding different types of projects through a variety of eligible categories. As well, amendments made to the NBCF program in 2016 provided further flexibility for PTMs.

Effectiveness (Design and Delivery)

There is concurrent programming between NBCF and programs in place before the NBCF was created. However, funding for the programs that preceded NBCF was, in a very large part, already allocated to projects being constructed before NBCF projects were approved.

To date, based on the number of approved projects, the impact of the 2016 program amendments of five additional categories has been limited. However, the change in the lowered volume requirements for the funding category of highways and roads has led to an increase in the number of approved projects.

NBCF's sub-components are very similar. This provides PTs with multiple options for funding a particular project. However, this also allows for a project to potentially be funded under multiple sub-components of different programs. Given that each sub-program has different application and project approval requirements, the NBCF design did not maximize effectiveness and efficiency. For example, 68% of projects for municipalities with less than 100,000 of population funded under NBCF-NRP were also eligible under NBCF-SCF. As the approval process for PTIC-NRP and PTIC-SCF are significantly different, i.e. business cases vs project list, this required additional operating resources for project proponents and for Infrastructure Canada for due diligence.

Effectiveness (Progress towards achievement of outcomes)

Given the limited data available at the early stage of implementation, it is difficult to determine whether the NBCF will achieve its outcomes. However, this evaluation did conclude that the projects approved under NIC and PTIC-NRP were supportive of the overall objectives of the NBCF program.

While improvements have been made over time, INFC continues to face challenges when trying to report on results. A lack of consistency in—as well as the number of performance indicators among the different programs and recipients—will continue to challenge INFC's ability to gather and evaluate performance data so it can fully evaluate the success of the program.

Efficiency

The administrative costs to deliver the NBCF program were lower than INFC's previous program. As well, project approval time decreased. However, it is not possible to determine the extent to which the planned resources aligned with the actual work. The NBCF funds almost all of INFC's activities but the costing and tracking of these resources was not the same. This limits INFC's ability to find ways to better allocate resources in future programs more efficiently or improve its decision-making abilities.

Recommendations

It is recommended that:

INFC should continue to explore ways to work with PTs to develop a standardized and consistent approach to the collection of performance measurement data.
INFC should review its approach to cost, capture and monitor information about resource allocation. This would allow the department to continue improving how it delivers programs and makes decisions. The level of effort to do so should be commensurate with internal capacity.

2.0 Program Background

New Building Canada Plan

In 2013, as part of the Budget, the Government of Canada announced a ten-year investment of $53 billion through the New Building Canada Plan (NBCP) to support infrastructure projects across the country. In 2014, the NBCP was approved and comprised of the following elements:

A New Building Canada Fund (NBCF) to support major economic infrastructure projects;
The Community Improvement Fund which provided a permanent source of funding for the Gas Tax Fund (GTF) and the incremental Goods and Services Tax Rebate for Municipalities Program. These initiatives fund roads, public transit and recreational facilities and other community infrastructure projects; and
A renewed Public-Private Partnership (P3) Canada Fund administered by PPP Canada.

New Building Canada Fund

The NBCF was allocated $14 billion to fund provincial, territorial and municipal infrastructure projects that supported the federal government's priorities of economic growth, job creation and productivity. It includes the following three sub-components:

The NBCF – NIC: a merit-based program with a total allocation of $4 billion to support projects of national significance. Its goal is to support projects that have broad public benefits, contribute to Canada's long-term economic growth and prosperity, and reduce potential economic disruptions or foregone economic activity.
The NBCF PTIC-NRP: an allocation-based program with a total allocation of $9 billion. Its goal is to provide funding to support primarily provincial, territorial or municipal infrastructure projects of national and regional significance that contribute to economic growth, a cleaner environment and stronger communities. The PTIC-NRP recognizes and supports the important role that PTMs play in building Canada's public infrastructure. Projects are allowing people and goods to move more freely, have increased Canada's potential for innovation and economic development, helped to improve the environment, and support stronger, safer communities.
The NBCF PTIC-SCF: an allocation-based program with a total allocation of $1 billion. It provides contribution funding for municipal infrastructure projects in small communities with populations of 100,000 or less as determined by Statistics Canada^Footnote1. Projects were selected based on their ability to move people and goods more freely, increase the potential for innovation and economic development, help to improve the environment and support stronger, safer communities.

New Building Canada Fund 2-16 Amendments

In 2016, NBCF underwent changes to the program design to align with the new government's objectives. In general, the amendments accelerated investments in communities and provided greater flexibility to PTs to commit all remaining funding by 2018-2019. The amendments included the:

addition of five new categories (culture, recreation, tourism, civic assets and municipal buildings, and passenger ferries) under PTIC-NRP and PTIC-SCF in the provinces (these were already eligible in the territories under the Northern Infrastructure category);
reduction of minimum traffic volume requirements for highway and road projects under PTIC-NRP and the removal of these requirements under PTIC-SCF;
new approach to federal reviews of PTIC-NRP projects; this reduced the information requirements and accelerated approvals for lower risk projects;
Ability of the Territories to transfer PTIC-NRP funds to PTIC-SCF; and
removal of P3 screening step.

Program Resources

Table 1 provides allocations for each NBCF program, the number of projects approved^Footnote2, the average project size, and the program contributions (as of March 31, 2017). As of April 1, 2016, Infrastructure Canada no longer accepted applications under the NIC

Table 1: Allocation of funds by programs, as of March 31^st, 2017

Programs	Budget Allocation ($000) (a)	Number of Projects (Approved) (b)	Program Contribution ($000) (c)	Average Project Size ($000) (d) = (c) ÷ (b)
NIC	$4,000,000	8	$1,647,998	$205,999
PTIC-NRP^Footnote3	$9,000,000	134	$4,238,052	$31,627
PTIC-SCF	$1,000,000	529	$867,360	$1,639
TOTAL	$14,000,000	671	$6,753,410	N/A

Table 2 shows how FTEs ^Footnote4 directly involved in program operations fluctuated between 2014-15 and 2016-2017.

Table 2: Number of direct program operations FTEs per sub-component

Fiscal Year	NIC	PTIC-NRP	PTIC-SCF
2014-2015	5.4	15.3	6.1
2015-2016	8.3	19.1	7.8
2016-2017	3.8	32.3	9.0
Average per program:	5.8	22.2	7.6

3.0 Evaluation Context

Objective and Scope

The objective of this evaluation was to assess the extent to which the NBCF addressed provincial, territorial and municipal (PTMs) infrastructure needs, the effectiveness of its design and delivery, and the efficiency of the program. While the focus of this evaluation was on the design and implementation aspects, it also sought to provide preliminary information on progress made toward achieving expected outcomes.

Methodology

This evaluation used a mix of qualitative and quantitative evidence such as:

document reviews;
administrative and financial data;
key informant interviews with INFC employees, other government departments and recipients;
comparative analysis; and
case file reviews.

The analytical methods used for this evaluation have been tailored to the nature and availability of the data in relation to the evaluation questions and the evaluation design, and effort has been calibrated with available INFC resources.

The multiple lines of evidence were triangulated to substantiate findings and to minimize potential bias.

For more information related to the evaluation matrix, including the evaluation questions, indicators and methods of data collection used, please refer to Annex A.

4.0 Major Findings

The following sections present the findings related to relevance, effectiveness and efficiency of the NBCF.

4.1 Relevance

This section addresses the issue of ongoing infrastructure needs. It examines the relationship between what INFC is funding and what stakeholders identified as their infrastructure priorities. It also examines whether or not there is duplication or complementarity between INFC's infrastructure programs and those of other government departments.

Finding 1: There is alignment between what is being funded by the NBCF and what stakeholders identified as their infrastructure priorities.

An analysis of PTs' priorities identified during the 2012-13 NBCF consultations^Footnote5 and program uptake pre- and post-2016 was undertaken. It revealed that the program's wide-range of categories provided the PT's with the flexibility they needed to address their changing needs. It also found that PTs' priorities have changed from the initial consultations to today. For instance, the eligible funding categories of public transit, waste water, drinking water, and culture have resulted in more PTs receiving funds than those who identified these categories as funding priorities during pre-program consultations. As well, innovation, disaster mitigation, tourism and brownfield remediation and redevelopment weren't identified as priorities in 2012-13 and yet PTs have been submitting project proposals for these areas.

Furthermore, the addition of the new 2016 funding categories of culture, tourism, recreation, civic assets/municipal buildings, and passenger ferries would suggest that the program's eligible categories are broad enough to meet PTs' changing priorities. PTs reported that the new funding categories provided more flexibility to address their jurisdictional priorities. For more information please refer to Annex C, Finding 1, which identifies changing priority funding categories by PTM.

A review of the Canadian Infrastructure Report Card (CIRC) report: Informing the Future: The Canadian Infrastructure Report Card ^Footnote6 provided an assessment of the health of municipal infrastructure as reported by cities and communities across Canada ^Footnote7. Survey results below demonstrate that roads, municipal buildings, sport and recreation facilities and public transit are the asset classes most in need of attention, as seen in Figure 1.

Figure 1: Summary of average physical condition rating

Text description of Figure 1

Source of data: Canadian Infrastructure Report Card 2016

The CIRC surveyed 120 municipalities (representing 56% of the Canadian population) asking them about their infrastructure priorities. As part of this evaluation, these results were compared against the projects approved through the NBCF. Although the CIRC study showed the areas of sport and recreation facilities needing attention, these were the areas where PTs received the least funding from INFC programs to date. It is worth noting, however, that these categories were added as eligible under NBCF in 2016 and, as such, this may evolve over time.

Finding 2: The NBCF and infrastructure programs managed by other government departments complimented one another.

This evaluation also looked at whether the NBCF and infrastructure programs in other government departments (OGDs) duplicated or complimented one another. Programs reviewed included:

Transport Canada's (TC) Gateways and Border Crossings Fund (GBCF) and Asia-Pacific Gateway and Corridor Transportation Infrastructure Fund (APGCI),
Indigenous and Northern Affairs Canada's (INAC) Transfer Payments for Infrastructure and Capacity
Canadian Heritage's (PCH) Canada Cultural Spaces Fund (CCS),
Natural Resources Canada (NRCAN) and Environment and Climate Change Canada (ECCC's) Green Municipal Fund (GMF),
Innovation, Science and Economic Development Canada's (ISED) Canada Foundation for Innovation (CFI) and Connecting Canadians Program, and
P3 Canada.

Most interviewees felt that that there was no overlap between NBCF and selected OGD infrastructure programs. In fact, they felt that their different objectives complimented each other. Some of the specific cases were mentioned during interviews are:

Several respondents saw some complementarity with Transport Canada (TC) and Innovation, Science and Economic Development Canada (ISED) programs.

For instance, the eligible project categories in TC's Gateways and Border Crossings Fund (GBCF) are comparable to NIC and PTIC-NRP. Similarly, the eligible categories under the Asia Pacific Gateway and Corridor Initiative (APGCI) are similar to PTIC-NRP. These two programs provide funding to both national and local/regional significant projects, as does NBCF. One difference, however, is that they both have an international component that enlarges their scope compared to NBCF. These two TC programs were approved prior to NBCF and were intended to be replaced by NIC.

Indigenous and Northern Affairs Canada's (INAC) Contributions to support Construction and Maintenance of Community Infrastructure, are similar to PTIC-SCF (wastewater, water, community building, community infrastructure services, drink water, etc.). INAC's program is targeted towards Indigenous recipients mainly on-reserve, while INFC's funding is targeted at municipalities, with the scope including Indigenous recipients.
PPP Canada has funded projects related to transit, highways, wastewater, green energy and solid waste management as does NBCF, but these projects must be delivered as public-private partnerships.

4.2 Effectiveness (Design and Delivery)

To assess the effectiveness of the NBCF's design and delivery, this evaluation considered:

whether it overlapped or complimented other INFC programs;
whether several amendments made to the NBCF program in 2016 impacted the effectiveness of its design and delivery; and
whether or not the different design elements for each of the three sub-components were being implemented as planned.

Finding 3: As a consequence of program life cycle, there is parallel programming between INFC's historic programs and NBCF.

This evaluation compared NBCF with INFC's historic programs such as the Building Canada Fund – Communities Component (BCF-CC) and the Building Canada Fund – Major Infrastructure Component (BCF-MIC).

Once approved, given their size or complexity, projects can take several years to build. As such, historically, new infrastructure programs are often created to provide additional funding to new projects before older programs are fully implemented (i.e. all the projects have been built/completed).

Not surprisingly, this evaluation found that there is parallel programming between INFC's historic programs and NBCF.

Specifically, it found that they both fund similar categories, the recipients who receive the funding are often the same, and that each generation of programs are about stronger communities and better and cleaner environment.

INFC's expected end dates for most of its historic programs is 2019-2020. NBCF's expected end date is 2023-2024. However, as per Figure 2, the majority of the available funding under INFC's historic programs such as BCF was allocated to specific projects before the NBCF program was launched in 2014-15.

Figure 2: Percent of Approved projects and Remaining Funding by Fiscal Year for Historic Programs and NBCF

Text description of Figure 2

Source of data: Infrastructure Canada's Program Database, as of March 31^st, 2017

Interviews with INFC's program officers, policy analysts and managers found that while INFC's historic programs are very similar in terms of eligible categories, there is no duplication. Interviewees spoke about parallel programming as an expected reality that comes with transitioning from older programs to newer ones. Differences in design and delivery between older and newer programs were also noted. For instance, historic programs were delivered through Regional Development Agencies.

Finding 4: Compared to INFC's historic programs, PTIC-SCF was slower to submit and approve funding particularly for water and wastewater.

The PTIF and CWWF programs were launched in 2016 to help accelerate municipal investments in the short-term, primarily for rehabilitation of public transit systems and community water and wastewater infrastructure. As such, significant additional funding became available for transit and water/wastewater projects. However, these categories were already eligible under the PTIC announced two years earlier. One of the main differences between them was that PTIF/CWWF had application deadlines, and projects were to be completed within a two or three-year period.

On the other hand, when PTIC was created in 2014, PTs had no deadline to submit their projects to INFC for consideration. (Note: a deadline was put in place in April 2016, but it was later than the PTIC/CWWF deadline). The PTIC is of longer duration, and typically for larger sized projects.

This evaluation examined how the launch of PTIF/CWWF impacted the roll out of PTIC. It did this by comparing the number of approved PTIC projects before and after PTIF/CWWF was launched.

PTs who were interviewed commented that they submitted project proposals to PTIF/CWWF before PTIC to ensure that they could benefit from the time-limited funding. This evaluation examined the PTIC program data for one year before (2015-16) and one year after (2016-17) the announcement of PTIF. Results of this analysis are shown in the table below:

Table 3: Number of approved projects under PTIC before and after the launch of PTIF/CWWF

PTIC
N/A	Pre-PTIF/CWWF (FY 2015-16)	Post-PTIF/CWWF (FY 2016-17)	Total
Public Transit	7	9	16
Drinking Water	126	51	177
Wastewater	143	37	180
Sub-Total	276	97	373

Also, a review of the number of approved projects for eligible funding categories other than those covered in PTIF/CWWF found an increase for PTIC-NRP from 32 projects to 91 projects. Most of this increase is for the highways and roads category. For PTIC-SCF, when considering all eligible categories, the decrease in water and wastewater projects was not offset by increases in other eligible categories. The review noted a decrease from 332 projects to 157 projects.

While it is not possible to say exactly why the number of approved projects under PTIC-NRP increased, it may be due to the fact that the program was maturing and therefore the ramp up time required for larger projects was reached. The decrease in overall projects under PTIC-SCF may be due to the fact that PTMs focused on securing funding under PTIF/CWWF given they were time-limited programs and small communities have limited capacity, as noted in interviews.

To further assess whether or not the introduction of PTIF/CWWF had an impact on PTIC, this evaluation conducted a comparative analysis of the remaining funding during the first three years^Footnote8 of both BCF-MIC and PTIC-NRP. The analysis revealed that at the end of the third year of the launch of BCF-MIC and PTIC-NRP, the proportion of the remaining funds for PTIC-NRP (72%) was significantly higher than that for BCF-MIC (33%) as seen in Figure 3.

Indeed, the launch of PTIF/CWWF could be a factor among others explaining why 72% of funds are remaining under PTIC-NRP during the program's third year (2016-17).

Figure 3: Percent of Remaining Funding for BCF-MIC and PTIC-NRP

Text description of Figure 3

Source of data: Infrastructure Canada Program Database, as of March 31^st, 2017

The differences in PT's uptake of NBCF when considering past programs (e.g., BCF) and new programs (e.g., PTIF and CWWF) demonstrates the importance of taking into account existing program life cycle when introducing new programming. Slower uptake may affect the timely achievement of NBCF outcomes.

Finding 5: As of March 31, 2017 only a limited number of projects were approved under the new categories made available for funding in 2016.

This evaluation examined the impact of several of the 2016 program amendments on the effectiveness of the program design and delivery. The amendments that were examined included the: 1) addition of five new categories under PTIC (culture, recreation, tourism, civic assets and municipal buildings, and passenger ferries; and 2) reduction of minimum traffic volume requirements for highway and road projects under NRP, and the removal of these requirements under SCF.

During interviews with PTs, it was mentioned that although the additional funding categories that were added following the 2016 amendments did not generally have an impact, they appreciated the additional flexibility that would have allowed them to address their regional/jurisdictional priorities. Some mentioned that they were unable to take advantage of the flexibility that the amendments provided as their funds were already approved. In other cases, PTs already had request for proposals underway and it was too late to make any changes.

An analysis of PTIC-NRP and PTIC-SCF program data^Footnote9 partially supported what we heard during our interviews with PTs. When we looked at the amount of funds still available for projects, it revealed that only some PTs had insufficient funds available to adjust or make additional proposals. Under PTIC-NRP, only two PTs had used all of their allocation funding and only one had only 2% remaining. Otherwise four PTs had between 13% and 46% and six had between 62% and 99% of their allocated funds remaining. For PTIC-SCF, four PTs had less than 1% of funding remaining while three had between 22% and 38% and six PTs had between 60% and 89% of funds remaining.^Footnote10 For those PTs that had funding remaining, as noted by some of the PTs interview, it is possible that their project identification process were too advanced to be able to include the new eligible categories. However, given the significant amount of funding remaining, it is possible that an increase in projects under the new eligible categories will materialize before all funding is committed. [Note: a total of 131 projects were approved in the culture, recreation and tourism categories from April 2017 to May 2018]

Finding 6: There was an increase of 79% in the number of approved highways and road projects after the PTIC-NRP and PTIC-SCF programs were amended in 2016.

As part of the 2016 program amendments, the NBCF's terms and conditions were revised to make it easier for PTs to apply for road and highway project funding. Under PTIC-NRP, the annual average daily traffic threshold was reduced from 10,000 to 1,000. It is believed this change helped increase the number of projects INFC was able to approve from 24 to 43. A detailed review of 15 business cases out of the 43 approved projects after 2016 found that the reduction in volume requirements actually provided flexibility, in that 5 of the 15 projects would not have been eligible under the previous threshold requirements.

For PTIC-SCF, the threshold was eliminated completely. A review of this sub-component's data illustrated a significant increase in the number of approved projects under highways and roads after the removal of the volume requirements. While it likely that the elimination increased the number of projects INFC was able to approve, PTs were no longer required to submit traffic volume data when submitting their projects for funding consideration.

Finding 7: Although the support provided by INFC to NIC and PTIC-NRP project proponents in developing business cases is good, the differentiating eligibility criteria between sub-components are few (e.g., population threshold), and some elements of the methodology by which business cases for NIC and PTIC-NRP are not-well defined (e.g., national significance not defined and business case assessment largely qualitative).

In terms of the effectiveness of the approval process, this evaluation found that there had been improvements noted in the development of business cases. Interviews with PTs and program staff indicated that it was easier for proponents to develop business cases for NBCF. Information about program eligibility requirements and guidelines for the application process that were made available on INFC's website were improved from the previous round of programs.

As detailed earlier in this report under Program Background, the NBCF was designed to have three different sub-components that would each focus on different areas of significance.

As outlined in Table 4, each one of the NBCF's sub-components had criteria for project eligibility and different assessment processes. For example, NIC was designed to be merit based and focus on federal priorities of national significance. NIC was also designed to fund large provincial, territorial and municipal infrastructure projects as well as other types of infrastructure typically excluded under INFC programs, such as ports. On the other hand, PTIC-NRP focused on provincial/territorial priorities that have national and regional significance, whereas PTIC-SCF focused on municipal projects in small communities.

Table 4: Project Eligibility Criteria and Assessment Process for NBCF's three sub-components

Key Criteria for Project Eligibility	NIC	PTIC-NRP	PTIC-SCF
Key Criteria for Project Eligibility	national significance contribute to Canada's long-term economic goals merit-based federal priorities "soft" threshold minimum project size of $100 million in total eligible costs	national or regional significance contribute to economic growth, a clean environment and strong communities allocation based projects prioritized by PTs medium to large in size	local significance contribute to economic growth, a clean environment and strong communities allocation based projects prioritized by PTs priorities small communities (less than a population of 100,000)
Assessment Process	require project proposals and business cases Aboriginal consultation and environmental assessments if required Reviewed by INFC's Program Review Panel (PRP)	require project proposals and business cases Aboriginal consultation and environmental assessments if required Reviewed by INFC's PRP	prioritized by PTs according to eligibility criteria provide INFC with the list of projects

This evaluation looked at the criteria for project eligibility and the assessment process for each sub-component. A number of areas for improvements have been identified.

For instance, the concept of "national significance" was not well defined in the Program's policy parameters or terms and conditions. According to interviews with both INFC's Program staff and PTs, a lack of a precise methodology and a clear definition led to confusion as to which projects fell under NIC and which ones fell under PTIC-NRP.

In order to remedy the situation, a guide was developed by INFC Staff to provide analysts with further direction. However, the direction in the guide was still vague and hard to quantify.

In terms of meeting NIC's objectives, projects must meet at least one of the four objectives to qualify under NIC. Three of these four objectives address PTIC's objective of supporting economic growth. Only the objective of “providing benefits that extend beyond the provinces or territories where the project would be located” is clearly national. As such, most projects eligible under NIC would also be eligible under NRP.

The program guide states that, "there is no single definitive methodology used generally by infrastructure project proponents to measure economic/productivity impact." A literature review conducted as part of this evaluation revealed that other jurisdictions had more rigorous methodologies when it came to assessing "national significance". They produced a more accurate and justifiable cost-benefit-analysis by monetizing costs and benefits.

A key difference between NIC and PTIC-NRP is that NIC was merit-based. This evaluation found, based on the above analysis, that the merit criteria used for NIC project reviews was limited given the review was qualitative in nature. Also, unlike typical merit-based programs, the projects were not compared/ranked.

A key difference between PTIC-NRP and PTIC-SCF is that PTIC-SCF was for small communities of less than 100,000 residents. All PTIC-SCF projects followed the criteria of having a population less than 100,000. For PTIC-NRP, 68% of the projects had a population of less than 100,000. Based on the above criteria of population size to define local significance, this evaluation found that the original program design concept of having two separate sub-components may not be as effective as possible.

Given that the submission requirements and approval process are different for the two sub-components, this may have notable resource implications for PTMs and INFC. The effectiveness of INFC's approval process for its own resources is described later in the report. Please refer to Annex A for more details on the approved projects under PTIC-NRP and PTIC-SCF.

4.3 Effectiveness (progress toward the achievement of outcomes)

This evaluation looked at the alignment between projects and NBCF's expected outcomes for NIC and PTIC-NRP projects. It was not possible to do the assessment on PTIC-SCF projects since most projects are not yet completed. For those that are, INFC has received limited performance data from PTMs. Given this, the evaluation also looked at challenges in gathering the performance data it needs to demonstrate that the NBCF is achieving its expected outcomes.

Finding 8: Under NIC, projects are expected to generate the positive economic activity for which they were designed.

The eight approved projects under NIC come from four provinces (British Columbia – 2, Alberta - 3, Québec - 2, and New Brunswick - 1). The province of Alberta received 38% of the projects and 50% of the total funding.

In general, NIC's funded projects meet the program's objective to contribute to "Canada's long-term economic growth and prosperity". Table 5 illustrates the expected total of jobs created for some NIC projects.

Table 5: NIC Projects and Expected Program Outcomes

Name of Project	Meet Expected Program Outcomes	Total expected of jobs creation (Employment)
Port of Montreal	Generate positive economic activity	2,500 jobs (direct and indirect) $340 million (additional annual economic activity)
Port of Saint-John	Generate positive economic activity Generate productivity gains for the Canadian/New Brunswick economy	6,300 jobs relating to port operation 10,870 total jobs, including jobs in engineering, construction and indirect employment
Fort McMurray Regional Airport	Generate positive economic activity	1,900 full-time equivalent (FTE) jobs (of which approximately 750 were employed directly at the airport site)
Yellowhead Trail Freeway Conversion Project	Generate positive economic activity	Creation of 6,000 direct, indirect and induced jobs in Alberta and 6,250 jobs in Canada as a whole over a 10 year design/construction period.
Southwest Calgary Ring Road project	Generate positive economic activity	The construction phase alone will generate nearly 12,000 person years of employment and $880 million of employment income.

Source of data: INFC project business cases during evaluation time frame

In terms of gathering data to demonstrate progress towards expected outcomes, targets and baselines are mentioned in the contribution agreements for projects submitted under NIC.

Findings 9: Under PTIC-NRP, approved projects are aligned with program objectives. It is difficult to predict the extent to which these projects tend to achieve the expected outcomes of the Program.

Most projects' expected results are in alignment with PTIC-NRP's immediate outcomes. Most funded projects aimed to contribute to at least one of the PTIC-NRP three intermediate outcomes.

According to INFC's Project Review Panel (PRP) documentation, there is a balanced distribution of projects: 56% are expected to contribute to increasing potential for innovation and economic development; 34% to help improve the environment; and 47% to support stronger and safer communities.

However, with only about 2% of projects completed (2 out of 97) as at March 31, 2017, it is too early to determine how well they contributed to the NBCF's expected outcomes.

Finding 10: Challenges exist with regards to INFC being able to collect performance information and report on progress as indicators are not consistent between projexts, and eat PT has its own method of collecting data.

According to INFC staff, the collection of data on results is a challenge as the department is reliant on PTs to provide performance information. Interviewees identified no consistency between project indicators. This poses a challenge when trying to draw conclusions about the program's overall results, especially as INFC moves towards focusing more on outcomes.

In a past engagement on performance measurement, INFC's Evaluation Services also found that NBCF's performance data strategy was designed to collect the specific outcomes of funded projects. As there are different indicators and methods to gather data for the same expected results, consolidation from these indicators to form conclusions on outcomes at the program level represents a real challenge.

That said, as the programs under the Investing in Canada Plan demonstrate, there have been significant efforts to design performance indicators for new programs that are easier to collect and are more aligned to program outcomes.

4.4 Efficiency

This evaluation looked at the time between submission of project proposals and the recommendation to approve projects as well as the Full-Time Equivalents (FTEs) and administrative costs of each sub-component.

Finding 11: The time between the submission of projects by recipients and when it is reviewed by INFC's PRP has been reduced since the NBCF was amended in 2016.

In order to assess program efficiency, this evaluation looked at the time between submission of project proposals and the recommendation to approve projects^Footnote11. The analysis covers the period from April 1, 2014 to March 31, 2017. During this period, there was an organizational change and as a result, Policy's responsibilities relating to project approvals were transferred to the Program Operations Branch. This evaluation also looked at how the 2016 amendments to remove the Public-Private Partnership (P3) screen and adopt a risk-based approach for reviewing PTIC-NRP projects impacted the program's efficiency.

A comparative analysis found that under PTIC-NRP the average project approval was reduced from 217 working days prior to the implementation of these changes to 86 working days post-implementation. This represents a reduction of almost 60%. In addition, the median time spent went from 198 working days to 77 working days, which represents a reduction of about 61%.

In terms of PTIC-SCF, because Provinces submit prioritized project lists, the amendments did not have a significant impact on the duration of the project approval. Average approval time of the PTIC-SCF projects before and after the implementation of these changes decreased from 23 working days to 21 working days.

In response to Blueprint 2020 and a red tape reduction exercise, INFC made concerted efforts to train staff in Lean continuous improvement. INFC targeted project review and approvals as well as payment processes. As well, it dedicated staff to implement working-level, day-to-day improvements to make processes more efficient. While this report cannot be definitive, these investments likely made the delivery of INFC programs more efficient, including NBCF.

INFC analysts and managers who were interviewed commented that the removal of the P3 screen has made things faster, except for PTs who have their own P3 requirements. The program has limited information on the time and level of effort required for each project's approval (such as the application date, when approval in principal was received, and contribution agreement signed date). As well, since the programs keeps limited notes on issues or challenges faced during the approval process, it is not possible to confirm the specific impact that removing the P3 screen may have had on the program's efficiency—other than noting an improvement in timelines.

In terms of using a risk-based approach for PTIC-NRP projects, this evaluation was unable to assess the impact of this amendment on program efficiency, as it was never fully implemented as envisaged.

In addition, INFC staff interviewed commented on delays in obtaining project approval as a result of underdeveloped or incomplete applications submitted. This caused a lot of back and forth with proponents to get the needed information. As noted earlier, due to a lack of documentation in terms of project status and resource utilization, it is not possible to identify specific areas of efficiencies and challenges that could be addressed.

Finding 12: Given NBCF's operating resources are allocated not only for NBCF but for the vast majority of other departmental activities, it is not possible to assess whether the resources allocated to NBCF aligned with actual spending.

Most federal departments rely on on-going, permanent funding known as "A-base" funding to meet their normal day-to-day obligations. For example, human resources (HR) management, technology, policy support, etc. And if they are given a new contribution program to manage on top of their usual mandate, they would also receive additional funding to manage that program. This type of funding is known as "B-base" and is usually 3-5% of the value of the new program. However, this is not how it works at INFC.

Since its inception, INFC has had virtually no A-base funding. Its operating budget is almost all funded by carving-out funds from the contribution programs it manages^Footnote12.

The majority of INFC's operating funding was set to end on March 31, 2014. Instead of receiving A-base funding, the department sourced its operating needs for the next 10 years (2014-15 to 2023-24) from the NBCF (with the exception of the resources set aside for bridges), based on a costing exercise completed in 2013. This meant that the NBCF—INFC's largest grants and contribution program at that time—was used to support operating funding needs not only of the NBCF, but also for the basic departmental activities of all of its programs.

In speaking with program staff and senior management, this approach created a number of challenges in terms of tracking FTEs. Program documentation recognized that this ad-hoc approach to operational funding "has created significant challenges in terms of efficient departmental business and HR planning."

The 2013 departmental costing exercise did not break down its resource needs by each of INFC's contribution programs. Instead, it identified each branches' FTE and operating cost needs. In addition, actual resource utilization tracking methodology is not aligned to contribution programs. While in Program Operations Branch (POB), resource utilization is largely based on the managers' estimation of the time each employee spends on each contribution program, in other branches—like Human Resources and Information Management/Information Technology—each employee's time and operating resources are generally not aligned with a specific program(s).

As such, it is not possible to assess the extent to which planned NBCF funding associated with the contribution program was actually spent on the contribution program versus ongoing operating requirements (e.g., A-Base). The reliability of administrative data collected at a program level is not unique to NBCF. It has been documented as an issue within INFC in past evaluations and advisory engagements. In particular, the CSIF-BIF joint evaluation (2014), recommended that, "in order to assess the efficiency of program delivery, INFC should establish a pilot-project to develop and test efficiency indicators."

To that end, INFC developed and tested efficiency indicators. However, INFC was not able to implement them because of the high costs and level of effort needed to collect the data. In 2014, INFC also used an efficiency ratio to estimate its ten-year, departmental-wide operating demand. However, this ratio did not provide details on resource needs at the various stages of each program.

A 2016 GIF evaluation recommended that, "in order to facilitate reporting on resource utilization and alignment and to support future program planning that INFC reinforce its approach to tracking the utilization of FTEs and administrative costs per program in the financial system to ensure that managers are aware of their responsibilities to provide reliable data and information at the program level." Management's response and action plan included three activities. Only the first activity was implemented in the period covered by this evaluation. INFC's Financial Management Advisors provided reminders and detailed procedures to fund centre managers asking them to include one or more functional area codes for each employee in the salary resource management system.

Despite progress made in forecasting FTE utilization, the analysis of resource utilization for evaluation and efficiency purposes still remains an issue, especially as planning and tracking methodology differ.

Finding 13: PTIC is being delivered more efficiently than BCF since it is being delivered solely by INFC.

A comparative analysis of PTIC-NRP and PTIC-SCF with similar INFC historic programs (i.e. BCF-MIC and BCF-CC) was conducted. These programs were selected based on their similarities: all programs supported projects with national and/or regional significance, and the funding approaches all allocated funds to PT's. For further information on the similarities and differences between the above-mentioned programs, please refer to Annex C, Table 9.

Our analysis of the efficiency of the program's delivery is based on a methodology used in previous evaluations^Footnote13. As seen in Table 6, the PTIC-NRP efficiency ratio to deliver the program in terms of a dollar of federal contribution funding is slightly higher than BCF-MIC. The BCF-MIC included an arrangement with Transport Canada (TC) to administer projects on the department's behalf. The total amount transferred to TC was $30,698,000. INFC does not have information about the actual expenditures incurred by TC in this regard and, as such, this is not accounted for in the efficiency ratio. Given the similar efficiency ratio of the two programs solely based on INFC resources, clearly the delivery mechanism for PTIC-NRP is more efficient then BCF-MIC.

Table 6: Efficiency ratios for BCF-MIC, PTIC-NRP, BCF-CC and PTIC-SCF- Three first-year program cost delivery

Period^Footnote14	Programs for projects with National/Regional significance		Programs for projects within small communities
Period^Footnote14	BCF-MIC^Footnote15	PTIC-NRP	BCF-CC^Footnote16	PTIC-SCF
Months 1-12	0.04	0.08	0.10	NA
Months 13-24	0.02	0.02	0.02	0.04
Months 25-36	0.02	0.01	0.03

For PTIC-SCF, this evaluation found that the 3-year average cost to deliver one dollar of contribution funding is two cents, which is lower than BCF-CC (five cents), a comparable program, as seen in Table 6.

The substantial difference in the efficiency ratio per project could be explained by the more complex delivery model associated with the BCF-CC. The PTIC-SCF consists of a list of projects with limited data that are submitted by PTs for INFC analysis and recommendation to the Minister. The BCF-CC required that Federal Delivery Partners (FDAs) review more detailed projects submissions one by one. INFC was only responsible for the program design and for negotiating agreements with the FDAs, while the partners were responsible for delivering the program with the PT's. INFC was paying an operating fund percentage to them.

The design of the BCF-CC included a more complex governance than PTIC-SCF given it involved multiple federal ministers. As such, it is possible to conclude that the delivery of PTIC-SCF was more efficient than the delivery of the BCF-CC.

5.0 Conclusions and Recommendations