Evaluation of the Smart Cities Challenge and the Smart Cities Community Support Program

Copyright

© His Majesty the King in Right of Canada, as represented by the Minister of the Office of Infrastructure of Canada, 2024.

Cat. No. T94-72/2024E-PDF
ISBN 978-0-660-73198-8

Table of Contents

• Executive Summary
• Program Overview
• Objectives and Scope
• Conclusions and Key Findings
• Recommendation
• Annex A: Detailed Findings
• Annex B: Management Action Plan
• Annex C: Methodology and Limitations
• Annex D: Program Outcomes

Executive Summary

Program Background

Under the Impact Canada Initiative, the Government of Canada committed to spending $300 million over 11 years to create Smart Cities Challenge . Led by Infrastructure Canada (INFC), the Smart Cities Challenge was developed to provide funding to address complex economic, environmental and social problems across communities in Canada through two streams, Stream One, the Smart Cities Challenge provides prize incentives, via a challenge-based funding model, to communities of all sizes to improve the lives of their residents through innovation, data and connected technology; and Stream Two, the Smart Cities Community Support Program, provides support measures to communities aimed at increasing their capacity to adopt smart cities approaches.

Objectives and Scope

In alignment with the 2016 Treasury Board Policy on Results, this formative evaluation assessed the relevance and effectiveness of the program design, progress towards achievement of the immediate outcomes and inclusion of the Gender-Based Analysis Plus.

The evaluation covered the timeframe of April 1, 2017, to March 31, 2022.

Methodology

To respond to the evaluation questions, multiple lines of evidence were employed to gather evidence to support the findings. This included a review of program documents, program data, literature review, and key informant interviews. A detailed methodology can be found in Annex A.

Overall key findings

The evaluation found that:

• The Smart Cities Challenge (SCC) was relevant and met the needs of Canadian communities.
• The SCC addressed eligibility barriers, promoted transparency and trust, facilitated involvement of diverse communities, and included the unique perspective of Indigenous peoples.
• The SCC design approaches were effective in achieving the intended program objectives; however, some aspects of the delivery and implementation of the program posed challenges for applicants.

Recommendation

The Program Management should develop a clear way to share lessons learned internally at INFC and explore collaboration options with Impact Canada to communicate lessons learned with federal and other Canadian institutions to help improve the design and implementation of challenge and outcomes-based programs.

Considerations for future rounds or challenge design programs

For the next round of the SCC and subsequent challenge programming, INFC should consider:

• Program Design: Options that would support expanding the ‘reach’ and ‘impact’ of the program. Explore options that would further support smaller and lower capacity communities to develop quality applications and consider modifying prize structures to support a greater number of projects in more communities. This would enable INFC to align with its broader mission of building a stronger and more inclusive Canada.
• Program Delivery: Explore Strategies to better maintain and improve internal capacity to support and sustain the effectiveness of a hands-on program delivery approach throughout the entire program cycle of the challenge.
• Performance Measurement: Provide applicants with a clear scope, sufficient time and standardized tools and templates to support the implementation of an outcome-based model, particularly for the development of performance measurement framework.

Program Overview

Under the Impact Canada Initiative, the Government of Canada committed to spending $300 million over 11 years to create Smart Cities Challenge.

Led by INFC, the SCC was developed to provide funding to address complex economic, environmental and social problems across communities in Canada. The Smart Cities Challenge built upon lessons learned from the United States’ Department of Transportation Smart City Challenge which tasked cities to collaborate in partnerships with citizens, businesses and civil society to develop smart city plans.

The SCC program within Canada was designed to be an open-ended model allowing applicants to explore community efforts to solve Canada’s big challenges. It emphasized working across departments and sectors and integrating community systems, services, and infrastructure in the problem and solution definition. Towards being more inclusive, Canada adopted the principle of “smart cities approach”, which is defined as one that delivers positive outcomes for communities by leveraging connected technology and data. According to a survey by Cisco, a lack of coordination and collaboration among city stakeholders, including conflicting community priorities, hinders community innovation capacity in smart city planning and results in a lack of clarity on the direction for smart cities. This prevents communities from forecasting desired results, and assessing the costs and benefits of proposed interventions. In response, INFC’s Smart Cities Challenge Program established a second stream, the Smart Cities Community Support Program, to increase communities’ capacity to adopt the smart cities approach.

Smart Cities Challenge Streams

Stream One: The Smart Cities Challenge stream provides prize incentives, via a challenge-based funding model, to communities of all sizes to improve the lives of their residents through innovation, data and connected technology.

Stream Two: The Smart Cities Community Support Program stream provides support measures to communities aimed at increasing their capacity to adopt smart cities approaches.

Combined goals of the two streams

The Smart Cities Challenge has the following combined goals:

• Realize positive outcomes for residents
• Forge new partnerships and networks
• Empower communities to innovate
• Spread the benefit to all Canadians
  

Stream One: Smart Cities Challenge

The Smart Cities Challenge stream used prizes to encourage communities to improve the lives of their residents through innovation, data and connected technology.

The first round of the Smart Cities Challenge used a tiered-competition prize approach with three (3) prize categories based on population size to encourage communities of different sizes to compete in the challenge.

Competition One was open to municipalities and local or regional governments established by or under provincial or territorial statute, and Indigenous communities, including First Nations, Inuit and Métis communities. The prize categories included;

• One prize of up to $50 million open to all communities, regardless of population
• Two prizes of up to $10 million open to all communities with populations under 500,000 people
• One prize of up to $5 million open to all communities with populations under 30,000 people.

Competition One of the Challenge received 130 applications from all regions of Canada from which the Jury selected 20 finalists that were approved by the Minister. The program provided each of the twenty finalist applicants with a $250,000 grant to support the elaboration of their final proposals. Four winners were selected by the Jury of experts based on the prize categories and approved by the Minister.

Stream Two: Smart Cities Community Support Program

The Smart Cities Community Support Program stream was established to build a support network towards facilitating shared vision and capacity on the adoption of smart cities approaches across communities by funding a non-governmental not-for-profit organization to provide advisory services and activities directly to communities—including small- and medium-sized municipalities, rural, remote, Indigenous, northern and isolated communities.

The Smart Cities Community Support Program was designed to operate in parallel to the Smart Cities Challenge and amplify and sustain its objectives directly by funding a not-for profit organization(s) to offer knowledge, expertise, experience and guidance to Canadian communities to help build internal capacity and navigate the smart cities landscape.

Selection eligibility

Eligibility of the program was restricted to not-for-profit organizations with the capability to operate at a pan-Canadian level in both official languages. Applicants were required to have established local community and/or Indigenous population networks, mandates that align with advancing smart city approaches, infrastructure in place to deliver expected services and experience working across multiple sectors. Applicants were able to submit to the Community Support Program as a single entity or as a group.

The Smart Cities Challenge responded to the 2017 Minister of Finance’s Advisory Council on Economic Growth recommendation which prompted the creation of the Impact Canada Initiative* to develop policy innovation relying on challenge-based or outcomes-based funding (Pay-for-Results model) to encourage collaboration across different sectors and recognize achievements. The shift towards using Pay-for-Results and challenge approaches with local Canadian communities diverges from usual practice, providing opportunity to meet the President of the Treasury Board’s mandate to devote a fixed percentage of government program funds to experimentation.

Key Program timelines

INFC initiated Competition one of the Smart Cities Challenge stream in November 2017, spanning till May 2019. With a call for application in June 2018, for stream two, the Smart Cities Community Support Program was initiated, and the Contribution Agreement signed in January 2019.

Figure 1: Competition One Timelines

Text description of Figure

Smart Cities Challenge Competition One Timeline (Stream 1)

Call for Applications - November 23, 2017

Application Submission Deadline - April 24, 2018

Finalist Announcement - June 1, 2018

Final Proposal Submission Deadline - March 5, 2019

Smart Cities Community Support Program Competition One Timeline (Stream 2)

Call for Applications - June 2018

Application Submission Deadline - July 20, 2018

Recipient Announced - October 1, 2018

Contribution Agreement signed with Recipient - January 15, 2019

*Impact Canada Initiative is a Privy Council Office innovation unit established to accelerate novel outcomes-based policy and program approaches to optimize Government of Canada policy, program, and service delivery.

As part of the program design, the Smart Cities Challenge used the experimental challenge-based and outcomes-based funding models with local Canadian communities. Throughout the program design and implementation, INFC committed to providing support to communities to help them in identifying their unique smart cities needs.

Challenge-Based Approach

This model provided a structured and open competition to solicit proposals from communities to solve challenges of varying levels of scope without prescribed themes. The challenge and the prizes were designed to encourage participation by the communities. Additionally, the challenge had application requirements and processes that were designed to encourage community engagement and mobilize internal and external resources to foster collaboration. For eligibility, communities were required to select a prize category and identify a challenge statement to define measurable and achievable project outcomes in their proposals.

These proposals were evaluated by an independent jury of experts that were selected based on knowledge or expertise in areas such as indigenous perspective, accessibility, urban planning, social innovation etc. Their recommendation of the finalists and winners was based on defined ‘evaluation merit criteria’ around the feasibility and potential impact of the proposed projects.

Outcomes-Based Approach

The Outcomes-based approach with a Pay-for- Results model is a performance-based approach to fund programs or projects where payments are based on the achievements of predetermined outcomes or results, rather than the general outputs achieved by funding.

An objective of the Smart Cities Challenge was to advance the outcomes-based programming model which linked payments for winning proposals to progress towards achieving community defined outcomes. The three key strategies supporting this model were the requirements to include a challenge statement in the application, develop a strong performance measurement framework with clear outcomes and the open nature of the competition.

Hands-On Program Delivery Mechanism

The Smart Cities Challenge provided direct support to applicants through regional leads who provided them with tailored responses and feedback regarding their enquiries about the application requirements. Additionally, Stream Two of the program was designed to provide support to the applicants throughout the process.

These support systems were designed to be available throughout the phases of the program. They were intended to provide guidance, communication, and outreach activities to help communities share their experiences in navigating the challenge and outcomes-based approaches, and explore new opportunities to engage with the department’s infrastructure programs.

Evaluation Objectives and Scope

Objective:
In alignment with the 2016 Treasury Board Policy on Results, this formative* evaluation assessed the relevance and effectiveness of the program design, progress towards achievement of the immediate outcomes and inclusion of the Gender-Based Analysis Plus.

The evaluation will also meet the legislative requirements of the Financial Administration Act that requires an assessment of relevance, and effectiveness of all INFC contribution programs once every five years.

Scope:
The scope of the evaluation covered the timeframe of April 1, 2017, to March 31, 2022, and was limited to the program’s immediate outcomes. The evaluation assessed the extent to which the program’s design and delivery were sufficient to achieve the expected results for competition one of the SCC.

Methodology:
To respond to the evaluation questions, multiple lines of evidence were employed to gather evidence to support the findings. This included a review of program documents, program data, literature review, and key informant interviews. A detailed methodology can be found in Annex C.

Evaluation Questions:
The evaluation questions were developed based on a review of program documentation and objectives.

1. To what extent are the Smart Cities Challenge and the Smart Cities Community Support Program relevant and responsive to the needs of Canadians?
2. To what extent have the Smart Cities Challenge and the Smart Cities Community Support Program design, delivery and implementation been effective in achieving program objectives?
3. To what extent have the Smart Cities Challenge and the Smart Cities Community Support Program’s immediate outcomes been achieved? (see Annex D for program outcomes)
4. To what extent has the design and delivery of the Smart Cities Challenge and the Smart Cities Community Support Program incorporated inclusiveness and accessibility?

*A formative evaluation assesses the feasibility and appropriateness of the program design before full implementation to adjust and improve effectiveness.

Evaluation Conclusion and Summary of Key Findings

The evaluation concluded that the Smart Cities Challenge design and delivery were responsive to the needs of Canadian communities, including those with diverse populations.

Key Findings	Why is it important?
1. The Smart Cities Challenge (SCC) was relevant and met the needs of Canadian communities.	The SCC is a challenge-based program that implemented a Pay-For-Results (outcomes-based funding) model. Its design was flexible and leveraged a bottom-up approach that encouraged communities to identify their shared persistent problems and create innovative ways to address them. The applicants selected Empowerment & Inclusion and Economic Opportunity as their top areas of needs based on six predetermined priorities by the program. The multi-sectoral and multi-stakeholder engagement approach of SCC design supported community ownership, thus positioning the proposed projects for success.
2. The SCC addressed eligibility barriers, promoted transparency and trust, facilitated involvement of diverse communities, and included the unique perspective of Indigenous peoples.	The SCC incorporated GBA plus considerations throughout its program design and implementation by integrating key strategies to mitigate barriers to access, promote transparency, trust and facilitated involvement of diverse and Indigenous communities through varied strategies: The tiered prizes approach facilitated the participation of diverse communities of all types and sizes in the Challenge e.g., flexibility and structuring of all prizes to be accessible to all Indigenous communities regardless of their size. Financial support was provided to finalists to support elaboration of their proposal, thus reducing the financial risks and encouraging participation. Stream two of the program, Smart Cities Community Support Program, served as a capacity building and support system for the applicants. The program engaged Indigenous communities and stakeholders during the program design and incorporated their unique perspectives and met their needs by designing multiple strategies to encourage their involvement. A requirement for communities to consider indigenous perspectives in their application and its incorporation in the Jury assessment system. Redirection of $15 million originally designated for an Indigenous-specific prize to Indigenous Services Canada as part of the Indigenous Homes Innovation Initiative to support identified priorities.
3. The SCC design approaches were effective in achieving the intended program objectives; however, some aspects of the delivery and implementation of the program posed challenges for applicants.	The challenge-based approach fostered innovative thinking and development of bold ideas through cross- sectoral collaboration by the communities. In addition, the outcomes-based approach was effective in allowing communities to think about transformative, impactful solutions and has the potential to lead to community-wide measurable positive outcomes. While the design was effective, the scope, timelines and requirements to develop project-specific performance measurement framework were challenging for applicants, due to lack of capacity and inadequate comprehension of the expectations. Some applicants perceive that there is a risk-reward imbalance in the program. This means the perception of a mismatch between the amount of work and cost of participating in the Challenge versus the potential reward. Unavailability of standardized tools to guide submissions by the applicants made addressing the program requirement more challenging. Although the Smart Cities Challenge provided support to the communities through a hands-on-program delivery approach by INFC staff, this support was mostly provided at the early phase of the program and not sustained throughout the program cycle. There is no defined way to use lessons learned to improve design and implementation of future challenges and outcomes-based programs, including: Ensuring that the support components of the program are in place from the beginning and maintained throughout all phases is essential; and Creation of a community of practice for knowledge-sharing between winning communities.

Recommendation and Considerations

Recommendation

The Program Management should develop a clear way to share lessons learned internally at INFC and explore collaboration options with Impact Canada to share lessons learned with federal and other Canadian institutions to help improve the design and implementation of challenge- and outcomes-based programs.

Considerations for future rounds or challenge design programs

For the next round of the SCC and subsequent challenge programming, INFC should consider:

Program Design: Options that would support expanding the ‘reach’ and ‘impact’ of the program. Explore options that would further support smaller and lower capacity communities to develop quality applications and consider modifying prize structures to support a greater number of projects in more communities. This would enable INFC to align with its broader mission of building a stronger and more inclusive Canada.

Program Delivery: Explore Strategies to better maintain and improve internal capacity to support and sustain the effectiveness of a hands-on program delivery approach throughout the entire program cycle of the challenge.

Performance Measurement: Provide applicants with a clear scope, sufficient time and standardized tools and templates to support the implementation of an outcome-based model, particularly for the development of performance measurement framework.

Annex A: Detailed Key Findings

Key Finding #1: The Smart Cities Challenge (SCC) was relevant and met the needs of Canadian communities.

SCC’s design was flexible and leveraged a bottom-up approach that encouraged communities to identify their shared needs and create innovative ways to address them

Based on evidence gathered through interviews with key stakeholders, including finalists, winners, and content experts, the SCC supported Canadian communities in identifying and addressing their needs and priorities by:

• Designing an open-ended program where communities were encouraged to compete for prizes towards solving persistent problems that they identified through multi-sectoral and multi-stakeholder engagement
• Requiring outcomes-focused project proposals to benefit the community at large towards delivering positive outcomes for its members
• Incentivising applicants to work locally to engage residents and multi-sectoral experts and define their own unique needs and priorities
• Funding a not-for-profit organization to provide guidance and support at a local level

In addition, the stakeholders interviewed expressed that the flexibility built into the design, delivery and implementation of the program had the potential to preserve ownership and encourage communities of different sizes to compete. This aligns with the intended benefits of the bottom-up approach in community programming identified in the literature.

The views of the respondents to the 2019 applicants’ survey align with the key informants on the potential of the program to empower communities to drive the needed change, based on their identified shared needs, and realize positive outcomes for the communities.

Figure 2: One out of three applicants (31%) identified Empowerment and Inclusion as their top priority.

Supporting an open-ended project scope based on six pre- determined focus areas:

Text description of Figure

31% Empowerment and inclusion

23% Economic opportunity

13% Environmental quality

13% Healthy living and recreation

12% Mobility

8% Safety and Security

Key Finding #2: The SCC addressed eligibility barriers, promoted transparency and trust, facilitated involvement of diverse communities, and included the unique perspective of Indigenous peoples.

Some design components of the challenge mitigated against barriers to participation throughout the competition

• The tiered-prize approach offered small communities the chance to participate in the challenge without being in direct competition with larger urban cities that have greater resources. Interviewed applicants found the tiered- prize design effective in ensuring the competition was accessible to communities of all sizes.
• Application guides and outreach support provided diversity and inclusion considerations to help communities develop intersectional approaches to their solutions.
• The application requirements for applicants to demonstrate efforts in incorporating Gender-Based Analysis Plus considerations in their consultations expanded the diversity of the proposed projects' reach within the challenge.
• The finalists were tasked with outlining plans to adhere to policy requirements regarding consultation with Indigenous groups, fulfilling modern Treaty obligations, and applying for Community Employment Benefits aimed at diversifying recruitment, training, and procurement practices. As part of the Community Employment Benefits criteria, winning communities were expected to regularly update their progress on these plans in their milestone tracking.

Financial support provided to finalists supported their application and increased accessibility of the challenge to all communities in Canada.

• Finalists indicated that the $250,000 finalist grant encouraged them to compete in the challenge, adding that it offered reduced risk perception of potential loss of resources.
• Additionally, this grant helped the communities to bridge capacity gaps that allowed the communities elaborate on their proposal and meet the program requirements for final submission.

Smart Cities Community Support Program provided support, education and networking opportunities that included capacity building for communities of all sizes.

• This stream was part of the plan to support the network to build a shared vision and capacity for the adoption of smart cities approaches across communities. Interview respondents from finalist and winning communities found the resources and strategies effective to address diverse, Indigenous and low-capacity needs.

Figure 3: The combination of strategies enable diverse communities to access the challenge. The SCC received 130 eligible applications, of which 44% were from communities with a population between 1k and 30k. These strategies could be sustained for future programming.^Footnote1

Text description of Figure

Rural < 1K

Small 1K-29K

Medium 30K-100K

Large Urban  >100K

The unique perspectives, needs and priorities of Indigenous communities were incorporated in the SCC through tailored multi-pronged approaches encouraging involvement.

INFC engaged Indigenous leaders, key stakeholders, communities and organizations to finalize the design of a competition specific to Indigenous communities to reflect their unique needs and priorities. Following this engagement, $15 million, originally reserved for Indigenous-specific SCC prizes, was redistributed to Indigenous Services Canada to be part of the Indigenous Homes Innovation Challenge, recognizing that this Challenge would provide more direct support to the needs and priorities identified through consultations. This program is outside the scope of the current evaluation.

The SCC (stream one) design addressed the needs and priorities of Indigenous communities by being flexible and ensuring accessibility to all prize categories.

The evaluation found that some key design components of the program worked synergistically and were effective in encouraging the involvement of Indigenous communities in the challenge;

•  The SCC finalists were required to consult, engage and incorporate the perspectives of Indigenous communities in their applications. This program requirement for building intercommunity collaboration encouraged consideration of the need to involve Indigenous communities.
• The SCC also integrated an Indigenous perspectives through a jury system with representation from Indigenous communities (3 of 13 jurors) to ensure Indigenous perspectives were considered in the evaluation of the applications.
• The Smart Cities Community Support Program (Stream Two) provided education and support focused on Indigenous support via handbook, toolkits and developed specific information sessions available to all communities. The Community Resources and Assistance activities provide support to members of rural, remote and Indigenous communities in supporting development of smart cities within their community.

Figure 4: Among the 130 eligible applicants, 20 (15%) were from Indigenous communities or focused on Indigenous populations. Additionally, 30% (6) out of the 20 finalists were from Indigenous communities or focused on Indigenous population, indicating a higher proportion of applicants representing Indigenous needs in the finalist round. ^Footnote2

Text description of Figure

Among the finalists, there were:

• 6 Indigenous Finalists
• 14 Non-Indigenous Finalists

Among the applicants, there were:

• 20 Indigenous Applicants
• 110 Non-Indigenous Applicants

Key Finding #3: The SCC design approaches were effective in the intended program objectives; however, some aspects of the delivery and implementation of the program posed challenges for applicants.

The Challenge-based approach fostered innovative thinking and the development of bold ideas through cross-sectoral collaboration among the communities

In terms of the design of the challenge-based model, all the external stakeholders, which includes the winners, finalists, and content experts interviewed, highlighted that this model stimulated cross-sectoral engagement at the community level for the identification of community-wide problems and demonstrable outcomes for their proposed projects.

Additionally, the evaluation found that the model was effective in generating the interest^Footnote3 of communities of all types to compete in the challenge, thereby receiving 149 applications of which 130 (87%) met the eligibility criteria. Further evidence shows that the model helped ensure that the problems identified, and the solutions (projects) proposed, had the buy-in of the community members. Based on the literature, a shared vision is needed for ownership and the realization of maximum benefits on a project.

Additionally, evidence from the evaluation showed that due to the competitive nature of the challenge and the need to develop bold ideas, the communities were innovative and ambitious with their proposed solutions. This highlights the potential suitability of this model to support policy innovation and resolve challenges in the communities.

Figure 5: The evaluation found, from the various lines of evidence, the following design elements to be strengths of the challenge-model

Text description of Figure

Strengths of the model: Open Nature, Community-level Stakeholder engagement, Multiple Prizes, Transparent Process, Community-led.

The outcomes-based approach was effective in allowing communities to think about transformative, impactful solutions and has the potential to lead to community-wide measurable positive outcomes

The flexible design of the outcomes-based approach allowed communities to be results- focused.

• All the winning communities interviewed had a positive perception about the flexibility and adaptability of the outcomes-based model, stating that it enabled them to adapt their projects and
• They added that it allowed them to build trust among investors and attract more investments to their projects because there were no constraints that were historically embedded in traditional funding models. Typically, eligible expenses were predefined by the program and payments were linked to the achievement of tangible project milestones and not direct community-level outcomes.

Applicants, including winners, expressed concerns regarding the model’s risk-based design and operationalization.

• Among the applicants including the winners interviewed, all particularly smaller communities expressed concerns about the design and operationalization of the model. They opined that it was risk-based with fears of potential loss of resources and political will for subsequent programming if not remunerated despite work done and milestones achieved.
• Their concern also hinged on the possibility of not achieving their proposed outcomes due to factors outside their control. The impact COVID-19 had on the planned projects confirmed the concerns of the applicants. According to interviews with the winners, the pandemic disrupted their project plans, including human resourcing.

Applicants found the requirement to develop performance measurement frameworks for their projects very challenging due to inadequate capacity and lack of tools leading to delays with timelines.

• The evaluation found that despite the system set up to support the applicants, they had limited knowledge of how to develop a performance measurement framework with tangible community-wide results linked to milestones for payments.
• The winners reported that this contributed to delays experienced in negotiating contracts and contribution agreements with INFC.
• Following the signing of the contribution agreements, there were delays with the submission of the milestone reporting and inadequate tracking of the outcomes by winners due to inexperience with the approach and lack of tools and templates to guide the submissions. They found the process very difficult, time- consuming and the cycle too short (frequent).

Outcomes-based model is more suited for projects with defined time frame for realization of outcomes and less suited for social-projects with longer term outcomes.

Being a formative evaluation, demonstration of the extent to which the model led to achieving measurable results at the community level was not within the scope and is recommended for subsequent summative evaluation to loop back learning into future programming.

However, in seeking to learn when the outcomes-based model works, for whom and in what conditions, the evaluation found the stakeholders hold the perceptions that the model, in the context of smart cities, works better for projects of fixed timelines and not rolling projects such as social projects with more long-term outcomes. This is due to the challenge of defining and measuring the outcomes required for payment.

The hands-on program delivery approach was successful in providing direct support to the applicants on the challenge at the early phase of the program

The hands-on program delivery approach was successful in providing direct support to the applicants at the early phase of the program.

• At the early phase of the program, the SCC team at INFC established a regional leads support system for the communities. This design component is in line with the best practices recommended in the literature on the benefits of establishing direct contact with experts for support and guidance in challenge-based approaches. These leads provided applicants with useful tailored responses and feedback regarding their enquiries about the application requirements.
• The majority (71%) of the applicants were very satisfied with the quality of their interaction with SCC team through emails, over telephone support (55%), information sessions (50%), and webinars (53%), based on evidence from the 2019 survey.
• Although the evaluators were only able to interview one of the sixteen finalists who did not emerge as a winner, the finalist expressed that INFC staff were accessible and thought that the guidebook and INFC webinars were helpful.
• Based on the evidence from interviews with finalists, jurors and applicants, these stakeholder groups found the direct support provided by Infrastructure Canada useful. These include:
• Robust program website with updated frequently asked questions
• Establishment of regional leads who provided tailored support
• Webinars
• Monthly check-in meetings with Infrastructure Canada
• Finalist grant of $250K
• Two-day check-in event for finalists

However, the SCC did not sustain the support provided to the winners at the later phase of the program.

• The majority of the winners interviewed had a shared opinion that the frequent staffing changes to their INFC points of contact was a major barrier to having the right information and receiving tailored guidance about the program requirements at this phase of the program^Footnote4 .
• Evidence from interviews with winners suggests that there were inconsistencies in the information received by the communities at the later phase of the program due to the staff turnover.
• The SCC needs to improve the effectiveness of its hands-on program delivery approach to ensure that the participants receive needed support from the launch of the program and throughout its lifecycle.

The program has opportunities to enhance impact through the early initiation of Stream Two and the implementation of an adequate system for sharing lessons learned among participants and other government departments.

The absence of the Smart Cities Community Support Program due to late initiation was a major drawback to the level of support that could have been provided to the applicants. In addition, the evaluation did not find any proof of lessons learned sharing with other federal departments.

The SCC emphasized the need for knowledge-sharing beyond the participants and winners of the competition, upholding Impact Canada Initiative’s goal for all Canadians to benefit from ideas and best practices found through its programs^Footnote5. While the applicants appreciated the support that was provided by the Smart Cities Community Support Program, they suggested that they could have benefited from an early launch during the competition phase of the program (see Figure 1- program timelines). Based on information from the program, delay with initiation was based on programmatic decision to prioritize rollout of the Stream One.

The winners interviewed acknowledged that although Stream Two did not support them during the application phase, it was helpful in connecting the winning communities for learning purposes. Some of the activities provided by the Smart Cities Community Support Program on initiation included six idea camps that supported three- to four-hour in-depth discussions between communities and experts; six collision days that provided community members with one-on-one sessions with experts; six community roundtables; and 75 virtual sessions.

The evaluation did not find any evidence of the program meeting its commitment as indicated in the logic model to share lessons learned among participants in addition to other federal government departments^Footnote6.

Lessons learned from key findings:

In general, evidence from interviews with winners, finalists, jurors and from 2019 survey results suggest that overall support for the Challenge competition one could have been strengthened by:

• Launching the Smart Cities Community Support stream concurrently with stream one, or prior to the application phase, to allow communities to leverage support
• Providing more clarity around the scope and expectations and definitions of the “challenge statement” and “outcome statement”
• Allowing for a longer timeframe for the implementation to support communities with resourcing challenges
• Developing more support to the winners including a tool-kit with templates; and
• Bringing the winners together to network and provide a platform to exchange ideas, best practices and learning for efficiency.

While the design and implementation of most components of the challenge- based approach were successful, some were not as successful. Lessons learnt can be applied to subsequent rounds and similar challenge-based models:

Most applicants hold positive opinion about SCC and would apply for subsequent rounds, a few applicants reported that they may not be open to compete in subsequent rounds if implemented frequently, indicating that perhaps annually may be too frequent. This is due to the reasons above and the overall risk-reward balance; greater risk of not getting funding and potential to lose political support for similar projects if they did not win.

Annex B: Management Action Plan

#	Recommendation	Management Action Plan	Deliverables	OPI and Due Date
1	The Program Management should develop a clear way to share lessons learned internally at INFC and explore collaboration options with Impact Canada to communicate lessons learned with federal and other Canadian institutions to help improve the design and implementation of challenge- and outcomes- based programs.	I. Develop, in consultation with Impact Canada, a presentation on the Smart Cities Challenge to share within INFC and possibly more broadly with other federal government departments (at the discretion of Impact Canada) lessons learned in designing and delivering a Gs&Cs program that is incentive (outcome) based funding.	Community Policy and Programs Branch (CPPB) develop presentation in consultation with Impact Canada Presentation within INFC including INFC Director Committee, Director General Management Committee (DGMC), and Deputy Minister Executive Committee (DMEC) Consult with Impact Canada on presenting to other federal departments	CPPB Fiscal Year (FY) Q4 2024-25 CPPB FY Q4 2024- 25 CPPB FY Q4 2024- 25
		II. Develop a presentation on the Smart Cities Community Support Program to share within INFC and possibly other federal departments and organizations. The presentation would be developed in consultation with Evergreen to highlight lessons learned concerning capacity-building supports for communities, including resources and tools to share information on smart cities approaches with communities.	Community Policy and Programs Branch (CPPB) develop presentation in consultation with Evergreen Presentation within INFC including INFC Director Committee, Director General Management Committee (DGMC), and Deputy Minister Executive Committee (DMEC) Presentation to other federal departments and organizations	CPPB Fiscal Year (FY) Q4 2024-25 CPPB FY Q4 2024- 25 CPPB FY Q4 2024- 25

Annex C: Evaluation Methodology and Limitations

Methodology

Lines of Evidence

The four lines of evidence used for this evaluation draw on qualitative data (e.g., document review, literature review and key informant interviews) and some quantitative data (e.g., administrative and survey data). The analytical methods used for this evaluation were tailored to the nature of the data available. The evaluation design and level of effort were calibrated with available INFC resources. The following paragraphs describe the lines of evidence used for data collection as well as the limitations encountered during data collection and analysis, and the mitigation strategies used to address those limitations.

Document Review
Review of documents was conducted to assess relevance, program’s design and delivery and progress towards expected outcomes. The document review included program outreach and promotion products, applicant proposals, finalist proposals, records of decisions, grants and contributions agreements, past lessons learned, progress reports, virtual and video conferences, online articles and press releases, and other relevant documents.

Literature Review
The evaluation reviewed relevant academic literature from similar smart cities challenge programs to gather evidence used to determine how smart cities are defined in addition to best practices and lessons learned from other programs. The evaluation considered studies from programs such as IBM Smarter Cities Challenge, the United States’ Department of Transportation Smart Cities Challenge, and Bloomberg Philanthropy’s Mayors’ Challenge, the United Nations and APEC case studies as well as other international experiences such as the European Commission’s recent mapping of smart city approaches.

Data Review
Internal and external performance data were examined to collect evidence on the program’s progress towards achieving outcomes. Sources including program data, the survey results with applicants and finalists in 2019, internal data hub and external data from winners' progress reports, and Statistics Canada’s data were examined. This included data from the winners’ milestone reports, as well as the Infrastructure Funding Reports and the Program Information Management System.

Interviews
Interviews were conducted to gather perspectives on the extent to which the two streams of the SCC was responsive, made progress towards expected outcomes, and the effectiveness of program design and delivery. The interviews were conducted with three INFC officials; two Privy Council Office officials; five representatives from the Smart Cities Community Support Program; one juror; three smart cities content experts; one finalist project; and three of the four winning projects including one for Indigenous communities.

Limitations and Mitigation Strategies

The evaluation project encountered few challenges that had minimal impact on the project implementation and its capacity to provide evidence to support the evaluation questions. The challenges and the mitigation strategies that were implemented have been described below:

Limited examples of previous research studies on the topic: As Smart Cities continues to emerge as a Canadian practice, most of the literature acquired for the evaluation was based on existing global contexts. As a result, there was limited literature available that covered Canadian smart city perspectives for Canadian-specific programming. To mitigate against this challenge, the evaluation utilized a line of evidence which included consultations and interviews with subject matter experts to gather evidence on their perception of the SCC programming in Canada.

Sub-optimal response to request for interviews by the stakeholders: The evaluation relied on gathering qualitative data to provide robust evidence towards responding to the evaluation questions. While efforts were made to reach the various stakeholders defined for the project, the evaluation encountered challenges with the lack of response to invitation for the key informant interviews. Specifically, none of the applicants that did not win a prize in the challenge responded to the invite.

This impacted the capacity of the evaluation to gather evidence on the perception of this group of stakeholders that could support program improvement. Additionally, only one jury member interview and no $50 million prize-winner interview, French-language interviews, or non- finalist applicant interviews held. The unavailability of updated contact information for the jury members contributed to this challenge. The evaluation mitigated against the impact of this challenge by utilizing multiple lines of evidence including results from a survey conducted by the program which had responses from most of the applicants.

Annex D: Program Outcomes

Immediate Outcomes (within scope of this evaluation)
1. Communities from all sizes and from across Canada develop and submit smart cities applications and proposals that demonstrate smart cities approaches which address challenges identified by the community. (Findings 1, 2)
2.  Winning communities are supported in implementing their projects. (Finding 3)
3. Knowledge/lessons learned are shared and transmitted across other government departments and stakeholders. (Finding 3)
4. Policy issues and knowledge gaps are identified. (Finding 3)

Intermediate Outcomes*
Communities across Canada, of all sizes, are encouraged to implement smart cities projects.
Broad dissemination of knowledge and lessons learned by winning communities.
Multi-stakeholder partnerships and networks are established and strengthened.
Sustained community engagement in community planning processes.
Challenge projects are replicated, transferred and scaled.
Community Support Program emerges as a primary resource for communities seeking assistance for smart cities initiatives.
SCC performance measurement frameworks are used to benchmark measurable and measured outcomes for participating and non- participating communities in smart city approaches (governance, economic, social, sustainable and inclusiveness).
Best practices from SCC are disseminated to the broader policy community.

Ultimate Outcomes*
Evidence to support whether a challenge prize is a useful policy tool and funding model for program delivery.
Increased community readiness and capacity for innovation across Canada.
Improved national and community economic growth, environmental sustainability, and/or social inclusion due to smart cities approach.
Informed policy and investment strategies that align infrastructure with technology/digital transformation.

* Not within scope of this evaluation to be covered in subsequent evaluation of the SCC