Evaluation of the EC Development Program

PDF Version (Size: 776.62 KB) (776 KB)

Help on accessing alternative formats, such as PDF, PPT and ZIP files, can be obtained in the alternate format help section.

Executive Summary

Program Overview

In 2009, Infrastructure Canada (INFC) implemented an EC Development Program (ECDP) to support the need for more working level employees to manage and deliver the 2009 Economic Action Plan. The ECDP was designed and implemented as a mechanism to attract and retain talented individuals to the Economic and Social Sciences Services (EC) occupational group to the department. The ECDP identified the competencies critical to INFC business and aimed to offer a simplified learning, development and promotions process to develop and promote EC-02s, EC-03s, and EC-04s through to the EC-05 level. As at 31 March 2015, the program included 24 active participants and 10 program graduates.

Evaluation Objective and Scope

The purpose of the evaluation is to provide a neutral assessment on whether there is continued need for the program. A particular focus was placed on whether the raison d'être of the program is still relevant and the extent to which the ECDP is achieving its objectives. Whether the program's design and implementation remains effective in relation to current departmental needs was also examined.

The evaluation considered all aspects of program activity from its inception in 2009 to 2015 and was conducted in alignment with the five core issues, as prescribed by the Treasury Board Policy on Evaluation, 2009. Both qualitative and quantitative information were collected through multiple lines of evidence including a series of key informant interviews with program representatives and a census of all participants and graduates from the program, and a comparative review of similar programs in other federal government departments.

Key Findings, Conclusions and Recommendations

With the exception of attracting individuals to INFC and engaging in learning and development, little evidence was found attributing the achievement of EC recruitment, development, and retention outcomes specifically to the program. Some core issues in the areas of governance and oversight, design, and information sharing were found to be impacting the overall delivery of the ECDP. Further efficiencies could likely be gained by reducing the level of and duplication of effort within some aspects of the program and improving communications.

It was found that the need to strategically attract, retain and develop ECs in order to sustain a strong working level capacity at INFC is pertinent. However, the medium- and long-term INFC business needs related to ECs appear to be evolving and are currently unclear. At the same time, human resources management has been evolving government-wide (i.e. linking performance objectives to competencies) since the program's inception. Therefore, the need for a strategy related to sustaining a capacity of strong ECs at INFC is relevant; however, it is difficult to conclude that the ECDP, as it is currently designed, is the right mechanism.

A summary of the findings and conclusions fully detailed in this evaluation report can be found at Annex A: Summary of Evaluation Findings and Conclusions.

Recommendations

In line with the priorities of the Clerk of the Privy Council, INFC is vying to become more strategic and innovative in employee recruitment and development. The importance of employee retention and development was echoed throughout this evaluation. Though not within the scope of this evaluation, it is important to note that all senior management interviewees indicated that recruitment, development and retention strategies are important to have in place for all occupational groups and not just for ECs and PEs, as is the case at INFC.

While some of the intended outcomes have been achieved, they cannot be clearly attributed to the ECDP. This, along with the overall performance findings, suggests that the program objectives and logic need to be reviewed. Simply revamping the program would not address the key issues impacting the achievement of outcomes.

Unprompted, interviewees from the participant- to Director General-level emphasized that changes to or cessation of the current ECDP will have a significant impact on the morale of the current participants and the EC community at INFC.

Given the above and that all current participants, should they meet the competency requirements upon completion of minimum time at level, could graduate by the end of fiscal year 2017-2018, the evaluation recommendations are as follows:

1. INFC senior management should identify a member of the management team from the 'business' to champion and monitor strategies related to EC people management.
2. INFC senior management should maintain the ECDP for current participants taking into consideration the findings in this evaluation and making short-term improvements, particularly:
1. Clarifying governance (program ownership, senior management involvement, monitoring).
2. Reviewing competencies and associated learning and development.
3. Improving understanding of the current ECDP by re-confirming and documenting program guidelines.
4. Reducing duplication of effort between the use of ECDP tools and the PS Performance Management Program (PSPMP).
5. Developing and implementing a plan to communicate changes or updates to ECDP and to ensure information is communicated to all regularly and consistently.
3. In assessing people management requirements for the EC community, senior management should determine if an EC development program or other strategies are needed to meet departmental business requirements and overall objectives.

Introduction

An evaluation of INFC's ECDP was undertaken by the Evaluation Directorate from March to September 2015.

This report was presented to the Departmental Evaluation Committee on November 17, 2015 and approved by the Deputy Minister on December 11, 2015.

The following presents the program profile, the evaluation methodology, the detailed findings and conclusions as well as the recommendations, management response and action plan.

Program Profile

In 2009, INFC was experiencing significant growth in departmental responsibilities in policy and programming; particularly with the management and delivery of the 2009 Economic Action Plan. There was a need for more working level employees with significant competencies and experience. At this time, upon consultation with management, the EC-05 level was targeted to be strengthened.

The ECDP was designed and implemented as a mechanism to attract and retain talented individuals to the Economic and Social Sciences Services (EC) occupational group to the department in order to meet the present and future needs of INFC. The ECDP identified the competencies critical to INFC business and aimed to offer a simplified learning, development and promotions process to develop and promote EC-02s, EC-03s, and EC-04s through to the EC-05 level.

The set of competencies were derived from the employee level of the Key Leadership Competencies for the Public Service (published by the Office of the Chief Human Resources Officer in 2005). They are as follows:

• Engagement: working effectively with people, organizations and partners. Exhibited by: Teamwork; Relationship building; and Communication (verbal and written).
• Strategic thinking: innovating through analysis and ideas. Exhibited by: Organizational awareness; and Analytical thinking.
• Excellence: action management and results. Exhibited by: Results-Oriented; Adaptability and flexibility; and Continuous learning.
• Values and Ethics: Exhibited by: Observing the Values and Ethics Code for the Public Service; Supporting/encouraging diversity and bilingualism; and Acting with transparency and fairness.

The program includes level-specific behavioural indicators for each of these competencies.

In 2010 and in 2011, the EC competencies, intake and promotions processes were reviewed and new guidelines were developed. The most noticeable changes were that the EC-03 level was added to the program and that to be promoted from an EC-04 to EC-05 candidates would undergo a full board assessment including: a written exam, interview and reference checks. Additionally, in 2011, all EC employees below the EC-05 level were enrolled into the program. The program was put on hold in June 2012, due to departmental budgetary constraints. The INFC People Management Committee (PMC) approved the re-launch of the ECDP effective for September 2013.

Objectives

The program has two objectives to (1) provide participants with meaningful development and learning opportunities to enable them to acquire the competencies needed at the EC-05 working level; and (2) simplify the promotion process for qualified participants.

For the purposes of the evaluation, expected program outcomes were further articulated with the evaluation working group. These are as follows:

Eligibility and Enrollment

There are no pre-determined ECDP positions within INFC. All new and indeterminate positions at the EC-02 to EC-04 levels are eligible. Upon creation of EC positions or deployments at these levels, managers individually consider whether to approve the position into the ECDP based on the nature of the work and the level required to perform it. All ECDP positions are intended to eventually be reclassified up to the EC-05 level. Participants who have been appointed to the ECDP receive notification through their letter of offer.

Learning and Development

Upon entry into the program, ECDP participants are to actively engage in their learning and development in order to gain the experience and competencies required to be promoted through the various levels up to the EC-05 level.

Together, participants and managers put in place a Learning and Development Plan (LDP). Per the program guidelines, LDP's should include a variety of learning and development activities linked to ECDP competencies. These activities are jointly determined by the manager and participant and the plan is tailored to the participant's needs.

Assessments, Promotions and Graduation

Once the minimum time-at-level requirements of the program have been met, participants can complete a Self-Assessment Profile where, for each competency, they must provide one concrete, specific example of achievement from their own experience that they feel demonstrates their proficiency.

If the Delegated Manager determines that the participant has demonstrated the acquisition of competencies, they will complete the assessment against the Statement of Merit Criteria. The participant will then undergo an assessment for promotion process, which depending on their level, can include presentations to the ECDP Steering Committee, written exams, interviews, and reference checks. The ECDP Steering Committee makes recommendations related to promotions and with the support of the participant's Assistant Deputy Minister, the DG Human Resources ultimately approves promotions within the program.

Successful participants are appointed, as per the Non-Advertised Appointment Policy and their existing position is re-classified. Upon appointment to the EC-05 level, participants are deemed to have graduated from the ECDP.

Key Roles and Responsibilities

Participants are expected to play an active role in learning and development. They are also responsible for completing the self-assessment profile and the necessary presentations, exams, and interviews within the assessment for promotion processes.

Supervisors assist in developing and reviewing learning plans with participants, monitor participant progress and provide feedback. The supervisor will also recommend participants for promotion to the delegated manager and present the case for promotion to the steering committee.

Delegated Managers consider the nature of work and level required to do it before approving the position into the program. They also approve the budget for ECDP learning and development and make recommendations for promotions.

Human Resources provide secretariat and program design support as well as support staffing activities (assessments, reclassification, appointment processes) related to ECDP promotions. HR representatives also participate as ECDP Steering Committee members and conducts reference checks upon a candidate's consideration for promotion. The Director General, HR has the authority to approve promotions.

The ECDP Steering Committee role is to ensure that the program is administered consistently. The Committee advises on program design, implementation and on learning and development plans to ensure support is in place. The Committee also participates in the assessment for promotions. Currently, the Committee has three members: one EX or senior EC (EX minus-1, with HR management responsibility) employee from each of the Policy and Communications, and Program Operations Branches, plus a PE with expertise in staffing.

INFC People Management Committee (PMC) makes decisions concerning key parameters of the program, including minimum time-at-level and the assessment process for promotion. The PMC also approves learning and development activities in excess of $5,000 per participant. Members include the Associate Deputy Minister, all EX-04 and EX-05 positions, the Director General of Human Resources and the Chief Audit and Evaluation Executive.

Program Resources

The ECDP does not have a budget allocated for program delivery. Rather, learning and development activities are funded from the Delegated Manager's learning and development budget. Human Resources Services dedicate a partial Full-Time Equivalent employee to provide secretariat support to the program and participating Branches provide an EX-level or Senior EC representative to the ECDP Steering Committee.

The EC community at INFC as at 31 March 2015

Per the table below, the EC group accounts for 34% of total INFC employees, followed by the Administrative Services (AS) group (20%) and the Information Services (IS) group (12%). Since 2009, the EC group has been the department's largest occupational group representing approximately 1/3 of employees and has had a slightly higher or comparable amount of years of service in INFC and in the Public Service to the AS and IS groups.

Table 1: Number of employees and average years in Public Service and at INFC by group, by Fiscal Year

Branch / Level	2009-10	2010-11	2011-12	2012-13	2013-14	2014-15
EC Group
Number of Employees	99 (32%)	105 (28%)	107 (30%)	96 (30%)	95 (31%)	116 (34%)
Average Years in PS	8.6	8.3	8.1	9.3	10.1	10.4
Average Years in INFC	2	2.6	3.4	4.4	4.7	4.4
AS Group
Number of Employees	60 (19%)	80 (21%)	74 (21%)	59 (19%)	59 (19%)	69 (20%)
Average Years in PS	8.4	7.4	8.4	9.1	9.6	9
Average Years in INFC	2	2.3	3.2	4.1	4.2	3.8
IS Group
Number of Employees	36 (12%)	54 (14%)	53 (15%)	51 (16%)	44 (14%)	40 (12%)
Average Years in PS	5.9	5.7	7.1	8.3	8.8	10.8
Average Years in INFC	1.7	2	3	4	4.9	5.5
Total INFC Employees	311 (100%)	375 (100%)	352 (100%)	317 (100%)	304 (100%)	340 (100%)

Source of data: INFC Human Resources Data from 2009-2010 to 2014-2015.

The majority of EC employees (80%) are distributed between the Program Operations (POB) (52%) and Policy & Communications (P&C) (28%) Branches. The remaining work within the Federal Montréal Bridges (FMB) (13%), Corporate Services Branch (CSB) (5%) and the Audit and Evaluation Branch (3%).

As per the following graph, at the time of the evaluation, the EC community was comparably representative in relation to the whole of the Public Service in regards to first official languages (FOL) and gender. The representation of Persons with Disabilities and Aboriginal People is smaller within the EC community at INFC than the whole of the Public Service while the representation of visible minorities is higher.

Source of Data: INFC Human Resources data pertaining to ECs as at 31 March 2015; Public Service Management Dashboard Data pertaining to the Public Service as at 31 March 2014.

ECDP as of 31 March 2015

At the onset of the evaluation, three of INFC's five branches were participating in the ECDP (Policy and Communications, Program Operations and Corporate Services). Given the limited size of the branch and the need for all levels of EC, the development program was not appropriate for the Audit and Evaluation Branch and the new Federal Montréal Bridges Branch had not yet participated.

Within the three participating branches, there were 26 EC-02,-03,-04 ECDP-eligible positions filled at INFC. Of these, virtually all of them (24) are ECDP positions.

Table 2: Filled EC-02,-03, -04 Positions and ECDP positions in INFC by level by Branch as at 31 March 2015

Branch	Positions in ECDP	ECDP EC-02	ECDP EC-03	ECDP EC-04	Positions not in ECDP	Total Filled EC-02,-03, -04 Positions
Policy & Communications	7	0	2	5	0	7
Program Operations	17	2	5	10	1	18
Audit & Evaluation	0	0	0	0	1	1
Corporate Services	0	0	0	0	1	1
Federal Montréal Bridges	0	0	0	0	5	5
Total	24	2	7	15	8	32

Source of data: ECDP Program Data

Since 2009, ten individuals have graduated from the program.

About the Evaluation

The purpose of the evaluation is to provide a neutral assessment with respect to supporting a DMC discussion on whether there is continued need for the program. A particular focus was placed on whether the raison d'être of the program is still relevant and the extent to which the EC Development Program is achieving its objectives. Whether the program's design and implementation remains effective in relation to current departmental needs was also examined.

The evaluation considered all aspects of program activity from its inception in 2009 to 2015. The assessment was conducted in alignment with the five core issues, as prescribed by the Treasury Board Policy on Evaluation, 2009. Both qualitative and quantitative information were collected through multiple lines of evidence including: document review; financial and administrative data review; secondary data from the 2014 Public Service Employee Survey results and the Public Service Management Dashboard; a series of key informant interviews with program representatives and a census of all participants and graduates from the program; and a comparative review of similar programs in other federal government departments. For a list of key informants and other federal government programs reviewed, Please see Annex B: Detailed Methodology.

It should be noted that the vast majority of EC-02, EC-03 and EC-04 positions are part of the program. It was not possible to establish a non-participant control group for further comparative analysis. For full methodology details including limitations and mitigation plans, please see Annex B: Detailed Methodology.

Detailed Findings

The following sections present the findings related to relevance, performance, design and delivery and the efficiency and economy of the ECDP at INFC.

Relevance

The evaluation assessed whether the raison d'être of the program remains relevant in the current context and whether the program is aligned and consistent with government priorities and responsibilities.

Extent to which the raison d'être of the ECDP is still relevant to INFC

Finding 1: ECs at INFC remain in their positions for a slightly longer period of time than ECs across the Public Service. However, EC-05, -06, and -07s represent the majority of EC departures in the last five years and, along with EX-01s, indicate that they intend to leave INFC in the next two years.

As per the graph below, on average, INFC has retained EC employees for comparable amounts of time to other government departments; in some cases INFC has retained ECs for longer intervals.

Source of data: INFC Human Resources Data from 2009-2010 to 2014-2015, Public Service Management Dashboard as at 31 March 2014.

Though retention has been stable overall, there is indication that INFC may have impending challenges in retaining ECs. The 2014 PSES results for ECs depicted in the following graph indicate that while ECs are slightly more satisfied at INFC with regards to support for training and career development than those in the Public Service, fewer ECs at INFC feel that their job is a good fit with their skills.

Further to this, INFC Human Resources data showed that EC-05, -06 and -07s have represented 79% (61 of 77) of total INFC EC departures from 2009-10 to 2014-15 and most (68%) departed for a lateral transfer. 2014 PSES results indicated that this trend may not reverse at INFC, as 43% of EC respondents and 75% of EX-01 respondents indicated they intend to leave their position in the next two years and most still intend to leave for a lateral transfer.

Finding 2: The organizational needs related to ECs at INFC are not fully known. The 'working level' may neither be the same across INFC Branches nor that which was defined upon program creation.

There is indication from both interviews and human resources data that the organizational needs related to ECs are not fully known and may vary across the organization.

On one hand, many senior managers (Director- and Director General-level) indicated that the working level remains for the most part at the EC-04 or EC-05 level. On the other hand, the bulk of EC employees, as per the following graph, have been spread across the EC-05, -06 and -07 levels since the onset of the program. Therefore, the working level may no longer be primarily at the EC-05 level.

Source of data: INFC Human Resources Data from 2009-2010 to 2014-2015.
Note: In February 2014, Federal Montréal Bridges responsibility and employees were transferred from Transport Canada to Infrastructure Canada (INFC).

Secondly, some management interviewees, including all Assistant Deputy Ministers, also indicated that a mix of ECs is required to deliver INFC business; that there is work specific to EC-02 or EC-03 level positions within INFC. The data suggests that there are challenges to identifying the needs related to ECs at the junior level. Either the business needs related to junior ECs evolve frequently or are not yet fully known. As per the graph above, the number of EC-01, -02, -03, and -04 positions has not trended consistently over the depicted years. To supply a development program to the EC-05 level, one would expect that a consistent amount of positions at the junior level would be needed and filled accordingly.

Alignment with Government of Canada and INFC Priorities and Responsibilities

Finding 3: The program objectives continue to be in line with Government of Canada and departmental priorities which include attracting top-quality candidates and high quality performers as well as offering flexible staffing options for a sustainable work force.

Some senior managers, from the Director- through Assistant Deputy Minister-levels, expressed a need to maintain a strong analytical capacity at INFC, as analysts work on programs of significant materiality and complexity. Some EC work is viewed as the backbone of INFC's business.

The objective of the ECDP - to attract, develop and retain analysts - is an evident area of focus for the Government of Canada and INFC.

The Clerk's Twenty-Second Report to the Prime Minister (2015) highlighted the need to reinforce the policy community as a profession and went on to mention that maintaining programs aimed at recruiting top-quality graduates as future leaders should continue to be a priority for the Public Service. The Blueprint 2020 Initiative (2014) supports this and aims to look at current employment models, jobs and organizations, and new competencies needed in leaders, managers and employees. It encourages new ways to empower and motivate employees, build internal capacity, and provide flexible means of working. The Prime Minister's Advisory Committee Ninth Report (2015) discussed the importance of learning and training as an investment in productivity and as a lever for recruitment and retention. These priorities are aligned to the program outcomes of attracting, developing, retaining and sustaining a working level capacity.

In alignment with the Government of Canada, in 2015-2016 INFC^Footnote1, is focusing on building and sustaining the capacity of a high-performing workforce. Particularly the department plans to continue to renew its workforce and support employee development. This includes providing targeted and relevant learning opportunities and developmental programs to INFC employees.

Finding 4: The use of a development program is an appropriate and common practice to support human resource strategies.

The use of development programs to sustain a working level and leadership pipeline in departments and across the Public Service is common practice in the Government of Canada.

Under the Financial Administration Act^Footnote2, Deputy Heads have direct authority for learning, training and development and this authority has been sub-delegated in INFC across the Supervisor- to Assistant Deputy Minister-levels.

The continued use of ECDPs by other federal government departments to attract, develop and retain ECs suggests that development programs can be a good mechanism to support people management plans and strategies.

Performance (Achievement of Outcomes)

The evaluation examined to what extent the ECDP has achieved its expected outcomes as of 31 March 2015. The findings in this section are organized by outcome as identified in the ECDP logic model depicted in Figure 1 under the Program Profile section of this report.

Extent to which key stakeholders are aware of and understand the ECDP

Finding 5: Though participation in the program is high, there is a varied level of understanding of the program.

The ECDP benefits from an active partixipation of 26 eligible^Footnote3 EC-02, -03 and -04 positions in the participating Branches; 24 are ECDP positions. Those who were INFC employees prior to becoming ECDP participants indicated at least knowing of the program prior to becoming enrolled within it. The majority of participants indicated having heard about the program by word of mouth or from their managers.

However, there is a varying degree of understanding across participants and management active in the program. Some participants still had questions related to the intent of the program including: “How does one become a participant?” and “Is the ECDP a talent management program?” Similarly, the degree of program knowledge varies across supervisory and management stakeholders. Though the program in general is known, not all interviewees in these groups were fully familiar with the program parameters and some wondered how the program really differs from the PSPMP.

Finding 6: The level of external awareness of the INFC ECDP is unknown. However, the program has been a deciding factor among external hires for joining INFC.

External awareness of the program was difficult to gauge as 21 of the 33 (63%) participants and graduates were already INFC employees when enrolled in the program; 14 of whom were already in an EC position that was later enrolled into the program. The remaining individuals either deployed internally to an EC/ECDP position or were working in non-EC positions that were converted into an EC/ECDP position.

Of the 12 participants who entered the program as new-hires, 8 learned of the ECDP during discussions on deployment. Only one applied knowingly to an ECDP position through an advertised poster.

External hires into the program and experiences from other government departments suggest that the program could be a good mechanism for attracting talent to INFC. Of the 16 participants and graduates who had to make a decision to come to or remain^Footnote4 at INFC, 15 indicated that the ECDP was a determining factor in accepting their offer. Many participants indicated that ECDP is a significant opportunity for development and the progression aspect was appreciated; it was viewed as INFC investing in their future.

The majority of the other government department programs reviewed undertake university recruitment. The few interviewees who commented on awareness indicated that hundreds of recent graduate candidates applied to their positions when advertising their development programs via the Public Service Commission's (PSC) University Recruitment.

Extent to which participants are engaged in continuous learning and development

Finding 7: All participants are aware of the ECDP Learning and Development aspects of the program and are involved in developing and implementing their learning and development plans. Participants view the identification of their learning needs as their responsibility.

Per the ECDP, learning and development plans (LDP's) have been put in place for each program participant. Most participants (16 of 24) indicated being proactive in identifying and suggesting learning with many indicating that they drove the process. Both supervisor focus groups agreed, indicating that ECDP participants are engaged and active in producing and maintaining their LDP's.

For the most part (80%), participants indicated that their LDP's were reviewed at least once in the 2014-2015 year with their manager. The review is mostly triggered by the Performance Management Program cycle; as suggested in the ECDP guidelines. Of those who did not have their plans reviewed, some reasons given included that they were on assignment and their current manager felt that their substantive manager should review it, or that there had been turnover amongst their managers and it had not yet occurred.

Extent to which participants have learning and development opportunities related to career development

Finding 8: Planned learning and development for participants is aligned with the program guidelines.

Most (9/10) LDP's reviewed align with the 2013 ECDP guidelines on learning and development which note that LDP's should include a variety of formal courses, assignments and other activities. It was found that the LDP's consisted of a variety of learning and development activities and were linked to ECDP competencies. Out of the 10 LDP's reviewed, 6 had a combination of 3 different types of learning (i.e. courses, assignments, and other activities), while 3 had at least 2 types of learning and development activities. Most of the LDP's reviewed (8 out of 10) also included the suggested learning activities outlined in the ECDP guide.

Finding 9: Though job-related learning is supported and occurring, development opportunities appear to be lacking. Learning and development opportunities within the ECDP are not perceived by participants as being better than those available outside the program.

Interviewees from both the participant and supervisor groups indicated that the program provides a framework to facilitate discussion on learning and career development. Overall, supervisors and Directors feel they can give good on-the-job experience through complex work, training, and coaching; in part due to the characteristics of their ECDP participants and in part due to the fact that they themselves are conscious of learning. Most participants agreed that there is an overall effort to provide support for learning and training. A couple of participants indicated that the ECDP may have been given as a rationale for approving certain learning activities, especially when budgets were constrained.

However, both participants and management agreed that what an ECDP participant experiences is not likely any different than what a non-participant experiences, particularly with regards to development. Supervisors indicated that they actively support both participants and non-participants in the same way.

Most participants agreed. Of 23 participants that responded^Footnote5, only three indicated that the ECDP provided them with better learning and development opportunities. Whereas 13 indicated a negative response and 7 were unsure. Some participants felt that they received more complex work simply because they have proven they can perform it. Many participants cited that development experiences, such as assignments or secondments, to acquire a breadth of experience or organizational awareness, were lacking or not supported in the spirit of the program. The few participants that did go on assignment indicated that it was due to their own initiative and not through the structure of the ECDP. Other participants identified assignment opportunities but their request to participate in them was denied. Supervisors and Directors cited that operational requirements are usually the reasons given for the inability to host or release a participant temporarily. Finally, some participants and supervisors indicated that they were expecting that assignments, coaching, group training, etc. would be built into an established program structure or curriculum.

Extent to which ECs have developed the competencies critical for INFC's business

Finding 10: Overall, it appears that participants are acquiring competencies and doing so within the minimum time-at-level requirements of the program. However, competencies gained are not likely attributable to the ECDP.

Since 2009, there have been 18 promotions. Overall, 75% of attempts at promotion were successful. All unsuccessful attempts have occurred at the EC-05 board and since the program was re-launched in 2013. As at 31 March 2015, 40% of the unsuccessful candidates have since re-attempted the EC-05 board.

Participants, graduates, supervisors and Directors felt that ECDP participants are acquiring competencies and are doing so within the minimum time-at-level required. As per the table below, when discounting the period for which the program was on hold^Footnote6, candidates were promoted shortly after the minimum time-at-level requirements were met:

Table 3: ECDP promotions and average time-at-level distributed by progression level, by program period from 1 April 2009 to 16 June 2015

Progression	Minimum Time-at-Level Requirement	Average Time-at-Level*	Number of Promotions
			Pre-Program Re-Launch	Post-Program Re-Launch
EC-02 to EC-04**	12 Months	22 Months	3	0
EC-03 to EC-04	12 Months	10 Months	1	5
EC-04 to EC-05	18 Months	19 Months	5	4
Total			9	9

Source of data: INFC Human Resources Data from 2009-2010 to 2014-2015.
*Average time-at-level may be shorter than the minimum time-at-level as participants were able to account for time-at-level while the program was on hold.
**All promotions of EC-02s to date have occurred before the program introduced the EC-03 level. Therefore, promotions to date have been from the EC-02 to EC-04.

Some supervisors and Directors also noted that some participants could even benefit from shorter timelines. They viewed ECDP participants as high performing individuals that are engaged and to whom they could give more complex work to.

However, supervisors, participants and graduates were for the most part unsure of whether the competencies gained are attributable to participation in the program. Just over half of participant and graduate respondents, who had participated in the program long enough to opine (n = 27), indicated that competencies were not likely gained as a result of the ECDP. A further 30% were unsure. Only 3 participants and 2 graduates attributed competencies gained to the ECDP.

Those that were unsure or did not feel that competencies acquired were attributable to ECDP stated that they gained competencies in part by being proactive in requesting learning opportunities and in part to having a great working relationship with their managers, who in turn provide good work and coaching. A few indicated having gained competencies while working across various parts of INFC, while others acquired them prior to coming to INFC. Three participants were employed at a higher level prior to becoming an ECDP participant.

Extent to which ECDP has enabled INFC to retain ECs, and manage and meet staffing/competency needs

Finding 11: On average, ECs are staying at INFC for periods of time comparable to other occupational groups. ECDP seems to be becoming less of a factor for remaining at INFC, particularly for newer participants and those at EC-04 level.

Since 2012-2013, the average number of years at INFC for the EC group has remained at similar levels ranging from 4.4 to 4.7 years. This is comparable over the same time period to other large occupational groups within the department where the average number of years at INFC ranged from 3.8 to 4.2 for the Administrative Services (AS) Group and from 4.0 to 5.5 for the Information Services (IS) Group.

Since program inception, only 4 participants have left prior to graduating from the program and 7 of 10 graduates remain at INFC. On average, graduates have stayed at INFC longer than the EC group as a whole. As per Human Resources data as at 31 March 2015, graduates had been at INFC for an average of 5.6 years.

Graduate interviewees currently with INFC all indicated that INFC still offers challenging and interesting work in a good environment. They cited that the work is still fresh and new and they get to do it with colleagues they enjoy working with. When asked if they plan to leave INFC in the next two years, one said yes, two were unsure, and four intended to stay. Those who indicated that they may leave or are planning to leave in the next two years did so more reluctantly. These interviewees mentioned that they have been at level or at INFC for a period of time and felt that for their own development it may simply be time to move on and acquire another experience.

While graduates seem more inclined to stay, the ECDP seems to becoming less of a factor for current participants to remain at INFC, particularly for newer participants. When asked, half of the current participants indicated that the ECDP was a factor for remaining at INFC to date. However, as per the table below, only 3 current participants plan to stay at INFC for the next 2 years. 11 intend to leave while 10 are unsure.

Of the participants (21) and graduates (3) that indicated they will or may leave in the next two years, 17 (71%) are individuals who became ECDP participants more recently; they were enrolled after the program was re-launched in September 2013. What is more, all but one EC-04 (14 of 15) intends to leave in the next two years or is unsure.

As at 31 March 2015, on average, the EC-03 and EC-04 participants have been at INFC for 4.3 and 4.0 years respectively. When asked why the ECDP has been or is a factor for remaining at INFC, current participants provided varying views. Participants intending to stay indicated that they enjoy INFC and hope to grow here. Some indicated that the program offered personal stability and hoped to leverage the program for further learning and development. Others indicated they have invested in the ECDP by having taken a demotion to enter the program.

Participants intending to leave also indicated that they enjoy working at INFC; particularly due to its people and environment. However, some would like to do different types of work and or explore a new department. Others feel they have been working at level, or even above level, for a period of time and are actively seeking opportunities for progression. A few indicated being significantly disappointed in the learning and development aspects of the ECDP; that the program did not unfold as it was sold.

Finding 12: There is a perception that the ECDP is in part helping maintain a stable working level at INFC; viewed mostly as the EC-05. However, there are concerns related to a need for positions at the junior levels and organizational top-heaviness.

Some Director-level and Director General-level interviewees perceive that the ECDP is in part helping develop and maintain a stable number of working level ECs, indicating that ECDP had helped sustain the EC-05 capacity of their directorates. It was viewed that the program was a good mechanism to attract and retain junior ECs. They felt that strong EC-05s have resulted from the program and that junior employees are quickly showing the capability to progress and work at the next level.

As per graph 4 discussed previously, the number of EC-05s in the department has remained fairly consistent - just below or above 25 - since the program launched in 2009. Currently, there are 28 EC-05s employed, which is only one more EC-05 than in 2009 when the program was established. While the number of EC-05s has remained consistent, as previously discussed in finding 2, to supply the development program, it would be expected that a consistent number of junior level EC positions would be required to be filled. The data shows fluctuations across the junior levels over the program period.

Some senior management interviewees also expressed concerns related to the organization increasingly becoming top heavy due to almost all positions being enrolled in the program; creating a concentration of employees at the EC-05 level, with equal or higher number of EC-06 and EC-07 positions.

Performance (Design and Delivery)

The evaluation considered the extent to which current aspects of the program design were appropriate and to what extent does the current delivery model support the achievement of the program's objectives.

Competency Profile

Finding 13: The ECDP competency profile was viewed as a good tool to facilitate discussion on progression and on the implementation of learning and development. However, there is indication that the current competencies may need to be reviewed.

Similar to other government departments, INFC identified competencies that aligned with those defined by TBS for the whole of the Public Service as well as those that are pertinent to the departmental needs.

Generally, participants, supervisors and Directors like having a framework for discussing competencies and for producing a plan for employee learning and development. They indicated that a development path, especially for junior employees, was good; it provides a concrete trajectory to follow.

However, there is indication that the program competencies could be reviewed. Firstly, the OCHRO Key Leadership Competency profile (2005) on which the program is partly based was recently updated and began implementation in 2015. The new profile includes competencies that underscore the current ones as well as two new competencies. Secondly, participants and supervisors indicated that it can be difficult to distinguish the real difference between levels for some competencies. Some competencies even have the same behavioural descriptor for various levels. Lastly, a few participants felt the competencies might be too generic and do not take into consideration the differences in the work that ECs do across the department (i.e. Program Operations work versus Policy work).

Levels Targeted

Finding 14: There is indication that different EC levels could be targeted by the ECDP at INFC.

The program includes the EC-02 through EC-05 levels, like most of the other federal Government Department programs reviewed. Intake is determined individually by delegated managers based on their staffing and capacity needs.

Based on an overview of program level data, there is indication that different EC levels could be targeted by INFC for development. Since the 2013 re-launch of the program, 80% of participants were enrolled as an EC-03 or EC-04. As shown in the table below, of the 20 participants enrolled in the program since 2013, 11 were enrolled at the EC-03 level, 5 at the EC-04 level, and 4 at the EC-02 level. While, over time there have been 8 EC-02s enrolled in the program, as at 31 March 2015, there were only two participants at the EC-02 level and only a total of 3 EC-02s in the department.

Table 4: Number of participants by ECDP entry level, by program period

ECDP Entry Level	Total	Intake before re-launch		Intake after re-launch (as of September 2013)
	Number	Number	%age	Number	Percentage
EC-02	8	4	31%	4	20%
EC-03	13	2	15%	11	55%
EC-04	12	7	54%	5	25%
Total	33	13	100%	20	100%

Source of data: ECDP Program Data

Finding 15: Having multiple levels targeted by the program was viewed as an effective way to support staffing needs.

Hiring Managers indicated that being able to individually intake at all levels provided flexibility and made the ECDP a responsive HR management tool. It was mentioned that the program is an attractive feature to recruit employees who are already experienced.

Learning and Development

Finding 16: Structured learning and development is not embedded within the program.

As supported by earlier findings in this report, the learning and development aspect of the ECDP was identified as an area of improvement by most interviewees across all groups.

As per the ECDP guidelines, learning and development is tailored to the individual. The program provides suggestions and guidance for how this could be undertaken.

Interviewees across all groups most often suggested to provide more structure to the program (built in mandatory learning, ECDP group learning, etc.) and to facilitate opportunities to gain broader experiences (mini-assignments, external assignments, etc.).

Without the structure, it was noted by interviewees, that some development aspects encouraged by the program, such as assignments, mentoring and coaching, were viewed as not being supported or as not really being part of the program.

An ECDP Steering Committee role is, in part, to review the LDP's to ensure support is in place. In practice, this does not appear to occur. Learning and development is overseen for each participant individually without an overarching program-level view. However this activity was viewed by the ECDP Steering Committee interviewees as one that could easily make a significant difference in the learning and development aspect of the program.

Minimum Time-at-Level Requirements

Finding 17: While the minimum time-at-level requirements appear to be sufficient for candidates to acquire the designated competencies per level, internal views on the effectiveness of this requirement vary.

The INFC ECDP minimum time-at-level requirements are in line with those of most other federal government department programs and, as discussed in Finding 10, appear to be sufficient. Candidates are for the most part acquiring the necessary competencies with the minimum time-at-level. However, the view on timelines is divided at INFC within and across the interviewee groups.

Some suggested that exceptions should be defined as some ECs progress faster than others; particularly at the more junior levels. This could minimize the risk of losing someone who can perform (or is already performing) more substantial work after a promotion.

Some interviewees suggested that time requirements should be removed all together to emphasize 'progress at your own pace.' This would allow for those who progress faster to do so and for those who need more time not to be set up to fail by attempting a board too early. Some indicated that the minimum time-at-level requirements are interpreted as the date at which a participant is expected to be ready for promotion. Finally, others indicated that the minimum time-at-level requirements may be too short citing that the work conducted by an EC-05 is significantly different and more complex than that of an EC-02 and that three and half years is a short period within which to make such a jump in responsibility and complexity.

Participant Assessment

Finding 18: The 'board' experience is unique to INFC. Whether it should be a formal assessment tool is debated among internal stakeholders.

The evaluation did not specifically probe on the use of a board (exam, interview and reference checks) for the assessment of participants seeking to progress from the EC-04 to EC-05 level. However, unprompted, some interviewees from each interview group commented on the subject. Views varied across and within interview groups.

On one hand, some human resources, supervisor, Director and graduate interviewees felt that having a board was important, indicating the program will fail participants if at the end they do not know how to 'compete' within or external to INFC upon ECDP graduation. Some interviewees indicated that a participant who was bridged into the department as a student could progress to the EC-05 level never having done an exam or an interview within a competitive process. Some interviewees also indicated that the exam and interview processes are developmental in themselves and whether the board is an official assessment tool or a development module, it should be included in the program for participants at some point. A few other federal government department program interviewees agreed, indicating that they may explore (or are exploring) the use of a board within the upcoming reviews of their programs.

On the other hand, some Director- and Director General-level interviewees indicated that the assessment and promotion process must be rigorous and consistent but should not be laborious. Otherwise, it's contradictory to the principles of a learning and development program with an element of progression. Some graduates and current participants felt that the work should speak for itself and, when monitored and documented, should be sufficient to use as evidence of competencies gained and progression readiness.

No issues were raised with regards of relying on a paper board when assessing the progression from the EC-02 to EC-03 level and from the EC-03 to EC-04 level.

Governance, Oversight and Communication

Finding 19: Program governance and oversight is not fully developed.

There is a sense from interviewees at all levels that the ECDP is perceived as an “HR program” and that ownership is unclear. Some Director General and Assistant Deputy Minister interviewees agreed; indicating that they should be more involved in the management of the program. Some interviewees from these groups indicated that the ECDP is not discussed regularly and that performance measurement of the program is not undertaken.

Further to this, some ECDP Steering Committee members indicated that roles are not always clear when it comes to committee business. One example given was that, at times, it can seem that committee discussions are occurring 'with HR Services' and not as a committee making both the committee chairmanship and decision-making process unclear. In addition, it was noted that non-HR Services ECDP Steering Committee Members are also the delegated managers of some participants, therefore there is potential for a real or perceived conflict of interest among these members.

Finally, there was acknowledgement by some ECDP Steering Committee member interviewees that the program could benefit from more committee involvement (i.e. greater oversight and discussion on the complement of participants, their needs and progression).

Finding 20: There are gaps in some program guidelines and processes.

Upon reviewing the process against the input provided during the interviews with INFC stakeholders and with other federal Government Departments, it was found that there are some gaps in program processes and guidelines. For example, there are no provisions for recourse when a candidate is unsuccessful during the assessment process, for candidate removal or exit from the program, for when the first LDP should be submitted, etc. There is also a lack of clarity on what criteria are applied to enroll a position or individual in the program.

Finding 21: There is a lack of clarity on some program guidelines and processes. Information sharing and communication were often cited as an area of improvement.

It was found that some program guidelines are not clear or are not documented in the current 2013 ECDP Guide, which is the only documented resource for stakeholders. One example of a rule that is applied, but not documented, is when a candidate is unsuccessful during an assessment, they cannot be re-assessed for a minimum of 6 months.

Interviewees across all groups indicated that information sharing could be improved both in the Guide and via other means. Participants sense that there are various amounts of or conflicting information among them. They cited examples of receiving conflicting guidance on the ability to use previous experience to demonstrate competencies gained or towards time-at-level, on having a second manager attest to their competencies, and on having to show only one or more concrete examples of how competencies are met. Many indicated that it would be of benefit to know who is in the program and to have facilitated group networking/information sessions to share information on learning and development and to be kept apprised in a consistent manner of program-related information, decisions or changes.

Supervisors and managers indicated that it would be helpful to know what tools are available and are being used by others within the program. For example, is the INFC mentorship program still active and is it (or can it be) linked to the ECDP?

Director General- and Assistant Deputy Minister-level interviewees indicated that continuous communication on what the program is and is not is needed in order to ensure it is being applied appropriately across the Branches and that participants are receiving common messaging.

Performance (Efficiency and Economy)

The evaluation considered whether there were more efficient ways to deliver the program or achieve the intended results. The following details the findings, while design alternatives from other government department programs potentially suitable for INFC, are found in Annex C.

Finding 22: Hiring Managers view ECDP as a cost-effective staffing tool.

Director-level interviewees found that the ECDP can at times create economies in that the program can sometimes attract individuals who are experienced at their level, which allows them to become fully activated on the job more quickly. Additionally, developed program participants who are familiar with INFC and the work at the next level can be promoted through a simplified process at the appropriate time upon qualifications being met. This was viewed as less costly (time, people, effort) than recruiting externally to the department and then, in turn, training and developing the new employee or than running a competition at each level each time. It is important to note that hiring managers did express that a balance of new and internal hires is needed, as external recruits bring new perspectives and experiences.

Finding 23: With the implementation of the new Performance Management Program, a duplication of effort with the ECDP has developed.

The LDP's were viewed by supervisors and participants as good for structuring discussions on and planning for learning and development. However, some interviewees in these same categories indicated that it is unclear how the PSPMP plans and activities and the ECDP complement one another. Since the inception of the ECDP, performance management practices government-wide have evolved. The new PSPMP was implemented in 2014. Supervisor and participant interviewees expressed that there is a duplication of effort in having to complete an ECDP learning plan and a PSPMP learning plan. Some mentioned that they use the ECDP plan as their PSPMP learning plan; which requires them to produce the same document in two different formats.

Furthermore, the PSPMP Performance Agreement now includes employee objectives related to competencies including core, functional (i.e. specific to analysts), and technical. The core competencies that everyone must demonstrate overlap in some cases with the ECDP competencies, which is expected given they are both sourced from TBS competency profiles. Also, the PSPMP employee learning and development plan encourages specific learning objectives that are job-, department- and Public Service-specific as well as learning to support employee achievement of their career development goals.

Finding 24: The assessment process is viewed as laborious due to documentation requirements and a lack of clarity on some administrative aspects.

Some participants conveyed that the assessment process was paper heavy indicating that they spend a lot of time completing their assessment documents and keeping them up to date. Some participants and supervisor interviewees also noted that the documents to be completed by the participant and the supervisor were similar and felt it was in part a duplication of effort.

Both participant and supervisor interviewees who had more recently prepared or participated in an assessment indicated that there were a lot of questions requiring assistance from Human Resources. For example, the 2013 guide indicates having to show only one concrete example; however, the participants' managers insisted they document three.

Finding 25: The lack of dedicated resources to the program creates ineffectiveness.

HR Services interviewees indicated that there are periods where the ECDP workload can be quite intensive, particularly during the assessment periods or when there are new candidates enrolled in the program.

Some participants and managers indicated that when requesting information, it can take time to receive responses from the program secretariat. They, however, acknowledged that there has been turnover within HR Services as well as reduction of personnel dedicated to the program. In addition, some participant, supervisor and HR Services interviewees indicated that ECDP Steering Committee meetings can be a challenge to organize on demand as it requires the coordination of various Director and, sometimes, Director General level members' schedules. This was noted as sometimes resulting in the overall assessment process occurring over a few months as there is a delay between when a candidate is put forward by their manager for assessment and when the ECDP Steering Committee participates in the assessment.

The ECDP Steering Committee members from the participating Branches also acknowledged that it would be beneficial to play a more active role (i.e. to review learning plans, discuss candidate progression and needs); however, they indicated that without support, they are not likely able to coordinate such efforts due to time constraints.

Conclusions

Relevance

In line with the priorities of the Clerk of the Privy Council and of Blueprint 2020, there is a responsibility to develop strong employees in the department and in the Public Service. INFC management indicated that there is a need to be more strategic and structured in developing and sustaining a strong EC community at INFC, and ECDP participants at INFC value development and career enhancement opportunities.

While the continued use of ECDPs by other federal government departments suggests that development programs can be a good mechanism to support people management plans and strategies, the organizational needs related to ECs at INFC appear to be changing and are not fully known. The 'working level' may no longer be that which was targeted upon program creation and may vary across branches. Further to this, while ECs at INFC have remained at INFC for periods comparable or longer to ECs elsewhere in the Public Service, the more senior ECs at INFC, which are not part of the ECDP, have been the largest share of departures and many at these levels have indicated their intention to leave INFC in the next two years.

The need to strategically attract, retain and develop ECs in order to sustain a strong capacity at INFC is relevant. However, given that the current INFC business needs related to ECs are unclear, it is difficult to conclude that the ECDP targeted to sustain the EC-05 level is needed.

Performance - Achievement of Expected Outcomes

With the exception of attracting individuals to INFC and engaging in learning and development, little evidence was found attributing outcomes related to EC recruitment, development and retention specifically to the program.

There is good level of engagement in learning and development and competent ECs are progressing through the program as per the timelines. However, learning and development would likely not be any different for participants if they were not in the program. Both learning and development and competencies gained were generally attributed to conscientious managers and proactive participants and not to the ECDP.

While the ECDP was partly identified as a factor for remaining at INFC to date - and most graduates have in fact remained - on average, ECs are staying at INFC for comparable periods of time to other large occupational groups at INFC and to other ECs in the Public Service. While graduates have been more inclined to stay, the ECDP seems to be becoming less of a factor to remain at INFC for newer participants as many are considering leaving INFC in the next two years, including most EC-04s.

While the number of EC-05s has remained consistent since program inception, with the limited achievement of outcomes directly linked to the program, the evaluation cannot evidence that the program is contributing to sustaining a working-level capacity of ECs.

Performance - Design and Delivery

The ECDP at INFC was found to align with the design of other government departments with regards to the identification of competencies, the levels targeted, and the minimum time-at-level requirements. It differs from them with regards to the assessment process and governance. Some aspects denoted as strong by interviewees were unique to INFC. For the most part, however, areas of improvement within many aspects of the program were identified. Some core issues were found to be impacting the overall delivery of the program, such as:

• That the ECDP does not a have a 'home' within INFC. Ownership of the program is unclear resulting in limited involvement by senior management and a lack of governance. Clarity on roles and responsibilities is needed.
• There is no 'program' approach to the ECDP. Enrollment, learning and development and assessment are delivered in silos by managers and supervisors across the organization, and learning and development is not embedded into the program.
• There are gaps in program parameters and inconsistent documentation and communication of program details resulting in a varied understanding about what the program's intent is and how it works.

Performance - Efficiency and Economy

Given the program does not have a budget allocated and costing information is not available, the evaluation cannot conclude wholly on the efficiency and the economy of the program. The evaluation found that efficiencies could likely be gained. Specifically, a reduction in the duplication of effort between the ECDP and PMP and to the level of effort required for the assessment process could be made. And it appears that reducing the time dedicated to obtaining clarity on some of the administrative aspects of the assessment process and improving timeliness of ECDP Steering Committee meetings could also create efficiencies.

Recommendations, and Management Response and Action Plan

In line with the priorities of the Clerk of the Public Service, INFC is vying to become more strategic and innovative in employee recruitment and development. The importance of employee retention and development was echoed throughout this evaluation. Though not within the scope of this evaluation, it is important to note that all senior management interviewees indicated that recruitment, development and retention strategies are important to have in place for all occupational groups and not just for ECs and PEs, as is the case at INFC.

While some of the intended outcomes have been achieved, they cannot be clearly attributed to the ECDP. This, along with the overall performance findings, suggests that the program objectives and logic need to be reviewed. Simply revamping the program would not address the key issues impacting the achievement of outcomes.

Unprompted interviewees from the participant- to Director General-level emphasized that changes to or cessation of the current ECDP will have a significant impact on the morale of the current participants and the EC community at INFC.

Given the above and that all current participants, should they meet the competency requirements upon completion of minimum time at-level, could graduate by the end of fiscal year 2017-2018, the evaluation recommendations are as follows:

Recommendation	Management Response and Action Plan	Planned Completion Date	Office of Primary Interest
INFC senior management should identify a member of the management team from the 'business' to champion and monitor strategies related to EC people management.	Agreed. The Assistant Deputy Minister, Policy and Communications, has been identified as the EC Development Program (ECDP) champion and will work through Infrastructure's governance structure, as the senior level committees will play key roles in how the ECDP will be managed moving forward. The appointment of the champion will be communicated across the department.	Dec. 31, 2015	ADM Policy and Communi-cations
INFC senior management should maintain the ECDP for current participants while taking into consideration the findings in this evaluation and making short-term improvements , particularly: Clarifying governance (program ownership, senior management involvement, monitoring). Reviewing competencies and associated learning and development. Improving understanding of the current ECDP by re-confirming and documenting program guidelines. Reducing duplication of effort between the use of ECDP tools and the PS Performance Management Program. Developing and implementing a plan to communicate changes or updates to the ECDP and to ensure information is communicated to all regularly and consistently.	Agreed. Senior management will continue to maintain the ECDP for current participants and will make short-term improvements to the program considering the findings from this evaluation.	Sept. 30, 2016	ADM Policy and Communi-cations
In assessing people management requirements for the EC community, senior management should determine if an ECDP or other strategies are needed to meet departmental business requirements and overall objectives.	Agreed. Senior management will review human resource requirements for the EC community and develop strategies as required to meet the departmental need.	Dec. 31, 2016	Depart-mental manage-ment Committee

Annex A: Summary of Evaluation Findings and Conclusions

Relevance
1.	ECs at INFC remain in their positions for a slightly longer period of time than ECs across the Public Service. However, EC-05, -06, and -07s represent the majority of EC departures in the last 5 years and, along with EX-01s, indicate that they intend to leave INFC in the next two years.	Conclusion: In line with the priorities of the Clerk of the Privy Council and of Blueprint 2020, there is a need at INFC to be more strategic and structured in sustaining a strong EC community. While development programs are commonly used and are appreciated by participants and management alike, the organizational needs and challenges related to ECs at INFC appear to be changing and are not fully known. Therefore, the need to strategically attract, retain and develop ECs in order to sustain a strong working-level capacity at INFC is relevant. However, it is difficult to conclude that an ECDP specific to developing individuals to the EC-05 level is currently needed.
2.	The organizational needs related to ECs at INFC are not fully known. The 'working level' may neither be the same across INFC branches nor that which was defined upon program creation.
3.	The program objectives continue to be in line with Government of Canada and departmental priorities which include attracting top-quality candidates and high-quality performers as well as offering flexible staffing options for a sustainable work force.
4.	The use of a development program is an appropriate and common practice to support human resources strategies.
Performance (Achievement of Expected Outcomes)
5.	Though participation in the program is high, there is a varied understanding of the program.		Conclusion: With the exception of attracting individuals to INFC and engaging in learning and development, little evidence was found attributing outcomes specifically to the program. Though candidates are progressing through the program and are considered to be highly competent, both learning and development and competencies gained were generally attributed to conscientious managers and proactive participants and not to the ECDP. ECs are staying for comparable periods of time to other large occupational groups at INFC and to other ECs in the Public Service. While graduates have been more inclined to stay, the ECDP seems to be becoming less of a factor remaining at INFC for newer participants. While the number of EC-05s has remained consistent over time, with the limited achievement of outcomes directly linked to the program, the fluctuation of EC positions at the junior levels over time and equal number of more senior EC positions, the evaluation cannot evidence that the program is contributing to sustaining a working-level capacity of ECs.
6.	The level of external awareness of the INFC ECDP is unknown. However, the program has been a deciding factor among external hires for joining INFC.
7.	All participants are aware of the ECDP Learning and Development aspects of the program and are involved in developing and implementing their learning and development plans. Participants view the identification of their learning needs as their responsibility.
8.	Planned learning and development for participants is aligned with the program guidelines.
9.	Though job-related learning is supported and occurring, development opportunities appear to be lacking. Learning and development opportunities within the ECDP are not perceived by participants as being better than those available outside the program.
10.	Overall, it appears that participants are acquiring competencies and doing so within the minimum time-at-level requirements of the program. However, competencies gained are not likely attributable to the ECDP.
11.	On average, ECs are staying at INFC for periods of time comparable to other occupational groups. The ECDP seems to be less of a factor for remaining at INFC, particularly for newer participants and those at EC-04 level.
12.	There is a perception that the ECDP is in part helping maintain a stable working level at INFC, viewed mostly as the EC-05. However, there are concerns related to a need for positions at the junior levels and organizational top-heaviness.
Performance (Design and Delivery)
13.	The ECDP competency profile was viewed as a good tool to facilitate discussion on progression and on the implementation of learning and development. However, there is indication that the current competencies may need to be reviewed.	Conclusion: Some design and delivery aspects of the ECDP were denoted as strong by interviewees and unique in comparison with other government departments. However, areas of improvement were found within all aspects of the program. Core issues identified to be impacting the overall delivery of the program include: Ownership of the program is unclear and there is a lack of clarity on roles and responsibilities. There is no 'program' approach to the ECDP. The program is mostly delivered in silos throughout the organizations. There are gaps in program design and inconsistent documentation and communication of program rules and processes.
14.	There is indication that different EC levels could be targeted by the ECDP at INFC.
15.	Having multiple levels targeted by the program was viewed as an effective way to support staffing needs.
16.	Structured learning and development is not embedded within the program.
17.	While the minimum time-at-level requirements appear to be sufficient for candidates to acquire the designated competencies per level, internal views on the effectiveness of this requirement vary.
18.	The 'board' experience is unique to INFC. Whether it should be a formal assessment tool is debated among internal stakeholders.
19.	Program governance and oversight is not fully developed.
20.	There are gaps in some program guidelines and processes.
21.	There is a lack of clarity on some program guidelines and processes. Information sharing and communication were often cited as an area of improvement.
Performance (Demonstration of Efficiency and Economy)
22.	Hiring managers view the ECDP as a cost-effective staffing tool.	Conclusion: Given the costs related to the program are not tracked nor does it have a budget, the evaluation cannot conclude wholly on the efficiency and the economy of the program. The evaluation found that a reduction in the duplication of effort between the ECDP and PMP and to the level of effort required for the assessment process could be made. Other efficiencies could be gained by improving clarity of the assessment process and timeliness of ECDP Steering Committee meetings as well as considering design alternatives implemented by other government departments.
23.	With the implementation of the new Performance Management Program, a duplication of effort with the ECDP has developed.
24.	The assessment process is viewed as laborious due to documentation requirements and a lack of clarity on some administrative aspects.
25.	The lack of dedicated resources to the program creates ineffectiveness.

Annex B: Detailed Methodology

The purpose of the evaluation is to provide a neutral assessment with respect to supporting a Departmental Management Committee discussion on whether there is continued need for the program. A particular focus was placed on whether the raison d'être of the program is still relevant and the extent to which the ECDP is achieving its objectives. Whether the program's design and implementation remains effective in relation to current departmental needs was also examined.

Scope, Approach and Methodology

The evaluation considered all aspects of program activity from its inception in 2009 to 2015.

The assessment was conducted in alignment with the five core issues, as prescribed by the Treasury Board Policy on Evaluation, 2009. The study addresses a number of questions related to the continued need for the program, its alignment with government priorities, its consistency with federal roles and responsibilities, the achievement of its expected outcomes, the effectiveness of the design and delivery and the extent to which it demonstrates efficiency and economy.

Both qualitative and quantitative information were collected through the following lines of evidence:

• A document review was conducted to understand and assess the current program design and delivery, alignment with departmental strategies and government priorities.
• ECDP administrative data was analyzed to inform progress towards achieving program outcomes and program delivery.
• Secondary Data from sources such as the Public Service Employee Survey and the Office of the Chief Human Resources Officer (OCHRO) Public Service Management Dashboard (PSMD) were analyzed to inform both program relevance and performance.
• Key informant interviews were conducted internally to inform all aspects of the evaluation. Approximately 54 internal interviews were conducted with 61 interviewees including: program management (ECDP Steering Committee, HR Resources personnel), Management with EC employees, and current participants and graduates. The breakdown is as follows:



Table B.1: Distribution of Interviewees and Interviews by Stakeholder Group

Interviewee Group	# of Interviewees per Group	Interviews Completed
Assistant Deputy Ministers	4	4
Director General	7	7
Directors	7	7
Supervisors	7	2 Group Interviews
Human Resources Services	3	1 Group Interview
Participants	24	24
Graduates	9	9
Total	61	54



• A comparative review of development programs in other federal government departments was conducted in order to identify alternative delivery models and more cost-effective ways of obtaining intended outcomes. Document reviews and interviews were undertaken where possible with the following programs:

Table B.2: Comparator Departments

Program Type/Department	Status	Document Review	Interview
EC-Specific Development Programs
1. Natural Resources Canada	Active
2. Statistics Canada	Active
3. Canada Border Services Agency	Active
4. Transport Canada	Active
5. Environment Canada	Active
6. Department of Finance	Active
7. Industry Canada	Active		N/A
8. Agriculture and Agri-Food Canada	Active
9. Employment and Social Development Canada	Discontinued
10. Treasury Board Secretariat	Discontinued		N/A
11. Office of the Comptroller General (Multi-Department, Advanced Policy Analyst Program)	Active
Other Development Programs and Initiatives
12. Analyst Learning and Development Strategy (TBS) (Multiple Occupational Groups)	Active		N/A
13. Financial Officer and Internal Auditor Recruitment and Development (FORD/IARD) Program (OCG, Multi-Department) (FI only)	Active

Limitations and mitigation plans

There were some limitations with the evaluation.

Lack of control group and potential interview bias: Given that virtually all eligible positions are enrolled in the program, there was not a control group of non-participants available to the evaluation team. In addition, it is likely that program participants (EC-02s, EC-03s and EC-04s) are very invested in the program and the expectations for development and advancement that it promises. Given this, there is risk that the interview findings related to program effectiveness and success could be biased towards a positive assessment.

Mitigation: To reduce this risk, interviews were conducted with program graduates who have experience in the program but are no longer personally invested in it as well as all participants, including those that were unsuccessful within the program. Multiple lines of evidence were also used to support the findings on program effectiveness and success.

Absence of program delivery cost Information and performance indicators: The program does not currently conduct performance measurement nor does it have information on program delivery costs. The program does not have a budget allocated to it. Therefore, the ability to conduct benchmarking of the program's efficiency and cost-effectiveness were limited.

Mitigation: To reduce this limitation, interviews with both internal stakeholders and with other government departments will be used to inform the assessment of program delivery efficiency and design alternatives.

Evaluation Working Group

To support the conduct of the evaluation, a working group was formed. The working group comprised representatives of the Evaluation Directorate and from Human Resources involved in the delivery of the program. The working group ensured: that issues addressed were consistent with TB policies and guidelines related to evaluation and program accountability; that results were based on reliable and defendable results, anchored in methodology that is the most appropriate in the context of the evaluation; and that the evaluation was carried out collaboratively by ensuring that all key stakeholders were represented and provided with an opportunity to present their ideas and opinions.

Annex C: Design alternatives that may be suited to INFC

Upon review and discussion with other government department programs, it was found that some design alternatives exist which may be worth exploring by INFC for the purposes of improving efficiency and cost-effectiveness. It is important to note that some of these programs differ significantly in design, that some have been in existence for over 30 years, and that some are delivered in other government departments with considerably larger EC populations. For consideration, some programs:

• Leverage and align to the Performance Management Program Talent Management component: Some programs reviewed have linked to the PSPMP. Rather than a time criteria, either development achievements or high performance ratings trigger assessment for promotion. Some other programs have the performance rating as part of their assessment for promotion within their INFC-similar programs or have a high performance rating as an indicator for assessment for an earlier promotion. These further linkages to the PSPMP reduce the duplication of effort across programs.


• Have a focus on learning and development rather than progression. Promotions occur as per organizational needs: Some programs' candidates, such as the Financial Officer and Internal Audit Development Programs (FORD/IARD), graduate from the program upon achieving on the learning and development goals and not upon promotion. For this program, as well as others, when a candidate is determined to have acquired enough competencies required for the next level, they are assessed and, if successful, put into a partially assessed pool where they are available for a full assessment for promotion on an as needed basis. For these programs, the focus is on the learning and development and not the progression aspect of the program for candidates. Progression occurs based on organizational demand and needs, and not based on candidate supply.


• Undertake a cohort approach to intake: Some programs, particularly with programs that exist as a strategy within their people management plans, undertake a cohort approach to intake based on a pre-determined number of program positions per year. This was found to be efficient by these organizations in that one staffing process occurs by the department (or in some cases, they leverage the PSC University Recruitment process) and that the candidates' entry, orientation, learning, etc. are coordinated and pre-planned per one timeline.


• Undertake a joint effort: The Advanced Policy Analyst Program (APAP) consists of multiple departments and central agencies that combine efforts to provide joint recruitment, learning and development for a small number of elite candidates. Candidates are dedicated to a 'home' department at the onset of the program and are required to return to their home department for a guaranteed period of time after successfully completing assignments in a combination of departments and central agencies. Recruitment responsibilities rotate across departments and all create and host assignments. The program was viewed as efficient and effective in that recruitment, selection, development and assessment responsibilities were shared across departments and that candidates received broad experience in a short period of time (2 years). Therefore, this could be a potential option should a portfolio approach be explored.

Annex D: List of Terms and Abbreviations

A&B	Audit and Evaluation Branch
ADM	Assistant Deputy Minister
AS	Administrative Services Group
CS	Corporate Services Branch
DG	Director General
EC	Economics and Social Science Services Group
ECDP	EC Development Program
FMB	Federal Montréal Bridges Branch
FPS	Federal Public Service
FTE	Full-Time Equivalent
HR	Human Resources
INFC	Infrastructure Canada
IS	Information Services Group
LDP	Learning and Development Plans
OCHRO	Office of the Chief Human Resources Officer
P&C	Policy and Communications Branch
PMC	People Management Committee
POB	Program Operations Branch
PS	Public Service
PSC	Public Service Commission
PSES	Public Service Employee Survey
PSMD	Public Service Management Dashboard
PSPMP	Public Service Performance Management Program
TBS	Treasury Board Secretariat of Canada

Annex E: References

Canada. Clerk of the Privy Council Office. Blueprint 2020. 2014.
Canada. Clerk of the Privy Council Office. Eighth Report of the Prime Minister's Advisory Committee on the Public Service. March 2014.
Canada. Clerk of the Privy Council Office. Ninth Report of the Prime Minister's Advisory Committee on the Public Service. March 2015.
Canada. Clerk of the Privy Council. Twenty-First Annual Report to the Prime Minister on the Public Service of Canada. March 2014.
Canada. Infrastructure Canada. Draft: Integrated Business Plan Fiscal Year 2015-2016.
Canada. Infrastructure Canada. EC Development Program. September 2013.
Canada. Infrastructure Canada. Integrated Business and Human Resources Plan Fiscal Year 2013-2014.
Canada. Infrastructure Canada. EC Development Program Re-Launch. Presentation to Managers. 25 September, 2013.
Canada. Infrastructure Canada. Instrument of Sub-Delegation of Human Resources Authorities. November 2014.
Canada. Infrastructure Canada. Integrated Business Plan 2014-2015. October 2014.
Canada. Infrastructure Canada. Report on Plans and Priorities 2014-2015. March 2014.
Canada. Infrastructure Canada. Report on Plans and Priorities 2015-2016. March 2015.
Canada. Infrastructure Canada. Training, Development and Learning Policy. 2004.
Canada. Justice Canada. Financial Administration Act, Section 12(a). 1985.
Canada. Office of the Chief in Human Resources Officer. A Leadership Development Framework for the Public Service of Canada. 2010.
Canada. Treasury Board of Canada Secretariat. 2011-12 Human Resources Management: Annual Report to Parliament. December 2012.
Canada. Treasury Board of Canada Secretariat. 2014 Public Service Employee Survey Results by Question for Infrastructure Canada. March 2015.
Canada. Treasury Board of Canada Secretariat. Demographic Snapshot of the Federal Public Service, 2013. March 2013.
Canada. Treasury Board of Canada Secretariat. Directive on Performance Management. 2013.
Canada. Treasury Board of Canada Secretariat. Policy on Learning, Training, and Development. 2008.
Canada. Treasury Board of Canada. Performance Management Program for Employees. March 2015.