Quality Performance in the Immigration Appeal Division 2019–2020

​​ Printable version (PDF, 1 MB)

​​Cat. No. MQ1-11E-​PDF ​

​​ISSN 2564-0291

​​​​​​​​​​​​​​Results Report

Prepared by:

Elaine Doyle and the Corporate Planning and Accountability Directorate
Immigration and Refugee Board of Canada

Table of contents

• 1.0 Context
• 2.0 Performance results
• 2.1 Timely and complete pre-proceeding readiness
• 2.2 Respectful proceedings
• 2.3 Focused proceedings
• 2.4 Reasons state conclusions on all determinative issues
• 2.5 Decisions provide findings and analysis necessary to justify conclusions
• 2.6 Reasons are transparent and intelligible
• Annex A– Checklist
• Annex B- SOGIE Checklist

1.0 Context

This study measures the quality of decision-making in the Immigration Appeal Division (IAD) by the Immigration and Refugee Board of Canada (IRB).

This study’s objective is to identify quality-related trends, patterns and issues in order to provide input into quality management at the divisional level. The results complement other available data sources—Federal Court decisions, evaluations, key indicator reports and quality measures from previous years—to understand the performance in the IAD in both qualitative and quantitative terms.

Sample methodology

The study reviewed 45 appeals randomly selected from a total of 62 appeals decided on their merits, after a hearing held before a single member, finalized between April 1, 2019 and September 30, 2019,. Members with less than six months’ experience hearing cases were excluded from the sample. All 21 experienced IAD members who finalized at least one appeal during this period are included in the sample.

All types of appeal for which a hearing was held and a decision delivered within the assessment period were included in the sample. All hearings of up to six hours were included.

Region, language of hearing, type of appeal, outcome (allowed or dismissed) and rate of representation by counsel are all proportionally represented in the sample.

The following diagrams illustrate the sample design:

Number of members

% of Hearings Per Region^{Footnote 1}​

Outcome

Appeal Type

Language of Appeal

Oral or written decision

Legal Representation

Assessment methodology

The person in charge of reviewing the files, also referred to in this report as the reviewer, was once a Division member. She examined all the evidence and the administrative documents in the file and listened to the complete audio recordings. She then assessed these against qualitative measures in a checklist developed by the Corporate Planning and Accountability Directorate and approved by the Chairperson in 2020 (see Appendix). The checklist has six categories:

1. Timely and complete pre-proceeding readiness
2. Respectful proceedings
3. Focused proceedings
4. Reasons include conclusions on all determinative issues
5. Decisions provide findings and analysis necessary to justify conclusions
6. Reasons are transparent and intelligible

Out of the 33 indicators, 17 apply to all hearings (universal indicators), whereas the others are assessed only as applicable. In six hearings, certain universal indicators could not be assessed, either because the appeal was withdrawn at the start of the hearing or, in some cases, there was a decision to allow the appeal taking into account the consent of counsel for the Minister. Each measure is assessed using a 1-to-3 rating scale or a categorical yes-no scale.

The 1-to-3 rating scale is as follows:

1=Does not meet expectations: The quality requirement was not met. The evidence showed one or more key instances where the proceeding or reasons would have markedly benefited had this requirement been met. There may have been an effort to apply the requirement but the level of achievement fell short of expectations.

2=Meets expectations: This is a level of acceptable achievement. On balance, the member satisfied this quality requirement though there is margin for minor improvement.

3=Exceeds expectations: This is a level of consistent, above-average achievement. The evidence shows a grasp of the quality requirement and an understanding of its importance to a high-quality proceeding or decision, as the case may be.

Results are also expressed as a percentage of hearings that meet expectations, scoring 2.0 or higher.

The sexual orientation and gender identity and expression (SOGIE) checklist was developed to evaluate how members apply the Chairperson’s Guideline 9: Proceedings Before the IRB Involving Sexual Orientation and Gender Identity and Expression. The checklist is used to assess the extent to which members treat appellants, claimants and applicants in cases involving SOGIE with respect and sensitivity throughout the hearing process.

The evaluation unit of the Corporate Planning and Accountability Directorate endeavoured​ to find hearings in which Guideline 9 applied by searching indicators in the case database. An online search of cases in the CanLII​ library using keywords was also conducted. Lastly, the reviewer received instructions to complete the checklist if the topic was addressed in a hearing or the member mentioned Guideline 9. No cases were identified. The checklist is appended for reference.​

Considerations and limitations

This study’s aggregate findings are accurate within a margin of error of ±6%, 18 times out of 20. This margin of error increases when data is broken down by region or case type. Therefore, the study does not advance any conclusions in cases where the sample size is insufficient to do so.

This study acknowledges the inherent limitations to qualitative research, which does not generate precise data like research involving quantitative parametres. To mitigate the inherent limitations of qualitative research, detailed performance indicators were provided to the assessor to help focus the assessment.

The findings of this report, including “What the Numbers Say”, Strengths, Areas for improvement and Recommendations sections remain those of the reviewer. The evaluation unit of the Policy, Planning and Corporate Affairs Branch provided the charts, statistics and “Considerations” for each section, along with the information in sections 1.1 “Context” and 2.0 “Performance Results.” The reviewer’s observations are necessarily subjective in nature and do not lend themselves to firm conclusions on legal matters such as the correct application of the law, the weighing of the evidence, or the fairness of the proceedings from a natural justice perspective. Only a court reviewing the case can arrive at such conclusions. This report aims to provide a perspective on how to improve the Division’s overall performance.

2.0 Performance results

What was measured

Each section of the results (2.1 to 2.6) has a table representing the number of hearings assessed for each indicator, the average score, and the percentage of hearings evaluated that scored 2.0 or higher. The average score is a finding that helps determine which indicators had strong or weak results, and helps inform the observations and recommendations. The number of hearings assessed is provided for reference and context only.

There are two performance targets for this evaluation:

• The first target is for each indicator to obtain a score of 2.0 or higher in 75% of all hearings assessed. This target was achieved for all 33 indicators, and is evident in the tables provided (see “% of hearings scoring 2.0 or higher” column in the table below). Even if an indicator has achieved this target, there may still be areas for improvement that are addressed in the reviewer’s observations below the table (see strengths, areas for improvement, recommendations).
• The second target is that 75% of hearings meet IRB quality expectations, which for the study is defined as an overall average score of 2.0 or higher for that hearing. To calculate this target, scores were compiled for each indicator assessed in each hearing, and averages per hearing were calculated. This target was achieved, with 100% of hearings assessed obtaining an average score of 2.0 or higher.​
  

2.1 ​Timely and complete pre-proceeding readiness

Why assess these indicators

The groundwork for quality is set before the hearing when the Registry prepares a timely, organized and complete file and the member understands the facts and key issues of the case

What was measured

Considerations

These indicators apply to all hearings.

What the numbers say

This study revealed certain differences in regional practices, including whether an “Exhibit List” document is used.

In the Central Region, members use the exhibit list to record the exhibits admitted into evidence, for each party, by assigning them an identifier (R-1, R-2, A-1, A-2, etc.), indicating information identifying the document (for example, “Certificate of Education”) and the number of pages in each exhibit. The Appeal Record is recorded as an exhibit filed by the Minister (R-1).

In the Western Region, members proceed the same way but do not record the Appeal Record as an exhibit filed by the Minister.

In the Eastern Region, this document is not used. In principle, parties filing exhibits attach an exhibit list, but they do not always do so. For example, for the cases under review, only slightly more than a third of the cases in this region had an exhibit list. As in the Western Region, the Appeal Record is not recorded as being an exhibit filed by the Minister.

I am of the opinion that having a document that lists the exhibits admitted into evidence is very helpful for keeping your place.

Strengths

• The ability to assign a case to a member a few days prior to the hearing is important in and of itself since it eliminates the need to postpone a hearing due to a member being unavailable.
• In the vast majority of cases, the file appears to have been given to the member in a timely manner. In a few rare cases (three hearings), the member was given the file a few days prior to the hearing, but it appears that this was due to a change of member.
• The members were familiar with the material in the file and were ready to proceed. The review shows that all members were ready for their hearings.

Areas for improvement

• Identification and filing of documents entered in evidence: The files were complete and contained all required documentation, with the exception of two files. In one case, the appellant and the member referred to a document sent by the appellant the weekend prior to the hearing. The member identified the document as “A 1​,” but the document does not appear to be in the file. Furthermore, no exhibits filed by the appellant were listed in the exhibit list. In another case, a pre-hearing conference was held three weeks prior to the hearing, but there was no document or Registry note in the file relating to the appellant’s notice to appear at that pre-hearing conference. The appellant was reached by telephone and appeared to be aware that he would be contacted.
• In relation to point 3, the cases reviewed reflect their status after the hearing and not before. The exhibits introduced into evidence in some files had no holes punched in them and/or were not attached to the file. In one case, many documents were filed into evidence: it would have been easier for document handling and storage to separate the file into two or three volumes as appropriate.
• In a few cases, members did not assign an identifier (e.g. A-1, A-2) to documents filed into evidence, or an identifier was assigned but not recorded on the document in question.
• In some cases, certain documents allowed in evidence were filed in the administrative section of the file (first part of the file) rather than in the evidence section (second part of the file).

Recommendation

• Remind members to assign an identifier to documents allowed in evidence (e.g. A-1, A-2) and to indicate this identifier on the documents.
• Ensure that all documents admitted into evidence (Appeal Record, evidence filed by the Minister, evidence filed by the appellant) are filed together.
• Make sure that all documents in a file are attached thereto.

2.2​ Respectful proceedings

Why assess these indicators

Individuals appearing before the IRB expect that they will be treated with sensitivity and respect. Any shortcoming in this regard potentially undermines tribunal integrity and public confidence.

What was measured

Considerations

Indicators #5 and #6 are considered universal—they apply to all hearings. Indicators #7, #8, and #9 are assessed as applicable.

What the numbers say

• In all cases analyzed, members treat participants with sensitivity and respect.
• In all cases, members ensure parties have an opportunity to present and respond to evidence and to make submissions. As a general rule, counsel for the appellant begins with their questions, followed by counsel for the Minister and the member, if necessary. When an appellant was unrepresented, the members explained to the appellant different possible ways of proceeding. For example, a proceeding could begin with the Minister’s questions and then the appellant is given an opportunity to complete their testimony afterwards; or if the appellant wished, they could begin by talking about their case and then counsel for the Minister would ask questions; or in some cases, the member would begin with general questions and then it would be counsel for the Minister’s turn to ask questions and the appellant would then be able to complete their testimony. In one case, the member did not ask the unrepresented appellant whether he had anything to add to his testimony after being questioned by counsel for the Minister; however, following the counsel for the Minister’s submissions, the appellant was invited to make submissions regarding his case.
• When the situation arose, members identified cases where the evidence had not adequately addressed an important issue. In one case, for example, this led the member to admit new evidence after the testimony was completed and to reopen questions on that document alone. In another case, the member suggested that counsel for the Minister discuss a concern raised during cross-examination with the counsel for the appellant, quickly narrowing the issue. In another case, the member explained to the unrepresented appellant what documents the appellant must obtain in order for the adoption to be considered valid within the meaning of the Immigration and Refugee Protection Act (IRPA).

Strengths

• Participants are treated with sensitivity and respect.
• All members ensure parties have an opportunity to present and respond to evidence and to make submissions, and identify cases where the evidence has not adequately addressed an important issue.

Areas for improvement

• Testimony with the assistance of an interpreter: As a general rule, members manage situations well when there is an interpreter at the hearing. However, it has happened that a witness who understood the official language used at the hearing answered questions before the interpreter would finish translating, meaning that the translation and the answer overlapped. Occasionally, the interpreter had difficulty translating everything a witness said because the witness spoke for too long without pausing.

Recommendation

• Remind members to step in quickly when a witness and an interpreter are speaking at the same time or when a witness testifies for too long without pausing, to ensure that the entire testimony is accurately translated.

2.3 ​Focused proceedings

Why assess these indicators

Proceedings that are efficient and well managed create favourable conditions for quality outcomes to emerge and support the IRB’s efforts to make the most effective use of its resources.

What was measured

Considerations

Indicators #10 to #12 are considered universal—they apply to all hearings. Indicators #13 to #22 are assessed as applicable. In certain hearings, universal indicators could not be assessed, either because the appeal was withdrawn at the start of the hearing, or the appeal was allowed on consent.

What the numbers say

• In all of the cases analyzed, the members ensured that the testimony and the documentation entered into evidence focused on relevant issues. In a number of cases, members were proactive and stepped in appropriately to limit questions that were not necessary for decision making and to clarify or narrow the testimony heard.
• For information purposes, the length of the appeal taken into account is the length of the audio recording. Naturally, in the majority of cases the actual length of the hearing was longer due to breaks and adjournments so that counsel and the appellant could talk to each other, etc. However, even taking this into account, almost all of the hearings were completed in or under the allotted time.
• In all cases, the questions asked by the members were relevant, and in some cases they helped to narrow the issue.
• When challenging situations arose, such as the presence of young children in the hearing room, they were well managed by the members.
• Where applicable, the members identified opportunities to narrow the issues with the consent of the parties: for example, on at least two occasions in a hearing, the member asked counsel for the Minister for his position following the evidence provided. The member also checked with counsel for the Minister to see if he was satisfied with the evidence submitted with respect to one of the grounds of refusal, which he was, so that questions on this point were limited.
• In all cases, oral applications made by the parties were dealt with properly by the members: late applications for filing of documents, preliminary applications, party’s objection, and requests for the addition of a ground of refusal.

Strengths

• Members show a strong tendency to be proactive by stepping in appropriately to ensure that the questions asked by the parties, either the appellant's counsel and/or the Minister's counsel, are relevant and related to the determinative issues of the appeal.
• One of the results of this strength is that the length of the vast majority of hearings were within, or even under, the allotted time.

Areas for improvement

• Setting the agenda issues: In the majority of cases, members took the time to carefully explain to appellants, particularly unrepresented appellants, how the hearing would proceed, the issues of the case and the criteria and/or factors that the member must consider in making his or her decision. However, in about 20% of the cases analyzed, the criteria on which the member must base his or her decision were not explained to the appellant, particularly when the appellant based his or her appeal on humanitarian and compassionate considerations. In some cases, this may have been an oversight, since the same member listed the factors in one case but did not do so in another.

Recommendation

• Although in the majority of hearings the members set the issues on the agenda, it is recommended that members be reminded of this so that any issue in the case is clearly explained to the parties at the start of the hearing and to ensure that appellants are consistently informed. It could be appropriate to discuss best practices in this regard at a national professional development day.

2.4 ​Reasons state conclusions on all the determinative issues

Why assess these indicators

The Supreme Court of Canada set the requirement for justifiability, intelligibility and transparency of the decisions of an administrative tribunal. Through indicators #23 and #24, this study applies the Court’s requirement in the context of IRB decision-making.

What was measured

Considerations

Indicators #23 and #24 are considered universal—they apply to all hearings. In certain hearings, universal indicators could not be assessed, either because the appeal was withdrawn at the start of the hearing, or the appeal was allowed on consent.

What the numbers say

• All reasons cover the issues of the case at the hearing, even if they were not clearly stated at the outset. For example, where it was stated that the hearing would deal with humanitarian and compassionate considerations but the factors to be considered were not listed, those factors were covered in the reasons.

Strengths

• The issues in the case that are relevant to the appeal were dealt with in the reasons.

Areas for improvement

• None identified

Recommendation

• No recommendation

2.5 ​Decisions provide findings and analysis necessary to justify conclusions

Why assess these indicators

The Supreme Court of Canada set the requirement for justifiability, intelligibility and transparency of the decisions of an administrative tribunal. Through indicators #25 to #30, this study applies the Court’s requirement in the context of IRB decision-making.

What was measured

Considerations

Indicators #25 to #27 are considered universal—they apply to all hearings. Indicators #28 to #30 are assessed as applicable. In certain hearings, universal indicators could not be assessed, either because the appeal was withdrawn at the start of the hearing, or the appeal was allowed on consent.

What the numbers say

• In general, members carefully explain why they give more weight to a particular piece of evidence, although some reasons do not clearly set out the factors raised that go against the decision made by the member. The reasons were clear and supported by the evidence.
• In some cases where counsel for the Minister consented to the appeal, there were no reasons, only the decision to allow the appeal with the Minister’s consent.

Strengths

• All of the reasons present clear, unambiguous findings of fact based on the documents admitted into evidence and the testimony heard.

Areas for improvement

• None identified

Recommendation

• No recommendation

2.6 ​Reasons are transparent and intelligible

Why assess these indicators

The Supreme Court of Canada set the requirement for justifiability, intelligibility and transparency of the decisions of an administrative tribunal. Through indicators #31 to #33, this study applies the Court’s requirement in the context of IRB decision-making.

What was measured

Considerations

Indicators #31 to #33 are considered universal—they apply to all hearings. In certain hearings, universal indicators could not be assessed, either because the appeal was withdrawn at the start of the hearing, or the appeal was allowed on consent.

What the numbers say

• The vast majority of the decisions and reasons in the cases reviewed were delivered orally, which shows that members were very well prepared for the hearings.
• The reasons were generally written in plain language.
• There was very little full-text citation, which helps to make the reasons more accessible and easy to read.

Strengths

• The reasons delivered orally were clear and coherent.
• The members made little use of full-text citation.

Areas for improvement

• Use of shorter subheadings and paragraphs: Some reasons would be clearer if subheadings were added in addition to headings dividing sections, such as when dealing with the factors analyzed for assessing humanitarian and compassionate considerations. Certain paragraphs could be shorter if they contained only one idea per paragraph.
• Sometimes information is repeated, especially in the “background” and “analysis” sections. Members should try to avoid repeating information in those sections as much as possible.

Recommendation

• Remind members, perhaps in a workshop held on a national professional development day:
• to write short paragraphs containing one idea each
• to use subheadings.

Annex A-​Checklist

Annexe B- SOGIE quality review checklist:

Performance indicator and rating guide