Quality performance in the Refugee Appeal Division for FY 2022 to 2023

​​​​​​​​​​This report describes the results of the measurement of quality in decision-making in the Refugee Appeal Division (RAD).

On this page

• 1. Overview
• 2. Summary of results
• 2.1 Reasons are complete
• 2.2 Reasons are transparent and intelligible
• 2.3 Supplementary questions
• 3. Results for specific populations
• 3.1 Consideration for sexual orientation, gender identity and expression, and sex characteristics
• 4. Recommendations
• 5. Management Response and Action Plan
• Annex A – RAD performance indicators.
• Annex B – SOGIESC quality review checklist: Performance indicators and rating guide

1. Overview

The study reviewed 70 paper appeals out of the 1065 that were finalized between October 1 – December 31, 2022 (assessment period) before the Refugee Appeal Division (RAD). The appeals were randomly selected in proportion to region, language and outcome. Members with less than 6 months of experience at the start of the assessment period were excluded from the sample.

The following charts illustrate the sampling makeup:

Assessment methodology

This qualitative assessment was performed by an independent reviewer, Paul Daly,​ who is the Research Chair in Administrative Law and Governance at the University of Ottawa, a leading commentator on Canadian administrative law, and an expert in decision-making in the administrative state. The reviewer examined all necessary evidentiary and administrative materials on file and assessed these against qualitative indicators in a checklist developed by the Strategic Planning, Accountability and Reporting (SPAR) Directorate and approved by the Deputy Chairperson of the RAD (see Annex A). The checklist assesses twenty-two indicators across three categories:

1. Reasons are complete
2. Reasons are transparent and intelligible
3. Supplementary questions

Eighteen of the indicators were mandatory for assessment, and four were assessed only when applicable. Each performance indicator is assessed along a 1-3 rating scale and, for supplementary questions related to SOGIESC cases, a categorical yes-no scale was used.

The 1-3 rating scale is as follows:

1=Does not meet expectations: The quality requirement was not met. The evidence showed one or more key instances where the proceeding or reasons would have markedly benefited had this requirement been met. There may have been an effort to apply the requirement, but the level of achievement fell short of expectations.

2=Meets expectations: This is a level of acceptable achievement. On balance, the member satisfied this quality requirement though there is margin for minor improvement.

3=Exceeds expectations: This is a level of consistent, above-average achievement. The evidence shows a grasp of the quality requirement and an understanding of its importance to a high-quality proceeding or decision, as the case may be.

Results are also expressed as a percentage of cases that meet expectations, by obtaining a score of 2.0 or higher.

In addition, as part of Gender Based Analysis Plus (GBA Plus), a second checklist was developed for cases related to Sexual Orientation, Gender Identity and Expression, and Sex Characteristics (SOGIESCSC). Within the sample, six cases related to Sexual Orientation, Gender Identity and Expression (SOGIESC) were further d through a particular SOGIESC lens. However, given the small sample size observations are not generalizable.

Several of the SOGIESC-flagged files did not involve consideration of SOGIESC issues but, rather, involved appeals that were resolved on procedural grounds that did not require consideration of SOGIESC issues. In future quality assessments, the assessor recommends that care be taken by the Audit and Evaluation Team to ensure that SOGIESC-flagged files are not procedural in nature.

To mitigate the inherent limitations of qualitative research, detailed performance indicators were provided to the reviewer to help focus the assessment. To ensure quality and consistency in the assessment, a reviewer was selected based on their in-depth knowledge of the IRPA, refugee and immigration matters, and administrative law. Moreover, a small sample size limits the inferences that may be made about the broader caseload. Where sample sizes are too small, observations or recommendations may still have been provided but these are not based on representative findings.

This report aims to provide a perspective to improve the Division's performance overall. T​he Audit and Evaluation Team provided the statistics found in the table accompanying each result section as well as the information in 1.0 “Context”. However, the findings in this report, including all strengths, areas for improvement and recommendations are solely those of the reviewer. Their observations do not lend themselves to firm conclusions on legal matters such as the correct application of the law, the weighing of the evidence, or the fairness of the proceedings from a natural justice perspective. Only a court reviewing the case can arrive at such conclusions.

2. Summary of results

The primary performance target for this assessment is for 80% of cases to meet quality standards. The quality standard is that the case achieves a score of 2.0 or higher in at least 80% of all standard indicators assessed within that case. This target was achieved with 87% (61 out of 70 cases), meeting the expectations. In other words, 87% of appeals reviewed either met or exceeded the quality standard.

Where an indicator had many cases that did not meet the target, it is addressed in the reviewer's observations following the table (“What we can improve”).

What we did well

The RAD performed well in the important task of producing reasons for decision that are justifiable, intelligible and transparent:

• Applying legal frameworks accurately (Indicators #1, #3, #4, #10)
• Ensuring clarity and completeness of factual analysis (Indicators #5, #6, #7, #9, #12)
• Evidencing commitment to best practices for contemporary administrative tribunal reasons, such as plain-language communication, issues-based analysis, point-first drafting and writing for the losing side (Indicators #13, #15, #17, #S1, #S2)
• Demonstrating cultural competence and sensitivity to intersectionality (Indicators #11, #S5)

What we can improve

The following areas were identified as areas that could be improved. The assessor made recommendations that could improve performance on these indicators:

• Providing succinct summaries of the basis for decision, in order to enhance the comprehensibility of members' reasons (Indicator #2).
• Encouraging members to conduct an independent assessment rather than an error-focused analysis, which will lead to shorter, more focused and more comprehensible decisions, thereby enhancing public confidence and achieving the RAD's mission of ensuring finality based on independent assessment of refugee claims (Indicators #8, #14, #16, #S3).
• Doing more to draft decisions with a trauma-informed practice lens: where there is evidence of trauma, especially medical evidence, members should deal with the evidence explicitly and explain briefly why, or why not, it made a difference to the member's analysis (Indicator #S4).

2.1 Reasons are complete

Why measure this

The Supreme Court of Canada has affirmed that justifiability, intelligibility, and transparency are important hallmarks of a reasonable administrative tribunal decision. Indicators 1 to 12 of this study apply these principles to the context of IRB decision-making.

Considerations

Indicator 11 is only assessed if applicable. Remaining indicators are considered in all paper appeals.

General Observations (and/or best practices)

Members in general performed excellently in terms of the completeness of their analysis.

Members meet best practices for comprehensive decision-writing in many respects: understanding and applying the test for the admission of new evidence and holding a hearing; identifying and confining their analysis to a determinative issue; making clear, unambiguous findings of fact; ensuring context-sensitive decision-making; and addressing all relevant issues.

Members generally ground their findings of fact in clear examples of probative evidence.

However, members struggled to succinctly summarize their decisions.

What we did well

Members engage in the best practice of 'writing for the losing side', that is, writing decisions that clearly explain to the unsuccessful party why their appeal failed, which enhances public confidence in members' decisions (Indicators #5, #9, #10 and #12).

Members write their decisions in a manner that is sensitive to social and cultural context, which makes decisions easier to accept for appellants (Indicator #11).

Members understand the test for the standard of review, the admission of new evidence and the requirement to hold a hearing and apply those tests with a high degree of accuracy, indicative of excellent training on the applicable legal frameworks (Indicators #1, #3 and #4).

Members are skilled in identifying determinative issues (i.e. those issues which conclusively resolve an appeal), which makes decisions shorter, clearer and easier to read (Indicator #6).

What we can improve

Members have been well-trained to make clear and unambiguous findings of fact (Indicator #7). However, although their performance remained excellent overall, members could do better at grounding their findings of fact in clear examples of probative evidence (Indicator #8). When members fell short in this respect, it was usually because they relied too heavily on findings made by the Refugee Protection Division: members should receive additional training on making independent assessments of appeals that are not unduly reliant on RPD findings.

Each decision should feature a one-paragraph summary of the decision, containing (1) an explanation of the circumstances of the appellant, (2) a description of the grounds of appeal and (3) a justification of the outcome. Most members succeeded in summarizing (1) and (2). However, members only summarized (1), (2) and (3) in roughly one-third of decisions. The value of such summaries is that they make decisions more comprehensible and also focus the mind of members on the key elements of the decision. Members should receive additional training on preparing one-paragraph summaries containing elements (1), (2) and (3).

2.2 Reasons are transparent and intelligible

Why measure this

The Supreme Court of Canada has affirmed that justifiability, intelligibility, and transparency are important hallmarks of a reasonable administrative tribunal decision. Indicators 13 to 17 of this study apply these principles to the context of IRB decision-making.

Considerations

All indicators are applied to all paper appeals.

General observations / and or best practices

Members use techniques of plain-language, point-first and issues-based writing. This is best practice in contemporary administrative tribunals.

In terms of plain language, members write decisions using short sentences, simple syntax and straightforward words, which enhances the comprehensibility of decisions and contributes to public confidence in decisions. In terms of point-first writing, each sentence and paragraph has a discrete point, which is then addressed clearly, enhancing the comprehensibility of decisions. In terms of issues-based writing, decisions are organized around key issues, making decisions easier to navigate and understand.

Members' commitment to best practice is evident. These best practices enhance the comprehensibility of decisions, which in turn enhances the legitimacy of decisions by making it easier for appellants to accept members' decisions.

However, some members spend too much time in their decisions summarizing findings made by the Refugee Protection Division and arguments about those findings, which is a departure from best practice.

What we did well

Members demonstrate sustained excellence in plain-language writing. Their decisions are clearly written, containing short sentences, simple syntax and straightforward words (Indicators #13 and #14).

Members also perform excellently in preparing reasons that are easily understandable and based on a logical chain of analysis (Indicator #15). This is facilitated by members' commitment to issues-based, point-first writing.

Members' success in using plain language and organizing decisions in issues-based, point-first form means that their reasons are generally likely to explain the decision to the subject of the appeal (Indicator #17).

What we can improve

Some members spend time summarizing the findings of the Refugee Protection Division (RPD) and the arguments made by an appellant.

Producing lengthy summaries means that reasons are less concise (Indicator #14) and that reasons are not as short and economical as they could be (Indicator #16). Producing lengthy summaries is not necessary to achieve best practice in contemporary administrative tribunal decision writing.

Instead of summarizing RPD findings and arguments, members should work RPD findings and arguments into their issues-based analysis.

For example, members should briefly summarize an RPD finding and the relevant argument about the finding before making their own independent assessment on the finding and argument. Members should receive training as necessary on integrating RPD findings and arguments into issues-based analysis.

2.3 Supplementary questions

Why measure this

The supplementary questions' section allows the RAD to incorporate new indicators to the annual RAD QMI review that may not fit into other sections, and contributes to a responsive, relevant, and robust assessment of quality.

Considerations

Indicator S4 is only considered if applicable. Remaining indicators are considered in all paper appeals.

General observations and/or best practices

These indicators are additional measures of quality in decision-making, which complement the best practices of plain-language, issues-based and point-first writing and seek to ensure that members are fulfilling the mission of the Refugee Appeal Division and maintaining or enhancing public confidence in their work.

In general, members perform excellently in producing high-quality decisions that are accessible to readers, in rendering decisions that determine definitively (where possible) the appellant's refugee status, in exhibiting cultural competence and in demonstrating sensitivity to the realities of trauma and intersectionality.

Taken together, a high level of success on these indicators contributes to the RAD's mission and to public confidence in the RAD.

What we did well

Members sometimes remit files to the Refugee Protection Division (RPD) where the RPD decision was defective. In these situations, members do an excellent job of providing appropriate direction to the RPD on how to remedy the defective decision (Indicator #S1). Members' best decisions, which are those that meet best practice standards in terms of plain-language, issues-based and point-first writing, also provide useful guidance for other Members and third parties who read decisions in order to better understand Canadian refugee law (Indicator #S1).

Members achieve standards of excellence in achieving finality, as they generally take a comprehensive view of the file appealed from the RPD and conduct an analysis that allows them to make a definitive determination of whether the appellant should be granted refugee status (Indicator #S2).

Members have evidently been well trained in cultural competence as they almost always avoid reliance on myths, stereotypes and assumptions (Indicator #S5).

What we can improve

Members generally conducted independent assessments of appellants' refugee status, which ensures finality and thereby contributes to the achievement of the RAD's mission (Indicator #S3). Sometimes, however, members focus more on the errors allegedly committed by the RPD than on whether the appellant should be given refugee status. This deviation occurs most often in decisions where members summarize RPD findings and the appellant's arguments. A focus on alleged RPD errors makes it more difficult to achieve finality and also deviates from best practices in relation to issues-based analysis. Members should be reminded of, and receive additional training on, the RAD's mission and the desirability of achieving finality in appeals.

Members for the most part were sensitive to issues of trauma and intersectionality in their decisions (Indicator #S4). Members were excellent at identifying files where intersectionality was relevant and explaining how they resolved the appeal by taking intersectionality into account: best practice here is explicit recognition of intersectionality and a brief explanation of how it factored into the member's decision. Members had relatively more difficulty with trauma. In files where there is evidence of trauma, especially medical evidence, members should deal with the evidence explicitly and explain briefly why, or why not, it made a difference to the member's analysis. Members' best practice in respect of intersectionality should be extended to trauma.

3. Results for specific populations

3.1 Consideration for sexual orientation, gender identity and expression, and sex characteristics

Why measure this

For the purposes of Gender Based Analysis Plus (GBA Plus), which is a priority for the IRB as well as within the Government of Canada as a whole, a second checklist was developed for cases involving sexual orientation, gender identity and expression, and sex characteristics (SOGIESC). The checklist assesses the application of Guideline 9: Proceedings Before the IRB Involving Sexual Orientation, Gender Identity and Expression, and Sex Characteristics.

What was measured

Six SOGIESC cases were identified within the sample, however, not all indicators were applicable to every case. N/A ratings were not included in final calculations for '# of cases assessed' or in '% of cases compliant.' Due to the small number of SOGIESCSC cases and indicators assessed, findings cannot be generalized to other SOGIESCSC cases and are included as observations only.

Target: 100% of cases scored as compliant

General observations and or best practices

There were few SOGIESC files in the sample. Accordingly, caution must be exercised in drawing general lessons from this review. With that caveat in mind, the treatment of SOGIESC issues was generally excellent.

What we did well

Members generally performed excellently with SOGIESC issues. Their cultural competence and sensitivity to trauma and intersectionality clearly informed their treatment of SOGIESC files, which contributes to public confidence in the work of the Refugee Appeal Division.

What we can improve

Although the sample size is small and thus significant caution is required in drawing general conclusions, members seemed to demonstrate less sensitivity to psychological issues in SOGIESC files, which is consistent with my remarks under the “what we can improve” heading in section 2.3 above.

4. Recommendations

The overall performance of the RAD is excellent. With a view to maintaining overall excellence and further improving performance in certain areas, I make the following recommendations:

1. Recommendation: Members should continue to receive training on plain-language, issues-based and point-first writing.

   This training could consist of workshops led by RAD team leaders or outside experts. RAD team leaders should consider creating a database of decisions that represent best practice in plain-language, issues-based and point-first writing across different thematic areas (e.g., credibility, internal flight alternative, state protection) to which members could regularly refer.

   Rationale: These are best practices in contemporary administrative tribunal decision-writing. Ongoing training is required to ensure that members continue to meet the high standards of excellence they are achieving in decision writing.

2. Recommendation: Members and team leaders should receive additional information on the importance of issues-based analysis to achieving the RAD's mission and ensuring public confidence in the RAD.

   Additional training is not required but the importance of issues-based analysis should be emphasized in regional and national meetings. Team leaders and members should be given information about the negative effects of deviation from best practice and reminded of the benefits of issues-based analysis.

   Rationale: Members deviate from best practice and risk undermining the RAD's mission and public confidence when they structure their decisions around the findings of the Refugee Protection Division (RPD) and the alleged errors in the RPD's decision. Decisions that give lengthy summaries of RPD findings and alleged errors do not explain to the reader why the RAD came to a particular conclusion; they often fail to ground findings of fact in clear examples of probative evidence; and they inhibit an independent assessment of the appellant's claim to refugee status, thereby compromising finality. Issues-based decision writing is more difficult than error-focused decision writing but any efficiency gains from error-focused decision writing are heavily outweighed by the negative effects on the RAD's mission and public confidence.

3. Recommendation: Members should receive additional training on how to summarize decisions.

   Additional training could consist of workshops led by RAD team leaders or outside experts in which members are asked to read hypothetical draft decisions and write summaries which can then be discussed with colleagues, team leaders and/or outside experts. RAD team leaders should also consider putting in place voluntary reasons review protocols specifically to ensure that in the opening paragraphs of decisions members identify and explain the outcome of the appeal.

   Rationale: Members fell short in summarizing their decisions. Shortcomings in respect of summarizing decisions can be easily identified in members' drafts and quickly corrected. There are significant potential gains available in terms of justifiability, transparency and intelligibility at relatively low cost.

5. Management Response and Action Plan

Annex A – RAD performance indicators

Annex B – SOGIESC quality review checklist: Performance indicators and rating guide