Paul Philip

Chief Executive Officer

The Solicitors Regulation Authority

By email: [email protected]

11 June 2020

Dear Mr Philip

The SRA's new Solicitors Qualifying Examination and its final application to the Legal
Services Board

We write prior to the Solicitors Regulation Authority's ("SRA") final application to the Legal
Services Board ("LSB") for the approval of the Solicitors Qualifying Examination ("SQE"), which
we currently understand is scheduled for late July 2020.

The Junior Lawyers Division (“JLD”) supports the decision of the SRA to overhaul the current
system of qualification and to introduce a standard centralised examination. The JLD has been
supportive of the ‘centralised assessment’ aspect of the SQE since it was first announced.
However we are keen to ensure that its broader implementation is fit for purpose, maintains high
standards and does not create an arbitrary barrier for entry to the profession.

We have continually raised concerns about the proposed mechanism for the SQE and the impact
it will have on legal education, social mobility, equality and diversity, and standards for entry to
the profession.

Overhauling the system for qualification should be a once in a generation event and, as such, it
is vital that the final form of the SQE is one that the legal profession can be proud to stand behind.
While we recognise the inevitable need for tweaks to be made once the new system is in place,
it is vital that the form of qualification agreed upon is as close to perfect as can be upon
implementation. It is not sufficient to say that what is proposed will be better than the current
process for qualification and accept that as a suitable standard.

We are concerned that the proposed SQE, as a process for qualification, is not yet ready for final
approval and that more time is needed to ensure it is fit for purpose. We believe that a further
delay is needed and that the SQE should not be brought in, in its current form, in 2021.
You will note that much of what we have set out below, we have said before1. However, we believe
it is vital that we lay out our concerns again, before such an important decision is made, to ensure
that the voices of our members are heard.

Executive summary of concerns

We note that the SRA's most recent announcement on the SQE, on 5 June 2020, confirms that
the two objectives the SRA set out for the SQE were:

1. delivering greater assurance of consistent, high standards at the point of admission.

This will protect consumers of legal services from all communities [our emphasis added];
and
2. encouraging the development of new and diverse pathways to qualification, which are
responsive to the changing legal services market and promote a diverse profession by
removing artificial and unjustifiable barriers.

The SRA also confirmed that the criteria for the final assessment model would be:

 Reliable
 Valid
 Manageable
 Cost effective

We are concerned that:

1. there has only been one Equality, Diversity and Inclusion (“EDI”) risk assessment ("Risk
Assessment") exploring the potential risks and benefits of the SQE to EDI in the profession.
The Risk Assessment outlined specific problems with the introduction of the SQE which
remain largely unanswered. While we understand this Risk Assessment is currently being
updated, we still have concerns about the scope of that assessment and these are set out
further below;

2. the removal of the requirement to complete a qualifying law degree ("QLD") or a graduate
diploma in law ("GLD") coupled with the SQE 1's reliance on multiple choice questions
("MCQs") to assess a candidate's functioning legal knowledge ("FLK"), dilutes the
internationally recognised and respected standards for qualifying as a solicitor in England
and Wales and gives rise to serious doubts that MCQs alone can authorise a candidate's
ability in all areas of law that the SRA grants a solicitor a licence to practice within;

3. by not mandating a set order in which the SQE stages are to be completed (i.e. with SQE 2
as a final assessment of a candidate's ability to qualify as a solicitor, at the point of
admission), the status-quo will remain, and the SQE will not be a bettering of the inequality

1 We include with this letter our joint letter with The Law Society dated 05 September 2019.
 within the profession, particularly in respect of diversity and social mobility. We believe that
it is vital that the SRA mandate a specific order of Qualifying Work Experience ("QWE")
before SQE 2 in order to ensure that becoming a solicitor does not remain a pay-to-access
profession, but is instead about attracting the best and the brightest talent, regardless of
their socio-economic background;

4. the pilot of SQE 2 has only recently been completed and its results only just made available
to the profession at large, with the assessment specification to be consulted on in late June
and July 2020. We do not believe that there is enough time for the profession to review and
comment on this fully before the LSB has to consult on the SRA's application (August
onwards); and

5. encompassing all of the above, we feel there is a general lack of transparency surrounding
the SQE. The failure to publish the raw data, on which both pilot reports and
recommendations are based, means the development process cannot be transparent.

The SQE is intended to be a new and cheaper route to qualification that will ensure consistent,
high standards for qualifying solicitors, alongside providing further flexibility in training and
increased access to the profession. For the SQE to meet these aims, it is essential that the
problems above, and as further detailed below, are given serious consideration and resolved prior
to the introduction of the SQE.

In its current form, the JLD does not believe that the SQE meets the SRA's intended aims and
objectives and the JLD cannot endorse the SQE as the new route to qualification as a solicitor in
England and Wales.

1. EDI

Concerns

1.1. The Risk Assessment, published in April 2017, brought to light a range of potential EDI
impacts that the introduction of the SQE could have. It is essential that the potential
difficulties outlined are recognised, with measured action taken to mitigate the EDI risks
posed by the SQE.

1.2. We have been left unconvinced that the necessary actions have been taken to ensure
that the SQE does not negatively impact EDI within the profession. We understand that
The Bridge Group have been asked to update the 2017 Risk Assessment and the JLD
have contributed to their research. We await the publication of that report but set out our
own concerns in the meantime.

1.3. Concerns have been raised about the accessibility of the type of testing that is
implemented. The increase in MCQs, rather than written exam style answers, poses
problems and adds to the burden for students with various disabilities. MCQs are often
 speed tests and if the student has visual impairments and must work through a scribe or
speech software, this increases the challenge. Extra time is arbitrary if the design of the
testing is wrong for the end user.

1.4. It is important that the special arrangements in place have independent oversight to
understand and ensure that the process provides fair and equal access to the
assessments. Often those in need of alternative arrangements do not speak up for
themselves; it is therefore imperative that the process for ensuring fair access is inclusive
and clear to candidates for the SQE.

1.5. We note the SRA cite that Kaplan are very well experienced in dealing with reasonable
adjustments for candidates in relation to the Qualified Lawyers Transfer Scheme.
However, the numbers of candidates coming through the SQE will be significant, in
comparison to those applying for qualification from another jurisdiction. This will no doubt
increase the range of adjustments required and the frequency of such requests for
adjustments. Accordingly, we cannot accept the reliance on Kaplan's previous
experience as an adequate answer.

1.6. Further, there is a significant concern surrounding accessibility to take both SQE 1 and
SQE 2, more so with SQE 2. This is due to the number and location of the tests centres;
for instance, we understand that SQE 2 will only be available in three locations to begin
with, increasing to five in total. This causes issues, particularly for disabled candidates
and some ethnic minority candidates, in attending the test centres which are not in a
candidate's local vicinity and the possible need for an overnight stay. This would also
impact all candidates in terms of the costs of overnight stay and / or travel, which again
does not improve access to the profession for those from lower income backgrounds.

1.7. Both pilot results have raised serious questions over the performance and pass rates of
different groups of candidates. As highlighted by the Kaplan report for SQE 1, ethnicity
and disability were both factors affecting performance. The SQE 1 pilot was reported to
have 18 candidates who were provided with alternative arrangements for the
assessment. However, it is unclear what measures were taken to provide alternative
arrangements. The report on the pilot by Kaplan also highlighted that there was a
difference in performance by disability in the pilot FLK test but the numbers who declared
themselves as disabled under the Equality Act 2010 were too few for conclusions to be
drawn.

1.8. All three SQE 1 pilot reports (from Kaplan, the SRA and the Independent Reviewer),
noted that Black, Asian, Minority and Ethnic (“BAME”) candidates did not perform as well
as other candidates, particularly in the skills part of SQE 1. It has therefore been
suggested that having skills tests as part of SQE 1 may set an unnecessary barrier to
qualification which disadvantages BAME candidates.

1.9. The SRA have now concluded that this aspect of SQE 1 simply be removed to tackle the
 issue. It is not clear whether the barrier will now simply reappear at SQE 2, and we note
particularly that no conclusion on this can be reached from the SQE 2 pilot either. We
have addressed this further below in relation to SQE 2.

1.10. We are concerned that not enough is being done before SQE implementation to assess
why BAME candidates did not perform as well as other candidates and that removing
skills testing from SQE 1 is a rushed solution. The testing methods must strike a balance
of being both rigorous and reliable, whilst remaining fair to all candidates.

1.11. According to the report, due to the sample size of candidates being too few, conclusions
on disability and ethnicity could not be properly drawn. It is essential that further analysis,
and if necessary further piloting, is undertaken in order to ensure that these findings do
not represent a wider problem within the SQE assessment methods.

1.12. We do not agree with the element of skills testing being removed from SQE 1.

Action

1.13. The SRA has previously stated that it will conduct another risk assessment once the SQE
has gone live. Further, in the SQE 1 pilot report it is stated that “we [the SRA] will be
reporting on performance by protected characteristic when the SQE assessments go
live”. It is wholly unacceptable that in the first few years of SQE being implemented,
candidates will be expected to pay for the SQE exams in full, despite these being the
"testing and learning" phase for the SRA.

1.14. Has the SRA considered the remedies that they will make available to candidates if
issues are found with the examinations which render them unreliable or inadequate?

1.15. Issues relating to the identified BAME gap need to be addressed and dealt with
beforehand, and not reported on afterwards. Far more work needs to be done on
understanding why there is a disparity and addressing ways to deal with this, without
jeopardising the standard expected of a solicitor in England and Wales.

1.16. It was anticipated that there would not be the same problems with the skills tasks in SQE
2 as was encountered in SQE 1. Yet it is also acknowledged that it is not known why
BAME candidates performed worse. If this is the case, it cannot be said that BAME
candidates will necessarily perform on par with other candidates in the skills
assessments in SQE 2. We find this assumption to be incredibly concerning.

1.17. The report of the second pilot does not address this issue in any detail and so it is not
clear the extent to which the SRA prioritised the consideration of whether BAME
candidates, or disabled candidates, would performed on par with non-BAME and non-
disabled candidates. We request that the SRA release the raw data from the SQE 2 pilot
and specifically address the question as to performance of disabled and BAME
 candidates.

1.18. We believe it is in the profession’s interest for both a further set of pilots (including enough
candidates with disabilities and from minority groups to draw statistically meaningful
conclusions) and the updated EDI risk assessment to be given priority and conducted
(allowing time for the report to be considered) before the SRA makes its final application
to the LSB. This will help to ensure that the SQE meets its aim of providing further access
to the profession and adequately safeguards EDI within the profession.

2. QUALITY

Concerns

2.1. The SQE 1 pilot emphasised the need to consider whether the SQE assessments will
adequately prepare candidates for working life as solicitor. It is a major concern that,
without a requirement for a QLD/GDL and a lack of testing at SQE 1, outside MCQs,
newly qualified solicitors (“NQs”) arriving on ‘day one’ are unlikely to have developed
specialist knowledge in different areas of practice in the way that they currently do
(without taking a traditional QLD or additional top-up courses, which will add to the cost
of qualification under the SQE route).

2.2. Legal research, analysis and writing skills in particular are essential to the work of a
solicitor. We agree that it is the role of organisations to take an active role in the training
of the solicitors of tomorrow, but legal organisations also expect their future solicitors to
arrive with a certain level of skill and knowledge, which goes beyond having passed a
MCQ exam. The removal of the requirement for a student to obtain a QLD/GDL, could
lead to a dilution of standards of legal knowledge and a substantial gap in learning and
skills that will need to be bridged by organisations or individuals themselves.

2.3. By way of example highlighting concerns, FLK in the SQE will be assessed over a total
of 11 hours, compared to circa 22 hours for a GDL (plus the further LPC assessments for
electives). Also, looking at the exam structure, SQE1 is two exams of 180 questions.
Exam one covers seven topics and exam 2 covers 6 topics. Presuming an equal share of
questions per topic and ignoring the fact that there are supposed to be ethics questions
pervading throughout, that means entire areas of law are going to have around 25-30
MCQs each. Examples such as this no doubt cause some degree of concern from the
profession about the SQE and its lack of requirement for a law degree or GDL.

2.4. To fill the gap which is perceived to be created by the lack of rigour involved in the SQE,
and with no requirement for a QLD/GLD, many City firms are already considering the
solution of providing bespoke training for their new recruits. We consider that this
approach from such firms, many of which have an international presence, sends a
negative message about the reliability and suitability of SQE to the wider world, risking
damaging the reputation of a solicitor of England and Wales internationally and harming
 England and Wales being a jurisdiction of choice.

2.5. This approach also creates a significant risk of a two-tier profession and does nothing to
improve accessibility and social mobility within the profession. Nor does it meet the SRA's
aim that organisations will stop needing to look behind the title of solicitor, considering
what University a candidate attended and the type of organisation in which they trained.
The SQE qualification will not be considered a universal marker of a solicitor's ability.

2.6. We are concerned that top-up courses and further learning will only be available to the
firms, or candidates, that can afford it. Under the current system, many firms cannot
afford to pay for a candidate's LPC costs, and so candidates must take the financial risk
upon themselves. If firms across the country require a higher standard of training than is
necessary under the SQE, because of their lack of faith in the quality of the SQE, we are
concerned that candidates will have to take the burden of this themselves. This means
that our profession will remain a "pay to access" profession for those that don't head to
the larger firms and/or create a two-tier standard. This does not meet the SRA's aim, for
the SQE, of creating a more accessible, diverse profession with consistent standards.

2.7. Those firms that cannot afford to pay for additional training are likely to bear the brunt of
criticism for not advancing social mobility if they choose to recruit those who have
pursued further education, at their own cost, instead of those from a larger, more diverse
pool of candidates, but which includes many that can't afford to take on additional studies.

2.8. In attempting to increase diversity, we fear that the approach the SRA is currently taking
will only increase elitism and make the importance of a candidate’s degree, University
attended, and any extra-curricular or additional education, of even more importance to
law firms. This was not the intention or aim of creating the SQE and in our view results
in the opposite outcome.

International element

2.9. Moreover, we have spoken with the Law Society's International Division and understand
that the world does not currently see the SQE as an improvement. It appears that the
wider world see the SQE as a dumbing down of the qualification of a solicitor and this
will mean that, for those solicitors wishing to practice in a foreign jurisdiction, the
education they have received and their route to qualification (University attended, degree
subject and qualification, type of legal organisation and work they have experienced) will
be scrutinised even more by foreign regulators.

2.10. This view internationally damages England and Wales being a jurisdiction of choice for
consumers of legal services, it damages the title of solicitor internationally for those
wishing to practice in a foreign jurisdiction and it damages the desirability of foreign
lawyers wishing to qualify as a solicitor of England and Wales. It could also make it harder
for English and Welsh qualified solicitors to move and practice abroad.
2.11. We note that in justifying the approach to MCQs only as a test of FLK, the SRA have
cited the fact that many other jurisdictions use MCQs and test candidates in a similar
way through foreign bar exams. However, these jurisdictions largely require that a
candidate also has a recognised law degree which the SRA is no longer requiring. Again,
this is perceived as a dumbing down and reduction in specialist legal knowledge for
solicitors of England and Wales.

MCQs

2.12. Furthermore, our concerns remain about the standard of the MCQs proposed. In
meetings with representatives of the JLD and The Law Society, the SRA has explained
the work that has gone and will go into the development of the MCQs to be used in SQE.

2.13. However, the sample questions the SRA released in December 2019, do not provide
reassurance. The JLD were contacted by a small number of individuals who had tested
some of these sample questions with worrying results:

2.13.1. Candidate 1, age 26, just started working as a paralegal, no formal legal training,
took 30 minutes to do the first 30 questions – result obtained 43%;

2.13.2. Candidate 2, age 24, working as a paralegal, has a law degree, not done the
LPC, took 30 minutes to do the first 30 questions – result obtained 60%;

2.13.3. Candidate 3, age 26, no degree whatsoever or legal training, resides in a foreign
jurisdiction, took 25 minutes to do the first 30 questions – result obtained 30%;

2.13.4. Candidate 4, age 30, has a non-legal PhD but no legal training, took 8 minutes
to do the first 30 questions – results obtained 73%; and

2.13.5. Candidate 5, age 23, working as a paralegal, has a law degree and is currently
studying the LPC, did questions 1-60 throughout the course of one day – results
obtained 60%.

2.14. This small number of examples produce troubling results and would suggest that the
MCQs are not overly difficult, confirming the current concerns of the profession, and that
the SRA is potentially setting the level of legal knowledge and/or question standard too
low. Undoubtedly in order to set a higher standard it would be necessary for individuals
to have the requisite legal knowledge to complete the assessment and we refer back to
our concerns about the lack of a requirement to complete a QLD/GDL. We consider that
further work needs to be undertaken in relation to MCQs and the level of legal knowledge
candidates are expected to have, and that the data from the SQE 1 pilot should be
released in a similar manner to our examples above.
2.15. Further, feedback we received from a candidate who actually participated in the first SQE
pilot was that the MCQs did not demonstrate a knowledge of the law, but more a
demonstration of common sense. It was noted that a law degree teaches students not
only the law, but also a way of thinking within the legal framework, to apply legal
knowledge and formulate arguments. This skill is simply not assessed at SQE 1.

2.16. We understand that the sample questions, published in December 2019, were updated
in April 2020 due to a number of errors in the first batch of samples (such as an incorrect
answer being stated to be the correct answer), however we do not believe the standard
of the actual questions being tested to have changed.

2.17. MCQs can demonstrate some retention of information but give no insight into how a
candidate analyses that information, applies it to a problem and communicates their
answer. Our concerns also remain about a candidate potentially failing questions on one
area of law, but scoring enough marks in other questions to ensure an overall pass. This
is a clear risk to consumers in that a solicitor in the future may be granted a licence to
practice in all areas of law, and may in fact practice in the area of law in which they failed
to demonstrate the required level of functioning legal knowledge.

2.18. To reiterate the points we have made on quality, we again remind the SRA of the
criticisms previously provided in response to the 2016 SQE consultation from other
stakeholders, as follows:

 The Association of Law Teachers wrote that they had ‘grave reservations… [MCQs do]
not replicate practice in any meaningful sense…we appreciate that computer-based
assessment is used in other jurisdictions, but we are unaware of one where it is the sole
means of assessment.’
 Clyde and Co wrote that ‘we have strong concerns on whether [legal knowledge] can be
adequately tested and assessed by using multiple choice assessments… we also have
strong concerns that individuals entering the profession under the new proposals may do
so knowing substantially less law [as] the proposed SQE1 assessments do not cover the
current breadth of the GDL, LPC and PSC combined.’
 Linklaters wrote that ’we fundamentally disagree that the proposed SQE is a robust and
effective measure of competence…other international jurisdictions generally include 50%
essay based questions and not merely MCQs…essay-based questions examine (1)
students’ critical reasoning skills, (2) their ability to analyse and assimilate information, (3)
their ability to recall information unprompted and (4) their ability to construct an argument
and present the issues and solutions they have identified in a coherent, convincing and
consumer-friendly form. Put simply, a multiple choice assessment alone cannot assess a
candidate’s suitability to practice as a solicitor.’
 Liverpool John Moores University wrote that ‘the literature relied on to support the
contention that [MCQs are] an effective method of assessment does not in reality do this…
[one article cited by the SRA in support] acknowledges that…[USA] law schools view
MCQs as less intellectually rigorous than essay questions and less realistic in their
 relationship to the actual practice of law…we share the concern of many legal educators
that the depth and breadth of legal knowledge, the intellectual skills, the value of qualifying
legal work experience and the level of professional practice skills required to pass the SQE
will be significantly less than is required at present.’
 The University of Oxford wrote that ‘[MCQ] testing is of no value in determining whether
an individual would be able to give competent advice in situations in which the law is
unclear…[this] can only be judged through more more sophisticated forms of testing in
which the candidate is permitted to explore the problem at length in writing.’

Action

2.19. The SRA has now concluded that SQE 1 will be MCQ questions only to assess a
candidates functioning legal knowledge. It is therefore necessary for the SRA to show
conclusively that MCQs, without the requirement for a QLD/GDL, can sufficiently prepare
all candidates to have the breadth of legal knowledge and the level required of a NQ
solicitor. We do not believe the SRA has provided such evidence.

2.20. In order to achieve this, we believe further pilots and analysis needs to be conducted and
that there needs to be more engagement with law firms, of all sizes, to ensure that they
are comfortable that candidates are being tested at the right level and have the right
amount of legal knowledge and skills, that currently comes from a QLD/GDL or
equivalent.

2.21. Alternatively, and our preference, would be that the requirement for a QLD/GDL (or
equivalent as in the case currently), to be held by all candidates, be implemented.

3. ORDER OF THE ELEMENTS OF SQE

Concerns

3.1. We are concerned that there is no current requirement for candidates to have completed
their QWE prior to taking SQE 2. We recognise the SRA's desire to create a system that
allows for greater flexibility in pursuing a route into the profession, however we believe
that this approach will have a detrimental impact to access to the profession.

3.2. The purpose of solicitors having undertaken two years’ training is to ready them for the
daily experiences faced by solicitors. This means that QWE must be meaningful,
providing candidates with the necessary skills to become a solicitor. The purpose of SQE
2 should be to ensure that candidates have received meaningful training and that they
have the skills to enter the profession – surely this is why the threshold to pass these
skills exams is set at the level of a competent newly qualified solicitor. However, without
a restriction in place as to when SQE 2 is taken, this will not necessarily be reflected in
practice. The SRA has provided no explanation of how it considers that a candidate will
 have achieved the competence of a newly qualified solicitor before they have undertaken
any QWE. We strongly suspect classroom based courses will begin to appear in the
market.

3.3. Further we note that the regulations do not appear to require that individuals demonstrate
competence or that QWE provides any obligation on firms to train candidates; candidates
only need to have had the ‘opportunity’ to develop some of (not all) the competencies
during work experience. This opens the door to a lowering of standards of the profession.
However the SRA states that it is the SRA's role to assess the competence, and
presumably this is assessed in SQE 2. So why is it that these skills could be assessed
before a candidate is ready to be certified as a qualified solicitor (i.e. because they still
need to then undertake QWE)? This is not ensuring standards at the point of admission
in line with the SRA's objectives.

3.4. A large number of City firms have already made it clear that they will take the approach
of requiring completion of SQE 1 and SQE 2 before taking on candidates for QWE. When
other firms follow suit, candidates will inevitably be forced to take further preparatory
courses (similar to the LPC) in order to pass SQE 2 so that they can secure QWE. SQE
2 will then become a test of the quality of these courses and not a test of the quality of
training from firms and the competence of a candidate at the point of admission. .

3.5. As such, we are concerned that the overall cost of the SQE will not be cheaper than the
status quo. We have been told that the SQE is to range from £3,000 to £4,500; but this
figure does not include the costs of training courses likely to be taken in preparation for
the SQE – as outlined above. This will compound the issues set out at 2.4 to 2.8 and
mean further costs and further financial discrimination within the profession.

Funding

3.6. The Risk Assessment outlined several risks associated with funding and the SQE,
including that the cost of preparatory training, combined with the cost of the SQE
assessment, could be the same as, or more than, the cost of the LPC. It also echoed our
fear that, because the preparatory training for the SQE would not be included within an
undergraduate or post-graduate degree, it would not be eligible for government- backed
student loans.

3.7. Without access to graduate student loans, the SQE's costs may prove to be too costly
for many without access to capital or a training contract at a City firm, to enter the
profession. This is particularly relevant at a time when the government has discontinued
the use of Professional Career and Development Loans which, in light of the current
financial climate, we believe is unlikely to be reversed.

3.8. This could encourage the uptake of high street bank loans, leaving applicants in high
interest debt on top of their undergraduate student loans. Further, this may force aspiring
 solicitors to save up the necessary funding, thus delaying their assessments. This would
leave many applicants in a disadvantaged position compared to those able to financially
fund the assessments immediately and this flies in the face of providing greater access
to the profession.

Action

3.9. We believe that the SRA should prescribe that SQE 2 can only be taken after a significant
period of the two years' QWE has been completed.

3.10. By prescribing an order for QWE and SQE 2, we believe this will help to ensure that SQE
2 acts as a test of not only a candidate's quality but also a test of a firm's quality, in
training solicitors. In doing so it will provide a greater level of consumer protection than
is present under the current system. Without this, the SRA simply continues the issue it
has with the current system; that there is no test of a candidate's ability and standard at
the point of entry to the profession.

4. SQE 2 PILOT

Concerns

4.1. The SQE 2 pilot took place in December 2020. Yet, the SRA released its report on this
pilot only a week ago, 5 June 2020, despite the fact it is now making final decisions on
elements that the pilot was to test, in order that an application for the final design of SQE
can be made to the LSB. In doing so, the SRA has provided almost no time at all for the
profession at large to respond to the outcomes of the pilot.

4.2. Prior to release of this report, the SRA discussed emerging findings' from the pilot with
key stakeholders. Those emerging findings, by the SRA's own admission, are not
particularly helpful or reliable due to a data set that was too small.

4.3. Our initial review of the formal pilot results indicates that there are several areas of
immediate concern.

4.3.1. Firstly, we note that in a similar fashion to SQE 1, it appears possible for a
candidate to 'perform poorly' (which we presume is a euphemism for ‘fail’) in one
SQE 2 area but for this to be compensated for in other areas, leading to an
overall pass. This would not be permitted under the current system, where a
failed paper would have to be re-sat and passed in order for the candidate to
continue. This cannot represent anything apart from a lowering of standards.
We accept that such circumstances may be rare, but in the context of thousands
of students sitting the examination, it should be impossible.
 4.3.2. Secondly, we note that the pilot data suggests there was no reliable link between
prior work experience and performance. Our understanding is that the SRA
intended that candidates would take SQE 2 towards the end of the two-year
period of QWE, in order to acquire the skills necessary to pass the assessment
(hopefully meaning candidates taking this approach would avoid the need of
costly preparation courses). The fact that prior work experience is no predictor
of performance can only suggest that the pilot questions were not adequately
calibrated to require experience. This does nothing for the profession's
confidence in the rigour of SQE and presumably explains why larger firms are
considering funding educational top-up courses for their candidates. It also adds
to the confusion for junior lawyers about how they might find the right kind of
work experience and/or training course to assist them with the necessary skills
and legal knowledge to be successful in SQE2.

4.3.3. Thirdly, we note that Kaplan themselves acknowledged that they have concerns
about the performance differential observed between BAME and white
candidates. Kaplan suggested some ways in which this differential might be
addressed, but in the absence of further pilot assessments these measures will
be untested before real candidates take the examination. This will clearly be
prejudicial to BAME candidates who are in the first cohorts to sit the
examination, and is unacceptable.

4.3.4. Finally, the low number of candidates with a disability who took part in the pilot
means that identifying, let alone addressing, problems for this group of
candidates will have to be done when the exam goes ‘live’ unless there are more
comprehensive pilots conducted first. As above, this will clearly be prejudicial to
people with a disability who are in the first cohorts to sit the examination, and is
unacceptable.

4.4. Since the SQE 2 pilot, the SRA has stated that the pilot was not actually to simulate a
live sitting of SQE 2 in practice at all, but simply a test to assist the SRA in deciding
whether candidates would be assessed in the SQE 2 skills in a common core context or
a specialist context. The actual examination, SQE 2 therefore, remains untested.

4.5. We fear that the first sittings of SQE 2 following implementation will in fact be the true
pilot of SQE 2 and this, not only given the fact that junior lawyers will be paying for the
privilege of sitting this exam at a cost of c. £2,000, is wholly unacceptable.

4.6. We are extremely concerned that in the Kaplan report, it is clear that "there is a cause
for concern" in the differential performance by binary ethnicity, particularly the differential
performance between BAME and white candidates. This appears to confirm our
concerns from the SQE 1 pilot, that by simply having all skills in SQE 2, the issue of
BAME candidates' performance in comparison to white candidates would not be rectified;
it is simply moved to the second, more expensive exam.
4.7. It is concerning that there is very little information considering this further from the SQE
2 pilot report and that the SRA is instead taking an approach of continuing to consider
this issue further, after finalising the design of the SQE. If the SRA subsequently conclude
that the differential performance is, in part, due to the design of the SQE, what then?
How will the SRA undo the work it is doing right now to have the design signed off? What
about those candidates who pay for the SQE and have a lower performance than may
have otherwise been the case had the design been different? It seems to us premature
to finalise the design of a national licensing examination, before understanding the
reason for the issues that have been highlighted in the pilots.

Action

4.8. The confused findings from the pilot can only raise concerns about the efficacy of the
SQE 2 assessment, should it be introduced in its current form. Furthermore, the lacuna
of understanding why BAME candidates performed poorly in relation to white candidates
(and the absence of analysable data in relation to candidates with disabilities) is a matter
of huge concern.

4.9. The SRA must pause its application to the LSB, rethink its approach to SQE 2 and ensure
that proper pilots have been run providing conclusive data to ensure the protection of
candidates and our profession. The broader legal sector must have an appropriate
opportunity to comment on such pilots before any application to the LSB is made.

5. TRANSPARENCY

Concerns

5.1. We call on the SRA to release the raw data from the SQE2 pilot. The published analysis
of the SQE1 pilot failed to deliver an acceptable standard of transparency due to the lack
of accompanying data with the report. The lack of data raises serious questions over the
reliability and validity of the claims and recommendations made by the report.

First pilot

5.2. Firstly, we were told that the candidates were ‘broadly representative’ of those that would
sit the live assessment. This vague description fails to clarify who sat the pilot, who
dropped out or failed to finish the pilot and what percentage of different groups were
involved. This is exemplified by the fact that it is declared that 18 candidates requested
special arrangements without any mention of what arrangements were made and how
such measures would be applied in the live assessments without increasing the
predictive costs of the SQE.
5.3. Secondly, without access to the raw data it is very difficult to gauge the fairness of the
assessment method. The changes to the FLK and removal of the legal skills element
have been based on data that is unavailable for the profession’s scrutiny. Without the
release of the transparent results and data it is impossible to make a reasonable
judgment on the fairness of the assessment.

5.4. Access to the data is all the more necessary because the report does not provide the
number of people failing to meet the pass mark, which again has been left unidentified,
and only mentioned to highlight a fail rate of 50 percent. It is worth mentioning that the
SQE would represent a very expensive resit.

5.5. The lack of data and transparency around the report also failed to adequately address
the potential discriminatory nature of the MCQ assessment. Although it is declared that
BAME candidates achieved lower results, it is stated that a bigger factor was between
those who came from a Russell Group University and those who did not. While this may
be true, it does nothing to explain why particular groups performed worse and whether
this is down to the particular sample of candidates in this pilot, or a more serious,
systemic problem with the SQE assessment itself. It is unacceptable to implement a live
assessment for thousands of candidates without answering this question definitively.

Second pilot

5.6. We have already addressed our concerns about the SQE 2 pilot above but believe that
the same raw data must be released to understand what work experience candidates
had and what result they obtained in the skills assessments. At present, it seems that the
second pilot exercise was not adequately controlled in order to provide reliable
conclusions. The raw data therefore must be available for proper consideration.

Action

5.7. Without the release of overall results, raw candidate feedback and detailed information
about the marking process, the reports can offer little reassurance that the SQE
assessments are ready for implementation. We urge the SRA to make the data from the
pilots publicly available for scrutiny and overall improvement of the live assessments.

5.8. For the pilots to be considered a transparent process, it is imperative that the analysis
outlined by the report is supported by the accompanying data. This will enable the
profession to evaluate the claims made by the pilot reports and build overall confidence
in the process of SQE policy development. This should also include further detailed
information on the pilots, including on the representativeness of the candidates and what
alternative arrangements were made for those who qualified under the Equality Act.
Conclusion

We believe that the decision to apply for LSB authorisation in summer 2020, with the first SQE 1
examinations occurring in 2021, should not proceed until the issues outlined have been
addressed. The SQE is of critical importance to the future of the profession and the SRA needs
to ensure that it is fit for purpose before it goes live.

In addition to the actions listed above requiring response, we are not clear exactly how the SRA
considers now that the proposed form of the SQE, which it intends to take to the LSB this month,
still adheres to the initial rationale for change. We are concerned that what we have on paper now
looks very different to what the SRA started with (although we acknowledge improvements have
been made since the very first proposal) and that it is no longer satisfying the original reasons for
change.

Again, referring back to the objectives:

 delivering greater assurance of consistent, high standards at the point of admission.

As stated, the profession do not have greater assurance in this new qualification producing
high standards and that seems to be largely due to MCQs being used to assess the breadth
of legal knowledge and that candidates no longer need to obtain a law degree or equivalent.

In addition, we do not accept that the SRA will be assessing a candidate's competence at the
point of admission if SQE2 is not the final stage of the SQE elements.

 encouraging the development of new and diverse pathways to qualification, which are
responsive to the changing legal services market and promote a diverse profession by
removing artificial and unjustifiable barriers.

For the reasons set out above, particularly for BAME and disabled candidates, we do not
currently believe the evidence shows that the SQE will promote a diverse profession or
remove barriers. We also believe there is a significant risk to the SQE creating an additional
barrier for those from lower socio-economic backgrounds. The reality is that the SQE is likely
to be the same price as the current system, with a lot of those costs being front loaded on
candidates, without access to suitable funding options.

Furthermore, particularly given the unreliability of the data from the SQE 2 pilot, we are not clear
how the SRA considers that its final assessment model can be reliable. SQE 2 has not been
tested as an assessment at all and so the first sitting of this, which candidates will pay for, will
effectively be its first pilot. The same can be said for SQE 1 which was piloted as three papers of
120 questions each (rather than the final design of two papers of 180 questions each) and
included skills (the final design no longer including skills).

We believe that centralised assessments of a candidate’s ability to be certified as a solicitor of

England and Wales is the right approach, and this would satisfy almost all of the cited benefits of
SQE (save for getting rid of the trainee bottleneck - although as we have previously pointed out,
we believe that this will shift to an NQ bottleneck).

We also believe that without sufficient progress on the matters above, the SQE will damage the
profession as a whole. Excellent education lies at the heart of the legal profession and its future
as a jurisdiction of choice around the world. Such an overhaul of the system therefore demands
a thorough, transparent and participative approach to its development.

Yours sincerely,

Charlotte Parkinson Chair, Junior Lawyers Division

of England and Wales

CC: Matthew Hill, CEO, Legal Services Board

Helen Phillips, Chair, Legal Services Board