HACT and the Joseph Rowntree Foundation (JRF) have launched a new report on the use of randomised control trials (RCTs) in driving evidence-based service design in the social housing sector.
The research, part-funded by JRF, draws on a research programme launched in 2018 which looked to enhance the evidence base about what works in the housing sector, particularly in relation to employment interventions.
The programme enabled housing providers to use RCTs to support the development of this sector-wide evidence base.
The report, entitled Changing research in social housing: the role of RCTs, outlines progress so far in the use of RCTs in the social housing sector – as well as providing guidance on when to use RCTs either as an alternative to, or in conjunction with, other research methodologies.
HACT has been working with JRF to support social housing organisation partners with RCT trials, as well as to encourage the wider use of RCTs.
Following this, barriers to RCT within organisations included:
- A lack of understanding about the RCT methodology, as well as a lack of confidence about the practicalities of designing and running an RCT
- Confusion about identifying a suitable intervention to test, as well as sensitivities about randomisation and withholding a potentially better service from the control group of service users
- Data management issues, including the introduction of new processes, reliability and quality control
- The costs of running trials, concerns about capacity issues and the challenge of convincing senior colleagues about the benefits of RCTs to the business
HACT has since worked with HACT has worked with social housing organisations to help them plan the resource and financial commitments required to run an RCT from design through to fieldwork and final analysis.
According to the report, typically, RCTs will be run over several months or, in some cases, years and there will need to be capacity to design, set up and run a trial, as well as resources for analysis, data management and possible intervention actions.
This might be provided internally, the report said, or it could be bought in from external consultants.
“To reduce initial costs, it is best to keep trials simple, testing services with small, non-controversial changes, and accept that existing measurement mechanisms might not be perfect”, it highlighted.
The report also recommends:
- Where you have the internal expertise, it is important to consider the capacity of staff and their ability to dedicate their time to the trial
- A plan for monitoring the data, a plan for collecting the data, as well as a process to react to adverse interim findings, and adequate resources to deliver all of these
- The development of a contingency plan, so you can react to any issues that might arise during a trial
The report also highlights that the traditional perception of RCTs is that they require “long testing periods” – but HACT have reinforced that they can also be used for shorter time periods.
“This can be useful for social housing organisations that are under pressure to deliver changes to services quickly”, the report said.
“If you have time, before you commit to your full RCT, consider using the methodology on a smaller scale over a shorter time period.
“This will help develop internal capability and provide you with proof as to whether the methodology meets your requirements.
“It will also build internal confidence about using the methodology and help to convince internal decision-makers about the use of RCTs to inform business planning”, the report added.
According to the report, RCT’s can tell you if something is working, but not necessarily why.
Therefore, it recommends combining RCT’s with qualitative research to help test assumptions – with insights used to refine interventions.
“Talking to staff and participants who are involved in the RCT can reveal why it didn’t work as expected, and help you think about how to adapt the service in the future”, the report said.
As highlighted, a key opportunity that has emerged is RCTs can be combined with complementary
qualitative research methods so it can be “better integrated” with existing research and performance analysis.
“While thinking in the sector is changing, there is still a preoccupation with operational performance data, which is unsurprising bearing in mind the pace and complexity of housing services.
“We hope this report helps social housing organisations to use RCTs as a complementary methodology to their existing research methods, so they can develop a more comprehensive profile and understanding of their services, their customers and their needs”, the report concludes.