European network for Health Technology Assessment Joint Action (EUnetHTA JA): a process evaluation performed by questionnaires and documentary analysis.

BACKGROUND

The European network for Health Technology Assessment Joint Action (EUnetHTA JA) project's overarching objective was to 'establish an effective and sustainable HTA [Health technology assessment] collaboration in Europe that brings added value at the regional, national and European level'. Specific objectives were to develop a strategy and business model for sustainable European collaboration on HTA, develop HTA tools and methods and promote good practice in HTA methods and processes. We describe activities performed on behalf of the National Institute for Health Research HTA programme; evaluating the project processes and developing a data set for a registry of planned clinical studies of relevance to public funders.

METHODS

Annual self-completion online questionnaires were sent to project participants and external stakeholders to identify their views about the project processes. Documentary review was undertaken at the project end on the final technical reports from the work packages to examine whether or not their deliverables had been achieved. The project's impact was assessed by whether or not the deliverables were produced, the objectives met and additional 'added value' generated. The project's effectiveness was evaluated by its processes, communication, administration, workings of individual work packages and involvement of external stakeholders. A two-stage Delphi exercise was undertaken to identify the data elements that should be included in a registry of planned clinical studies of relevance to public funders. The data set was validated by an efficacy testing exercise.

RESULTS AND DISCUSSION

High response rates were achieved for the questionnaires sent to project participants and this was attributed to the evidence-based strategy implemented. Response rates to questionnaires sent to external stakeholders were disappointingly lower. Most of the high-level objectives were achieved, although applying the developed tools in practice will be implemented in the European network for Health Technology Assessment Joint Action 2 (EUnetHTA JA2). Most work packages produced their planned deliverables. Networking emerged as one of the main benefits of the project and face-to-face meetings were important. However, the overarching objective did not appear to have been met because there will be a follow-up EUnetHTA JA2 project (reliant on project funding) before the establishment of any permanent network. Twelve organisations from three continents participated in the Delphi exercise to develop the data set. It was demonstrated that a registry for matching pragmatic clinical studies under consideration by funders could be built on a very small data set. This would include 10 unique items, of which five are required to describe a study and the rest are metadata. In the test sample the data set with an appropriate matching rule was able to deliver a sensitivity of between 50% and 100% and a specificity of between 43% and 86% for matching different elements.

CONCLUSIONS

A number of recommendations have been made for the next EUnetHTA JA2 project and its evaluation. This included that the evaluation of the EUnetHTA JA2 project should extend beyond the end of the project to allow assessment of its impact; that the quality, usability and cost-effectiveness of tools in 'real-world HTA practice' should be assessed and tangible benefits of international networking should be evaluated. It is worth proceeding to develop a database registry aimed at identifying trials in development based on the data set developed.

FUNDING

The study was funded by the National Institute for Health Research Health Technology Assessment programme (50%) and the European Union Commission (50%).