Evaluating the usability of a semantic health data framework: approach and study

This manuscript was submitted as 'full paper' and should be reviewed along the usual dimensions for research contributions which include (1) originality, (2) significance of the results, and (3) quality of writing. Please also assess the data file provided by the authors under “Long-term stable URL for resources”. In particular, assess (A) whether the data file is well organized and in particular contains a README file which makes it easy for you to assess the data, (B) whether the provided resources appear to be complete for replication of experiments, and if not, why, (C) whether the chosen repository, if it is not GitHub, Figshare or Zenodo, is appropriate for long-term repository discoverability, and (4) whether the provided data artifacts are complete. Please refer to the reviewer instructions and the FAQ for further information.

Synopsis
The authors present a semantic health data framework which allows for the interlinking of health events with environmental risk factors particularly in the domain of rare diseases yielding favorable usability testing outcomes.

Major comments
English editing is required. There were several grammatical and wording issues throughout that would benefit from a thorough proofing to make reading smoother. Many of this had to do with word choice, and at other times several grammatical issues. The authors should consider the use of HL7 FHIR compliant standards to ensure the interoperability of the proposed semantic health data framework by offering links to medical terminologies from existing ontologies like the SNOMED-CT, LOINC, ATC, as well as, ICD10/11 terminologies. A separate paragraph should be added to highlight the FAIRness of the proposed framework. It is not clear whether the authors developed an ontology (semantic data model) for the disease(s) under investigation. If so, please clarify and add related description in the Evaluation approach. Since the proposed framework is based on the extraction of semantic knowledge, the authors should compare it with similar approaches in the literature, such as, the SORTA tool (a system for ontology-based re-coding and technical annotation of biomedical phenotype data).

summary
authors present a usability study of a tool named SERDIF (already published) that aim at facilitating the process of manual data linking. The particular task considered in the study is the linking between healthcare data and environmental data (i.e., weather and pollution data).

main remarks
This work is very original in that it evaluates a data linking tool, not based on the fact that it provides valid linking, what KG matching usually aims at doing, but rather by evaluating how the help provided to medical experts in the task of linking health and environmental data is usable to them.
The article is well written and illustrated.

I think that the choice of the use case “health + environmental data” needs to be clearly motivated:
I found confusing that the introduction mentions a use case that is a physician wondering if he should better change the dose of a treatment. And, next the use case is environmental studies and linking health and environmental data. And I never found the link in the article. Also, I am not sure to understand how linking these data my help conducting environmental studies.
This may be due to the lake of description of healthcare data authors have in mind. It seems that the article wants these data to stay general, but this makes the scenario very large, and the usefulness questionnable.

Authors mention the reproducibility of the experiment. It is clear that effort is made in this direction, what is both original and laudable.
I regret that very few is said about the subjectivity, and biais that may arise at the different steps of the study: its design (choice of codes, themes for instances) and in the alignement between themes, codes and users answers, or in the interpretation of quantitative metrics. I wonder how much the results would differ, with distinct experimenters (but same sample of users).

In this same vein, authors conclude on “the usability testing approach
was able to validate the usefulness of the framework for HDR”,
but it seems to me that some elements of interpretation of the Fig. 4 tends to the opposite, and for this reason that this conclusion needs to be moderated.
Example of negative reported results:
“Furthermore, 14 out of 17 participants found the
query process and query inputs complex at least once”

“Most of the expert participants needed
assistance when completing T4, T5, T6 and T7”

“While the plots were also useful to explore the data from different perspectives,
they need an additional explanation to clarify
some of the elements and guide the participants in
what they should be looking for”

“the navigation was complex as supported
by the type of assists for tasks T4, T5, T6 and
T7 (Fig. 4B) […]”

I agree that data linking may open doors to new studies and experiments. I also agree that usability studies for SW technologies are of great importance, and that in some sense authors illustrate that their tool is useful to guide data linking. But in the end, I miss too much the biomedical application that is the core of the call for the special issue.

detail remarks
Fig 2 does not display a polar plot.
I found the detailed reporting of the various steps followed during the experiment rather tedious.

OVERALL EVALUATION:

This paper presents a detailed usability evaluation on a semantic data framework focused on the health domain.
In general the paper is well-written and thus it is easy for the reader to follow.
This work actually extends previous works by the authors. In [8] they present their dashboard (tool), in [9] they made their first experiments and now
they perform an extended evaluation of their tool through a qualitative and quantitative analysis.
The paper is sound technically good, the followed methodology is based on state of the art,
the evaluation setup and other choices are well justified.
The results of the evaluation seem promising although the sample is not so big (17 Health Data Researchers - HDR).
My only concern is about the delta of the current work as comparing with their published work.
The authors provide a github repository with the code , a public SPARQL endpoint and an online demo.
Some more documentation is required on how to use the endpoint (e.g. a running example is missing) and how to run the online demo from github in a Windows machine.
The presentation of related work is good enough and it is interestingly relates state-of-the-art with their proposed approach/work.
The authors are encouraged to summarize in a separate section (e.g., Discussion) their lessons learned from evaluation and associate them with their recommendations and open research topics.

SUGGESTIONS FOR IMPROVEMENT:
1. In the abstract the health context is missing. The authors should explain why they focus on this domain...
2. Table 1 could analyse the type of requirement e.g. Querying, Understanding...
3. The second paragraph of introduction discuss about visualizations, RDF, FAIR data but their connection with the paper story is not so clear.
4. In the introduction the problems that Health Data Researchers face could be declared explicitly and connected as well with the paper contribution.
5. Instead of gdpr-info.eu it would be better to use the official link from European Commission for GDPR https://eur-lex.europa.eu/eli/reg/2016/679/oj
6. The authors should declare explicitly their contributions in comparison with their previous works.
7. The last paragraph of Related Work contains limitations of the present study so it would be better to be presented in a different subsection and not in the RW.
8. The UI URL takes too much time to load (https://serdif-example-dash.herokuapp.com/)

STRONG POINTS:
-Fits well with the special issue of the SWJ
-Clear contribution
-Meaningful insights for SW Meets Health Data Researchers
-Reproducibility
-Solid methodology and interesting findings

WEAK POINTS:
-Delta is not clear from previous works
-The evaluation sample is small
-The evaluation revealed that more guidance was needed for users to accomplish some of their tasks
-The applicability of the proposed evaluation methodology to other domains or use cases is not so obvious. In any case it would be better if discussed at the end of the paper.