advertisement

Editors Selection IGR 15-3

Comment: Extraordinary claims require extraordinary evidence

Comment by Giovanni Montesano on:

In their research, Zhao et al. report on the effects of a virtual-reality (VR) training protocol on 54 patients with glaucoma. The hypothesis was that, through neuroplasticity, their training protocol could induce structural changes in the retina and, crucially, improvements in visual function, tested with white-on-white perimetry (24-2 SITA fast). This is an attractive proposition, because it would offer a non-invasive approach to improving visual function in patients with glaucoma. They show a small borderline statistically significant increase in the thickness of the superior macular ganglion cell and inner plexiform layer (mGCIPL) and a similarly small improvement in the average sensitivity of the central 12 locations of the 24-2 test. They conclude that “VR visual training has some positive effects on retinal ganglion cells and the central visual sensitivity of glaucoma patients.”

This might well be true. However, several methodological shortcomings make their evidence far from conclusive. Perimetry, or static visual field (VF) testing, is the most important test to assess functional damage and its progression in glaucoma patients and has been successfully used to demonstrate the effect of therapeutic intervention in landmark clinical trials. However, it is also affected by an important learning effect, by which the measured sensitivity can improve over time simply because of improved performance with experience.^1,2 This makes showing true improvement challenging. The authors mention that patients were required to have produced at least two reliable VF tests prior to recruitment. However, this does not prevent the influence of learning, whose effect has been shown to carry on, on average, for up to 7 tests.^1,2 A similar critique could be made for research claiming to show improvement after pressure lowering interventions.^3,4 As for many other challenging scientific questions, the answer comes from the magic of randomization:^4,5 the authors should have randomised an equal number of patients to either receiving their VR training or a sham treatment, comparing the results in the two groups. This procedure would have equated patients’ characteristics and, importantly, the learning effect between the two groups, allowing the authors to isolate the true effect of their protocol on functional improvement. There are also technical shortcomings in their choice of the testing approach, such as the use of a SITA Fast algorithm (instead of Standard) and the use of a 24-2 grid to investigate macular sensitivity (instead of a 10-2). These choices reduce the precision of the measurements and the confidence in the results.

The structural assessments suffer from similar issues. Although structural data are not affected by systematic learning, they are influenced by noise, especially when relying only on a single scan per patient at each time-point. Moreover, similarly to their VF testing, the authors did not adapt their methodology to target the specific scientific question. Their hypothetical explanation for possible changes in mGCIPL increase was related to plastic changes in the retina after training. This could have been better investigated by analysing the GCL and IPL thickness separately, since the latter would be more influenced by hypothetical synaptic changes induced by their VR training protocol. Once again, any refinement of their methodology would not overcome the strong limitation of lacking a control group.

There are additional methodological issues in their approach to the analysis and reporting of their results. Despite being registered as a clinical trial, the registered protocol does not report any pre-specified analysis or outcome. This is concerning, because their statistically significant findings resulted from multiple comparisons, with no attempt to correct for the increased false discovery rate derived from multiple testing. In fact, even their smallest p-value would not stand up to a relatively conservative multiple test correction (in Table 4, p = 0.024 is < 0.05, but the threshold should be lowered to at least 0.017). An even stronger point of concern is that the original published protocol does mention a plan to recruit a control group. The authors not only do not mention it in the final paper, but explicitly state that a control group was not recruited.

The topographical concordance between the increased superior macular sensitivity and the inferior mGCIPL thickness is, however, promising and might warrant additional investigation into the potential of VF training for glaucoma patients. The evidence presented in this paper is, however, insufficient to support the author’s extraordinary claims.

• ← Previous
• Next →