Evaluating Variability and Biases among Reviewers of ROV-Collected Video, and Optimizing Video Review Protocols

Since 2004, the Washington Department of Fish and Wildlife (WDFW) has utilized a remotely operated vehicle (ROV) to survey groundfish and benthic habitats within Puget Sound. Video imagery collected during these surveys requires substantial post-processing, with reviewers undergoing extensive training to systematically collect data on substrate, biocover, and species of interest. After initial training by senior staff to ensure they are following standardized protocols, individuals review videos independently. However, inherent reviewer biases stemming from various sources can influence data collection and ultimately skew species abundance estimates produced from video data. To gauge variability and biases among reviewers, select ROV-collected video transects are independently analyzed by two or more reviewers, and data are directly compared to assess differences in habitat classification, species detection, and species identification. Results suggest higher variability between reviewers’ substrate and biocover classifications than identification of species. Recognizing these variations allows reviewers with contrasting biases to be paired such that the overall error rate and bias is minimized, saving time and improving accuracy. Additional time savings may result from optimized subsampling of video recordings following a standardized procedure. Results of the analysis and protocol improvements to maximize accuracy and minimize video review time will be discussed.