Importance sampling of delta-AUC: A basis for active learning for improved keyword search

Kerri Barnes; Matthew Snover; Man-Hung Siu; Herbert Gish

ICASSP 2016accepted0 citations

Importance sampling of delta-AUC: A basis for active learning for improved keyword search

Kerri Barnes, Matthew Snover, Man-Hung Siu, Herbert Gish

Abstract

We present an importance sampling based approach to the active learning problem of selecting additional training data to supplement a seed model. Our proposed Δ-AUC selection optimizes AUC improvement in keyword search and is evaluated on the Spanish Fisher corpus. We show that over different training data sizes, Δ-AUC selection consistently outperforms random sampling by 1.05% to 2.69% absolute AUC and requires no more than 60% of the transcriptions needed by random sampling to achieve the same AUC. On terms not seen in the original seed model training, the proposed algorithm achieves a 3.47% better AUC and 4.66% reduction in word error rate. We also introduce a regression analysis model that can refine our Δ-AUC strategy in the future.

BibTeX

@inproceedings{icassp2016_importancesampli,
  title = {Importance sampling of delta-AUC: A basis for active learning for improved keyword search},
  author = {Kerri Barnes and Matthew Snover and Man-Hung Siu and Herbert Gish},
  booktitle = {ICASSP 2016},
  year = {2016}
}