Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Selective Annotation via Data Allocation: These Data Should Be Triaged to Experts for Annotation Rather Than the Model (2405.12081v2)

Published 20 May 2024 in cs.CL

Abstract: To obtain high-quality annotations under limited budget, semi-automatic annotation methods are commonly used, where a portion of the data is annotated by experts and a model is then trained to complete the annotations for the remaining data. However, these methods mainly focus on selecting informative data for expert annotations to improve the model predictive ability (i.e., triage-to-human data), while the rest of the data is indiscriminately assigned to model annotation (i.e., triage-to-model data). This may lead to inefficiencies in budget allocation for annotations, as easy data that the model could accurately annotate may be unnecessarily assigned to the expert, and hard data may be misclassified by the model. As a result, the overall annotation quality may be compromised. To address this issue, we propose a selective annotation framework called SANT. It effectively takes advantage of both the triage-to-human and triage-to-model data through the proposed error-aware triage and bi-weighting mechanisms. As such, informative or hard data is assigned to the expert for annotation, while easy data is handled by the model. Experimental results show that SANT consistently outperforms other baselines, leading to higher-quality annotation through its proper allocation of data to both expert and model workers. We provide pioneering work on data annotation within budget constraints, establishing a landmark for future triage-based annotation studies.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (68)
  1. Jonathan F Bard. 2013. Practical bilevel optimization: algorithms and applications, volume 30. Springer Science & Business Media.
  2. A curriculum learning method for improved noise robustness in automatic speech recognition. In 2017 25th European Signal Processing Conference (EUSIPCO), pages 548–552. IEEE.
  3. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  4. Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793–805, Vancouver, Canada. Association for Computational Linguistics.
  5. Tools impact on the quality of annotations for chat untangling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 215–220, Online. Association for Computational Linguistics.
  6. Robust bounds for classification via selective sampling. In Proceedings of the 26th annual international conference on machine learning, pages 121–128.
  7. Worst-case analysis of selective sampling for linear classification. Journal of Machine Learning Research, 7(7).
  8. Language resource efficient learning for captioning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1887–1895, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  9. Jit2r: A joint framework for item tagging and tag-based recommendation. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, pages 1681–1684.
  10. A survey on automatic image annotation. Applied Intelligence, 50(10):3412–3428.
  11. Ike-an interactive tool for knowledge extraction. In Proceedings of the 5th Workshop on Automated Knowledge Base Construction, pages 12–17.
  12. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pages 1184–1193. PMLR.
  13. Ai assisted data labeling with interactive auto label. In AAAI Conference on Artificial Intelligence.
  14. Semi-automated data labeling. In Proceedings of the NeurIPS 2020 Competition and Demonstration Track, volume 133 of Proceedings of Machine Learning Research, pages 156–169. PMLR.
  15. Semi-automated data labeling. In NeurIPS 2020 Competition and Demonstration Track, pages 156–169. PMLR.
  16. Is GPT-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173–11195, Toronto, Canada. Association for Computational Linguistics.
  17. Active Learning for BERT: An Empirical Study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7949–7962, Online. Association for Computational Linguistics.
  18. Yansong Feng and Mirella Lapata. 2008. Automatic image annotation using auxiliary text information. In Proceedings of ACL-08: HLT, pages 272–280, Columbus, Ohio. Association for Computational Linguistics.
  19. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056.
  20. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR.
  21. Annollm: Making large language models to be better crowdsourced annotators.
  22. Anea: distant supervision for low-resource named entity recognition. the Practical Machine Learning For Developing Countries Workshop at ICLR.
  23. Zero-shot clinical entity recognition using chatgpt. arXiv preprint arXiv:2303.16416.
  24. Rebecca Hwa. 2000. Sample selection for statistical grammar induction. In 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 45–52.
  25. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144.
  26. To trust or not to trust a classifier. NeurIPS, 31.
  27. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431. Association for Computational Linguistics.
  28. The INCEpTION platform: Machine-assisted and knowledge-oriented interactive annotation. In COLING: System Demonstrations, pages 5–9, Santa Fe, New Mexico. ACL.
  29. Data augmentation for hypernymy detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1034–1048.
  30. Oyku Deniz Kose and Yanning Shen. 2022. Fairness-aware selective sampling on attributed graphs. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5682–5686.
  31. Interactive information extraction with constrained conditional random fields. In AAAI, volume 4, pages 412–418.
  32. Inconsistencies in crowdsourced slot-filling annotations: A typology and identification methods. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5035–5046.
  33. Promptiverse: Scalable generation of scaffolding prompts through human-ai hybrid knowledge graph annotation. In CHI Conference on Human Factors in Computing Systems, pages 1–18.
  34. Boosting few-shot learning with adaptive margin loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12576–12584.
  35. Coannotating: Uncertainty-guided work allocation between human and large language models for data annotation.
  36. Vinery: A visual ide for information extraction. Proceedings of the VLDB Endowment, 8(12):1948–1951.
  37. A comprehensive evaluation of chatgpt’s zero-shot text-to-sql capability. arXiv preprint arXiv:2303.13547.
  38. Continuous quality control and advanced text segment annotation with WAT-SL 2.0. In Proceedings of the 13th Linguistic Annotation Workshop, pages 215–219, Florence, Italy. ACL.
  39. A general framework for uncertainty estimation in deep learning. IEEE Robotics and Automation Letters, 5(2):3153–3160.
  40. Predict responsibly: improving fairness and accuracy by learning to defer. NeurIPS, 31.
  41. Semi-automatic discourse annotation in a low-resource language: Developing a connective lexicon for nigerian pidgin. In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 84–94.
  42. Samuel Marcos-Pablos and Francisco J García-Peñalvo. 2020. Information retrieval methodology for aiding scientific database search. Soft Computing, 24(8):5551–5560.
  43. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pages 15630–15649. PMLR.
  44. Hussein Mozannar and David Sontag. 2020. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning, pages 7076–7087. PMLR.
  45. On the calibration of multiclass classification with rejection. NeurIPS, 32.
  46. Differentiable learning under triage. NeurIPS, 34.
  47. Glove: Global vectors for word representation. In EMNLP, pages 1532–1543.
  48. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.12220.
  49. A survey of deep active learning. ArXiv, abs/2009.00236.
  50. Inclusive yet selective: Supervised distributional hypernymy detection. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1025–1036.
  51. Analysis of automatic annotation suggestions for hard discourse-level tasks in expert domains. In 57Th Annual Meeting of the Association for Computational Linguistics (Acl 2019), pages 2761–2772. Association for Computational Linguistics.
  52. Agam Shah and Sudheer Chava. 2023. Zero is not hero yet: Benchmarking zero-shot performance of llms for financial tasks. arXiv preprint arXiv:2305.16633.
  53. Learning emphasis selection for written text in visual media from crowd-sourced label distributions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1167–1172, Florence, Italy. Association for Computational Linguistics.
  54. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 761–769.
  55. Selective annotation makes language models better few-shot learners.
  56. Ying-Peng Tang and Sheng-Jun Huang. 2019. Self-paced active learning: Query the right thing at the right time. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 5117–5124.
  57. Web interface for exploration of latent and tag spaces in music auto-tagging. In Proceedings of the 37 th International Conference on Machine Learning. Vienna, Austria. ICML.
  58. Collaborative topic regression with social regularization for tag recommendation. In Twenty-Third International Joint Conference on Artificial Intelligence.
  59. Boosting active learning via improving test performance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8566–8574.
  60. Xin Wang and Siu Ming Yiu. 2021. Classification with rejection: scaling generative classifiers with supervised deep infomax. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 2980–2986.
  61. Large margin few-shot learning. arXiv preprint arXiv:1807.02872.
  62. Breaking the curse of kernelization: Budgeted stochastic gradient descent for large-scale svm training. The Journal of Machine Learning Research, 13(1):3103–3131.
  63. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  64. Learning to complement humans. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 1526–1533.
  65. FreeAL: Towards human-free active learning in the era of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14520–14535, Singapore. Association for Computational Linguistics.
  66. Beyond labels: Empowering human with natural language explanations through a novel active-learning architecture.
  67. An interactive neural network approach to keyphrase extraction in talent recruitment. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2383–2393.
  68. Learning with local and global consistency. In Advances in Neural Information Processing Systems, volume 16. MIT Press.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Chen Huang (88 papers)
  2. Yang Deng (115 papers)
  3. Wenqiang Lei (66 papers)
  4. Jiancheng Lv (99 papers)
  5. Ido Dagan (72 papers)
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now