Learning with noisy labels and crowdsourcing

An umbrella for our work on labels that cannot be trusted: blind ensemble learning, robust crowdsourcing in the presence of spammers and adversaries, and constrained crowdsourcing that exploits side information.

Overview

Contemporary supervised learning relies on vast amounts of curated data \(\{\mathbf{x}_n,y_n\}_{n=1}^{N}\), where \(\mathbf{x}_n\) denote data features of the \(n\)th data point and \(y_n\) the corresponding label. While data features \(\mathbf{x}_n\) can be readily obtained from sensors, cameras, or other connected devices, labels require significant and often expensive effort to procure. Crowdsourcing has emerged as a cost-efficient way to obtain labels for large datasets: a dataset is dispatched to a “crowd” of \(M\) workers, and the \(m\)th of them provides a response \(\check{y}^{(m)}_n\in\{1,\dots,K\}\) for the \(n\)th data point assigned to them, where the superscript \((m)\) indexes the annotator and the subscript \(n\) the data point. These responses are subsequently fused into estimates \(\hat{y}_n\) of the ground-truth labels. The same mathematical problem arises when annotators are algorithms rather than people, as in blind (unsupervised) ensemble learning, where \(\check{y}^{(m)}_n\) is the decision of the \(m\)th learner on \(\mathbf{x}_n\), and a meta-learner, or fusion center, combines the decisions of all \(M\) learners, without access to ground-truth labels and without knowing how reliable each learner is. A related setup appears in distributed detection, where sensors decide which of \(K\) hypotheses is in effect and transmit their decisions to a fusion center; a similar task is known as the CEO problem or multiterminal source coding.

Annotators, or learners, observe the data \(\mathbf{x}_n\) and the meta-learner fuses their responses \(\check{y}^{(1)}_n,\dots,\check{y}^{(M)}_n\) into a final decision \(\hat{y}_n\).

What renders such fusion possible, in the absence of any ground truth, is a model of annotator behavior. Each annotator \(m\) is typically characterized by a confusion matrix \(\mathbf{H}^{(m)}\in\mathbb{R}^{K\times K}\), whose \((k,k')\)th entry \(h^{(m)}_{kk'}:=\Pr\big(\check{y}^{(m)}_n=k'\,\big|\,y_n=k\big)\) quantifies their per-class reliability and is unknown a priori. Under the standard assumption that annotator responses are conditionally independent given the true label, the statistics of the observed responses are determined by \(\{\mathbf{H}^{(m)}\}_{m=1}^{M}\) and the label priors, and low-order moments of the responses suffice to recover both, without any labeled data. Such setups arise in diverse disciplines including medicine, biology, team decision making and economics, and have gained traction with the advent of crowdsourcing platforms such as Amazon Mechanical Turk.

Annotator responses are assumed conditionally independent given the true label \(y_n\).

Conditional independence, however, is frequently violated in practice. Data may exhibit temporal or network structure, so that labels of successive or neighboring data points are dependent; annotators may be correlated with each other, for instance when they rely on common training, tools, or sources of information. Our work accounts for such dependencies by modeling sequential and networked data, by identifying groups of dependent annotators, and by casting label fusion as learning over the bipartite graph that connects annotators to data points. When a limited labeling budget is available, active learning complements this by prescribing which few data points are worth labeling by an expert.

Robust and constrained crowdsourcing

Not all deviations from the above model are benign. Spammers respond at random, and thus contribute no information about the label; adversaries deliberately provide incorrect responses, and may act in concert, in which case their responses are abnormally correlated, and the colluding subset manifests itself as a dense subgraph of the annotator similarity graph. Aggregation schemes that presume merely imperfect annotators can be markedly degraded by such behavior, which motivates detecting and discounting these annotators, either as a preprocessing step or jointly with the estimation of the remaining reliabilities.

Responses \(\check{y}^{(m)}_n\) reach the aggregator from annotators of differing intent; some may be spammers or adversaries.

In the other direction, side information about the data is often available at little or no extra cost, and can be brought to bear on the aggregation. Such information may take the form of label constraints, that is, ground-truth labels for a small subset of the data, or of instance-level constraints, which indicate whether a pair of data points shares the same label without revealing what that label is. Bayesian algorithms based on variational inference incorporate both regimes of semi-supervision into crowdsourced classification, and analogous constrained formulations apply to decision fusion over networks.

Key papers

Overview

1. S. Ibrahim, P. A. Traganitis, X. Fu and G. B. Giannakis, “Learning from Crowdsourced Noisy Labels: A Signal Processing Perspective,” IEEE Signal Processing Magazine, vol. 42, no. 3, pp. 84–106, 2025. arXiv

Blind ensembles/Label fusion

2. P. A. Traganitis and G. B. Giannakis, “Unsupervised Ensemble Classification with Sequential and Networked Data,” IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 10, pp. 5009–5022, 2022. arXiv
3. P. A. Traganitis, A. Pagès-Zamora and G. B. Giannakis, “Blind Multiclass Ensemble Classification,” IEEE Transactions on Signal Processing, vol. 66, pp. 4737–4752, 2018. arXiv
4. P. A. Traganitis and C. I. Kanatsoulis, “A Graph Autoencoder Approach to Crowdsourcing,” Proc. of IEEE SAM Workshop, Corvallis, OR, 2024.
5. H. Wang and P. A. Traganitis, “Learning from Multiple Noisily Optimal Demonstrators in Stochastic Multi-Armed Bandits,” Proc. of IEEE MLSP, Istanbul, Turkey, 2025.
6. P. A. Traganitis, D. Berberidis and G. B. Giannakis, “Active Learning with Unsupervised Ensembles of Classifiers,” Proc. of ICASSP, Barcelona, Spain, 2020.
7. P. A. Traganitis, “Blind Ensemble Classification of Sequential Data,” Proc. of IEEE Data Science Workshop, Minneapolis, MN, 2019.
8. P. A. Traganitis and G. B. Giannakis, “Blind Multi-class Ensemble Learning with Dependent Classifiers,” Proc. of EUSIPCO, Rome, Italy, 2018.
9. P. A. Traganitis, A. Pagès-Zamora and G. B. Giannakis, “Learning from Unequally Reliable Blind Ensembles of Classifiers,” Proc. of GlobalSIP, Montreal, Canada, 2017.

Robustness and constraints

10. A. Karaaslanli, P. A. Traganitis and A. Konar, “Dense Subgraph Detection for Crowdsourcing with Colluding Adversaries,” IEEE Transactions on Signal Processing (submitted).
11. A. Karaaslanli, P. A. Traganitis and A. Konar, “Identifying Adversarial Attacks in Crowdsourcing via Dense Subgraph Detection,” Proc. of ICASSP, Hyderabad, India, 2025.
12. P. A. Traganitis and G. B. Giannakis, “Identifying Dependent Annotators in Crowdsourcing,” Proc. of Asilomar Conf. on Signals, Systems, and Computers, 2022.
13. P. A. Traganitis and G. B. Giannakis, “Detecting Adversaries in Crowdsourcing,” Proc. of IEEE ICDM, Auckland, New Zealand, 2021. arXiv
14. P. A. Traganitis and G. B. Giannakis, “Bayesian Constrained Decision Fusion,” Proc. of IEEE SPAWC, Lucca, Italy, 2021.
15. P. A. Traganitis and G. B. Giannakis, “Bayesian Crowdsourcing with Constraints,” Proc. of ECML-PKDD, Bilbao, Spain, 2021.
16. P. A. Traganitis and G. B. Giannakis, “Identifying Spammers to Boost Crowdsourced Classification,” Proc. of ICASSP, Toronto, Canada, 2021.