Crowdsourcing & Human-in-the-Loop

Quality-aware human-in-the-loop systems for data collection, processing, and analysis in data-intensive applications.

Large-scale data management cannot rely on machines alone. Human judgment is essential for handling ambiguity, ensuring quality, and interpreting domain-specific context. My research develops human-in-the-loop frameworks that integrate crowd contributions intelligently at each stage of the data pipeline.

Data collection and quality

I have developed models and tools for evaluating and improving data quality in crowdsourced settings, including crowd-based annotation tasks and quality metrics that account for worker expertise and bias (Bozzon et al., 2013) (Bozzon et al., 2015).

Crowdsourcing platforms and patterns

Early work at Politecnico di Milano produced pattern-based specifications for crowdsourcing applications (Bozzon et al., 2014), community-based crowdsourcing models (Brambilla et al., 2014), and adaptive platform designs (Brambilla et al., 2015).

Applications

Human-in-the-loop methods have been applied across several domains:

Music transcription — microtask crowdsourcing for music score error detection (Samiotis et al., 2020)
Energy consumption — user-generated content analysis for smart city energy behavior (Mauri et al., 2018) (de Kok et al., 2019)
Social research — complementing studies on vulnerable youth with social media data (Mauri et al., 2021)
Privacy — human-in-the-loop approaches to image privacy preservation (Mauri & Bozzon, 2021)

References

2021

CHItaly

Complementing Studies on Vulnerable Youths with Reddit Data

Andrea Mauri, Achilleas Psyllidis, Alessandro Bozzon, and 4 more authors

In 14th Biannual Conference of the Italian SIGCHI Chapter (CHItaly), 2021

Bib

@inproceedings{mauri2021complementing,
  title = {Complementing Studies on Vulnerable Youths with Reddit Data},
  author = {Mauri, Andrea and Psyllidis, Achilleas and Bozzon, Alessandro and Lee, Ju-Sung and Pridmore, Jason and van Zoonen, Liesbet and Giest, Sarah},
  booktitle = {14th Biannual Conference of the Italian SIGCHI Chapter (CHItaly)},
  pages = {1--8},
  year = {2021},
}

IIR

Towards a Human in the Loop Approach to Preserve Privacy in Images

Andrea Mauri and Alessandro Bozzon

In 11th Italian Information Retrieval Workshop (IIR), 2021

Bib

@inproceedings{mauri2021towards,
  title = {Towards a Human in the Loop Approach to Preserve Privacy in Images},
  author = {Mauri, Andrea and Bozzon, Alessandro},
  booktitle = {11th Italian Information Retrieval Workshop (IIR)},
  year = {2021},
}

2020

ISMIR

Microtask Crowdsourcing for Music Score Transcriptions: an Experiment with Error Detection

Ioannis Petros Samiotis, Sihang Qiu, Andrea Mauri, and 3 more authors

In International Society for Music Information Retrieval Conference (ISMIR), 2020

Bib

@inproceedings{samiotis2020microtask,
  title = {Microtask Crowdsourcing for Music Score Transcriptions: an Experiment with Error Detection},
  author = {Samiotis, Ioannis Petros and Qiu, Sihang and Mauri, Andrea and Liem, Cynthia C.S. and Lofi, Christoph and Bozzon, Alessandro},
  booktitle = {International Society for Music Information Retrieval Conference (ISMIR)},
  pages = {901--907},
  year = {2020},
}

2019

Energies

Automatic Processing of User-Generated Content for the Description of Energy-Consuming Activities at Individual and Group Level

Roos Kok, Andrea Mauri, and Alessandro Bozzon

Energies, 2019

Bib

@article{de2019automatic,
  title = {Automatic Processing of User-Generated Content for the Description of Energy-Consuming Activities at Individual and Group Level},
  author = {de Kok, Roos and Mauri, Andrea and Bozzon, Alessandro},
  journal = {Energies},
  volume = {12},
  number = {1},
  pages = {1--28},
  year = {2019},
  publisher = {MDPI},
}

2018

WWW

Social Smart Meter: Identifying Energy Consumption Behavior in User-generated Content

Andrea Mauri, Achilleas Psyllidis, and Alessandro Bozzon

In Companion Proceedings of The Web Conference (WWW), 2018

Bib

@inproceedings{mauri2018social,
  title = {Social Smart Meter: Identifying Energy Consumption Behavior in User-generated Content},
  author = {Mauri, Andrea and Psyllidis, Achilleas and Bozzon, Alessandro},
  booktitle = {Companion Proceedings of The Web Conference (WWW)},
  pages = {195--198},
  year = {2018},
  organization = {International World Wide Web Conferences Steering Committee},
}

2015

JWE

Designing Complex Crowdsourcing Applications Covering Multiple Platforms and Tasks

Alessandro Bozzon, Marco Brambilla, Stefano Ceri, and 2 more authors

Journal of Web Engineering, 2015

Bib

@article{bozzon2015designing,
  title = {Designing Complex Crowdsourcing Applications Covering Multiple Platforms and Tasks},
  author = {Bozzon, Alessandro and Brambilla, Marco and Ceri, Stefano and Mauri, Andrea and Volonterio, Riccardo},
  journal = {Journal of Web Engineering},
  volume = {14},
  number = {5&6},
  pages = {443--473},
  year = {2015},
}

IEEE IC

Adaptive and Interoperable Crowdsourcing

Marco Brambilla, Stefano Ceri, Andrea Mauri, and 1 more author

IEEE Internet Computing, 2015

Bib

@article{brambilla2015adaptive,
  title = {Adaptive and Interoperable Crowdsourcing},
  author = {Brambilla, Marco and Ceri, Stefano and Mauri, Andrea and Volonterio, Riccardo},
  journal = {IEEE Internet Computing},
  volume = {19},
  number = {5},
  pages = {36--44},
  year = {2015},
  publisher = {IEEE},
}

2014

ICWE

Pattern-based Specification of Crowdsourcing Applications

Alessandro Bozzon, Marco Brambilla, Stefano Ceri, and 2 more authors

In International Conference on Web Engineering (ICWE), 2014

Bib

@inproceedings{bozzon2014pattern,
  title = {Pattern-based Specification of Crowdsourcing Applications},
  author = {Bozzon, Alessandro and Brambilla, Marco and Ceri, Stefano and Mauri, Andrea and Volonterio, Riccardo},
  booktitle = {International Conference on Web Engineering (ICWE)},
  pages = {218--235},
  year = {2014},
  organization = {Springer International Publishing},
}

WWW

Community-based Crowdsourcing

Marco Brambilla, Stefano Ceri, Andrea Mauri, and 1 more author

In Proceedings of the 23rd International Conference on World Wide Web Companion (WWW), 2014

Bib

@inproceedings{brambilla2014community,
  title = {Community-based Crowdsourcing},
  author = {Brambilla, Marco and Ceri, Stefano and Mauri, Andrea and Volonterio, Riccardo},
  booktitle = {Proceedings of the 23rd International Conference on World Wide Web Companion (WWW)},
  pages = {891--896},
  year = {2014},
  organization = {ACM},
}

2013

WWW

Reactive crowdsourcing

Alessandro Bozzon, Marco Brambilla, Stefano Ceri, and 1 more author

In Proceedings of the 22nd International Conference on World Wide Web (WWW), 2013

Bib

@inproceedings{bozzon2013reactive,
  title = {Reactive crowdsourcing},
  author = {Bozzon, Alessandro and Brambilla, Marco and Ceri, Stefano and Mauri, Andrea},
  booktitle = {Proceedings of the 22nd International Conference on World Wide Web (WWW)},
  pages = {153--164},
  year = {2013},
  organization = {International World Wide Web Conferences Steering Committee},
}