ABSTRACT
Crowdsourcing has created a variety of opportunities for many challenging problems by leveraging human intelligence. For example, applications such as image tagging, natural language processing, and semantic-based information retrieval can exploit crowd-based human computation to supplement existing computational algorithms. Naturally, human workers in crowdsourcing solve problems based on their knowledge, experience, and perception. It is therefore not clear which problems can be better solved by crowdsourcing than solving solely using traditional machine-based methods. Therefore, a cost sensitive quantitative analysis method is needed.
In this paper, we design and implement a cost sensitive method for crowdsourcing. We online estimate the profit of the crowdsourcing job so that those questions with no future profit from crowdsourcing can be terminated. Two models are proposed to estimate the profit of crowdsourcing job, namely the linear value model and the generalized non-linear model. Using these models, the expected profit of obtaining new answers for a specific question is computed based on the answers already received. A question is terminated in real time if the marginal expected profit of obtaining more answers is not positive. We extends the method to publish a batch of questions in a HIT. We evaluate the effectiveness of our proposed method using two real world jobs on AMT. The experimental results show that our proposed method outperforms all the state-of-art methods.
- http://www.mturk.com.Google Scholar
- O. Alonso, D. Rose, and B. Stewart. Crowdsourcing for relevance evaluation. In SIGIR Forum, volume 42, pages 9--15. ACM, 2008. Google ScholarDigital Library
- A. Feng, M. Franklin, D. Kossmann, T. Kraska, S. Madden, S. Ramesh, A. Wang, and R. Xin. Crowddb: Query processing with the vldb crowd. VLDB, 4(12), 2011.Google Scholar
- M. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. Crowddb: answering queries with crowdsourcing. In SIGMOD, pages 61--72, 2011. Google ScholarDigital Library
- S. Guo, A. Parameswaran, and H. Garcia-Molina. So who won?: dynamic max discovery with the crowd. SIGMOD, pages 385--396. ACM, 2012. Google ScholarDigital Library
- P. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. SIGKDD workshop, pages 64--67. ACM, 2010. Google ScholarDigital Library
- G. Kazai, J. Kamps, M. Koolen, and N. Milic-Frayling. Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking. SIGIR, 2011. Google ScholarDigital Library
- A. Kittur, E. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. SIGCHI, pages 453--456. ACM, 2008. Google ScholarDigital Library
- X. Liu, M. Lu, B. Ooi, Y. Shen, S. Wu, and M. Zhang. Cdas: a crowdsourcing data analytics system. VLDB, 5(10):1040--1051, 2012. Google ScholarDigital Library
- A. Marcus, E. Wu, D. Karger, S. Madden, and R. Miller. Crowdsourced databases: Query processing with people. CIDR, 2011.Google Scholar
- A. Marcus, E. Wu, D. Karger, S. Madden, and R. Miller Demonstration of Qurk: a query processor for humanoperators. SIGMOD, pages 1315--1318. ACM, 2011. Google ScholarDigital Library
- A. Parameswaran, H. Garcia-Molina, H. Park, N. Polyzotis, A. Ramesh, and J. Widom. Crowdscreen: Algorithms for filtering data with humans. SIGMOD, pages 361--372. ACM, 2012. Google ScholarDigital Library
- A. Parameswaran and N. Polyzotis. Answering queries using humans, algorithms and databases. CIDR, 2011.Google Scholar
- A. Parameswaran, A. Sarma, H. Garcia-Molina, N. Polyzotis, and J. Widom. Human-assisted graph search: it's okay to ask questions. VLDB, 4(5):267--278, 2011. Google ScholarDigital Library
- V. Raykar, S. Yu, L. Zhao, G. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. JMLR, 11:1297--1322, 2010. Google ScholarDigital Library
- J. Selke, C. Lofi, and W. Balke. Pushing the boundaries of crowd-enabled databases with query-driven schema expansion. VLDB, 5(6):538--549, 2012. Google ScholarDigital Library
- J. Wang, T. Kraska, M. Franklin, and J. Feng. Crowder: crowdsourcing entity resolution. VLDB, 5(11):1483--1494, 2012. Google ScholarDigital Library
- P. Welinder and P. Perona. Online crowdsourcing: rating annotators and obtaining cost-effective labels. CVPR workshop, pages 25--32. IEEE, 2010.Google ScholarCross Ref
- T. Yan, V. Kumar, and D. Ganesan. Crowdsearch: exploiting crowds for accurate real-time image search on mobile phones. MobiSys, pages 77--90, 2010. Google ScholarDigital Library
Index Terms
- An online cost sensitive decision-making method in crowdsourcing systems
Recommendations
Online decision making in crowdsourcing markets: theoretical challenges
Over the past decade, crowdsourcing has emerged as a cheap and efficient method of obtaining solutions to simple tasks that are difficult for computers to solve but possible for humans. The popularity and promise of crowdsourcing markets has led to both ...
Incentivizing Distributive Fairness for Crowdsourcing Workers
AAMAS '19: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent SystemsIn a crowd market such as Amazon Mechanical Turk, the remuneration of Human Intelligence Tasks is determined by the requester, for which they are not given many cues to ascertain how to "fairly'' pay their workers. Furthermore, the current methods for ...
Truthful Team Formation for Crowdsourcing in Social Networks: (Extended Abstract)
AAMAS '16: Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent SystemsThis paper studies complex task crowdsourcing by team formation in social networks (SNs), where the requester wishes to hire a group of socially connected workers that can work together as a team. Previous social team crowdsourcing approaches mainly ...
Comments