<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="shen-etal-2019-modelling">
    <titleInfo>
        <title>Modelling Uncertainty in Collaborative Document Quality Assessment</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Aili</namePart>
        <namePart type="family">Shen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Daniel</namePart>
        <namePart type="family">Beck</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Bahar</namePart>
        <namePart type="family">Salehi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jianzhong</namePart>
        <namePart type="family">Qi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Timothy</namePart>
        <namePart type="family">Baldwin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2019-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Wei</namePart>
            <namePart type="family">Xu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alan</namePart>
            <namePart type="family">Ritter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tim</namePart>
            <namePart type="family">Baldwin</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Afshin</namePart>
            <namePart type="family">Rahimi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Hong Kong, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In the context of document quality assessment, previous work has mainly focused on predicting the quality of a document relative to a putative gold standard, without paying attention to the subjectivity of this task. To imitate people’s disagreement over inherently subjective tasks such as rating the quality of a Wikipedia article, a document quality assessment system should provide not only a prediction of the article quality but also the uncertainty over its predictions. This motivates us to measure the uncertainty in document quality predictions, in addition to making the label prediction. Experimental results show that both Gaussian processes (GPs) and random forests (RFs) can yield competitive results in predicting the quality of Wikipedia articles, while providing an estimate of uncertainty when there is inconsistency in the quality labels from the Wikipedia contributors. We additionally evaluate our methods in the context of a semi-automated document quality class assignment decision-making process, where there is asymmetric risk associated with overestimates and underestimates of document quality. Our experiments suggest that GPs provide more reliable estimates in this context.</abstract>
    <identifier type="citekey">shen-etal-2019-modelling</identifier>
    <identifier type="doi">10.18653/v1/D19-5525</identifier>
    <location>
        <url>https://aclanthology.org/D19-5525</url>
    </location>
    <part>
        <date>2019-11</date>
        <extent unit="page">
            <start>191</start>
            <end>201</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Modelling Uncertainty in Collaborative Document Quality Assessment
%A Shen, Aili
%A Beck, Daniel
%A Salehi, Bahar
%A Qi, Jianzhong
%A Baldwin, Timothy
%Y Xu, Wei
%Y Ritter, Alan
%Y Baldwin, Tim
%Y Rahimi, Afshin
%S Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)
%D 2019
%8 November
%I Association for Computational Linguistics
%C Hong Kong, China
%F shen-etal-2019-modelling
%X In the context of document quality assessment, previous work has mainly focused on predicting the quality of a document relative to a putative gold standard, without paying attention to the subjectivity of this task. To imitate people’s disagreement over inherently subjective tasks such as rating the quality of a Wikipedia article, a document quality assessment system should provide not only a prediction of the article quality but also the uncertainty over its predictions. This motivates us to measure the uncertainty in document quality predictions, in addition to making the label prediction. Experimental results show that both Gaussian processes (GPs) and random forests (RFs) can yield competitive results in predicting the quality of Wikipedia articles, while providing an estimate of uncertainty when there is inconsistency in the quality labels from the Wikipedia contributors. We additionally evaluate our methods in the context of a semi-automated document quality class assignment decision-making process, where there is asymmetric risk associated with overestimates and underestimates of document quality. Our experiments suggest that GPs provide more reliable estimates in this context.
%R 10.18653/v1/D19-5525
%U https://aclanthology.org/D19-5525
%U https://doi.org/10.18653/v1/D19-5525
%P 191-201

Markdown (Informal)

[Modelling Uncertainty in Collaborative Document Quality Assessment](https://aclanthology.org/D19-5525) (Shen et al., WNUT 2019)

• Modelling Uncertainty in Collaborative Document Quality Assessment (Shen et al., WNUT 2019)

ACL

• Aili Shen, Daniel Beck, Bahar Salehi, Jianzhong Qi, and Timothy Baldwin. 2019. Modelling Uncertainty in Collaborative Document Quality Assessment. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 191–201, Hong Kong, China. Association for Computational Linguistics.