Main

Protein glycosylation, the attachment of complex carbohydrates (glycans) to discrete sites on proteins, plays diverse roles in biology1. The system-wide analysis of intact glycopeptides, or glycoproteomics, aims to study glycan structures, modification sites and protein carriers at scale within a single experiment2,3. Facilitated by the recent advances in separation science, mass spectrometry (MS) and informatics, glycoproteomics has matured over past decades and is now ready to tackle biological questions and generate new insights into the heterogeneous glycoproteomes of biological systems4,5,6,7.

While glycoproteomics studies now routinely report thousands of N- and O-glycopeptides8, accurate identification of glycopeptides from large volumes of mass spectral data remains a bottleneck. The annotation process of glycopeptide MS/MS data is highly error prone due to the challenging task of correctly assigning the glycan composition, modification site(s) as well as the peptide carrier9,10,11. As a result, glycopeptides reported in glycoproteomics publications are frequently misidentified or suffer from ambiguous annotation even in studies attempting to control the false discovery rate (FDR) of assignments.

Diverse fragmentation modes including resonance-activation collision-induced dissociation (CID), beam-type CID (higher-energy collisional dissociation; HCD) and electron-transfer dissociation (ETD) have proved valuable for glycoproteomics12,13,14,15. When applied in concert—now possible, for example, on Orbitrap Tribrid mass spectrometers—these fragmentation strategies provide complementary structural information on glycopeptides. Briefly, HCD-MS/MS informs on the peptide carrier and produces useful diagnostic glycan fragments, enabling glycopeptide classification and deduction of generic glycan compositions, ETD-MS/MS reveals in favorable cases the modification site and peptide identity, while resonance-activation CID-MS/MS informs primarily on the glycan composition, sequence and topology16,17. Hybrid-type fragmentation strategies including electron-transfer/collision-induced dissociation (ETciD) and electron-transfer/higher-energy collision dissociation (EThcD) are becoming popular given their ability to generate information-rich glycopeptide fragment spectra containing multiple fragment types18. Accurate mass measurements (<5–10 ppm) at high resolution of precursor and product ions, available on most contemporary instruments, are essential in glycoproteomics. Despite these exciting advances, unambiguous glycopeptide identification remains challenging. Informatics advances are therefore required to ensure accurate glycoproteome profiling to further the field19.

Glycoproteomics has seen the development of diverse commercial and academic software showing promise for precise annotation and identification of glycopeptides from MS/MS data20,21. While some of these tools are already well established and widely applied in glycoproteomics22, the relative performance of software available to the community remains untested, leaving a critical knowledge gap that hinders rapid progress in the field.

Facilitated by the HUPO Human Glycoproteomics Initiative (HGI), we here perform a comprehensive community-based evaluation of existing informatics solutions for large-scale glycopeptide analysis. While informatics challenges undoubtedly still exist in glycoproteomics, our study highlights that several computational tools, some already demonstrating high performance and others showing considerable potential, are available to the community. Importantly, key performance-associated search parameters and high-performance search strategies were identified that may help software developers and users to improve glycoproteomics data analysis in the immediate future.

Results

Study design and overview

Two glycoproteomics data files (Files A and B) were generated using HCD-ETciD-CID-MS/MS and HCD-EThcD-CID-MS/MS of N- and O-glycopeptides from human serum, respectively (Fig. 1a). A synthetic N-glycopeptide was included as a positive control. Serum is a well-characterized biospecimen displaying profound heterogeneity of N- and O-glycoproteins23,24,25. Thus, Files A and B displayed characteristics (file size, complexity, type) similar to those of data typically encountered in glycoproteomics26,27,28,29 and were compatible with most search engines.

Fig. 1: Study overview.
figure 1

a, Two glycoproteomics data files of human serum (Files A and B) were generated and shared with participants. b, Participants comprising both developers (orange) and users (blue, team identifiers indicated) employed diverse search engines to complete the study. c, Teams returned a common reporting template capturing details of the applied search strategy including key search settings (SS1–SS13) and search output (SO1–SO9, Table 1) and their identified glycopeptides. d, Complementary performance tests (N1–N6, O1–O5; Table 2) were used to comprehensively evaluate the ability of teams to identify N- and O-glycopeptides. e, The performance profiles were used to score and rank the developers and users separately. Diverse team-wide and search engine-centric (Byonic-focused) approaches were employed to identify performance-associated variables and high-performance search strategies.

Full size image

Files A and B were shared with all 22 participating teams, who classified themselves as developers (9 teams) or users (13 teams) of glycoproteomics software (Fig. 1b and Extended Data Fig. 1a,b). All teams identified N- and O-glycopeptides from Files A and B and reported their approaches and identifications in a standardized reporting template. Most developers (5 teams) and users (8 teams) were experienced in glycoproteomics (>10 years). Participants were from North America, South America, Europe, Asia and Oceania (Extended Data Fig. 1c,d).

Unlike File A, which was processed by 20 of 22 teams (90.9%), File B was processed by all teams (Extended Data Fig. 1e). While most participants reported on spectra acquired with multiple fragmentation methods, a few teams used only HCD- or EThcD-MS/MS for the identifications (Extended Data Fig. 1f,g).

Participants used diverse search engines (Fig. 1b and Extended Data Fig. 1h). Some search engines were used as stand-alone tools or with other software while others were applied with pre- or postprocessing tools to aid the identification (Extended Data Fig. 1i). The developers used nine different glycopeptide-centric search engines, including the following: team 1: IQ-GPA v2.530; team 2: Protein Prospector v5.20.2331; team 3: glyXtoolMS v0.1.432; team 4: Byonic v2.16.1633; team 5: Sugar Qb34; team 6: Glycopeptide Search v2.0alpha35; team 7: GlycopeptideGraphMS v1.036/Byonic33; team 8: GlycoPAT v2.037 and team 9: GPQuest v2.038. Among the 13 users, 10 teams (~75%) used Byonic (teams 10, 11, 13, 15–18 and 20–22), while a few teams used Protein Prospector (team 12), SugarQB/Sequest HT (team 14) and Mascot (team 19) (Supplementary Table 1).

Files A and B contained 8,737/9,776 HCD-MS/MS scans, of which 5,485/6,148 (~63%) spectra contained glycopeptide-specific oxonium ions (e.g., m/z 204.0867) used for ETciD/EThcD/CID-MS/MS triggering (Extended Data Fig. 2a,b). Among all potential glycopeptide MS/MS spectra (Files A and B, 16,445/18,444, considering all fragmentation modes), 3,402/4,982 (20.7%/27.0%) nonredundant (unique) glycopeptide-to-spectrum matches (glycoPSMs) were collectively reported by participants. Most teams reported on HCD- and EThcD-MS/MS data, while only a few teams used CID- and ETciD-MS/MS. Similar charge distribution (most frequently quadruply charged precursors) was observed for glycopeptides reported from different fragmentation modes (Extended Data Fig. 2c,d).

A wealth of data was collected via a comprehensive reporting template. The team reports covered intricate details of the employed search strategies and identified glycopeptides (Fig. 1c). Details of the applied search settings were captured including permitted peptide modifications, mass tolerance, postsearch filtering criteria (Supplementary Table 1) and the applied glycan search space (Supplementary Table 2). The search settings (SS1–SS13, Table 1) varied considerably across teams.

Table 1 Overview of important study variables including key search settings (SS1–SS13) and search output (SO1–SO9)
Full size table

Diverse output data arising from the glycopeptide identification process were captured (Supplementary Table 3). The output data also varied notably across teams (Supplementary Table 4). Analysis of key search output variables (SO1–SO9, Table 1) revealed that the reported N- and O-glycopeptides, as expected, showed different characteristics (e.g., liquid chromatography (LC) retention time, glycan mass) while other characteristics (e.g., observed precursor m/z) were similar between the two analyte classes (Extended Data Fig. 3). Analysis of SO1–SO9 data also demonstrated that some teams reported highly discrepant outputs. For example, and without being able to link these observations to performance, the developers of Glycopeptide Search (team 6) and GlycopeptideGraphMS (team 7) reported glycopeptides with unusually low (z = ~3+) and high (~5.5+) charge states relative to other teams (~4.5+) (Extended Data Fig. 3c). These output data comparisons may be valuable for developers to better understand, further develop and ultimately improve their software.

The team performance was assessed using orthogonal performance tests that served to comprehensively evaluate the glycopeptide identification accuracy (specificity) and glycoproteome coverage (sensitivity), two key performance characteristics in glycoproteomics (Fig. 1d). Six (N1–N6) and five (O1–O5) performance tests were carefully designed to assess the relative performance for N- and O-glycopeptide data analysis across teams (Table 2 and Supplementary Tables 5–15). First, the ability to detect the synthetic N-glycopeptide in the datasets was assessed (N1). Further, the glycan compositions (N2, O1) and source glycoproteins (N3, O2) of the reported glycopeptides were compared to the established serum glycome and against known serum glycoproteins4,23,39,40,41,42. To validate the use of the literature to score teams, we performed manual site-specific glycoprofiling of four serum glycoproteins—α-1-antitrypsin (A1AT), ceruloplasmin (CP), haptoglobin (HP) and immunoglobulin G1 (IgG1)—and showed an excellent agreement (R2 = 0.85–0.99) with relevant literature on healthy human serum43,44,45,46 (Extended Data Fig. 4). The glycoproteome coverage, on the other hand, was simply the reported nonredundant glycopeptides (N4, O3). Finally, the ability to identify glycopeptides commonly reported by most teams (‘consensus glycopeptides’) (N5, O4) and glycopeptides free of NeuGc and multi-Fuc features (N6, O5) was also scored. We ensured that NeuGc and multi-Fuc glycopeptides, unexpected glycofeatures in human serum23,47,48,49, were indeed absent or rarely detected in Files A and B (discussed below) allowing these to be deemed putative false positives for the purpose of scoring teams (Extended Data Fig. 5).

Table 2 Overview of the performance tests applied to establish the relative team performance for glycopeptide data analysis
Full size table

The performance tests were used to score and rank teams (Fig. 1e and Supplementary Table 16). The developer and user groups were not compared because they received different study instructions. The team scoring was validated using an independent glycoprotein-centric site-specific profiling test (Supplementary Table 17). Finally, performance data from both team-wide and search engine-centric approaches revealed performance-associated search variables and led to improved glycoproteomics search strategies (Supplementary Tables 18 and 19).

Overview of the reported glycopeptides

The following analyses were carried out using data reported from File B processed by all teams. The total N-glycoPSMs (49–2,122) and source N-glycoproteins (9–168) reported by the 22 teams varied dramatically (Fig. 2a and Supplementary Table 3). In line with the literature on human serum N-glycosylation23,24, the reported N-glycopeptides carried mainly complex-type N-glycans (92.6%, average across teams). Relatively few oligomannosidic (6.4%) and truncated (herein defined as Hex<4HexNAc<3Fuc<2 or biosynthetically unusual N-glycans) (1.0%) N-glycopeptides were reported. The applied N-glycan search space spanned an equally wide range (23–381 compositions) comprising mostly complex-type N-glycans (89.1%) and the less heterogeneous oligomannosidic (5.9%) and truncated (5.0%) N-glycans. No associations were found between the size of the N-glycan search space and reported N-glycoPSM counts (Pearson R2 = 0.115). Unexpected glycan compositions including NeuGc and multi-Fuc-containing complex-type N-glycans, which are negligible features of human serum glycoproteins23,47,48,49, were not only included in the glycan search space (up to 26.5% and 28.9%, respectively), but also reported (up to 20.6% and 5.0%) by some teams. The absence of NeuGc and the rarity of multi-Fuc glycopeptides in the shared data was supported by a lack of diagnostic fragment ions for NeuGc (m/z 290/308), scarcity of antenna Fuc ions (m/z 512/803) and the frequent mis-annotation of MS/MS spectra claimed to correspond to NeuGc and multi-Fuc glycopeptides (Extended Data Fig. 5). While only infrequently detected, multi-Fuc glycopeptides were, however, evidently present in our data as supported by manual spectral annotation (Extended Data Fig. 5d).

Fig. 2: Glycopeptides reported across teams.
figure 2

a, Reported N-glycoPSMs (bars), unique source N-glycoproteins (dots) and the N-glycan search space applied (mirror bars) by each team. See key for N-glycan classification. b, Proportion of N-glycopeptides, source N-glycoproteins and N-glycan compositions commonly reported by teams. c, Reported O-glycoPSMs, unique source O-glycoproteins and the O-glycan search space applied by each team. Teams 8 and 9 did not perform O-glycopeptide analysis. See key for O-glycan classification. Multi-feature N- and O-glycans fitting into several of these classes were for this purpose classified in a prioritized order of multi-Fuc–NeuGc–NeuAc; see Supplementary Tables 2 and 3 for data. d, Proportion of O-glycopeptides, source O-glycoproteins and O-glycan compositions commonly reported by teams. The high-confidence ‘consensus’ N- and O-glycopeptides have been made publicly available (GlyConnect Reference ID 2943).

Full size image

Collectively, 2,556 unique N-glycopeptides (defined herein as unique peptide sequences and glycan compositions), covering 320 different source N-glycoproteins and 424 different N-glycan compositions, were reported across teams (Fig. 2b and Supplementary Tables 6, 7 and 9). Of these, only 43 N-glycopeptides (1.7%), 26 source N-glycoproteins (8.1%) and 28 N-glycan compositions (6.6%) were commonly reported by at least 75% of teams (see Extended Data Fig. 6a for an example of congruent spectral annotation across teams). Most glycopeptides, however, were commonly reported by only a few teams, probably due to frequent mis-annotation of the spectral data (Extended Data Fig. 6b).

Notably fewer, but equally discrepant, O-glycopeptides (5–578 O-glycoPSMs) were reported by participants (Fig. 2c and Extended Data Fig. 3i). As expected, most reported O-glycopeptides carried Hex1HexNAc1NeuAc1-240,50. The applied O-glycan search space also varied dramatically (3–223 glycan compositions). Similar to the N-glycopeptide analysis, no association was observed between the O-glycan search space and reported O-glycoPSM counts (Pearson R2 = 0.118). Instead, many other associations were identified (discussed below). While seven teams included NeuGc in the applied O-glycan search space (up to 9.0%), only four teams reported NeuGc O-glycopeptides (up to 7.3%). In addition, 12 teams included multi-fucosylated glycans in the O-glycan search space (up to 43.5%); 11 of those teams reported multi-fucosylated O-glycopeptides (average of 28.6%, up to 61.8%). Both NeuGc and multi-fucosylated O-glycans are negligible features of human serum O-glycoproteins as supported by the literature40,50 and our own analyses (above). The reported multi-fucosylated O-glycan compositions could, in principle, in some cases arise from multiple discrete O-glycans residing on the same peptide. As O-glycosylation sites were inconsistently and/or ambiguously reported by most teams (below) we were not able to assess this aspect further.

Collectively, 1,192 unique O-glycopeptides covering 231 different source O-glycoproteins and 288 different O-glycan compositions were identified, but surprisingly few O-glycopeptides were commonly reported across teams. Only three O-glycopeptides (0.3%), six source O-glycoproteins (2.6%) and seven O-glycan compositions (2.4%) were commonly reported by at least half the teams (Fig. 2c). Most O-glycopeptides were reported by a single or few teams.

Despite the discrepant reporting, high-confidence lists spanning 163 N- and 23 O-glycopeptides commonly reported by teams could be generated. Importantly, these consensus glycopeptides mapped to expected serum glycoproteins; for example, α-2-macroglobulin (UniProtKB, P01023) and haptoglobin (P00738) and carried expected serum N-glycans; for example, Hex5HexNAc4Fuc0-1NeuAc2 (GlyTouCan IDs, G09675DY/G22754FQ) and O-glycans; for example, Hex1HexNAc1NeuAc1-2 (G65285QO/G84906ML) that were biosynthetically related (Extended Data Fig. 7), devoid of NeuGc and poor in multi-Fuc, further supporting their correct identification. These high-confidence glycopeptides form an important reference to future studies of the human serum glycoproteome and have therefore been made publicly available (GlyConnect Reference ID 2943).

High-performance informatics solutions for N-glycoproteomics

The relative team performance for N-glycoproteomics was comprehensively assessed using six independent performance tests (N1–N6) (Table 2 and Supplementary Tables 5–10). Among these performance tests, N1 scored the ability to accurately identify a synthetic N-glycopeptide in the sample (Extended Data Fig. 8). Similar to the other performance tests, N1 was used to establish the relative team performance. Founded on a ‘ground truth’, the N1 data including the 12 manually annotated spectra all corresponding to the synthetic N-glycopeptide are particularly informative and may aid developers train algorithms and improve software to annotate N-glycopeptide spectral data better. The N1 data also supported observations made across the entire dataset (Extended Data Fig. 2) confirming that glycopeptides were preferentially identified in charge state 4+ using HCD- and EThcD-MS/MS even when high-quality MS/MS data from other charge states and fragmentation modes were available.

In line with the literature2,8, most teams employed HCD- and/or EThcD-MS/MS for glycopeptide identification. While these two fragmentation modes displayed similar performance in tests scoring the glycan composition (N2, O1) and glycoproteome coverage (N4, O3), higher scores were achieved for EThcD-based relative to HCD-based identifications in the source glycoprotein tests (N3, O2) (Extended Data Fig. 9). Importantly, accurate glycosylation site localization, not tested with this study (discussed below), is a recognized strength of EThcD-MS/MS data5,12.

The performance tests were used to score and rank developers and users (Fig. 3a). At a glance, the scorecard pointed to considerable team-to-team variations in the performance profiles suggesting that the applied software and search strategies exhibit markedly different strengths and weaknesses for N-glycoproteomics. As an example, IQ-GPA (team 1) and GlycoPAT (team 8) performed well (relative to other developers) in the N-glycan composition test (N2), while Protein Prospector (team 2) and Byonic (team 4) performed well in tests scoring the source N-glycoproteins (N3) and N-glycoproteome coverage (N4).

Fig. 3: Team scoring/ranking and identification of performance-associated variables.
figure 3

a, Heatmap representation of normalized scores (range 0–1) from the N-glycopeptide performance tests (N1–N6, Table 2). See Supplementary Tables 5–16 for performance data. #The top third performing teams (white font) were placed in a high-performance band. The team scoring was later validated (Extended Data Fig. 10). *Performance could not be determined. b, Many variables (search settings, search output) showed associations (negative or positive) with N-glycopeptide performance. See Table 1 for variables. See Supplementary Table 18 for statistics. See d for key to symbols. c, Scores from the O-glycopeptide performance tests (O1–O5). Teams 8 and 9 did not return O-glycopeptide data. d, Many associations between the search variables and O-glycopeptide performance were observed.

Full size image

Overall, Protein Prospector (team 2, overall score 0.682), Byonic (team 4, 0.676) and GlycoPAT (team 8, 0.647) were found to be high-performance software solutions for N-glycoproteomics. Notably, our scoring did not separate these three developers by any substantial margin, but their overall performance was slightly higher than that of IQ-GPA (team 1, 0.621) and glyXtoolMS (team 3, 0.580) and substantially higher than that of the other software (score range 0.296–0.366).

Supporting our scoring method, an independent assessment method based on the match between reported and actual site-specific N-glycoforms of four serum glycoproteins (A1AT, CP, HP, IgG1, thus founded on a ‘ground truth’) recapitulated the scoring profile across teams (R2 = 0.82) (Extended Data Fig. 10). Further supporting the top ranking of Byonic and Protein Prospector, the best performing user teams employed Byonic (teams 11, 13, 15, score range 0.687–0.777) and Protein Prospector (team 12, 0.709). Their overall performance scores were marginally higher than seven other Byonic users (teams 10, 16–18, 20–22, 0.593–0.679), but markedly higher than teams using SugarQb (team 14, 0.172) and Mascot (team 19, 0.413). Despite the similar overall performance among most user teams, not least the ten Byonic users, their performance profiles differed markedly across the six performance tests.

We then explored the scorecard for software-independent performance-associated variables including the search settings (SS1–SS13) and search output (SO1–9) using seven different statistical methods (Fig. 3b). Many statistically strong relationships were found revealing key performance-associated variables that either positively or negatively correlated with the glycopeptide identification efficiency. As an example, the use of decoy/contaminant databases (SS13) showed associations with performance in the synthetic N-glycopeptide test (N1) and high N-glycoproteome coverage (N4). Search strategies that allowed for a relatively high diversity and number of nonglycan variable peptide modifications (SS8, SS10) and few glycans per peptide (SS9) were also associated with high N-glycoproteome coverage (N4). As expected, allowing multiple missed peptide cleavages (SS7) and variable nonglycan modifications (SS10) in the search strategy correlated with higher glycopeptide FDRs as indicated by higher rates of NeuGc and multi-Fuc identifications (low N6 scores) (Supplementary Table 18).

The association analyses also identified many interesting relationships between the search output and performance (Fig. 3b). Intuitively, teams that reported many N-glycoPSMs (SO9) performed well in the synthetic N-glycopeptide test (N1), had a higher N-glycoproteome coverage (N4) and identified more consensus N-glycopeptides (N5). Further, teams that reported glycopeptides featuring a relatively high glycan mass (SO8) more often identified the correct glycan composition (N2), while teams that reported glycopeptides exhibiting relatively high molecular masses (SO5) more often identified the correct source N-glycoproteins (N3). Glycopeptides displaying relatively high molecular masses (large glycans and/or peptides) are less likely to be incorrectly identified due to fewer theoretical glycopeptide candidates (fewer potential false positives) in the higher mass range. In addition, early LC retention time (SO1), high charge (SO3), high glycopeptide mass (SO5), high actual mass error (low mass accuracy, SO6) and high glycan mass (SO8) were search output linked to high N-glycoproteome coverage (N4). Teams reporting N-glycopeptides with high molecular masses (SO5) more often identified consensus glycopeptides (N5). Finally, low actual mass error (SO6) was, as expected, associated with better identification accuracy.

High-performance informatics solutions for O-glycoproteomics

Protein Prospector (team 2) displayed the highest performance in tests scoring the source O-glycoproteins (O2) and consensus O-glycopeptides (O4) (Fig. 3c and Supplementary Table 16). Conversely, IQ-GPA (team 1) and glyXtoolMS (team 3) were the best performing software in tests scoring the O-glycan compositions (O1) and O-glycoproteome coverage (O3), respectively. Overall, Protein Prospector (team 2, overall score 0.613) and glyXtoolMS (team 3, 0.522) were found to be high-performance software for O-glycoproteomics. Among the users, four Byonic teams (teams 13, 15, 17, 18, overall score range 0.473–0.701) were ranked in the high-performance band.

Correlation analyses showed that accurate identification of the O-glycan compositions (O1) associated with approaches using a focused (narrow) O-glycan search space (SS2) and permitting only few missed peptide cleavages (SS7) (Fig. 3d and Supplementary Table 18). In addition, search strategies permitting incorrect precursor selection (SO4) were commonly used by teams scoring well in the O-glycan composition test (O1). Interestingly, employing a broad O-glycan search space (SS2) was associated with accurate identification of source O-glycoproteins (O2), high O-glycoproteome coverage (O3) and better identification of consensus O-glycopeptides (O4). Further, teams reporting identifications with low mass error (SO6) scored well in the O-glycan composition test (O1), but, notably, at the cost of lower O-glycoproteome coverage (O3) and fewer consensus O-glycopeptides (O4).

Search engine-centric analysis

We then explored the impact of different search strategies on the glycoproteomics data output for the popular Byonic search engine used by 11 teams. The Byonic teams employed highly diverse search strategies; except for the common use of decoy/contaminant databases (SS13) and monoisotopic correction (SS14), the search settings varied considerably across these teams (Fig. 4a). Undoubtedly, this search diversity and different output filtering methods used by the Byonic teams (e.g., Byonic score >100, PEP-2D <0.001, FDR <1%) contributed to the dramatic variation in reported glycopeptides (Fig. 4b and Supplementary Table 1). Unsurprisingly, therefore, the relative specificity (accuracy) and sensitivity (coverage) scores (established from N1–N6/O1–O5) showed different performance profiles of the Byonic teams particularly for the O-glycopeptide analysis (Fig. 4c and Supplementary Table 16). Teams achieving better than average sensitivity scores (e.g., teams 15 and 17), typically under-performed with respect to specificity. Other teams achieved higher than average specificity scores at the cost of sensitivity (e.g., teams 18 and 22), confirming the intuitive reciprocal relationship between these performance metrics.

Fig. 4: Search engine-centric (Byonic-focused) analysis of search strategies for high-performance glycoproteomics data analysis.
figure 4

a, Overview of the search settings employed by Byonic teams. Default: search strategy used by most teams (yellow). Custom: variations from the default search strategy (green). #Data for SS14, a setting not included in the team reports, were adopted from SO4 data. b, The glycoproteome coverage (unique glycopeptides, File B) varied among Byonic teams. c, Specificity (accuracy) and sensitivity (coverage) scores for (i) N-glycopeptides and (ii) O-glycopeptides for Byonic teams. d, Controlled (in-house) searches for N-glycopeptides using Byonic (File B). Individual search settings were systematically varied (iteration level 1) and output assessed for performance gains (specificity, sensitivity). Search settings showing performance gains (shaded circles/diamonds) without unacceptable costs in specificity (SS13) or search time (SS8/SS10, SS9; see examples in e) (gray stars) were collectively tested for synergistic performance gains (iteration levels 2 and 3, dark green). See e for shared symbol key. e, Byonic-centric O-glycopeptide searches. See d for details. f, Recommended Byonic-centric search strategies for ‘high accuracy’, ‘high coverage’ and ‘balanced’ (between accuracy and coverage) glycoproteomics data analysis. ^The recommended search strategies showed relative performance gains as determined using an independent glycoprotein-centric score (Supplementary Table 19b). Search time and glycoproteome coverage (unique glycopeptides) are also indicated.

Full size image

The individual search variables were then investigated through a series of controlled (in-house) searches using Byonic. For this purpose, the search settings were systematically varied from the ‘default’ search strategy used by most teams while keeping other parameters constant. Several search settings showed performance gains in terms of improved specificity (e.g., literature-guided narrow glycan search space, SS1–SS2) or sensitivity (e.g., decoy database disabled, SS13) but often at the expense of other performance characteristics (Fig. 4d,e and Supplementary Table 19a). While reduced sensitivity (glycoproteome coverage) may be an acceptable compromise for higher specificity (identification accuracy), the opposite arguably does not hold true. Thus, the considerable sensitivity gains and concomitant loss in specificity achieved by disabling the decoy database (SS13) did not benefit the data analysis. Instead, increasing the permitted glycans per peptide (SS9), tightening the allowed mass error (SS11–SS12) and relaxing the protease specificity benefitted both specificity and sensitivity. Search settings that showed cost-less performance gains were combined for subsequent rounds of iterative searches. Importantly, these efforts led to improved ‘high accuracy’, ‘high coverage’ and ‘balanced’ (accuracy >< coverage) search strategies for N- and O-glycoproteomics (Fig. 4f). None of the Byonic teams had utilized these combinations of search settings. When assessed using the independent glycoprotein-centric scoring method, these three recommended search strategies showed improved performance (specificity, sensitivity) relatively to the default strategy and strategies used by Byonic teams (Supplementary Table 19b). Notably, the high-coverage searches dramatically expanded the search time, a metric here not considered beyond logistic constraints.

Finally, we explored the performance of different fragmentation methods by systematically varying the spectral input (HCD/EThcD/CID) in Byonic while keeping search settings and output filtering constant. The highest performance was achieved when HCD and EThcD were jointly searched (Supplementary Table 19c). Our analysis also suggested that low-resolution CID data do not benefit the Byonic search performance when used alone or with HCD/EThcD data, an observation supported by the Byonic team comparison (Supplementary Table 19d).

Discussion

This community study has objectively discerned the performance of current informatics solutions for glycoproteomics data analysis. Excitingly, several high-performance glycoproteomics software and search strategies were identified. Among the nine developer teams, Protein Prospector (team 2) was identified as the top performing software for both N- and O-glycoproteomics. Byonic (team 4) also displayed high performance for N-glycopeptide data analysis, and while this developer only demonstrated moderate performance for O-glycoproteomics, four Byonic user teams (teams 13, 15, 17 and 18) displayed the highest performance for O-glycopeptide data analysis. Protein Prospector51 and Byonic33, developed 10–20 years ago, have pioneered the glycopeptide informatics field and are search engines already commonly used in glycoproteomics8,31,33.

Protein Prospector is an academic (free) tool recognized for its ability to identify modified peptides and modification site(s) from LC-MS/MS data using a probability–difference-based scoring system31. Protein Prospector is often a preferred search engine in studies addressing the challenging site annotation of O-glycopeptides, in particular when ET(hc)D-MS/MS data are available5. However, Protein Prospector does not estimate the FDR of the glycan components of glycopeptides, which it regards as nondescript post-translation modifications with an exact mass, and the software may appear less user-friendly than competing tools.

Facilitated by a user-friendly interface, precise spectral annotation and useful output reports of identified glycopeptides/proteins, the commercial Byonic search engine has gained considerable popularity (as illustrated herein) as it enables relatively straightforward identification of peptides with known and unknown modifications including glycosylation from different MS/MS data. Byonic features useful fine control options that enable tailored glycopeptide searches and postsearch filtering of output based on prior knowledge. Byonic scores and annotates multiple types of glycopeptide fragments to deduce the peptide carrier, glycan and modification site, but the FDRs, calculated identically for nonglycosylated peptides and glycopeptides, primarily address the correctness of the peptide rather than the glycan and the site localization33.

Notably, GlycoPAT37 (team 8) and glyXtoolMS32 (team 3) were in our study also identified as high-performance N- and O-glycoproteomics software, respectively. Furthermore, IQ-GPA30 (team 1) demonstrated merit for both N- and O-glycopeptide data analysis. While all three software packages handle high-resolution HCD-, ETciD- and EThcD-MS/MS data, IQ-GPA and GlycoPAT also identify glycopeptides based on high-resolution CID-MS/MS data and apply postsearch filtering based on advanced peptide and glycan decoy methods to estimate both peptide and glycan FDRs of glycopeptide candidates. The software glyXtoolMS instead uses oxonium ions, Y1-ions (peptide-HexNAc), other glycopeptide-specific fragments and peptide-specific b-/y-ions to control FDR. These three academic tools were recently developed (<5 years ago) and, thus, hold a considerable potential in the field.

We used both team-wide and search engine-centric approaches to uncover performance-associated search variables for glycoproteomics data analysis. The team-wide correlation analyses revealed many search settings and search output linked to performance. Backed by robust statistics, these ‘universal’ relationships existing across search engines will widely benefit glycoproteomics software developers and users aiming to improve N- and O-glycopeptide data analysis (Table 3). This knowledge may aid tackling existing challenges in glycoproteomics, among the most critical, reducing the FDR of glycopeptide candidates carrying glycans with similar (e.g., NeuAc-R versus Fuc2-R, Δm = 1.0204 Da) or identical (e.g., NeuAc1Hex1-R versus NeuGc1Fuc1-R, Δm = 0 Da) masses. Our study indeed confirms that glycopeptides displaying such ‘difficult-to-identify’ features (NeuAc, NeuGc, multi-Fuc, Met oxidation, Cys carbamidomethylation) are frequently mis-annotated with current search engines (Extended Data Fig. 5). We therefore recommend that efforts should be invested in improving tools to allow for accurate identification of such challenging glycopeptides.

Table 3 Overview of software-independent search variables important for high-performance glycoproteomics data analysis (see Fig. 3b,d, Tables 1 and 2 and Supplementary Table 18 for study variables and associations)
Full size table

Meanwhile, our search engine-centric approach involving systematic Byonic searches revealed search settings impacting the performance of this widely used search engine and highlighted that specificity (accuracy) and sensitivity (coverage) are competing performance characteristics challenging to achieve in a single search. This suggests that glycoproteomics data may benefit from being interrogated using multiple orthogonal search strategies that are subsequently combined or by approaches that strike a balance between accuracy and coverage. To this end, we here recommend a set of improved ‘high accuracy’, ‘high coverage’ and ‘balanced’ search strategies that should be selected (and further tailored/optimized) according to the sample and research question being investigated (Fig. 4f).

Although not a focus here, our study also showed that the search strategy dramatically impacts the search time. While the spectral input type and data output filtering represent other critical variables that also need further exploration, our study indicates that HCD and EThcD are currently the most informative spectral types in glycoproteomics, and that knowledge-guided filtering and curation of data output is critically required to lower FDRs.

The study also highlighted informatics challenges still associated with large-scale glycopeptide data analysis, as illustrated by the discrepant reporting of glycopeptides across teams. Notably, high discordance of reported glycopeptides was even found between participants using the same software, confirming that search variables other than the search engine also substantially impact the glycopeptide data analysis. While the ten Byonic user teams reported a marginally higher rate of consensus glycopeptides, their spread in terms of search output data, reported glycoPSMs and overall performance scores was of similar magnitude as the variance observed among other user teams. While the (self-reported) team experience in glycoproteomics was not found to be an accurate predictor of team scoring and ranking, the variability in the spectral data input, search settings and, importantly, at the postsearch filtering stage were identified as key factors contributing to the discrepant reports. Concertedly, these observations point to the importance of using both the most informative spectral data, powerful search engines, tailored search settings and knowledge-driven postsearch filtering to achieve high-performance glycoproteomics data analysis.

Despite the considerable team-to-team variation, this study produced consensus lists of 163 N-glycopeptides and 23 O-glycopeptides from serum glycoproteins commonly reported by teams. Importantly, these high-confidence glycopeptides carried biosynthetically related glycans that were devoid of NeuGc and poor in multi-Fuc features in line with the literature23,40 and mapped to known high-abundance serum proteins4,23,39,41,42. The consensus lists have been made publicly available (GlyConnect ID 2943) as they form an important reference for future studies of the human serum glycoproteome.

The study design including the sample type/preparation and data collection method was chosen to mimic conditions typically encountered in glycoproteomics while also aiming to accommodate most informatic solutions and appeal to users in the field. Multiple orthogonal performance tests and separate validation were applied to ensure a fair and holistic scoring of search engines and teams. Despite these efforts, it cannot be ruled out that some software or users may have been unintentionally disadvantaged and/or excluded by the chosen experimental design and scoring system. The team scoring and ranking should be viewed in light of these constraints and limitations common to most community-based comparison studies founded on communal data.

In addition to reporting on the peptide and glycan components of identified glycopeptides, teams were requested to report on site(s) of modification where possible. As most tryptic N-glycopeptides only comprise a single sequon, site localization is primarily a challenge related to O-glycoproteomics5,16. Most teams indeed returned data of the O-glycosylation site(s), but due to highly discrepant and often inconclusive reporting of sites and a paucity of literature on serum O-glycosylation sites, we were unable to score glycosylation site localization.

Most software currently available for glycoproteomics data analysis participated in this study. However, several glycopeptide search engines; for example, pGlyco52, MSFragger-Glyco53, O-Pair Search54 and StrucGP55, were unfortunately not represented due to LC-MS/MS data incompatibility or due to their development after the study period. Thus, this study is essentially a snapshot of the performance of software available at the time the data analysis was performed. Highlighting the rapid progress in glycoproteome informatics, most of the software solutions participating in this study have been improved and new versions released after the evaluation period. For example, GPQuest v2.0, GlycoPAT v1.0 and Protein Prospector v5.20.23 tested herein have been superseded by more recent versions: namely, GPQuest v2.1, GlycoPAT v2.0 and Protein Prospector v.6.2.2. Thus, a limitation of this study is that newer tools are available at the time of publication that were not compared in our analysis. Follow-up studies comparing the performance of these latest glycoproteomics software upgrades and informatics solutions not included in this study are therefore warranted. Beyond testing the ability of participants to identify the peptide and glycan components of glycopeptides from glycoproteomics data, such future comparative studies should ideally also test the ability to accurately quantify (relative, absolute) and report on modification sites of identified glycopeptides and could explore other relevant parameters not addressed herein including the use of alternative proteases, tandem mass tag-labeling and stepped-HCD-MS/MS data among other experimental conditions gaining popularity in glycoproteomics.

In summary, this community study has documented that the field has several high-performance informatics solutions available for glycoproteomics data analysis and has elucidated key performance-associated search strategies that will serve to guide developers and users of glycoproteomics software.

Methods

Study design and participants

Calls to join this study as a developer (academic/commercial) or user of glycoproteomics software were made widely across the proteomics and glycomics community. In total, 25 teams signed up for the study, out of which 22 teams comprising nine developer and 13 user teams completed the study. All teams identified N- and O-glycopeptides from two communal glycoproteomics LC-MS/MS data files (Files A and B), and reported their findings using a common reporting template (PXD024101, PRIDE repository). While the user teams were guaranteed anonymity, the developers were informed that their software (hence, potentially their identity) would be disclosed on publication. The user teams were free to use any search engine(s) at their disposal including manual annotation/filtering of search output. Developers returned the identified glycopeptides directly from their own software without manual postsearch filtering. The developers employed the following search engines: team 1: IQ-GPA v2.530; team 2: Protein Prospector v5.20.2331; team 3: glyXtoolMS v0.1.432; team 4: Byonic v2.16.1633; team 5: Sugar Qb34; team 6: Glycopeptide Search v2.0alpha35; team 7: GlycopeptideGraphMS v1.0/Byonic36; team 8: GlycoPAT v2.037 and team 9: GPQuest v2.038 (see Supplementary Table 1 and below for overview of software and pre- and postprocessing tools used by all participants). The relative team performance was compared within (not between) the developer and user groups as these two groups were given slightly different instructions (above).

Synthetic N-glycopeptide

An Asn building block carrying a disialylated, biantennary N-glycan (Hex5HexNAc4NeuAc2) was purified from chicken egg yolk powder. Previous studies have confirmed that a disialylated, biantennary N-glycan carrying only α2,6-linked NeuAc residues is the major component of the chicken egg yolk hexapeptide56,57. In short, this glycosylated hexapeptide was subjected to extensive proteolysis to generate a glycosylated Asn, which was then converted into a fluorenylmethoxycarbonyl (Fmoc) protected building block as described earlier57,58. Using this glycosylated Asn building block, a synthetic glycopeptide carrying a homogenous N-glycan (Hex5HexNAc4NeuAc2) was generated using an established method for solid phase peptide synthesis58,59,60. The synthetic peptide sequence mimicked a tryptic N-glycopeptide from human vitamin-K-dependent protein C present in human serum (UniProtKB, P04070, 284EVFVHPNYSK293). The structure, purity and integrity after deprotection and purification were confirmed using reversed-phase LC-MS/MS as described earlier59.

Study sample

Human serum from a commercial source was used for this study (product no. 31876, Thermo Fisher Scientific). As a positive control, 52 fmol of the synthetic N-glycopeptide from human vitamin-K-dependent protein C (see details above) was spiked into 5 µg human serum before digestion. Proteins were cysteine reduced and alkylated before protein digestion using 1:100 (w/w, enzyme:protein substrate) sequence-grade trypsin for 16 h, 37°C in 20 mM aqueous ammonium bicarbonate, pH 8.0. Undigested protein material and large peptides were removed by filtration using a 30 kDa molecular weight cut-off membrane (product no. 88502, Thermo Fisher Scientific). The membrane was washed using 30% (v/v) methanol in 0.1% (v/v) aqueous trifluoroacetic acid (TFA). The flow-through fraction was collected, evaporated using a SpeedVac, and then resuspended in 200 μL 50% (v/v) acetonitrile (ACN) in 0.1% (v/v) aqueous TFA. Glycopeptide enrichment was performed using Hypersep Retain AX columns (product no. 60107-403, Thermo Fisher Scientific). The columns were prepared according to the manufacturer’s instructions and were additionally washed with 100 mM aqueous triethylammonium acetate before equilibration with 95% (v/v) ACN in 1% (v/v) aqueous TFA. The sample was diluted in 3 mL 95% (v/v) ACN in 1% (v/v) aqueous TFA, applied to the columns, and then washed with an additional 3 mL 95% (v/v) ACN in 1% (v/v) aqueous TFA before the glycopeptides were eluted with 1 mL 50% (v/v) ACN in 0.5% (v/v) aqueous TFA. The enriched glycopeptide mixtures were dried using a SpeedVac and resuspended in 0.1% (v/v) aqueous TFA for LC-MS/MS analysis.

Mass spectrometry

The glycopeptides were separated by reversed-phase nanoLC using a Thermo Scientific EASY-nLC 1200 UPLC system connected to a C18 LC column (50 cm length × 75 µm inner diameter, Thermo Scientific EASY-Spray). Separation was achieved using a 75 min 6–45% (v/v) and 3 min 45–95% (v/v) gradient of solvent B consisting of 80% (v/v) ACN in 0.1% (v/v) aqueous formic acid in solvent A consisting of 0.1% (v/v) aqueous formic acid at a 300 nL/min flow rate. The separated glycopeptides were detected using a Thermo Scientific Orbitrap Fusion Lumos Tribrid mass spectrometer connected directly to the LC. Approximately 1 μg peptide material was injected on the LC column per run. The same glycopeptide sample was analyzed twice using two slightly different acquisition methods producing two related data files (Files A and B).

For both methods, MS1 scans were acquired from m/z 350–1,800 in the Orbitrap at a resolution of 120,000 and with an automatic gain control (AGC) of 4 × 105 and an injection time of 50 ms. Data-dependent HCD-MS/MS was performed for the ten most intense precursor ions selecting the highest charge state and the lowest m/z in each MS1 full scan. The HCD-MS/MS fragment ions were recorded in the Orbitrap at a resolution of 30,000 and with an AGC of 5 × 104, injection time of 60 ms, normalized collision energy (NCE) of 28% and a quadrupole isolation width of 2 Th. Already selected precursors were dynamically excluded for 45 s. Product-dependent ion triggered re-isolation and fragmentation of precursor ions were enabled on detection of at least one of three selected glycan oxonium ions (m/z 138.0545, 204.0867 and 366.1396) if the diagnostic ion(s) was among the top 20 fragment ions within each HCD-MS/MS spectrum. For File A, product-dependent-triggered ETciD- and CID-MS/MS events were scheduled. The ETciD-MS/MS fragments were detected in the Orbitrap at a resolution of 60,000 with an AGC of 4 × 105, injection time of 250 ms, CID NCE of 15% and a quadrupole isolation width of 1.6 Th. Charge-dependent ETD calibration was enabled. The CID-MS/MS fragments were detected in the Orbitrap at a resolution of 30,000 with an AGC of 5 × 104, NCE of 30%, injection time of 54 ms and a quadrupole isolation width of 1.6 Th. For File B, product-dependent-triggered EThcD- and CID-MS/MS events were scheduled. The EThcD-MS/MS fragments were detected in the Orbitrap at a resolution of 60,000 with an AGC of 4 × 105, injection time of 250 ms, HCD NCE of 15% and a quadrupole isolation width of 1.6 Th. Charge-dependent ETD calibration was enabled. The CID-MS/MS fragments were detected in the ion trap at unit resolution using a rapid scan method with an AGC of 1 × 104, injection time of 70 ms, NCE of 30% and a quadrupole isolation width of 1.6 Th. Files A and B were provided to all participants as .raw data files (File A: 684 MB, File B: 811 MB) or as three separate .mgf files containing peak lists of the fragment spectra from the three different fragmentation modes used for Files A and B (23.9 MB–65.6 MB). Conversion to .mgf was performed using ProteoWizard61.

Search instructions and reporting template

The participants were requested to use a protein search space provided by the study organizers comprising the entire human proteome (20,201 UniProtKB reviewed sequences, downloaded July 2017) for their search. In contrast to the fixed protein search space, the participants were free to choose the N- and O-glycan search space. To limit the number of study variables, participants were asked not to include xylose and any glycan substitutions (e.g., phosphate, sulfate and acetylation) in the glycan search space. The participants were requested to report their team details, identification strategy and the identified glycopeptides in a common reporting template organized as five separate sheets in an Excel file comprising the following categories of information: (1) Team and contact details; (2) Identification strategy and other study information; (3) N- and O-glycan search space; (4) List of identified N- and O-glycopeptides and (5) Summary of identified glycopeptides and glycoproteins. The returned reports were carefully checked for compliance with the study guideline. See PXD024101 via the PRIDE repository62 for the common reporting template and the deidentified reports from all participants forming the foundation of this study.

Search engines and pre- and postprocessing tools used for the glycopeptide identification

A total of 13 search engines was used for glycopeptide identification: IQ-GPA v2.530, Protein Prospector v5.20.2331, glyXtoolMS v0.1.432, Byonic v2.16.1633, Sugar Qb34, Glycopeptide Search v2.0alpha35, GlycopeptideGraphMS v1.0/Byonic36, GlycoPAT v2.037, GPQuest v2.038, Mascot v2.5.163 or v2.2.07, MS Amanda v1.4.14.824364 and Sequest-HT (in Proteome Discoverer v2.2) (Extended Data Fig. 1h). These tools were used as stand-alone tools or in combinations. Some of the search engines were applied with pre- or postprocessing tools, including OMSSA v2.1.8, Preview v2.13.2, Protein Prospector MS-filter, MS-GF + /PepArML and pParse v.2.0 (Extended Data Fig. 1i).

Compilation and comparison of participant reports

Information of the participating teams was compiled from the returned reports (Supplementary Table 1–2). The lists of intact N- and O-glycopeptides reported by the 22 teams were compiled into a single table with a unique header (Supplementary Table 3). Additional columns were manually added to the compiled table with the purpose of standardizing some of the reported text variables and generating unique identifiers (IDs) for the reported glycopeptides and their glycan compositions and source glycoproteins. The glycan composition ID was written as the generic monosaccharide composition as Hex*HexNAc*Fuc*NeuAc*, where * represents the number of the individual monosaccharide residues. Glycopeptides adducted with Na+ and K+ were considered and reported by some teams. The adducted glycopeptides were combined with the corresponding nonadducted monosaccharide compositions. UniProtKB IDs were used as the source protein IDs. The glycopeptide IDs were written as the peptide sequence followed by the generic glycan composition.

The comparisons between the generic glycan compositions, source proteins and glycopeptide IDs reported by the 22 teams were performed using the pivot table tool available in Excel, where the ID type was placed in ‘rows’, and the team ID in ‘columns’. The variables from each ID type were compared as summed counts across the 22 teams.

Performance testing of teams and software

The relative team and software performance for glycopeptide data analysis was in this study determined via three different methods as detailed below. In short, all teams were first scored and ranked based on a comprehensive assessment method involving multiple complementary performance tests (1). Subsequently, the scoring of teams was validated using an independent glycoprotein-based assessment score (2). Finally, for the search engine-centric analysis and optimization of the search strategies for Byonic, the relative performance was evaluated based on a scoring method that produced relative specificity and sensitive scores (3).

Scoring and ranking of teams via multiple performance tests (N1–N6 and O1–O5)

The relative team performance was assessed using a scoring system composed of multiple independent tests designed to score the accuracy (specificity) and coverage (sensitivity) of the reported N- and O-glycopeptides in orthogonal ways. The raw scores from the individual tests (N1–N6 and O1–O5, described below) were normalized within the range 0–1. These normalized scores were used to establish an overall performance score (range 0–1), measuring the ability to perform accurate and comprehensive N- and O-glycopeptide analysis. The overall performance score was utilized to separately rank the developer and user teams.

  1. a.

    The synthetic N-glycopeptide test (N1): All MS/MS spectra corresponding to the synthetic N-glycopeptide from human vitamin K-dependent protein C (peptide sequence: EVFVHPNYSK, glycan composition: HexNAc4Hex5NeuAc2) were manually retrieved and annotated from Files A- and B. In total, nine MS/MS spectra corresponded to the nonadducted synthetic N-glycopeptide in charge state 3+ and 4+ spanning the four applied fragmentation modes (HCD-, ETciD-, EThcD- and CID-MS/MS) (Extended Data Fig. 8a,b). A further three MS/MS spectra (HCD-, EThcD- and CID-MS/MS) corresponded to the K+-adducted synthetic N-glycopeptide in charge state 5+. The sensitivity of the test was determined as the proportion of the 12 MS/MS spectra mapping to the synthetic N-glycopeptide that was reported by each team adjusting for the type of fragmentation mode(s) included in their respective search strategies. The specificity was calculated by the proportion of correctly reported glycoPSMs corresponding to the synthetic glycopeptide that matched the 12 annotated MS/MS spectra, again adjusting for the type of fragmentation mode(s) included in the applied search strategies. The test score was calculated by multiplying the sensitivity and specificity (Extended Data Fig. 8c,d and Supplementary Table 5).

  2. b.

    The glycan composition test (N2 and O1). The N-glycan composition score was calculated based on the Pearson correlation (R2) between the expected distribution of N-glycans carried by human serum glycoproteins as reported by Clerc et al23 and the observed N-glycan distribution reported by each team. The O-glycan composition score was calculated based on the Pearson correlation (R2) between the expected distribution of O-glycans carried by human serum glycoproteins as reported by Yabu et al40 and the observed O-glycan distribution reported by each team. The distribution of the N- and O-glycan compositions was calculated based on the glycoPSM count of each unique glycan ID relative to the total glycoPSM count reported by each team.

  3. c.

    The source glycoprotein test (N3 and O2). The source glycoprotein score was determined from the accuracy (specificity) and coverage (sensitivity) of the reported source glycoproteins relative to the glycoproteins expected in human serum. Reported N-glycoproteins previously identified in human serum by both Clerc et al23 and Sun et al39 received a score of 2, whereas N-glycoproteins only identified by Sun et al received a score of 1. Source glycoproteins not identified by any of the two studies received no score. Further, reported O-glycoproteins previously identified in human serum by Darula et al41, Yang et al42 and Ye et al4 received a score of 3, 2 or 1 according to the number of papers identifying the specific O-glycoprotein. The source glycoproteins not reported by any of these three studies received no score. For both the serum N- and O-glycoproteins, the number of glycoPSMs reported by each team was multiplied by the respective source glycoprotein score for each unique glycoprotein ID. The specificity of the test was calculated based on the summed glycoprotein score divided by the highest possible total score (number of unique glycoproteins reported by each team multiplied by the highest theoretical glycoprotein score). The sensitivity of the test was calculated based on the summed number of glycoproteins with score >0 divided by the number of unique source glycoproteins reported in the selected literature.

  4. d.

    The glycoproteome coverage test (N4 and O3): The N- and O-glycoproteome coverage was calculated based on the number of unique glycopeptides (unique peptide sequence and glycan composition) reported by each team.

  5. e.

    The commonly reported (‘consensus’) glycopeptide test (N5 and O4): The consensus N-glycopeptide score was calculated based on the proportion of glycopeptides commonly reported by at least 50% of the 22 teams returning N-glycopeptide data. The consensus O-glycopeptide score was calculated based on the number of glycopeptides commonly reported by at least 30% of the 20 teams returning O-glycopeptide data.

  6. f.

    The NeuGc and multi-Fuc glycopeptide test (N6 and O5). The number of reported N- and O-glycoPSMs corresponding to NeuGc and multi-Fuc (Fuc ≥2) containing glycopeptides was normalized to the total glycoPSMs reported by each team. Separate N- and O-glycopeptide scores were then calculated based on the average of non-NeuGc and non-Fuc ≥2 containing glycoPSMs for teams that included NeuGc and multi-Fuc containing glycan compositions in their glycan search space.

The overall performance scores for N- and O-glycopeptide analysis were established separately by averaging the scores of the individual performance tests (N1–N6 and O1–O5, respectively).

Orthogonal glycoprotein-based scoring to validate the team scoring and ranking

To validate the scoring and ranking of teams based on the multiple performance tests described above (“Scoring and ranking of teams via multiple performance tests (N1–N6 and O1–O5)”), an orthogonal glycoprotein-centric scoring method was devised. The method, founded on a ‘ground truth’ as opposed to inference from the literature, evaluated the quantitative match of the glycoPSMs reported by the teams to the actual site-specific N-glycosylation of selected high-abundance glycoproteins including A1AT, CP, HP and IgG1. For this purpose, two metrics were developed (specificity and sensitivity) to score the match to the actual site-specific glycoform distribution. First, the site-specific distribution of N-glycans covering Asn70, Asn107 and Asn271 from A1T1, Asn138, Asn358 and Asn762 from CP, Asn184 and Asn241 from HP and Asn180 from IgG1 was manually determined using area-under-the-curve (AUC)-based glycopeptide quantitation (see below for details) and also determined for each team based on spectral counting of reported glycoPSMs. The specificity score was then calculated by multiplying the site-specific glycoform distributions reported by teams by the relative abundance of the actual site-specific glycoforms. The site-glycoform specificity scores were summed within each protein and normalized across the teams (best coverage set to 1). The ‘overall specificity score’ was calculated by averaging the normalized scores from A1AT, CP, HP and IgG1. The sensitivity score was calculated by the proportion of reported nonredundant (unique) glycoforms covering the expected site-specific glycoforms of the four glycoproteins based on robust literature23,43,44,45,46. The site-glycoform sensitivity scores were summed within each protein and normalized across the teams (best coverage set to 1). The ‘overall sensitivity score’ was determined by averaging the normalized scores from A1AT, CP, HP and IgG1. Combined scores (‘glycoprotein-centric score’) were established by averaging the overall specificity scores and the overall sensitivity scores. The combined scores were then compared to the overall N-glycopeptide scores generated from the performance tests N1–N6 (see “Scoring and ranking of teams via multiple performance tests (N1–N6 and O1–O5)” above) using Pearson correlation (R2). The data underpinning this scoring method can be found in Supplementary Table 17.

Search engine-centric scoring of the sensitivity and specificity of Byonic search strategies

To establish the performance of various Byonic search strategies, a scoring method that assessed the relative sensitivity (coverage) and specificity (accuracy) of the search engine was devised. For this purpose, the multiple performance tests already established for the scoring and ranking of teams (N1–N6 and O1–O5; see “Scoring and ranking of teams via multiple performance tests (N1–N6 and O1–O5)”) were used, but in a slightly different manner. The individual sensitivity and specificity scores from the synthetic N-glycopeptide test (N1) and source glycoprotein test (N3 and O2) were namely separately considered and grouped with several sensitivity and specificity-centric performance tests to establish ‘global sensitivity scores’ and ‘global specificity scores’ that could be compared between searches. The global sensitivity score for N-glycopeptides was determined by averaging the normalized sensitivity scores from the synthetic glycopeptide (N1), source N-glycoprotein (N3), N-glycoproteome coverage (N4) and commonly reported (‘consensus’) N-glycopeptide (N5) tests. The global specificity for N-glycopeptides was determined by averaging the normalized specificity score from the synthetic N-glycopeptide (N1), glycan composition (N2), source N-glycoprotein (N3) and non-NeuGc/multi-Fuc glycopeptide (N6) tests. The global sensitivity score for O-glycopeptides was determined by averaging the normalized sensitivity score from source O-glycoprotein (O2), O-glycoproteome coverage (O3) and commonly reported (‘consensus’) O-glycopeptide (O4) tests. The global specificity for O-glycopeptides was determined by averaging the normalized specificity score from the O-glycan composition (O1), source O-glycoprotein (O2) and non-NeuGc/multi-Fuc glycopeptide (O5) tests.

Manual quantitative glycoprofiling of select serum N-glycoproteins

A comprehensive quantitative site-specific analysis of the N-glycosylation of four high-abundance serum N-glycoproteins including α-1-antitrypsin (A1AT, UniProtKB, P01009, three N-glycosylation sites: Asn70, Asn107 and Asn271), ceruloplasmin (CP, P00450, Asn138, Asn358 and Asn762), haptoglobin (HP, P00738, Asn184 and Asn241) and immunoglobulin G1 (IgG1, P01857, Asn180) was manually performed to allow for a quantitative comparison of the studied glycoprotein sample to glycoprofiling data in the literature43,44,45,46, and thus validate the literature-based performance tests (N2–N3 and O1–O2) used to score teams (above). The site-specific N-glycoprofiling data were also used as a ‘ground truth’ to validate the scoring and ranking of teams in an orthogonal manner (see above for details).

For the quantitative site-specific glycoprofiling, the HCD- and EThcD-MS/MS data from File B were first searched using Byos v3.9–7 (Protein Metrics Inc.)33,65. The ‘default’ search strategy for N-glycopeptides commonly used by teams in this study was employed for the Byos search (see details below). The Byos-identified N-glycopeptides (PEP-2D <0.001 was used as a general confidence threshold) were manually confirmed, and the Byos output and the LC-MS/MS raw data were carefully inspected for any additional N-glycoforms expected based on the literature of the selected glycoproteins43,44,45,46, or based on known biosynthetic rules using Xcalibur v3.0.63 (Thermo Fisher Scientific) and with support from protein sequence handling software GPMAW v9.51 (Lighthouse)66. This comprehensive approach ensured that all relevant N-glycopeptides belonging to these four source glycoproteins were included in the quantitative analysis. The relative abundance of all observed N-glycopeptides from the four selected source glycoproteins was manually determined using EIC-based area-under-the-curve measurements of all observed charge states of the monoisotopic precursor ions using Xcalibur v3.0.63 (Thermo Fisher Scientific). The relative abundance of each glycoform was determined as the percentage of the peak intensity of the individual glycopeptide forms relative to the peak intensity of all glycopeptides spanning each glycosylation site, an approach commonly employed in quantitative glycopeptide analysis67,68,69.

Analysis and optimization of the search strategies used for the Byonic search engine

A Byonic-centric analysis and optimization of the search strategies were performed through a series of controlled in-house searches in which the search settings were systematically varied and the output assessed for performance. For this purpose, only the HCD- and EThcD-MS/MS data from File B were used and searched on an ordinary desktop computer (Windows 10, 64-bit, 16 GB RAM, Intel Core i7-8700 at 3.20 GHz). The ‘heavy’ multicore parameter option was selected for all searches. Fragment spectra from these two dissociation methods were searched in concert (‘HCD/EThcD’ setting enabled) using Byonic v3.9.4 (Protein Metrics Inc.) using a series of search strategies in which the diverse search settings were sequentially changed. The search strategy used by most teams employing Byonic in this study is herein referred to as the ‘default’ search strategy. The default search strategy employed a predefined glycan database containing either 309 mammalian N-glycans or 78 mammalian O-glycans available within Byonic, allowed up to one glycan per peptide as a ‘rare’ variable modification, considered only peptides with tryptic cleavage patterns with a maximum of two missed tryptic cleavages per peptide, allowed up to 10/20 ppm deviation of the observed precursor/product ion masses from the expected values, considered up to one Met oxidation (+15.994 Da) per peptide (variable ‘common’ modification), used monoisotopic correction (error check equals ± floor (mass in Da/4,000)) and employed a decoy and contaminant database available in Byonic. One or more of the search settings (SS1–SS2 and SS6–SS14) used for the default search strategy were then systematically changed; these alternative settings were selected based on the literature and by taking inspiration from search strategies used by the high-performance teams. For the N- and O-glycan databases (SS1–SS2), customized glycan databases of 25 N-glycans expected in human serum (Clerc et al23) or 13 O-glycans expected in human serum (Yabu et al40) were used. The systematic searches also explored the output when allowing up to two glycans per peptide as a ‘rare’ variable modification (SS9), when considering semispecific trypsin cleavages (SS6) with a maximum of one missed cleavage per peptide (SS7), when allowing 5/10 ppm deviation of the observed precursor/product ion masses to their expected values (SS11/SS12), when considering up to four variable ‘common 1’ modifications per peptide, including Met oxidation (+15.994 Da), Asn/Gln deamidation (+0.9840 Da), Gln → pyro Glu (−17.0265 Da) (SS8 and SS10) and, finally, also when no error check for monoisotopic correction (SS14) and no decoy/contaminant database (SS13) were employed. Cys carbamidomethylation (+57.021 Da) (fixed modification) and the protein search space (20,201 UniProtKB reviewed sequences, downloaded July 2017) remained constant across all searches and none of the searches employed spectral recalibration. Glycopeptides were filtered to 0% FDR at the peptide level by manually removing glycopeptides identified in the decoy or contaminant database after a general confidence score threshold was applied to the data output (Byonic score >100). The resulting lists of glycopeptides identified from each of these Byonic-centric searches were subjected to the devised performance tests for N- and O-glycopeptides (N1–N6 and O1–O5, respectively) and the relative sensitivity and specificity scores were determined as described above. All sensitivity and specificity scores were normalized to the scores arising from the default search strategy (set to 1). A detailed summary of the search settings and performance scores generated from these systematic searches can be found in Supplementary Table 19a.

In addition to the analysis and optimization of the search settings used for Byonic, we analyzed the impact of the data input on the performance of this search engine. For this purpose, series of controlled searches were carried out by systematically changing the fragmentation type considered for the searches while keeping the search settings and data output filtering constant. The File B raw data file was used as input for all searches and the fragmentation type for each search was specified within the Byonic interface. The ‘default’ search strategy for N-glycopeptides was used for all searches (see above for details). Fragment mass tolerance of 0.5 Da and 20 ppm were considered for CID- and HCD/EThcD-MS/MS data, respectively. The following fragmentation types and combinations thereof were tested in individual searches: CID only, HCD only, EThcD only, HCD/EThcD (in concert), HCD/CID (in concert) and HCD/EThcD/CID (in concert). Glycopeptides were filtered using the same criteria described above. Sensitivity and specificity scores were determined for the identified N-glycopeptides from each of the searches. Data from these additional searches can be found in Supplementary Table 19c.

Statistical analysis

Scores from each performance test (N1–N6 and O1–O5) and the overall team performance scores were tested for associations with the search settings (SS1–SS13) and search outputs (SO1–SO9) (average of selected variables). Specifically, seven statistical methods were applied to identify search settings and search output characteristics that were associated with high performance scores including: (1) a multiple linear regression model applied with a significance threshold of P < 0.05 to identify association between search variables (predictors) and performances scores (response variable); (2) a ridge linear regression model applied using an induced smoothing paradigm for hypothesis testing70; (3) a Lasso linear model for variable selection71; (4) a least angle regression exploiting exact postselection inference to identify associations72,73; (5) a forward stepwise linear regression applied using selective inference to identify association74; (6) a random forest algorithm (an ensemble learning model for regression) applied using a variable of importance score to identify association75 (a permutation strategy on augmented set of noise variables was exploited to define the variable importance cut-off); and (7) a gradient boosting tree algorithm (an ensemble of decision trees for prediction) applied using a similar strategy as the random forest algorithm to select important associations76,77. R packages v.1.2, 1.2.5 and 2.1.8 were used for these analyses. Only associations commonly observed across a minimum of three different statistical methods were considered in this study.

Unpaired two-sided t-tests were applied to compare N-glycopeptide (n = 22) against O-glycopeptide (n = 20) search output data from all teams (Extended Data Fig. 3) and to compare the performance scores based on HCD-MS/MS data (n = 17 or n = 16) against EThcD-MS/MS data (n = 13 or n = 10) (Extended Data Fig. 9). The confidence interval was set to 95% and statistical significance was indicated as *P < 0.05, **P < 0.01 and ***P < 0.001.

Pearson correlations (R2) were used to determine (1) the quantitative match between the observed site-specific N-glycan distribution of four selected glycoproteins in the investigated sample and the site-specific glycoform distribution reported by the literature and (2) the similarity between the overall team scores and the glycoprotein-based scores for all 22 teams.

Reporting Summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.