1 Introduction

Despite the success of the standard model (SM) of particle physics at energies up to the electroweak scale and its recent completion with the discovery of a Higgs boson at the large hadron collider [1, 2], it fails to describe phenomena such as the fermion mass hierarchy, the baryon asymmetry and the fine-tuning problem [3]. The existence of heavy vector-like quarks [4] would allow for the cancellation of quadratic divergences that occur in loop corrections to the Higgs-boson mass, solving the fine-tuning problem. Vector-like quarks are defined as coloured (under SU(3)\(_{\mathrm {c}}\)) fermionic states that have left-handed and right-handed components that both transform in the same way in the SM gauge group and therefore their masses are not obtained by a Yukawa coupling to the Higgs boson. Their existence is, for example, predicted in Little Higgs models [57], top-colour assisted technicolour [810] or composite Higgs models [1118].

In this paper, a search for single production of heavy vector-like Q quarks decaying into Wb is presented. An example of a leading-order (LO) Feynman diagram is shown in Fig. 1. The search targets the process \(pp \rightarrow qQb\) with subsequent \(Q \rightarrow Wb\) decay, where Q can be either a T quark with charge \(+2/3\) or a Y quark with charge \(-4/3\). Heavy exotic fermions, such as vector-like quarks, are added to the SM in isospin multiplets. T quarks can belong to any multiplet, while Y quarks cannot exist as singlets. The interpretation used in this paper focuses on Y quarks from a (YB) doublet and on singlet T quarks. For such T quarks, the branching ratios (BRs) for T are model dependent and mass dependent, but in the high-mass limit converge towards 2:1:1 (Wb:Zt:Ht). The \(Y \rightarrow Wb\) BR is 100 %.

Fig. 1
figure 1

Leading-order Feynman diagram of single \(Q=T,Y\) production and decay into Wb

The single production of vector-like quarks is enabled by their coupling to the SM quarks. At higher masses, single production can become the dominant production process at the LHC depending on the strength of this coupling. This dependence requires an interpretation of the results that relies on the formulation of the Lagrangian embedding these new interactions. In this paper, two such interpretations are pursued, namely that in Ref. [19] where a mixing term between the SM and vector-like quarks is introduced in a renormalisable extension of the SM, and in Refs. [20, 21] which uses a phenomenological Lagrangian parameterised with coupling terms but which, however, is non-renormalisable. When considering the phenomenology of these approaches, the two main differences are the additional terms allowed in Refs. [20, 21], which allow for larger production cross-sections, and the complete description of the multiplet-dimension dependence of the BR in Ref. [19]. The formulation of Ref. [19] also implies sensitivity to indirect electroweak constraints, such as the ratio \(R_{b}\) of the partial width for \(Z \rightarrow b\bar{b}\) to the total hadronic Z-boson width and the oblique parameters S and T [22].

In this paper, the interpretation of the search for the single production of vector-like quarks is presented in terms of \(\sin \theta \) and \(c^{Wb}\), corresponding to the mixing and coupling terms introduced by Ref. [19] and Refs. [20, 21], respectively. A comparison of their respective Lagrangians yields a simple relationFootnote 1 between \(\sin \theta \) and \(c^{Wb}\) given by \(c^{Wb}=\sqrt{2}\sin \theta \). For the interpretation in terms of \(c^{Wb}\), assumptions must be made about the \(Q \rightarrow Wb, Q \rightarrow Zt\) and \(Q \rightarrow Ht\) BRs, whereas \(\sin \theta \) fully determines those BRs for any given heavy quark mass. Therefore, in this paper, both interpretations are presented independently. The relative contribution of the left- and right-handed components of the mixing and coupling also depends on the dimension of the multiplet. For T singlets, the left-handed components (\(\sin \theta _{\mathrm {L}}\) and \(c_{\mathrm {L}}^{Wb}\)) are dominant. For Y quarks from a doublet, results are presented in terms of the magnitude of the total coupling \(\sqrt{{c_{\mathrm {L}}^{Wb}}^2 +{c_{\mathrm {R}}^{Wb}}^2}\), while for the interpretation in terms of mixing, this can be simplified to just the contribution of the right-handed (\(\sin \theta _{\mathrm {R}}\)) component [19].

The ATLAS and CMS collaborations have published searches for pair-production of vector-like T quarks in all decay channels [2328]. The best observed limits on the T-quark mass are \(m(T)> 0.855\) TeV for Ht [23], 0.810 TeV for Zt [24] and 0.920 TeV for Wb [27] decay channels at the 95 % confidence level (CL), where a BR of 100 % is assumed to the corresponding decay channel. For single T-quark production, searches for T quarks with decays into Zt [24] have been carried out by the ATLAS Collaboration using the 8 TeV dataset, but for the \(T \rightarrow Wb\) decay channel no mass limits have been set so far.

The analysis presented here is performed in the lepton+jets channel, characterised by the presence of exactly one electron or muon, and two or more jets. The outgoing light quark in the process depicted in Fig. 1 typically produces a jet in the forward region of the detector. One of the jets is a b-jet originating from the Q decay. The b-jet and the charged lepton are back-to-back in the transverse plane since both originate from the decay of a heavy object. The second b-jet originates from the gluon splitting and may be observed in either the forward or central region. Since this b-jet is soft, it often falls outside the detector acceptance. The dominant backgrounds are W+jets, top-quark pair and single top-quark production. At higher \(p_{\text {T}}\) of top quarks and W bosons, their decay products are more collimated. They can be identified as one high-mass jet with a large radius parameter (R). Events with high-mass large-R jets are vetoed to improve the suppression of the large \(t\bar{t}\) \(\rightarrow WbWb\) background process where one W-boson decays hadronically and the other leptonically.

2 ATLAS detector

The ATLAS detector [29] is a forward–backward symmetric multi-purpose detector and covers almost the full solid angle.Footnote 2 The inner detector (ID) is installed closest to the beam pipe, covering the pseudorapidity range \(|\eta |<\) 2.5. The ID comprises a silicon pixel detector and a silicon microstrip detector up to \(|\eta |<\) 2.5 and a transition radiation tracker up to \(|\eta |<\) 2.0. The ID is immersed in an axial 2 T magnetic field provided by a superconducting solenoid. Outside the solenoid magnet is the electromagnetic liquid-argon (LAr) sampling calorimeter, which has high granularity and covers up to \(|\eta | =\) 3.2. The central part of the hadronic calorimeter (up to \(|\eta |<\) 1.7) uses scintillator tiles as the active medium, while the forward part is a sampling calorimeter using LAr (1.5 \(< |\eta |<\) 4.9). The outer part of the ATLAS detector is the three-layer muon spectrometer which is immersed in a magnetic field provided by a large air-core toroid system.

The muon tracks are measured in \(|\eta |<\) 2.7 using monitored drift tubes and cathode-strip chambers, while resistive-plate and thin-gap chambers are used in the trigger system for \(|\eta |<\) 2.4.

Events are selected using a three-level trigger system [30]. In the first step (Level-1), the event rate is reduced to 75 kHz using hardware-based triggers. The High-Level Trigger (Level-2 and Event Filter) is software based and reduces the rate to 400 Hz.

3 Data and simulation samples

The search presented in this paper uses pp collision data at \(\sqrt{s} = 8\) TeV that were collected with the ATLAS detector in 2012. The data used for this analysis were taken under stable beam conditions and with all relevant ATLAS subdetector systems operational. The integrated luminosity of the data sample corresponds to 20.3 ± 0.6 fb\(^{-1}\) [31]. The events were selected using single-electron and single-muon triggers. Monte Carlo (MC) samples are generated in order to model the signal and background processes. In the MC simulation, multiple pp interactions in the same and neighbouring bunch crossings (pile-up) are taken into account. A weighting procedure is used to correct the simulated events such that they have the same pile-up distribution as the data. Geant4 [32] is used to simulate the full ATLAS detector [33] for the generated data. The simulated events and the ATLAS data are processed with the same reconstruction software.

The signal MC samples are based on the model described in Ref. [34] and are generated with MadGraph v5 [35] using a UFO model [36, 37] and the CTEQ6L1 parton distribution functions (PDFs) [38]. The samples are generated in the t-channel using the \(2 \rightarrow 3\) process \(pp \rightarrow qQb\), with Q decaying exclusively into Wb and W decaying inclusively into all the available modes. In the case that a branching ratio of 50 % is used, the corresponding signal yields are scaled by a factor of 0.5. Other decay modes of Q are assumed to be negligible and are not taken into account. The events are interfaced with Pythia8 [39] for parton showering, hadronisation and particle decay. Signal samples are generated with different Q masses in the range 0.4–1.2 TeV in steps of 0.1 TeV. All signal samples are produced using the narrow-width approximation with a width of \(\Gamma /m = 7\,\%\). Additional samples with \(\Gamma /m\) varying from 2 to 46 % are used to examine the dependence of the vector-like quark width on \(c_{\mathrm {L}}^{Wb}\).

The dominant backgrounds are \(t\bar{t}\), W+jets and single top-quark production. Smaller background contributions are Z+jets, diboson and multijet production. The \(t\bar{t}\) and single top-quark processes are modelled using the next-to-leading-order (NLO) Powheg-Box generator r2330.3 [40] using the CT10 PDFs [41]. Powheg-Box is then interfaced with Pythia v6.4 [42] with the Perugia 2011C set of tuned parameters [43] and the CTEQ6L1 PDFs. The top-quark mass is set to 172.5 GeV in all samples. The Alpgen v2.13 [44] LO generator and the CTEQ6L1 PDF set are used to simulate W/Z production. Parton showers and hadronisation are modelled with Pythia v6.4. The W/Z samples are generated with up to five additional partons, separately for W/Z+light-jet, W/\(Z+b\bar{b}\), W/\(Z+c\bar{c}\) and Wc. To avoid double-counting of partonic configurations generated by both the matrix-element calculation and the parton-shower evolution, a parton-jet matching scheme (MLM matching) [45] is employed. The overlap between W/\(Z+q\bar{q}\) (\(q=b,c\)) events generated from the matrix-element calculation and those generated from parton-shower evolution in the W/Z+light-jet samples is avoided via an algorithm based on the distance in \(\eta -\phi \) space between the heavy quarks: if \(\Delta R (q,\bar{q}) > 0.4\), the matrix-element prediction is used, otherwise the parton-shower prediction is used. Diboson samples with at least one leptonically-decaying boson are produced using Herwig v6.52 [46] and Jimmy v4.31 [47] using the CTEQ6L1 PDFs. Multijet production is modelled from data as described later.

A control region is used to obtain the normalisations and corresponding uncertainties for the \(t\bar{t}\) and W+jets contributions. Theoretical calculations of cross-sections are used to normalise the predictions of the smaller backgrounds. The inclusive Z+jets cross-section is calculated to next-to-next-to-leading-order (NNLO) accuracy using FEWZ [48]. The single top-quark production cross-sections are calculated at NLO+NNLL (next-to-next-to-leading-logarithmic) precision in QCD. The largest contribution comes from t-channel production, with a corresponding uncertainty of +3.9/−2.2 % [49]. Additional samples are generated to model the systematic uncertainties of the dominant backgrounds. The effect of initial-state radiation (ISR) and final-state radiation (FSR) on the \(t\bar{t}\) background is estimated using the LO AcerMC v3.8 [50] generator interfaced with Pythia v6.4 and using the CTEQ6L1 PDFs. A measurement of \(t\bar{t}\) production with a veto on additional central jet activity [51] is used to determine the ranges within which the parameters related to ISR and FSR are varied in Pythia.

The effect of using different models for hadronisation and factorisation is taken into account with a sample generated with Powheg-Box but interfaced to Herwig v6.52 using the CT10 PDFs in the matrix-element. The uncertainty due to the choice of \(t\bar{t}\) generator is modelled by comparing the default sample to a MC@NLO v4.03 [52, 53] sample interfaced with Herwig v6.52 using the CT10 PDF set and a sample produced with the multi-parton generator Alpgen+Herwig v6.52 (with up to three additional jets) using the CTEQ6L1 PDFs. For the evaluation of the single-top-quark modelling uncertainty, the default t-channel sample is compared to a sample generated with MadGraph5_aMC@NLO [54] and Herwig v6.52 using the CT10 PDF set.

4 Object definition

The search for vector-like Q quarks and the reconstruction of the Q-candidate mass relies on the identification of jets, electrons, muons and missing transverse momentum \(E_{\text {T}}^{\text {miss}}\). Jets are reconstructed with the anti-\(k_{t}\) algorithm [55] with radius parameters of \(R =\) 0.4 (small-R jets) and \(R =\) 1.0 (large-R jets). Locally calibrated topological clusters of calorimeter cells [56, 57] are calibrated to the energy scale of particle-level hadrons and are used as input to the jet clustering algorithm. Small-R jets are required to have a \(p_{\text {T}}\) greater than 25 GeV for \(|\eta |<\) 2.4, while for forward jets, with 2.4 \(< |\eta |<\) 4.5, \(p_{\text {T}}\) > 35 GeV is required. The higher jet \(p_{\text {T}}\) threshold for forward jets is used to mitigate pile-up effects. Large-R jets are required to have \(p_{\text {T}}\) > 200 GeV and \(|\eta |<\) 2.0. To reduce the influence of pile-up and of soft QCD radiation on large-R jets a trimming procedure is used [58], where the jet constituents are clustered into subjets using the \(k_{t}\) algorithm [59] with \(R =\) 0.3. These subjets are removed from the large-R jet if they fulfil \(p_\mathrm {T}^{\rm subjet} < 0.05 \, p_\mathrm {T}^{{\mathrm {large}{\text {-}}{R}}\, \mathrm {jet}}\) and the kinematics of the large-R jet are recalculated.

In order to further suppress jets originating from pile-up, a requirement on the jet vertex fraction (JVF) [60] is made. The JVF is defined as the summed scalar \(p_{\text {T}}\) of tracks associated with both the reconstructed primary vertex and the small-R jet, divided by the summed scalar \(p_{\text {T}}\) of all tracks associated with the jet. For jets with \(p_{\text {T}}\) < 50 GeV and |\(\eta \) | < 2.4, a JVF \(\ge \) 0.5 is required. When the small-R jets are built, the jets and electrons are not distinguished. Hence, an electron will also be reconstructed as a jet. To remove these objects, the jet closest to a selected electron is removed if \(\Delta R({\mathrm {jet}},e)<\) 0.2.

Jets containing b-hadrons are identified (b -tagged) using properties specific to these hadrons, such as a long lifetime and a large mass. This analysis uses a multivariate discriminant [61] that is based on displaced vertices and the impact parameters of tracks associated with the jet. The algorithm has an efficiency of 70 % to select b-jets and rejection factors of 5 and 135 for c-jets and light-quark or gluon jets, respectively, when assessed in a \(t\bar{t}\) simulated sample.

To reconstruct electrons, ID tracks are matched to energy deposits in the electromagnetic calorimeter [62, 63]. Only electrons with a transverse energy, \(E_\mathrm {T} = E_{\rm cluster} / \cosh (\eta _{\rm track})\), greater than 25 GeV are considered in the analysis. The \(p_{\text {T}}\) threshold of the offline lepton is higher than the momentum threshold of the trigger to ensure a trigger efficiency that is uniform in \(p_{\text {T}}\) for the selected leptons. The energy cluster of the electron candidate must have a pseudorapidity \(|\eta _{\rm cluster}|<\) 2.47. Electrons in the transition region between the barrel calorimeter and the endcap calorimeter (1.37 \(\le |\eta | \le \) 1.52) are rejected. To reject electrons originating from heavy-flavour hadron decays, electrons within a cone of size \(\Delta R = \) 0.4 around a jet are removed from the event. For calorimeter-based isolation, a requirement on the energy deposited in clusters within a \(\Delta R = \) 0.2 cone around the electron is made. The energy of the electron is subtracted and pile-up corrections are applied. A similar procedure is used for track-based isolation, using \(\Delta R=0.3\). Calorimeter-based and track-based isolation criteria which are dependent on \(E_{\mathrm T}\) and \(\eta \) ensure 90 % isolation efficiency at all electron \(p_{\text {T}}\) values for \(\Delta R =\) 0.2 and 0.3, respectively. A requirement on the longitudinal impact parameter \(z_0\) is made to the electron track, requiring \(|z_0|<\) 2 mm.

For the identification of muons, tracks from the ID and the muon spectrometer are combined [64]. Muons are required to have a \(p_{\text {T}}\) larger than 25 GeV and \(|\eta |<\) 2.5. Muons are required to be isolated from other high-\(p_{\text {T}}\) tracks within a small cone around the muon track. The size of the cone varies as a function of the muon \(p_{\text {T}}\) according to \(\Delta R =\) 10 GeV / \(p_{\text {T}}\) [65]. The muon is considered to be isolated if the scalar sum of the \(p_{\text {T}}\) from all other tracks in the cone is less than 5 % of the muon \(p_{\text {T}}\). This requirement has an average efficiency of 97 %. To reject muons originating from heavy-flavour decays, muons within a \(\Delta R =\) 0.4 cone around a jet are removed. The longitudinal impact parameter of the muon track has to fulfil \(|z_0|<\) 2 mm.

The neutrino from the leptonic W-boson decay cannot be observed directly, but its presence leads to \(E_{\text {T}}^{\text {miss}}\). To reconstruct the \({{\vec {E}}_\mathrm{{T}}^\mathrm{{miss}}}\), the vectorial sum of the momenta of all reconstructed electrons, muons and jets as well as all additional energy deposits in the calorimeters is calculated [66, 67]. The energy of clusters in the calorimeters matched to electrons, muons or jets is corrected according to the nature of the associated object.

Fig. 2
figure 2

Comparison of data to expected background for the variables used in the event selection. Each distribution is shown for events satisfying the preceding steps. The signal yields are shown for \(c_{\mathrm {L}}^{Wb}\) = 1 and for BR(\(T \rightarrow Wb\)) = 0.5. These are scaled up, in order to improve their visibility. Scale factors are chosen to ease a shape comparison between the signal samples shown. The distributions are shown here for the combined e+jets and \(\mu \)+jets channels

5 Event selection

This section defines the signal region (SR) and control regions (CRs). The event selection presented here is based on the strategy proposed in Ref. [68]. The preselection of events in the SR requires each event to have exactly one isolated lepton (electron or muon) as defined in Sect. 4. Furthermore, this lepton must be matched to the lepton that was reconstructed by the trigger. At least two small-R jets and at least one large-R jet are required; however, the large-R jet may contain one of the small-R jets.Footnote 3 The event must have a reconstructed primary vertex with at least five tracks with \(p_{\mathrm T}>\) 400 MeV. To suppress multijet background, the \(E_{\text {T}}^{\text {miss}}\) needs to be larger than 20 GeV and the sum of the \(E_{\text {T}}^{\text {miss}}\) and the W-boson transverse mass, \(m_{\mathrm {T}} (W) = \sqrt{2 p_\mathrm {T}^{\ell } E_{\mathrm {T}}^{\mathrm {miss}} (1 - \cos \phi (\ell , \vec {E}_\mathrm {T}^{\mathrm {miss}}))}\), must be larger than 60 GeV. The angle between the transverse momentum of the lepton and the \(\vec {E}_\mathrm {T}^{\mathrm {miss}}\) vector is defined as \(\phi (\ell , \vec {E}_\mathrm {T}^{\mathrm {miss}})\).

Several discriminating variables are used to further optimise the selection and define the SR. These requirements are explained in the following. Since T quarks are excluded for masses below 0.7 TeV, the optimisation of the selection criteria is done for the 0.7 TeV mass point. The sequence of the final selection is illustrated in Fig. 2b–f, for the combined e+jets and \(\mu \)+jets channels, following the order in which each criterion is applied. After the preselection, the final sequence of requirements is:

  1. (a)

    The highest-\(p_{\text {T}}\) (leading) large-R jet \(p_{\text {T}}\) must be greater than 250 GeV.

  2. (b)

    Events with massive large-R jets (\(m > 70\) GeV) are rejected.

  3. (c)

    At least one b-tagged jet matched to the large-R jet, \(\Delta R (\)large-R jet, b-tagged jet\()<0.8\), is required.

  4. (d)

    The azimuthal separation between the lepton and the large-R jet is required to be larger than 1.5.

  5. (e)

    Events with any jet with \(p_{\text {T}} > 75\) GeV and \(|\eta | < 2.4\) outside the large-R jet are rejected.

  6. (f)

    At least one forward jet is required in the event.

For the MC signal samples used, the combined acceptance times efficiency is 1.4 % for both \(m(T)=0.7\) TeV and \(m(T)=0.9\) TeV.

6 Background estimation

The multijet background is obtained from data using a matrix method [69] which predicts the shape and normalisation of the background process. This method relies on differences between the probability of a “real” (prompt) lepton and that of a “fake” (non-prompt or misidentified) lepton to fulfil certain selection criteria. The “fake” lepton efficiencies are measured in data using background-enriched control regions and are parameterised for different values of \(p_{\text {T}}\) and \(\eta \) of the charged lepton candidate. The “real” lepton efficiencies are measured in \(Z \rightarrow \ell \ell \) samples containing prompt leptons.

All other background shapes are obtained from simulation, using the samples discussed in Sect. 3. A fit control region (FitCR) is defined in order to estimate the normalisation of the \(t\bar{t}\) background and of the W+jets background from data. Two additional W+jets-enriched CRs are defined to validate the modelling (W1CR and W2CR).

In order to suppress the \(t\bar{t}\) contribution in the W2CR, the following requirement is made:

  1. (g)

    The invariant mass of the charged lepton and the b-tagged jet should be be larger than 175 GeV.

This requirement is not applied in any other region. All CRs are orthogonal to the SR, which is achieved by inverting requirement (e) as defined in Sect. 5. Therefore, instead of applying the jet veto, events are required to have a jet in that regime. The relation between the requirements used to define these CRs and the SR are summarised in Table 1.

Table 1 Differences in the event selections applied in the SR and CRs. A checkmark (\(\checkmark \)) is shown if the specific requirement is applied in the region, the cross (\(\times \)) shows that a requirement is not applied. Requirements (a) and (d) are applied in the SR and all CRs
Fig. 3
figure 3

Comparison of data to the expected background for the leading large-R jet mass in the FitCR, both for the electron (left) and muon (right) channels, after applying the W+jets and \(t\bar{t}\) normalisation correction factors

The \(t\bar{t}\) and W+jets normalisations are obtained from a fit to the large-R jet mass distribution in the FitCR. The large-R jet mass distribution for the W+jets contribution has a steeply falling shape, while the \(t\bar{t}\) fraction grows for values around the W-boson and top-quark masses. First, other small backgrounds, contributing less than 12 %, are subtracted from data. Normalisation correction factors are then obtained from the FitCR for the two background processes and the modelling is tested in the W1CR and the W2CR. Figure 3 shows the large-R jet mass distribution in the FitCR, including the corrections to the \(t\bar{t}\) and W+jets backgrounds.

The obtained correction factors with respect to the theoretical predictions for the muon (electron) channels are 0.874 (0.909) and 0.951 (0.947) for \(W\) +jets and \(t\bar{t}\) respectively.

After applying these corrections, a residual mismodelling of the W-boson \(p_{\text {T}}\) spectrum is observed at high \(p_{\text {T}}\) in all CRs. To correct for this mismodelling, corrections are obtained in the FitCR and W2CR for both \(t\bar{t}\) and W+jets events as a function of the W-boson \(p_{\text {T}}\). For \(t\bar{t}\) events, the derived correction factor is compatible with unity within the statistical uncertainties, and is therefore not applied. For W+jets, the correction factor is approximately unity for W-boson \(p_{\text {T}}\) below 300 GeV, decreasing to 0.6 for 500 GeV and 0.4 for 600 GeV.

7 Analysis procedure

After the event selection described in Sect. 5 and applying the correction factors obtained in Sect. 6, the Q candidate is reconstructed. The first step is the reconstruction of the W-boson candidate by summing the four-momenta of the charged lepton and the neutrino. To obtain the z-component of the neutrino momentum, the lepton–neutrino invariant mass is set to the W-boson mass and the resulting quadratic equation is solved. If no real solution exists, the \({{\vec {E}}_\mathrm{{T}}^\mathrm{{miss}}}\) vector is varied by the minimum amount required to produce exactly one real solution. If two real solutions are found, the one with the smallest \(|p_z|\) is used. The W-boson candidate and the small-R b-jet, which is matched to the large-R jet, are then used to reconstruct the Q candidate. Hence, no large-R jet information is used directly for the reconstruction of the discriminant, which reduces the dependence of the final result on the systematic uncertainties of the large-R jet kinematics. In Fig. 4 the distribution of the Q-candidate mass in the SR is compared to the SM background prediction and the signal distributions for \(m(Q) =\) 0.7 and 0.9 TeV.

Fig. 4
figure 4

Distribution of the Q-candidate mass for the electron and muon channels before the likelihood fit. The signal yields are shown for cross-sections corresponding to \(c_{\mathrm {L}}^{Wb}\) = 1 and for BR(\(Q \rightarrow Wb\)) = 0.5. These are scaled up, in order to improve their visibility. The uncertainty band includes all the uncertainties listed in Sect. 8, which are taken as fully uncorrelated between different sources

A binned maximum-likelihood fit to the distribution of the Q-candidate mass is carried out using the HistFactory [70] tool, which is part of the HistFitter [71] package. In the absence of signal, a profile-likelihood ratio is used to set an upper limit on the cross-section times BR at the 95 % CL. This is done using the CL\(_s\) method [72, 73]. A combined fit to the electron and muon channels is performed. The systematic uncertainties are taken into account as nuisance parameters. The likelihood is then maximised using the nuisance parameters and the signal strength \(\mu \) as parameters in the fit. The systematic uncertainty corresponding to each nuisance parameter is used as an a priori probability. These priors are assumed to follow a Gaussian distribution and constrain the nuisance parameters. The systematic uncertainties affecting both channels are treated as correlated across the channels.

8 Systematic uncertainties

The shape and normalisation of the distribution of the Q-candidate mass is affected by various systematic uncertainties. The sources of uncertainty are split into two categories: (1) uncertainties due to the modelling of the signal and background processes; (2) experimental uncertainties on the calibration and efficiency for reconstructed objects. The impact of each source on the total signal and background normalisation is summarised in Table 2.

Table 2 Summary of the impact of the systematic uncertainties on signal and background normalisations in percent. The values given for the signal are those corresponding to the 0.7 TeV mass point. If the uncertainties resulting from the up and down variations are asymmetric, the larger deviation is shown here

8.1 Modelling uncertainties

The uncertainties are propagated from the FitCR to the SR, resulting in a background prediction uncertainty of 15 % in the SR due to the statistical uncertainty in the FitCR. The \(t\bar{t}\) and W+jets normalisations are derived in the FitCR separately for each additional up and down variation accounting for a systematic uncertainty and applied in the SR. Therefore the uncertainties are taken to be fully correlated between the FitCR and SR.

The uncertainties due to QCD initial- and final-state radiation modelling are estimated with samples generated with AcerMC interfaced to Pythia6 for which the parton-shower parameters are varied according to a measurement of the additional jet activity in \(t\bar{t}\) events [51]. The impact of the \(t\bar{t}\) modelling is evaluated using three different simulation samples described earlier in Sect. 3. The uncertainty due to the choice of parton shower and hadronisation model is evaluated by comparing samples produced with Powheg+Pythia6 and Powheg+Herwig. For another comparison, the NLO matrix-element generator is changed simultaneously with the parton-shower model using samples generated with Powheg+Pythia6 and MC@NLO+Herwig. Finally, the Powheg+Pythia6 sample is compared to the LO sample generated with Alpgen+Herwig. The largest impact on the normalisation is observed when comparing Powheg+Pythia6 and MC@NLO+Herwig. The total \(t\bar{t}\) modelling uncertainty is 4.9 %.

The dominant single-top-quark process is the t-channel production. In order to estimate the impact of using different models for this process, the nominal Powheg+Pythia6 sample is compared to a sample produced with MadGraph5_aMC@NLO+Herwig. The change in the background acceptance is 6.3 %.

To account for the shape uncertainties in the multijet background estimates, alternative methods are used in the evaluation of the real and fake rates for the matrix method. For the electron channel, the systematic uncertainties on the fake efficiencies are obtained by changing the parameterisation. For the muon channel, the fake efficiencies obtained in two different control regions are compared. The uncertainty on the real efficiency is estimated by comparing the values obtained from the tag-and-probe method with those from an alternative method, where very tight requirements are applied on \(E_{\text {T}}^{\text {miss}}\) and \(m_{\mathrm T}(W)\). An additional uncertainty is applied by varying the background normalisation in the control region for the fake estimate by 30 %, which corresponds to the uncertainty on the Z+jets and W+jets backgrounds in that control region. The resulting uncertainty on the background acceptance is 2.6 %.

To account for the mismodelling of the W-boson \(p_{\text {T}}\), a polynomial fit is applied to obtain a continuous function for the reweighting. This fit is repeated with different polynomials and the mean value of these functions is used as a nominal weight. Polynomials of degrees starting from one up to the maximum allowed by the number of degrees of freedom are used. The largest deviation of the functions from the nominal weight in each bin is taken as a systematic uncertainty. The change in the background acceptance is 2.4 %.

To evaluate the PDF uncertainty, the uncertainties of three different PDF sets (NNPDF2.3 NLO [74], MSTW2008nlo [75] and CT10 NLO) and their eigenvectors are considered. Based on the PDF4LHC recommendation [76], the envelope of all uncertainties is taken and symmetrised. The resulting uncertainty on the background acceptance is 7.4 %.

8.2 Experimental uncertainties

The detector response is affected by several sources of uncertainty which influence the object reconstruction and hence lead to a change in the selection efficiency. The effect on the signal yields is quoted for a Q candidate with \(m(Q) =\) 0.7 TeV. In order to model the uncertainty on the b-jet identification, the b-tagging and mistagging scale factors are varied [61]. Large statistical fluctuations for high-momentum b-jets cause the corresponding systematic component to have a large impact on the total normalisation. The b-tagging uncertainties affect the background by 1.5 % and the signal acceptance by 8 %. This difference arises because the impact of b-tagging uncertainties on the background is strongly mitigated by the use of the FitCR to determine the background normalisation.

The jet energy resolution is measured using in situ methods and the corresponding systematic uncertainty is about 10 % for jets with 30 \(\le p_\mathrm {T} \le \) 500 GeV [77]. The total impact is 0.3 % on the background yields and 0.7 % on the signal yields. Pile-up suppression is achieved by applying a requirement on the JVF as described in Sect. 4. The JVF uncertainties affect the signal by < 0.1 % and the background yields by 0.2 %.

The small-R jet energy scale [78] uncertainty affects the background yield by 3.6 % and the signal acceptance by 3.3 %. Although the large-R jet is not directly used in the reconstruction of the Q candidate, uncertainties related to the large-R jet energy scale and resolution affect the event yields. The uncertainty on the large-R jet energy resolution and jet mass resolution results in an uncertainty of 6.8 % on the background yield and an uncertainty of 4.0 % on the signal acceptance. The large-R jet energy scale uncertainty has a larger effect: 9.7 % on the background acceptance and 7.2 % on the signal yield.

Uncertainties on trigger, reconstruction and identification efficiencies are evaluated in addition to uncertainties on lepton momentum scale and resolution. The impact of these uncertainties is <0.3 % on the background and 2.3 % on the signal acceptance. All experimental uncertainties are propagated consistently to the evaluation of the missing transverse momentum. The corresponding change in the event yields is smaller than 0.5 %.

The uncertainty on the integrated luminosity is 2.8 %. It is derived, following the same methodology as that detailed in Ref. [31].

9 Results

The event yields obtained in the signal region for an integrated luminosity of 20.3 fb\(^{-1}\) are displayed in Table 3. The expected yields for signal masses of 0.7 and 0.9 TeV are shown alongside the background prediction, which includes the normalisation of the \(t\bar{t}\) and W+jets event yields obtained in the FitCR and the number of events observed in data.

Table 3 Comparison of the observed number of events with the expected number before the fit in the signal region after applying the corrections and the full event selection. The normalisation of the \(t\bar{t}\) and W+jets backgrounds was obtained in the FitCR. The statistical and systematic uncertainties on the MC predictions are presented here and are symmetrised. The signal yields are shown for \(c_{\mathrm {L}}^{Wb}\) \(=1\) and BR(\(T \rightarrow Wb\)) \(=\) 0.5

No significant deviation from the SM background prediction is found. In the electron channel there is a tendency for the number of events in data to exceed the expectation for candidate masses above 0.9 TeV. The local \(p_0\)-value for the observed data to agree with the background-only hypothesis reaches its smallest value of 5.2 % (corresponding to 1.6 standard deviations) at 1 TeV. Mass-dependent exclusion limits in steps of 0.1 TeV are set on the cross-section times BR of the Q candidate as explained in Sect. 7. A simultaneous maximum-likelihood fit is performed to the electron and muon distributions. In Fig. 5 the mass distributions before (black) and after (red) the nuisance parameter fit (background-only hypothesis) are compared. The narrower uncertainty band for the post-fit distribution shows that the overall uncertainty is reduced in the nuisance parameter fit.

Fig. 5
figure 5

Distribution of the Q-candidate mass for the electron (left) and muon (right) channels, both before and after the nuisance parameter fit. The fit was performed using a background-only hypothesis. The error bands include the full statistical and systematic uncertainty before and after the fit. The bottom panels show the ratio between the observed data and the SM prediction before (black squares) and after (red triangles) the nuisance parameter fit

Fig. 6
figure 6

Observed and expected limits on the cross-section times BR for the single production of a vector-like quark \(Q \rightarrow Wb\) as a function of the Q mass. The limits are shown compared to three theoretical predictions: the NLO cross-section prediction in the composite-Higgs-model scenario [21] (brown dot-dashed line), and the maximum cross-sections for Tbj (red dashed line) and Ybj (black dashed line) [19]

The observed and expected 95 % CL limits on the cross-section times BR of singly produced Q candidates is shown in Fig. 6 for different candidate masses. The expected upper limit on the cross-section is determined using pseudo-data constructed from a background-only model built from the nuisance parameters fitted to real data. The limits include full statistical and systematic uncertainties and are compared to the maximum allowed cross-sections for Tbj and Ybj from electroweak constraints [19] and the NLO cross-section prediction for \(c_{\mathrm {L}}^{Wb} =1\) [21]. The observed direct limits are less stringent than the indirect limits on the maximum cross-sections from Ref. [19], but rely on fewer assumptions about the new physics that would produce T or Y quarks.

Fig. 7
figure 7

a Observed and expected limit (95 % CL) on the coupling of the vector-like quark to the SM \(W\) boson and b-quark as a function of the Q mass, where the BR(\(T\rightarrow Wb\)) is assumed to be 50 %. The excluded region is given by the area above the solid black line. b Observed and expected limit (95 % CL) on the mixing of a singlet vector-like T quark to the SM sector, where the BR(\(T\rightarrow Wb\)) is assumed to be that of a singlet. The excluded region is given by the area above the solid black line. The limits are shown compared to the indirect electroweak constraints from Ref. [19] (green and red line). In addition, the observed limits from pair-production searches by ATLAS [23] (olive) and CMS [27] (blue) are shown

Fig. 8
figure 8

a Observed and expected 95 % CL upper limits on the coupling of the vector-like Y quark to the SM \(W\) boson and b-quark as a function of the Q mass. b Observed and expected 95 % CL upper limits on the mixing of a vector-like Y quark to the SM sector in a (YB) doublet model. In addition, the indirect electroweak constraints from Ref. [19] are shown. For both a and b BR(\(Y\rightarrow Wb\)) is assumed to be 100 %

More events than predicted are observed for the higher mass values, leading to a less stringent observed limit for masses above 0.8 TeV. These differences are, however, within the 1\(\sigma \) uncertainty band. The mass limit is obtained from the intersection of the NLO prediction with the curve for the observed cross-section times BR limit. The observed (expected) limit on the Q-candidate mass obtained for this scenario is 0.95 (1.10) TeV.

9.1 Interpretation for singlet vector-like T quarks

The limit set on the cross-section times branching ratio can be translated into a limit on \(c_{\mathrm {L}}^{Wb}\), using the relation

$$\begin{aligned} |c_{\mathrm {L}}^{Wb} |&= \sqrt{\frac{\sigma _{\rm limit}}{\sigma _{\rm theory}}} \end{aligned}$$
(1)

and the theoretical predictions from Ref. [21]. For the theoretical prediction the value of \(c_{\mathrm {L}}^{Wb}\) was set to 1.0. The expected and observed limits are shown in Fig. 7a. These limits exclude couplings above 0.6 for masses below 0.7 TeV and above \(c_{\mathrm {L}}^{Wb}\) = 1.2 for a T quark with a mass of 1.2 TeV. The limits on the mixing angle between the vector-like quark and the SM sector are derived in a similar fashion and are shown in Fig. 7b. For lower masses, mixing angles from 0.4 to 0.5 are excluded, while the limit increases up to 0.81 for a T quark with a mass of 1.2 TeV.

As shown in Formula B1 of Ref. [21], the width of the vector-like quark is proportional to \({c_{\mathrm {L}}^{Wb}}^2\). Therefore, a larger width is expected for higher values of \(c_{\mathrm {L}}^{Wb}\). As described in Sect. 3, a narrow-width approximation is used in the production of the signal samples. To test the validity of the limits shown in Fig. 7, the limits were recalculated for signal samples with \(\Gamma /m\) values up to 0.46, using the same theoretical cross-section prediction. For all masses and \(\Gamma /m\) the observed limit is found to be more stringent than, or equal to, the value obtained for the narrow-width approximation. For \(m(Q)=0.9\) TeV the cross-section times BR limit decreases by 15 % (20 %) for \(\Gamma /m =\) 0.3 (\(\Gamma /m = 0.46\)) and for \(m(Q)=1.2\) TeV the limit decreases by 13 % (21 %) for \(\Gamma /m = 0.3\) (\(\Gamma /m= 0.46\)). Hence, the limits presented in this paper constitute a conservative estimate regarding the assumptions about the width of vector-like quarks.

9.2 Interpretation for a vector-like Y quark from a doublet

The limits on cross-section times BR are used to set limits on the couplings \(c_{\mathrm {L}}^{Wb}\) and \(c_{\mathrm {R}}^{Wb}\) for a vector-like Y quark. Using the theoretical cross-section and the general vector-like quark model discussed in Ref. [21] as well as the BR(\(Y \rightarrow Wb\)) = 1, a limit on \(\sqrt{{c_{\mathrm {L}}^{Wb}}^2 +{c_{\mathrm {R}}^{Wb}}^2}\) is set. Due to the higher BR of the vector-like Y quark, this limit as shown in Fig. 8a is more stringent, by a factor of \(1/\sqrt{2}\), than the limit on \(|c_{\mathrm {L}}^{Wb} |\) for single T production. The cross-section limit is also translated into a limit on the mixing parameter \(|\sin \theta _{\mathrm {R}} |\) in a (YB) doublet model. This is done as a function of the Y mass as discussed in Ref. [19]. Figure 8b shows the resulting limit on \(|\sin \theta _{\mathrm {R}} |\) for the (YB) doublet as a function of m(Y), including also the limit on \(|\sin \theta _{\mathrm {R}} |\) for a (YB) doublet model from electroweak precision observables taken from Ref. [19].

10 Summary

A search for the production of a single vector-like quark Q with subsequent decay into Wb has been carried out with the ATLAS experiment at the LHC. The data used in this search correspond to 20.3 fb\(^{-1}\)of pp collisions at a centre-of-mass energy of \(\sqrt{s} = 8\) TeV. The selected events have exactly one isolated electron or muon, at least two small-R jets, at least one large-R jet, one b-tagged jet and missing transverse momentum. Events with massive large-R jets are vetoed to reduce the \(t\bar{t}\) and W+jets background processes. The Q candidate is fully reconstructed and its mass is used as discriminating variable in a maximum-likelihood fit. The observed data distributions are compatible with the Standard Model background prediction and no significant excess is observed. Upper limits are set on the cross-section times branching ratio as a function of the T-quark mass using \(c_{\mathrm {L}}^{Wb}\) = 1 and BR(\(T \rightarrow Wb\)) = 0.5. The observed (expected) exclusion limit for T quarks is 0.95 TeV (1.10 TeV) at the 95 % confidence level. Using theoretical predictions, the cross-section limits are translated into limits on the QWb coupling \(c_{\mathrm {L}}^{Wb}\) and the mixing angle of the T quark with the SM sector. The results are also interpreted as the coupling of a vector-like Y quark to the SM W boson and b-quark as well as a limit on the mixing parameter \(|\sin \theta _{\mathrm {R}} |\) in a (YB) doublet model.