1 Introduction

Several theories beyond the Standard Model (SM) [1–3] involve enhanced symmetries that introduce new charged vector currents carried by new heavy gauge bosons, usually called \(W'\) bosons. For instance, Grand Unified Theories [4–7] extend fundamental symmetries of the SM, in which a massive right-handed counterpart to the SM \(W\) boson may occur. \(W'\) bosons can appear in phenomenological models involving extra space-time dimensions such as Kaluza-Klein excitations of the SM \(W\) boson [8] or in technicolor models [9]. Also Little Higgs theories [10] predict several new particles, including a \(W'\) boson. In order to interpret a direct experimental search independently of the details of particular models predicting a \(W'\) boson, it is advantageous to rely on an effective model describing the couplings of the \(W'\) boson to fermions [11].

The search for a \(W'\) boson decaying to a top quark and a \(b\)-quark (\(W'\rightarrow tb\))Footnote 1 explores models potentially inaccessible to \(W'\rightarrow \ell \nu \) searches. Also, in the right-handed sector, it is assumed that there is no light right-handed neutrino to which a \(W'\) boson could decay, and, hence, only hadronic decays are allowed [11, 12]. In some theories beyond the SM, new physics couples more strongly to the third generation than to the first and second [9]. Searches for \(W'\) bosons decaying to \(tb\) have been performed at the Tevatron [13–15] and at the LHC [15, 16], in leptonic top-quark decay channels excluding a \(W'\) boson with purely right-handed couplings (referred to as \(W'_{R}\)) with mass less than 2.13 TeV at 95 % Confidence Level (CL).

This document describes the first search for the \(W'\rightarrow tb \) process in the fully hadronic final state of the top-quark decay. For high \(W'\) masses, the final state signature consists of one high-momentum \(b\)-quark and another \(b\)-quark close to the two light-quarks from the \(W\)-boson decay. The distinct signature of high-momentum top quarks is exploited to isolate the signal from the copious hadronic multijet background making use of novel jet substructure techniques to identify boosted hadronically decaying top quarks. This allows for particularly good sensitivity at high \(W'\) masses. 95 % CL exclusion limits are presented on the \(W'\)-boson coupling as a function of the \(W'\) mass in an effective model.

2 The ATLAS detector

Charged particles in the pseudorapidityFootnote 2 range \(|\eta | < 2.5\) are reconstructed with the inner detector (ID), which consists of several layers of semiconductor detectors (pixel and strip) and a straw-tube transition-radiation tracker, the latter extending to \(|\eta | < 2.0\). The inner tracking system is immersed in a 2 \(\mathrm {T}\) magnetic field provided by a superconducting solenoid. The solenoid is surrouned by sampling calorimeters, which span the pseudorapidity range up to \(|\eta |\) = 4.9. High-granularity liquid-argon (LAr) electromagnetic calorimeters are present up to \(|\eta |\) = 3.2. Hadronic calorimeters with scintillating tiles as active material cover \(|\eta | <\) 1.74 while LAr technology is used for hadronic calorimetry from \(|\eta |\) = 1.5 to \(|\eta |\) = 4.9. Outside the calorimeter system, air-core toroids provide a magnetic field for the muon spectrometer (MS). Three stations of precision drift tubes and cathode strip chambers provide a measurement of the muon track in the region of \(|\eta | < 2.7\). Resistive-plate and thin-gap chambers provide muon triggering capability up to \(|\eta | < 2.4\).

3 Data and Monte-Carlo simulation samples

3.1 Data samples

The data used for this analysis was collected in \(pp\) collisions in 2012 at a centre-of-mass energy of \(\sqrt{s} = 8{\mathrm {\ TeV}}\). All candidate events must satisfy data-quality requirements that include being recorded during the LHC stable-beam periods and proper functioning of the detector and trigger subsystems. After the trigger and data-quality requirements, the amount of data used by this analysis corresponds to an integrated luminosity of \(20.3\;\text{ fb }^{-1}\) with an average number of interactions per bunch-crossing of 20.7.

3.2 Signal modelling

The right- and left-handed \(W'\) boson (denoted as \(W'_{R}\) and \(W'_{L}\), respectively) models are implemented in MadGraph 5 [17] using FeynRules [18, 19], which is used to generate events at leading-order (LO) in \(\alpha _s\) through Drell-Yan like production. MadGraph also simulates the decay of the top quark taking spin correlations into account. Pythia 8.165 [20] is used for parton showering and hadronisation. CTEQ6L1 [21] parton distribution functions (PDFs) are used for the event generation.

The \(W'_{R}\) and \(W'_{L}\) cross sections times branching ratios to the \(tb\) final state are obtained from next-to-leading order (NLO) QCD calculations [11, 22] and are shown for different \(W'\) masses in Table 1. The mass of a possible right-handed neutrino is assumed to be larger than the mass of the \(W'_{R}\) boson, allowing only hadronic decays of the \(W'_{R}\). In the case of a \(W'_{L}\) boson, leptonic decays are allowed. Dedicated Monte-Carlo (MC) simulation samples with interference effects between \(W'_{L}\) and SM \(W\) included have been used to estimate the change in the number of expected signal events. In the high mass signal region the change in event yield is less than 1 % after kinematic requirements. Interference effects with the SM \(s\)-channel single-top quark process are ignored. All simulated samples are normalised to these NLO calculations using NLO/LO \(k\)-factors ranging from 1.15 to 1.35 depending on the mass and the chirality of the \(W'\) boson. The models assume that the \(W'\)-boson coupling strength to quarks is the same as for the SM \(W\) boson: \(g'_R = g_\mathrm{{SM}}\) and \(g'_L = 0\) (\(g'_R = 0\) and \(g'_L = g_\mathrm{{SM}}\)) for \(W'_{R}\) (\(W'_{L}\)) bosons, where \(g_\mathrm{{SM}}\) is the SM \(SU(2)\) coupling.

Table 1 NLO cross sections times branching ratio to \(tb\) for different \(W'\) masses for the left-handed and for the right-handed model [11, 22]
Full size table

3.3 Background samples

The background estimate in this analysis is derived from a fit to data. However, an initial background estimate is introduced in Sect. 5.2, which uses a data-driven technique based on sideband regions for the multijet process and MC simulation samples for top-quark pair production (\(t\bar{t}\)). For this purpose, \(t\bar{t}\) production is simulated using the Powheg-Box generator [23, 24] coupled to Pythia 6.426 [25, 26] for parton showering and hadronisation. This sample uses the CTEQ6L1 PDF set. The \(t\bar{t}\) samples are normalised to the next-to-next-to-leading order (NNLO) calculations in \(\alpha _s\) including resummation of next-to-next-to-leading logarithmic soft gluon terms with top++2.0 [27–32]: \(\sigma _{t\bar{t}}= 253^{+14}_{-16}\) pb. PDF and \(\alpha _s\) uncertainties are calculated using the PDF4LHC prescription [33] with the MSTW2008 68 % CL NNLO [34, 35], CT10 NNLO [36, 37] and NNPDF2.3 [38] PDF sets, added in quadrature to the scale uncertainty. An uncertainty on the top-quark mass of \( 1{\mathrm {\ GeV}}\) is also considered.

For the optimisation of the \(W'\) top-tagger (Sect. 4.1), MC samples are generated with Pythia 8.160 using the AU2 tune [39] and the CT10 [36] PDF set.

After event generation, all signal and background MC samples are passed through a full simulation of the ATLAS detector [40] based on GEANT4 [41] and then reconstructed using the same algorithms as for collision data. All MC processes are simulated with pile-up interactions included and re-weighted to match the conditions of the data sample.

4 Physics objects and boosted top identification

This analysis relies on the reconstruction and identification of jets. Jets are built from energy depositions in the calorimeters with the anti-\(k_T\) algorithm [42] using locally-calibrated topological clusters as inputs. Jets are further calibrated using energy and \(\eta \)-dependent correction factors derived from simulation and with residual corrections from in-situ measurements [43]. Events with jets built from noisy calorimeter cells or non-collision backgrounds are removed [44]. In this analysis two radius parameters are used for jet reconstruction: a small-\(R\) radius of 0.4 and a large-\(R\) radius of 1.0. Small-\(R\) jets are required to have \(p_{\mathrm {T}}> 25{\mathrm {\ GeV}}\) and \(|\eta | < 2.5\). To minimise the impact of energy depositions from pile-up interactions the large-\(R\) jets are trimmed [45]. The trimming algorithm reconstructs jets using the \(k_T\) jet algorithm with \(R=0.3\) built out of the constituents of the original large-\(R\) jet. Constituent jets contributing less than 5 % of the large-\(R\) jet \(p_{\mathrm {T}}\) are removed. The remaining energy depositions are used to calculate the jet kinematics and substructure properties. Large-\(R\) jets are required to have \(p_{\mathrm {T}}> 350{\mathrm {\ GeV}}\) and \(|\eta | < 2.0\).

In order to identify small-\(R\) jets which originate from \(b\)-quarks, this analysis uses a neural-network based \(b\)-tagging algorithm [46]. Different observables based on the long lifetime of \(B\) hadrons are used as inputs and are able to discriminate between \(b\)-jets, \(c\)-jets and light-quark jets.

Events with reconstructed high-quality electrons [47] or muons [48] are vetoed in order to ensure orthogonality to analyses using the leptonic decay of the top quark [16]. Electrons and muons with transverse momenta above 30 GeV are considered for this veto.

4.1 The \(W'\) top-tagger

This analysis searches for \(W'\) bosons in the high mass (\(m_{W'}> 1.5{\mathrm {\ TeV}}\)) region, where the top quark and bottom quark have high transverse momentum. The average distance between the decay products of the top quark falls with increasing top-quark \(p_{\mathrm {T}}\), and their hadronic showers begin to overlap. This high-\(p_{\mathrm {T}}\) topology, where the decay products of a massive particle can be captured in one single large-\(R\) jet, is referred to as “boosted” [49–52].

The discrimination of large-\(R\) jets originating from hadronic top-quark decays from large-\(R\) jets originating from other sources using calorimeter information is termed top-tagging. The \(W'\) top-tagging algorithm is a cut-based algorithm using different large-\(R\) jet substructure properties developed to efficiently select large-\(R\) jets from \(W'\) signal events over the dominant background from multijet production featuring light-quark, \(b\)-quark and gluon-initiated jets. The procedure uses three substructure variables: the one-to-two \(k_T\)-splitting scale \(\sqrt{d_{12}}\) [53] and two ratios of \(N\)-subjettiness (\(\tau _N\)) variables [54, 55] \(\tau _{32} = \tau _3 / \tau _2\) and \(\tau _{21} = \tau _2 / \tau _1\).

The splitting scale \(\sqrt{d_{12}}\) distinguishes jets containing top-quark decays, which are relatively \(p_{\mathrm {T}}\)-symmetric in the top-quark rest frame, from \(p_{\mathrm {T}}\)-asymmetric light jets. It is calculated by reclustering the constituents of the large-\(R\) jet using the \(k_T\) algorithm, where the reclustering procedure is stopped at the last merging step. Since the \(k_T\) algorithm clusters the hardest objects last, the last clustering step corresponds to the merging of the two hardest subjets, and \(\sqrt{d_{12}}\) is defined as the corresponding scale:

$$\begin{aligned} \sqrt{d_{12}} = \text {min}(p_{\mathrm {T},1},p_{\mathrm {T},2}) \times \sqrt{\left( \Delta \eta _{12} \right) ^2 + \left( \Delta \phi _{12} \right) ^2}, \end{aligned}$$

where \(p_{\mathrm {T},1}\) and \(p_{\mathrm {T},2}\) are the transverse momenta of the two remaining subjets, and \(\Delta \eta _{12}\) and \(\Delta \phi _{12}\) are the distances in \(\eta \) and \(\phi \) between these two subjets. For jets from hadronic top-quark decays the \(\sqrt{d_{12}}\) distribution is expected to peak at approximately half the top-quark mass. For jets initiated by light quarks, \(b\)-quarks and gluons, the \(\sqrt{d_{12}}\) distribution is expected to peak near zero.

\(N\)-subjettiness is a measure of the compatibility of a large-\(R\) jet with a given number of subjets. The \(\tau _N\) are calculated by reclustering the large-\(R\) jet constituents with the \(k_T\) algorithm requiring exactly \(N\) subjets to be found. The \(\tau _N\) are then defined by:

$$\begin{aligned} \tau _N = \frac{1}{d_0}\sum _{k} p_{\mathrm {T}k} \times \text {min}(\delta R_{1k}, \dots ,\delta R_{Nk}), \end{aligned}$$

with \(d_0 = \sum _k p_{\mathrm {T}k} \times R\), where the sum runs over all constituents of the jet, \(p_{\mathrm {T}k}\) is the \(p_{\mathrm {T}}\) of the \(k\mathrm{{th}}\) constituent, \(R\) is the radius parameter of the original jet, and the variable \(\delta R_{ik}\) is the distance in \(\eta \)-\(\phi \) space from the \(i\mathrm{{th}}\) subject to the \(k\mathrm{{th}}\) constituent. Ratios of the \(\tau _N~(\tau _{ij}=\tau _{i}/\tau _{j})\) are then defined to discriminate if a jet is more \(i\)- or \(j\)-subjet-like. The \(\tau _{ij}\) distributions peak closer to 0 for \(i\)-subjet-like jets and closer to 1 for \(j\)-subjet-like jets.

The optimisation procedure for the \(W'\) top-tagger aims for an optimal compromise between the efficiency for jets originating from hadronically decaying top quarks and the rejection of jets originating from QCD-multijet production. First, an optimal requirement on \(\sqrt{d_{12}}\) is applied and then, selection criteria on the \(N\)-subjettiness variables are determined. The MC samples used are the 2 TeV \(W'_{L}\rightarrow tb\) signal sample and a high-\(p_{\mathrm {T}}\) QCD-multijet sample with a similar range in transverse momentum. It has been checked that changing the order in which \(\sqrt{d_{12}}\), \(\tau _{32}\) and \(\tau _{21}\) are optimised yields very similar results.

Figure 1 shows distributions of \(\sqrt{d_{12}}\) (top), \(\tau _{32}\) with the \(\sqrt{d_{12}}\) requirement applied (centre), and \(\tau _{21}\) with both \(\sqrt{d_{12}}\) and \(\tau _{32}\) requirements applied (bottom) for jets originating from hadronically decaying top quarks in 2 TeV \(W'_{L}\) and \(W'_{R}\) MC simulations. These are compared to the distributions for jets originating from light-quark, \(b\)-quark and gluon jets from QCD-multijet MC simulations. The optimised top-tagging requirements are \(\sqrt{d_{12}} > 40{\mathrm {\ GeV}}\), \(\tau _{32} < 0.65\) and \(0.4 < \tau _{21} < 0.9\). While \(\tau _N\) is an infrared- and collinear-safe observable [54], infrared-safety of \(\tau _{32}\) is ensured by the requirements on \(\tau _{21}\). The selection efficiency for jets originating from hadronic top-quark decays is estimated in MC simulations to be larger than 50 % for jet \(p_{\mathrm {T}}\) above 500 GeV, while the probability to falsely tag a light-quark, \(b\)-quark or gluon jet is below 10 % [50]. For jet \(p_{\mathrm {T}}\) below 800 GeV, where the sample size is sufficient, the top-tagging efficiency is cross-checked in data using single lepton \(t\bar{t}\) events, and the top-tagging efficiency is found to be consistent between data and MC.

Fig. 1
figure 1

Distributions in simulated samples of the substructure observables used for the \(W'\) top-tagger for trimmed large-\(R\) jets originating from QCD-multijet production and originating from hadronic decays of top quarks from 2 TeV \(W'_{L}\) and \(W'_{R}\) boson decays. The top figure shows the \(\sqrt{d_{12}}\) distributions and the middle (bottom) figures show the \(\tau _{32}\) (\(\tau _{21}\)) distribution after requiring \(\sqrt{d_{12}} > 40{\mathrm {\ GeV}}\) (\(\sqrt{d_{12}} > 40{\mathrm {\ GeV}}\) and \(\tau _{32} < 0.65\)). All distributions are normalised to unity. The arrows indicate the cut values used in the \(W'\) top-tagger

Full size image

5 Analysis

5.1 Event selection

Candidate events are triggered by requiring the scalar sum of the \(E_{\mathrm {T}}\) of the energy deposits in the calorimeters at trigger level to be at least 700 GeV. In order to perform the offline analysis in the fully efficient regime of this trigger, the scalar sum of the transverse momenta of reconstructed small-\(R\) jets with \(p_{\mathrm {T}}> 25 {\mathrm {\ GeV}}\) and \(|\eta | < 2.5\) is required to be at least 850 GeV. Candidate events must have at least one primary vertex with at least five tracks associated to it and have exactly one large-\(R\) \(W'\) top-tagged jet (top candidate) and one small-\(R\) \(b\)-tagged jet (\(b\)-candidate) each with \(p_{\mathrm {T}}> 350~{\mathrm {\ GeV}}\) and an angular separation \(\Delta R = \sqrt{\left( \Delta \eta \right) ^2 + \left( \Delta \phi \right) ^2}\) larger than 2.0 between the flight direction of the top candidate and the \(b\)-candidate. The invariant mass of the dijet system must be at least 1.1 TeV in order to avoid turn-on effects from the kinematic selection. The events are divided into two categories: the one \(b\)-tag category and the two \(b\)-tag category. For the two \(b\)-tag category, an additional \(b\)-tagged small-\(R\) jet with \(p_{\mathrm {T}}> 25{\mathrm {\ GeV}}\) has to be present close to the top candidate by requiring \(\Delta R\) between the small-\(R\) \(b\)-jet and the top candidate to be less than the large-\(R\) jet radius parameter 1.0. Figure 2 shows the acceptance times selection efficiency as a function of \(tb\) invariant mass at truth level in the one and two \(b\)-tag categories and the total signal efficiency corresponding to their sum. The difference in the efficiencies observed in the \(W'_{L}\) and \(W'_{R}\) models originates from the different top-tagging efficiencies, which is due to the preferred flight directions of the top-quark decay products in the top-quark rest frame for the two chiralities.

Fig. 2
figure 2

Selection acceptance times efficiency as a function of \(tb\) invariant mass at truth level for left- and right-handed \(W'\) MC. The total efficiency curves correspond to the sum of the efficiencies of the one \(b\)-tag and two \(b\)-tag categories

Full size image

5.2 Initial background estimate

An initial background estimate is performed to choose the fit function which is used to estimate the background from data. It is also used to derive the associated systematic uncertainties on the background modelling.

Multijet events are the dominant background comprising 99 % (88 %) of the events in the one (two) \(b\)-tag category. The estimate of the contribution of multijet events is based on a data-driven method which categorises events based on top- and \(b\)-tagging. Requiring the high-\(p_{\mathrm {T}}\) small-\(R\) jet to fail the \(b\)-tagging requirement or the large-\(R\) jet to fail the top-tagging requirement provides three orthogonal control regions in each \(b\)-tag category that are dominated by multijet events. These control regions are then used to make an estimate of the multijet contribution in the signal region, as defined by the event selection.

The other significant background is top-quark pairs, contributing 11 % in the two \(b\)-tag category as estimated using MC simulations. Other backgrounds, such as single-top, \(W\)-boson\(+\)jets, and \(Z\)-boson\(+\)jets production are found to have a very small contribution.

Table 2 reports the number of data and expected background events in the signal region for the one \(b\)-tag and two \(b\)-tag categories.

Table 2 Event yields in the signal region for SM processes using the initial background estimate compared to the yield observed in data. The uncertainties quoted for multijet and for \(t\bar{t}\) production contain statistical as well as systematic uncertainties (Sect. 6). The contributions from other background sources (single-top, \(W\)-boson\(+\)jets and \(Z\)-boson\(+\)jets production) have only been estimated approximately, but were found to be smaller than 0.4 % of the expectation for multijet production. An uncertainty of 100 % is quoted here in order to reflect the approximations made
Full size table

5.3 Statistical analysis

An unbinned likelihood fit to the \(m_{tb}\) distributions combining the one \(b\)-tag category and the two \(b\)-category is performed, where the range considered is 1.1–4 TeV. The lower bound of \(m_{tb}\) is due to turn-on effects of the \(m_{tb}\) distribution originating in the kinematic selection. This allows \(W'\) signals of \(m_{W'} \ge 1.5{\mathrm {\ TeV}}\) to be tested, because for lower values of \(m_{W'}\), the peak of the signal shape is only partially contained in the \(m_{tb}\) range considered. The upper bound of \(m_{tb}\) is motivated by the low expected number of events in the two \(b\)-tag category at such high values. Hence, \(W'\) signals of \(m_{W'} \le 3.0{\mathrm {\ TeV}}\) can be tested, in order to constrain the background function parameters also for high \(m_{tb}\) values. A profile likelihood-based test statistic is used for the evaluation of \(p\)-values for observations as well as a \(CLs\) test statistic for setting 95 % CL exclusion limits, where asymptotic formulas [56] are used. The expected and observed limits are corrected for small differences observed using toy experiments.

The reconstructed \(m_{tb}\) spectrum for the \(W'\) signal is parametrised using the sum of a skew-normal [57] and a Gaussian function. The skew-normal accounts for the asymmetric shape of the resonant \(W'\) signal and is the product of a Gaussian and a Gaussian error function. Non-resonant off-shell \(W'\) production is accounted for with the additional Gaussian distribution. This allows the signal shape to be fully parametrised. In order to search for a \(W'\) signal over the full mass range, the signal shape parameters, as well as the signal acceptance and the expected cross sections are interpolated between the generated mass points. Figure 3 shows the parametric fits to the \(W'\) signal distributions in the two \(b\)-tag category for \(W'_{L}\) masses between 1.5 and 3 TeV overlaid to the corresponding MC distributions.

Fig. 3
figure 3

Parametric fits to the \(W'\rightarrow tb \rightarrow qqbb\) signal distributions in the two \(b\)-tag category for \(W'_{L}\) masses between 1.5 and 3 TeV overlaid to the corresponding MC distributions. While unbinned fits are performed, the events from MC simulation are shown as a histogram for presentational purposes

Full size image

Additional contributions from \(W'\rightarrow tb \rightarrow \ell \nu b b\) are between 3.5 and 10 %. The leptonic contribution is taken into account by fitting the reconstructed \(m_{tb}\) distribution with a double-Gaussian function and interpolating between generated masses analogously to the treatment of the \(W'\rightarrow tb \rightarrow qqbb\) signal. Large-\(R\) jets can falsely be top-tagged in events with leptonic top-quark decays due to several effects including hard gluon radiation, calorimeter activity from non-identified electrons and hadronically decaying tau leptons.

Fig. 4
figure 4

Fit to the \(m_{tb}\) distribution from the initial background estimate in the one \(b\)-tag category (left) and in the two \(b\)-tag category (right). The ratio background/fit is also shown, where the uncertainties are the control-region and MC statistical uncertainties and the grey shaded band shows the statistical uncertainty in each bin as expected for \(20.3\;\text{ fb }^{-1}\)

Full size image
Table 3 Systematic uncertainties on the event yields of a 2 TeV \(W'\) boson in both categories in percent and on the background modelling in numbers of expected \(W'\) boson events
Full size table
Fig. 5
figure 5

\(m_{tb}\) distributions in data in the one \(b\)-tag (left) and the two \(b\)-tag category (right). Background-only fits are shown, and the bottom plots show the ratio of the data and the fit. The left plot shows an extrapolation of the background fit into the region 4–5 TeV. The ratio plot, however does not show the three data points in this range, because they are beyond the range considered for this analysis. Potential \(W'_{L}\) signal shapes in the hadronic top-quark decay channel with \(g'=g_\mathrm{{SM}}\) are also overlaid for resonance masses of 1.5, 2.0, 2.5 and 3.0 TeV

Full size image
Fig. 6
figure 6

Observed \(p\)-values for a left-handed (left) and right-handed (right) \(W'\) model as a function of the \(W'\) mass

Full size image
Fig. 7
figure 7

Limits at 95 % CL on the cross section times branching ratio to \(tb\) for the left-handed (left) and for the right-handed (right) \(W'\) model. The expected cross section for \(W'\) production with \(g'=g_\mathrm{{SM}}\) is also shown

Full size image
Fig. 8
figure 8

Observed and expected 95 % CL limits on the ratio of coupling \(g'_L/g_\mathrm{{SM}}\) (\(g'_R/g_\mathrm{{SM}}\)) of the \(W'_{L}\) (\(W'_{R}\)) model as a function of the \(W'\) mass

Full size image

The sum of all backgrounds is fitted with an analytic function. For each of the two categories, a function is chosen among several tested functions following a procedure based on the \(m_{tb}\) distribution obtained from the initial background estimate (Sect. 5.2). For each function under study the corresponding fake signal bias is quantified by fitting the \(m_{tb}\) distribution from the initial background estimate with a background-plus-signal model. The maximal extracted fake signal observed over the full range of \(W'\) masses is chosen as systematic uncertainty on the background modelling (Sect. 6). The function with the least number of free parameters is chosen out of all tested functions giving similarly small bias. This procedure yields an exponential function with a polynomial of order \(n\) as its argument: \(\exp \left( \sum _{k=1}^{n} c_k m_{tb}^k \right) \), with \(n = 4\) (\(n = 2\)) in the one (two) \(b\)-tag category. Figure 4 shows background-only fits to the initial background estimate in the one and two \(b\)-tag categories using the chosen functions. The ratio of the distribution and the fit is also shown, where the uncertainties are the control-region and MC statistical uncertainties and the grey shaded band shows the statistical uncertainty in each bin as expected for \(20.3\;\text{ fb }^{-1}\). Deviations from 1 in the ratio plot are much smaller than the expected statistical uncertainty and the chosen background functions are hence shown to be flexible enough to describe the background distribution in the signal region.

6 Systematic uncertainties

Systematic uncertainties may change the acceptance and shape of the potential \(W'\) signal, and are included as nuisance parameters in the likelihood function. Table 3 shows the impact of the systematic uncertainties on the event yield of a 2 TeV \(W'\) boson in the one and two \(b\)-tag categories. The largest sources of uncertainty come from the uncertainties associated with \(b\)-tagging, top-tagging and background modelling.

Uncertainties on the \(b\)-tagging efficiency and mistag rates are estimated from data using \(t\bar{t}\) di-lepton decays [46, 58]. The \(b\)-tagging (mistagging) uncertainties are increased for high \(p_{\mathrm {T}}\) and reach up to 34 % (60 %) per jet. Uncertainties on the \(W'\) top-tagger performance are evaluated based on the data-MC agreement as shown in Refs. [51, 52]. They are derived comparing the ratio of each variable from jets built from calorimeter clusters and the corresponding jet built from tracks in the ID. The observed differences between data and MC are taken as variations on the substructure variables and are translated into an uncertainty on the efficiency of the \(W'\) top-tagger. Within the kinematic reach, it has been shown with \(t\bar{t}\) events in the single-lepton channel that these uncertainties cover any possible disagreement between the efficiency observed in data and MC simulations. The jet energy scale (JES) uncertainty [43] depends on the \(p_{\mathrm {T}}\) and \(\eta \) of the reconstructed jet and includes the uncertainty on the \(b\)-jet energy scale. The JES of the two jet types are assumed to be correlated. The impact of the jet energy resolution uncertainty is evaluated by smearing the jet energy in the simulation to increase the nominal resolution. The uncertainty on the integrated luminosity is 2.8 % as derived from beam-separated scans [59]. Theoretical uncertainties are included by evaluating the change in the expected number of signal events. The deviations from varying the CTEQ6L1 PDF eigenvectors are summed in quadrature with the uncertainty from \(\alpha _s\), the renormalisation scale, and the change in acceptance at LO and NLO. In addition, the uncertainty on the beam energy [60] is included.

Uncertainties due to background mismodelling are quantified as discussed in Sect. 5.3. This uncertainty amounts to 28 (24) events in the two \(b\)-tag category and 44 (45) events in the one \(b\)-tag category for the \(W'_{L}\) (\(W'_{R}\)) model.

7 Results

Figure 5 shows the observed \(m_{tb}\) spectra in the two categories. The highest mass event in the two (one) \(b\)-tag category is at 3.25 TeV (4.68 TeV). A background-only fit to the spectra is also shown and good agreement is observed between the fit and the data.

Figure 6 shows the observed \(p\)-values for background plus left-handed or right-handed \(W'\) model as a function of the \(W'\) mass allowing the background parameters to float. The \(p\)-value from both categories combined is shown taking into account all systematic uncertainties. The maximum local significance is 1.4\(\sigma \). Hence, no significant excess over the background-only hypothesis is observed and 95 % CL limits are derived on the cross section times branching ratio to \(tb\) as shown in Fig. 7. The observed limits agree with the expected limits roughly within 1\(\sigma \) over the whole range for both \(W'\) models and range from 0.16 pb to 0.33 pb for left-handed \(W'\) bosons and from 0.10 pb to 0.21 pb for \(W'\) bosons with purely right-handed couplings. The relative mass independence of the expected sensitivity and observed limits is a unique feature of this search arising from the flat signal efficiency when combining the one and two \(b\)-tag categories.

For \(g' = g_\mathrm{{SM}}\), the limits on the cross section times branching ratio translate to observed (expected) limits on the mass to be above 1.68 TeV (1.63 TeV) and 1.76 TeV (1.85 TeV) in the left- and right-handed models, respectively. The observed cross-section limits are also interpreted as limits on other values of the couplings \(g'_{L/R}\). For \(g'_{L/R}/g_\mathrm{{SM}}< 2\), the reconstructed \(m_{tb}\) distributions are dominated by the experimental width. The results obtained for \(g'_{L/R} = g_\mathrm{{SM}}\) can hence be interpreted as limits in the \(g'_L\)-\(W'_{L}\) mass (\(g'_R\)-\(W'_{R}\) mass) plane, making use of the approximately quadratic dependence of the \(W'\) production cross section on \(g'\). The observed and expected limits on the ratio of couplings \(g'_L/g_\mathrm{{SM}}\) (\(g'_R/g_\mathrm{{SM}}\)) of the \(W'_{L}\) (\(W'_{R}\)) model as a function of \(W'\) mass are shown in Fig. 8 and amount to \(g' < 0.70\) (\(g' < 0.55\)) for a 1.5 TeV \(W'_{L}\) (\(W'_{R}\)) and to \(g' < 2\) for a 2.18 (2.29) TeV \(W'_{L}\) (\(W'_{R}\)) boson.

8 Summary and conclusion

A search for \(W'\rightarrow tb \rightarrow qqbb\) was presented using \(20.3\;\text{ fb }^{-1}\) of 8 TeV proton-proton collisions data taken with the ATLAS detector. The analysis makes use of jet substructure tagging optimised to select large-\(R\) jets coming from hadronically decaying top quarks and \(b\)-tagging of small-\(R\) jets. The observed \(m_{tb}\) spectrum from data is consistent with the background-only prediction and exclusion limits at 95 % CL are set on the \(W'\) boson production cross section times branching ratio to \(tb\). The use of novel jet substructure techniques allows cross-section limits to be set at high \(W'\) masses, which are similar to the limits at lower masses and range from 0.16 pb to 0.33 pb for left-handed \(W'\) bosons, and from 0.10 pb to 0.21 pb for \(W'\) bosons with purely right-handed couplings. In addition, limits are set at 95 % CL on the \(W'\)-boson effective couplings as a function of the \(W'\) mass.