1 Introduction

In the Standard Model (SM), electroweak symmetry breaking (EWSB) occurs through a single complex scalar doublet field and results in a single physical state, the Higgs boson [13]. A particle with characteristics of the SM Higgs boson has been discovered by both ATLAS [4] and CMS [5]. Beyond the SM, many models have been proposed, extending the Higgs sector to explain EWSB. The newly discovered boson is compatible with many of these models so that discovering its true nature is crucial to understanding EWSB. Two Higgs-doublet models (2HDM) [6] are simple extensions of the SM with five observable Higgs bosons, of which two are charged (H + and H ) and three are neutral (h 0, H 0 and A 0). The discovery of a charged Higgs boson would be a signal for new physics beyond the SM.

The Minimal Supersymmetric Standard Model (MSSM) [7] is an example of a 2HDM. At tree level, the MSSM Higgs sector is determined by two independent parameters, which can be taken to be the mass \(m_{H^{+}}\) and the ratio of the two Higgs doublet vacuum expectation values, parametrised by tanβ. In the MSSM, a light H + (defined as \(m_{H^{+}}<m_{t}\)) decays predominantly to \(c\bar{s}\), \(b\bar{b}W^{+}\), and τ + ν, with the respective branching ratios depending on tanβ and \(m_{H^{+}}\). Charge conjugated processes are implied throughout this paper. For tanβ<1, \(c\bar{s}\) is an important decay mode with \(\mathcal{B}(H^{+} \rightarrow c\bar{s})\) near 70 % [8, 9] for \(m_{H^{\pm}} \simeq110~\mathrm{GeV}\), whereas for tanβ>3, H +τ + ν dominates (90 %). For higher H + masses at low tanβ, the decay mode \(H^{+}\rightarrow Wb\bar{b}\) can be dominant. A light MSSM charged Higgs boson is viable at a relatively low tanβ≈6 in certain MSSM benchmark scenarios [10] that take into account the discovery of a Higgs boson with a mass of 125 GeV at the LHC.

The LEP experiments placed lower limits on \(m_{H^{+}}\) in any type-II 2HDM [11] varying between 75 GeV and 91 GeV [1216] depending on the assumed decay branching ratios for the charged Higgs boson. At the Tevatron, searches for charged Higgs bosons have been extended to larger values of \(m_{H^{+}}\). No evidence for a H + was found and upper limits were set on the branching ratio \(\mathcal{B}(t \rightarrow H^{+}b)\) varying between 10 % and 30 % for a light H + under the assumption of \(\mathcal{B}(H^{+} \rightarrow c\bar {s})=100~\%\) [17, 18]. The discovery of a Higgs boson at the LHC is a weak constraint on many 2HDMs, and is compatible with the existence of a light charged Higgs decaying to two jets, especially in type I 2HDMs [19, 20].

In this paper, a search for a charged Higgs boson produced in \(t\bar{t}\) decays is presented, where one of the top quarks decays via tH + b with the charged Higgs boson subsequently decaying to two jets (\(c\bar{s} \)), where again a 100 % branching fraction is assumed. The other top quark decays according to the SM via \(\bar{t} \rightarrow W^{-}\bar{b}\) with the W boson decaying into a lepton (e/μ) and the corresponding neutrino. The signal process therefore has the same topology as SM \(t\bar{t}\) decays in the lepton+jets channel, where one W decays to two jets and the other to a lepton and corresponding neutrino, but the invariant mass of the two jets from the H + peaks at \(m_{H^{+}}\). The search is performed by comparing the dijet mass spectrum in the data with the prediction from SM top-quark decays and with the expectation of a top quark having a non-zero branching ratio for decay to H + b.

2 Detector description and event samples

The data used in the analysis were recorded by the ATLAS detector in proton–proton (pp) collisions at a centre-of-mass energy of \(\sqrt {s}=7~\mathrm{TeV}\) during the 2011 data-taking period of the Large Hadron Collider (LHC) [21]. Events were required to pass a high-transverse momentum (p T) single-lepton (e/μ) trigger, and to have been recorded when all detector systems critical to muon, electron, and jet identification were operational. The lepton triggers required in the different data taking periods had varying p T thresholds: 20–22 GeV for the electron trigger and 18 GeV for the muon trigger. The resulting dataset corresponds to an integrated luminosity of 4.7 fb−1 [22, 23].

The ATLAS detector [24] consists of an inner tracking system immersed in a 2 T axial magnetic field provided by a thin solenoid; electromagnetic and hadronic calorimeters; and a muon spectrometer (MS) embedded in a toroidal magnet system. The inner detector tracking system (ID) comprises a silicon pixel detector closest to the beamline, a silicon microstrip detector, and a straw tube transition radiation tracker. The electromagnetic (EM) calorimeters are high-granularity liquid-argon sampling calorimeters with lead as the absorber material in the barrel and endcap regions, and copper in the forward region. The hadronic calorimetry uses two different detector technologies. The barrel calorimeter (|η|<1.7)Footnote 1 consists of scintillator tiles interleaved with steel absorber plates. The endcap (1.5<|η|<3.2) and forward (3.1<|η|<4.9) calorimeters both use liquid argon as the active material, and copper and tungsten respectively as the absorber. The MS consists of three large superconducting toroids each with eight coils, and a system of precision tracking and fast trigger chambers.

The largest background to the charged Higgs boson signal is the SM production and decay of \(t\bar{t}\) pairs. Additional background contributions (referred to as non-\(t\bar{t}\) backgrounds) arise from the production of a single top quark, of a W or Z boson with additional jets, of QCD multi-jets, and of dibosons.

Top-quark pair and single top-quark events (Wt-channel and s-channel) were generated using the mc@n-lo 4.01 [2528] Monte Carlo (MC) generator coupled to Herwig 6.520.2 [29] to provide the parton showering and hadronisation using the AUET2-CT10 [30, 31] tune; Jimmy [32] was used to model the underlying event. Single top-quark events in the t-channel were generated using AcerMC 3.8 [33] coupled to Pythia 6.425 [34] with the AUET2-MRST2007LO** [30, 35] tune. W/Z+jet and diboson events were generated using the leading-order (LO) Alpgen 2.13 [36] generator interfaced to Herwig with the AUET2-CTEQ6L1 [30, 37] tune. The W/Z+jet simulated data include dedicated samples for heavy-flavour production (\(b\bar{b}, c\bar{c}\) and c). Signal samples of \(t\bar{t}\to H^{+}bW^{-}\bar{b}\) were generated using Pythia 6.425 for seven different H + masses from 90 GeV to 150 GeV.

The data are affected by the detector response to multiple pp interactions occurring in the same or neighbouring bunch crossings, known as pile-up. Minimum-bias interactions generated by Pythia 6.425 [34], which has been tuned to data [38], were overlaid on the simulated signal and background events. The events were weighted to reproduce the distribution of the number of interactions per bunch crossing observed in the data. A Geant4 simulation [39, 40] is used to model the response of the ATLAS detector, and the samples are reconstructed and analysed in the same way as the data.

3 Physics objects and event selection

Jets are reconstructed from topological clusters of calorimeter cells [41] using the anti-k t  algorithm [42, 43] with a radius parameter R=0.4. Topological clusters are built using an algorithm that suppresses detector noise. Jets are corrected back to particle (truth) level using calibrations derived from Monte Carlo simulation and validated with both test-beam [44] and collision-data studies [45]. Events are excluded if they contain a high-p T jet that fails quality criteria rejecting detector noise and non-collision backgrounds [46]. To suppress the use of jets originating from secondary pp interactions, a jet vertex fraction (JVF) algorithm is used. Inner detector tracks, with p T>1 GeV, are uniquely associated with jets using \(\Delta R(\mathrm{jet, track}) < 0.4\), where \(\Delta R\equiv\sqrt{( \Delta\phi)^{2} + ( \Delta\eta)^{2}}\). The JVF algorithm requires that at least 75 % of the sum of the p T of the tracks associated with the jet is from tracks compatible with originating from the primary vertex of the event. Tagging algorithms identify jets originating from b-quark decays by selecting jets with tracks from secondary vertices or those with a large impact parameter significance. A multivariate algorithm (MV1) [47], which uses a neural network to combine the weights from multiple tagging algorithms, is used to identify jets originating from b-quarks. Jets passing the MV1 selection are referred to as b-tagged jets. The selection on the discriminating variable of the algorithm achieves an average per-jet efficiency of 70 % to select b-jets in \(t\bar{t}\) events, with a probability to incorrectly tag light jets of less than 0.1 % [48]. Studies have shown that this working point has a 20–40 % efficiency to tag a c-jet, depending on the p T of the jet [49].

Muons are required to be identified in both the ID and MS, and their momentum is obtained through a combined fit of all hits in both systems. Muons are also required to satisfy isolation criteria to reject those originating from heavy-flavour decays and hadrons misidentified as muons. The sum of the transverse momenta of ID tracks within a cone of ΔR=0.3 around the muon, excluding the muon track itself, is required to be less than 2.5 GeV. The transverse energy measured in the calorimeters within a cone of ΔR=0.2, excluding the energy associated with the muon, is required to be less than 4 GeV. In addition, muons are removed if they are found within ΔR<0.4 of a jet that has p T>25 GeV [50, 51].

The reconstruction of electron candidates starts from a seed cluster in the second layer of the EM calorimeter. The cluster is matched to a track found in the ID and a set of selection criteria are applied to reject electron candidates originating from jets [52]. Electrons are required to be isolated in order to suppress the QCD multi-jet background. The calorimeter isolation is performed using a cone of ΔR=0.2 and the track isolation uses a cone of radius ΔR=0.3. The calorimeter and track isolation cut values are chosen to achieve 90 % efficiency with respect to selected electron candidates [53]. As in the case of muons, the electron itself is excluded from the sum over the isolation cone.

Energy deposits in the calorimeter are expressed as four-vectors (E,p), where the direction is determined from the position of the calorimeter cluster and the nominal interaction point (x=y=z=0). The clusters are formed assuming E=|p|. The missing transverse momentum (\(E_{\mathrm{T}}^{\mathrm{miss}}\)) is given by the negative of the vector sum of the calorimeter four-momenta, projected into the (x,y) plane. The \(E_{\mathrm{T}}^{\mathrm{miss}}\) calculation uses the energy scale appropriate for each physics object described above. For muons, the momentum measured from the combined tracking is used as the energy. The remaining calorimeter cells not associated with any physics object are included at the electromagnetic energy scale of the calorimeter [54].

A set of requirements is imposed to select events containing \(t\bar{t}\) decays in the lepton+jets channel [50]. First, events are required to contain a primary vertex with at least five associated tracks to suppress non-collision backgrounds. Exactly one electron with a large transverse energy (E T>25 GeV) and |η|<2.5, excluding the barrel–endcap transition region 1.37<|η|<1.52, or one muon with large transverse momentum (p T>20 GeV) and |η|<2.5 is required. The selected lepton must match a lepton trigger object that caused the event to be recorded. Jets present in W/Z+jet events tend to originate from soft gluon emissions. These backgrounds are therefore reduced by requiring at least four jets with p T>25 GeV and |η|<2.5. At least two jets must be identified as originating from a b-decay using the MV1 algorithm. To suppress backgrounds from QCD multi-jet events, the missing transverse momentum is required to be \(E_{\mathrm{T}}^{\mathrm {miss}}> 20 (30)~\mathrm{GeV}\) in the muon (electron) channel. Further reduction of the multi-jet background is achieved by requiring the transverse massFootnote 2 (m T) of the lepton and \(E_{\mathrm{T}}^{\mathrm{miss}}\) to satisfy m T>30 GeV in the electron channel and \((E_{\mathrm{T}}^{\mathrm{miss}}+ m_{\mathrm{T}}) > 60~\mathrm {GeV}\) in the muon channel. These requirements favour the presence of a W boson, decaying to ℓν, in the final state. The selections are more stringent in the electron channel because of the larger multi-jet background.

4 Kinematic fit

In the selected events, the two jets originating from the decay of the H + must be identified in order to reconstruct the mass. A kinematic fitter [17] is used to identify and reconstruct the mass of dijets from W/H + candidates, by fully reconstructing the \(t\bar{t}\) system. In the kinematic fitter, the lepton, \(E_{\mathrm {T}}^{\mathrm{miss}}\) (assumed to be from the neutrino), and four jets are assigned to the decay particles from the \(t\bar{t}\) system. The longitudinal component of the neutrino momentum is calculated from the constraint that the invariant mass of the leptonic W boson decay products must be the experimental value (80.4 GeV) [55]. This leads to two possible solutions for this momentum. When complex solutions are returned, the real part of the solution is used in the fit. The fitter also constrains the invariant mass of the two systems (bℓν,bjj) to be within Γ t =1.5 GeV of the top-quark mass 172.5 GeV, which is consistent with the measured top-quark mass [56]. When assigning jets in the fitter, b-tagged jets are assumed to originate from the b-quarks. The best bbjj combination is found by minimising a χ 2 for each assignment of jets to quarks and for the choice of solution for the longitudinal neutrino momentum, where the five highest-p T jets are considered as possible top-quark decay products. Since the b-jets are only allowed to be assigned to the b-quarks, and the two untagged jets are assigned to quarks from the same charged boson, there are two possible jet configurations overall for events with four jets, two of which are b-tagged. For events with at least five jets, the two highest-p T jets are always assumed to be from the top-quark decay products (W/H + boson or b-quark) to reduce the combinatorics in the fit procedure. The combination with the smallest χ 2 value, \(\chi^{2}_{\mathrm{min}}\), is selected as the best assignment. The function minimised in the fit is:

(1)

In the first term, the fitted transverse momenta of the lepton and the four jets currently under consideration are allowed to vary around the measured values using the corresponding measured resolutions (σ i ). In the fit only the magnitudes of the object p Ts are varied; the angles of the jets and leptons are assumed to be measured with good precision. The vector sum of the momenta of the remaining jets (p T>15 GeV) in the event, labelled SEJ, is allowed to vary in the second term. The resolution for this term is taken from the nominal jet resolution. Letting the SEJ vary allows the \(E_{\mathrm{T}}^{\mathrm{miss}}\) to be recalculated from the fitted values of its dominant components. Jets with lower p T and energy from calorimeter cells not associated with any physics object are both minor contributions to the \(E_{\mathrm{T}}^{\mathrm{miss}}\) and are held fixed in the re-calculation of the \(E_{\mathrm{T}}^{\mathrm{miss}}\). The third term constrains the hadronic (jjb) and leptonic (bℓν) top-quark candidates to have a mass close to the top-quark mass.

The \(\chi^{2}_{\mathrm{min}}\) distribution for selected events in the data agrees well with the expectation from the simulation (see Fig. 1). Events are required to have \(\chi^{2}_{\mathrm{min}} < 10\) to remove poorly reconstructed \(t\bar{t}\) events. This selection has an efficiency of 63 % for SM \(t\bar{t}\) events. The fit results in a 12 GeV dijet mass resolution, as shown in Fig. 2. This is a 20–30 % improvement, depending on the mass of the boson studied, compared to the resolution obtained when the same jets are used with their original transverse momentum measurements. After the fit, there is better discrimination between the mass peaks of the W boson from SM decays of \(t\bar{t}\) and a 110 GeV H + boson in this example.

Fig. 1
figure 1

Comparison of the distribution of \(\chi^{2}_{\mathrm{min}}\) from the kinematic fitter for data and the expectation from the background estimates for the combined electron and muon channels. The MC simulation is normalised to the expectation for the SM (\(\mathcal{B} (t\to H^{+}b)=0\)). The uncertainty shown on the background estimate is the combination in quadrature of the ±1σ systematic uncertainties. The final bin also contains the overflow entries

Fig. 2
figure 2

Comparison of the dijet mass distribution before (upper part) and after (lower part) the kinematic fit and the χ 2<10 selection criterion. The distribution is shown for MC simulations of SM \(t\bar{t}\) decays and the \(m_{H^{+}} = 110~\mathrm{GeV}\) signal (\(t\bar{t} \rightarrow H^{+}bW^{-}\bar{b}\)). The curves are normalised to the same area

Table 1 shows the number of events observed in the data and the number of events expected from the SM processes after the selection requirements. The SM \(t\bar{t}\) entry includes events from both the lepton+jets and dilepton \(t\bar{t}\) decay modes, where the dilepton events can pass the event selection if the events contain additional jets and the second lepton is not identified. Good agreement is observed between the data and the expectation. The table also shows the number of signal events expected for \(\mathcal{B}(t \rightarrow H^{+}b)=10~\%\). The signal prediction accounts for acceptance differences due to the different kinematics of the tH + b events relative to the SM tWb events.

Table 1 The expected numbers of events from SM processes, integrated over the full range of dijet masses and the observed number of events in the data after all the selection requirements. The expected number of events in the case of a signal with \(m_{H^{+}}=110~\mathrm{GeV}\) and \(\mathcal{B}(t\to H^{+}b)=10~\%\) is also shown. The \(t\bar {t}\rightarrow W^{+}b W^{-}\bar{b}\) numbers include both the lepton+jets and dilepton decay channels. The uncertainties are the sum of the contributions from statistics and systematic uncertainties

5 Systematic uncertainties

The background estimates and the estimate of the signal efficiency are subject to a number of systematic uncertainties. The QCD multi-jet background is estimated using a data-driven method [57] that employs a likelihood fit to the \(E_{\mathrm{T}}^{\mathrm{miss}}\) distribution in the data, using a template for the multi-jet background and templates from MC simulations for all other processes. The uncertainty on the QCD multi-jet background is evaluated to be 50 % by studying the effect of pile-up events on the fit results and by performing likelihood fits on the m T(W) distribution. The dijet mass distribution of multi-jet events is obtained from a control region in the data, where leptons are required to be semi-isolated, such that the transverse momentum of the inner detector tracks in a cone of radius ΔR=0.3, excluding the lepton, satisfies \(0.1<p_{\mathrm{T}}^{\Delta R = 0.3}/p_{\mathrm{T}}(e,\mu)<0.3\). Leptons in the control region are also required to have a large impact parameter with respect to the identified primary vertex (0.2 mm<|d 0|<2 mm) and an impact parameter significance \(|d_{0}|/\sigma_{d_{0}}>3\).

The rate of W+jets events is estimated by a data-driven method [58] that uses the observed difference in the number of W + and W bosons in the data and the charge asymmetry (W +W )/(W ++W ), which is calculated to good precision by the MC simulation of W+jets events. The heavy flavour fraction of the W+jets MC simulation is calibrated using W+1 jet or W+2 jets events in the data. The uncertainty on the W+jets background is 26 % (28 %) for the electron (muon) channel, which includes the uncertainty from the charge asymmetry and heavy flavour fraction components. The shape of the m jj distribution for W+jets events is obtained from simulation.

Uncertainties on the modelling of the detector and on theory give rise to systematic uncertainties on the signal and background rate estimates. The following systematic uncertainties are considered: integrated luminosity (3.9 %) [22, 23], trigger efficiency (3.5 %/1 % for electron/muon), jet energy scale (1–4.6 %) [45], jet energy resolution (up to 16 % smearing) [59], and b-jet identification efficiency (5–17 %). The last three uncertainties depend on the p T and η of the jets. Uncertainties on lepton reconstruction and identification efficiency are determined using a tag and probe method in samples of Z boson and J/ψ decays [60]. The momentum resolution and scales are determined from fits to samples of W boson, Z boson, and J/ψ decays [53, 61]. Additional p T-dependent uncertainties are placed on the b-jet (up to 2.5 %) and c-jet (up to 1.3 %) energy scales [45]. Uncertainties on the modelling of the \(t\bar{t}\) background are estimated using a second MC generator (Powheg [6264]) and comparing the effect of using Pythia and Herwig to perform the parton showering and hadronisation. Uncertainties on initial and final state radiation (ISR/FSR) are assessed using AcerMC interfaced to Pythia and examining the effects of changing the ISR/FSR parameters in a range consistent with experimental data [65]. The predicted SM \(t\bar{t}\) cross-section for pp collisions at \(\sqrt {s} = 7~\mathrm{TeV}\), obtained from approximate next to next to LO QCD calculations, is \(\sigma_{t\bar{t}} = 167^{+17}_{-18}~\mathrm{pb}\) for a top-quark mass of 172.5 GeV [66]. The uncertainty on the predicted value includes the uncertainty in the renormalisation and factorisation scales, parton density functions, and the strong coupling constant. An additional uncertainty on the \(t\bar{t}\) cross-section (4.5 %) is included due to the uncertainty on the top-quark mass. The uncertainty on the top-quark mass is 0.9 GeV from the combined measurement [56] at the Tevatron. However, this result would be biased in the presence of a \(H^{+}\rightarrow c\bar{s}\) signal in the lepton+jets channel, so a larger uncertainty of 1.5 GeV is taken, which is consistent with the latest top-quark mass measurement in the dilepton channel from the CMS experiment [67]. Changing the top-quark mass leads to altered event kinematics, which results in a final uncertainty on the event rate of 1.9 %. The effects of these systematic uncertainties on the overall normalisation are listed in Table 2. The jet energy calibration, b-jet identification, \(t\bar{t}\) background modelling, and ISR/FSR uncertainties also modify the shape of the dijet mass distribution and are therefore determined as a function of m jj . The systematic uncertainties that affect the shape of the m jj distribution (top half of Table 2) are more important than the shape-independent uncertainties. The effects of the systematic uncertainties are comparable, within 10 % , between the SM and signal \(t\bar{t}\) samples. The combined uncertainty on the single top-quark and diboson backgrounds is 15 %, which comes mostly from the uncertainties on the cross-section, jet energy scale, and b-tagging. The total uncertainty on the overall normalisation of the non-\(t\bar{t}\) backgrounds is 30 %.

Table 2 Effect of the systematic uncertainties on the event rate of \(t\bar{t}\) background and signal (\(m_{H^{+}}=110~\mathrm{GeV}\)) events before any reduction from the likelihood fit, described in Sect. 6

6 Results

The data are found to be in good agreement with the distribution of the dijet mass expected from SM processes (see Fig. 3). The fractional uncertainty on the signal-plus-background model is comparable to the background only model. Upper limits on the branching ratio \(\mathcal{B}(t\rightarrow H^{+}b)\) are extracted as a function of the charged Higgs boson mass. The upper limits are calculated assuming the charged Higgs always decays to \(c\bar{s}\). The following likelihood function is used to describe the expected number of events as a function of the branching ratio:

$$ \mathcal{L}(\mathcal{B}, {\mathbf{\alpha}}) = \prod _i \frac{\nu _{i}(\mathcal{B} ,\alpha )^{n_i} e^{-\nu_{i}(\mathcal{B},\alpha)}}{n_i!} \prod_j \frac {1}{\sqrt {2\pi}} e^{-\frac{\alpha_j^2}{2}}, $$
(2)

where n i  is the number of events observed in bin i of the dijet mass distribution and j labels the sources of systematic uncertainty. The number of expected signal plus background events in each bin, \(\nu_{i}(\mathcal{B},\alpha)\), is given by

(3)

where \(n^{N}_{i}\) is the expected number of non-\(t\bar{t}\) background events, \(\sigma_{t\bar{t}}\) is the cross-section for \(t\bar{t}\) production, \(\mathcal{L}\) is the integrated luminosity, \(\mathcal{B}\) is the branching ratio of tH + b, and \(A^{H^{+}}\) and A W are the acceptances for signal (\(t\bar {t}\to H^{+}b\ell\nu\bar{b}\)) and SM \(t\bar{t}\) (\(t\bar{t}\to jjb\ell\nu \bar {b}\) and \(t\bar{t}\to\ell\bar{\nu} b\ell\nu\bar{b}\)) events respectively. The decay mode \(t\bar{t}\rightarrow H^{+}bH^{-}\bar{b}\) does not contribute to the expectation because this mode does not produce a single isolated lepton and hence has a negligible efficiency to pass the selection requirements. The \(S_{i}^{H^{+}}\) (\(S_{i}^{W}\)) parameter describes the shape of the m jj spectrum (normalised to one) for H + (W) boson production. It gives the relative number of events in bin i according to the normalised m jj distribution. The α j variables are nuisance parameters representing the systematic uncertainties, which are constrained via the Gaussian terms in Eq. (2). The effect of the systematic uncertainties on the non-\(t\bar{t}\) background can be obtained by calculating the effect of each source of uncertainty on each non-\(t\bar{t}\) background component, and combining them in quadrature. Since this sum is dominated by the uncertainties on the data-driven W+jets and multi-jet background estimates, the combined variation is treated as a single nuisance parameter (α b , bj) and is assumed to be uncorrelated from the other systematic uncertainties. The ρ ji functions account for the effect of nuisance parameters on the yields and are defined such that ρ ji (α j =±1σ) represents the 1±1σ fractional change in the number of entries in bin i of the dijet mass spectrum due to systematic uncertainty j. The physics measurement involves a sufficiently large number of events that this likelihood can constrain the α j parameters beyond the precision of the subsidiary measurements. The effects of systematic uncertainties are applied coherently in signal and background distributions. The subsidiary measurements of the α j parameters are taken to be uncorrelated. The fit uses 17 nuisance parameters in total. None of them are shifted by more than one sigma compared to the original values obtained in subsidiary measurements. Maximal reduction of uncertainty is obtained for the jet energy scale parameter which is reduced by 50 %.

Fig. 3
figure 3

The dijet mass distribution from data and the expectation from the SM (\(\mathcal{B}=0\)). The error bars represent the statistical uncertainty on the data. The uncertainty shown on the background estimate is the combination in quadrature of the ±1σ systematic uncertainties, accounting for the constraint from the profile likelihood fit. The first and last bins contain the underflow and overflow events respectively

The limits on the branching ratio are extracted using the CLs technique at 95 % confidence level [68, 69]. The consistency of the data with the background model can be determined by comparing the value of the test statistic (a profile likelihood ratio based on Eq. (2)) in the data with the expectation from background-only Monte Carlo simulated experiments. The corresponding probability (p-value) for the background to produce the observed mass distribution varies from 67 % to 71 % as a function of \(m_{H^{+}}\), indicating that there is no significant deviation from the background hypothesis. The expected and observed limits, shown in Table 3 and Fig. 4, are calculated using asymptotic formulae [68]. The expected limits on \(\mathcal{B}\), including both statistical and systematic uncertainties, vary between 1–8 % depending on \(m_{H^{+}}\); if only the statistical uncertainty is considered these limits are 1–3 %. The observed limits, including both statistical and systematic uncertainties, vary between 1–5 %. The extracted limits are the most stringent to date on the branching ratio \(\mathcal{B}(t\to H^{+}b)\), assuming \(\mathcal{B}(H^{+}\to c\bar{s})=100~\%\). These results can be used to set limits for a generic scalar charged boson decaying to dijets in top-quark decays, as long as the width of the resonance formed is less than the experimental dijet resolution of 12 GeV.

Fig. 4
figure 4

The extracted 95 % CL upper limits on \({\mathcal{B}}(t \rightarrow H^{+}b)\), assuming that \(\mathcal{B}(H^{+}\rightarrow c\bar{s}) = 100~\% \), are shown for a range of charged Higgs masses from 90 GeV to 150 GeV. The limits shown are calculated using the CLs limit-setting procedure

Table 3 Expected and observed 95 % CL limits, including systematic uncertainties, on the branching ratio for a top-quark to decay to a charged Higgs boson and a b-quark, assuming that \(\mathcal{B}(H^{+}\rightarrow c\bar {s}) = 100~\%\). The limits shown are calculated using the CLs limit-setting procedure

7 Conclusions

A search for charged Higgs bosons decaying to \(c\bar{s}\) in \(t\bar{t}\) production has been presented. The dijet mass distribution is in good agreement with the expectation from the SM and limits are set on the branching ratio \(\mathcal{B}(t\to H^{+}b)\), assuming \(\mathcal{B}(H^{+}\to c\bar{s})=100~\%\). The observed limits range from \(\mathcal{B}=5~\%\) to 1 % for \(m_{H^{+}}=90\) GeV to 150 GeV. These are the best limits to date on charged Higgs boson production in this channel.